Telegram soft-throttles sustained sequential getMessages calls from a
user account with growing per-call latency (no FLOOD_WAIT, so our retry
wrapper never sees it). On a large package count this made the
once-per-startup destination-message verification pass take 90+ minutes
and block the scheduler/fetch-listener from starting. Running up to 3
batches concurrently cuts wall-clock time well under real per-account
rate limits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP
detectArchive() had multipart patterns for ZIP and RAR but none for 7z's
own volume-split naming, so files like "Name.7z.001" matched nothing and
were dropped before ever reaching grouping or the skipped-package
bookkeeping — no Package row, no SkippedPackage row, no log line.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP
files/temp is TDLib's own redundant download cache — the worker
already prunes it after every ingestion run (see the recent
optimizeTdlibStorage fix), and its content still lives in the
source/destination Telegram chats regardless. With the cache grown
back to ~59GB between prune cycles, tarring it made today's backup
run for 4+ hours straight, fighting the actively-ingesting worker for
disk I/O and degrading the whole host. Excluding it keeps the backup
to what's actually irreplaceable: the DB dump and the TDLib session
state itself.
tryForwardArchiveSet only attempted the no-download forward path for
ZIP/RAR/7z archives, and bailed to download+reupload whenever the
ranged listing failed — even though the channel already grants
forwarding permission and forwarding a message costs nothing
regardless of what's inside it. Standalone DOCUMENT/STL/3MF
attachments never got a chance at the forward path at all.
Now any file on a forwarding-enabled channel is forwarded directly;
when there's no listing to derive a content hash from (non-archive
types, or ranged-listing failures), dedup falls back to
remote.unique_id identity — deriveForwardContentHash, crcFingerprint,
and compareFingerprints already degrade to this safely for
empty/incomplete entries, so there's no risk of unrelated files
colliding as false duplicates. Inner-file indexing is simply skipped
for these cases, matching the existing accepted limitation for
RAR/7z rebuild placeholders.
Several channels with allowsForwarding=true were still falling back to
download+reupload for specific RAR archives after the signature-bytes
fix, with zero log output to explain why — walkRarVolume and
readRarListingRanged return null silently on signature-detection
failure, the MAX_RAR_BLOCKS/MAX_RAR_HEADER_BYTES guards, and a
non-positive advance. Log each case so the next occurrence identifies
the actual cause instead of requiring another guess-and-redeploy cycle.
readRarListingRanged read the archive signature only to detect the RAR
version, then walkRarVolume began harvesting header regions at
pos = sigLen — the signature bytes themselves were never captured.
The reconstructed sparse file therefore started with zero bytes
instead of "Rar!", so every real unrar invocation rejected it as
"not RAR archive". This silently forced every RAR archive through the
expensive download+reupload fallback regardless of the source
channel's forwarding permission — defeating the forward-priority path
for the RAR-heavy channels it matters most for. The existing tests
didn't catch this because their assertions accepted either a null or
non-null result as passing, deferring real verification to production.
After the bot's TDLib session was rebuilt from scratch (following a
disk-full corruption on 2026-08-05), sendMessage started failing with
"Chat not found" for every previously-known user — a fresh TDLib
database has no cached chat/peer info until createPrivateChat
explicitly resolves it. Call it before every send.
TDLib keeps a permanent local copy of every file it downloads or
uploads (via inputFileLocal) with no automatic cleanup. Across the two
worker accounts this had grown to ~270GB, filling the host disk to 91%
and triggering a cascading disk-full failure in the bot's TDLib
session on 2026-08-05. Call optimizeStorage after every ingestion run
to clear it; a 5-minute immunity_delay protects files an in-flight
operation might still reference.
Task-by-task plan for the design in
docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md,
grounded against current worker.ts/provenance-backfill.ts/schema
signatures so a fresh agent can execute it without prior context.
Prioritize native Telegram forwarding over download+reupload for
source channels that allow it, reusing the ranged archive-listing
readers to keep indexing complete without a local download. Falls
back to the existing download+reupload pipeline per-channel (when
forwarding is blocked) and per-archive (when ranged listing fails).
Add MAX_RAR_HEADER_BYTES constant to prevent unbounded ranged reads when
a RAR block's HeaderSize is bogus. Real RAR block headers are far smaller;
this guards against amplification attacks on corrupt/desynced archives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prevent silent masking of short reads by validating buffer bounds before
accessing the first byte. Continuation-byte overflow was already caught,
but a short read that leaves pos at/past buffer.length would return {0, pos+1}
instead of throwing, masking the error from callers' try/catch handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live spike showed start+end sparse reconstruction is insufficient for
encoded-header 7z; fetch the mid-file packed header region as a 3rd
region (parse PackInfo). Adds Task 4b.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cheap listing without full download for RAR/7z placeholders: harvest header
regions via ranged reads, sparse-reconstruct, list with native 7z/unrar CLIs.
Full-download fallback (size-capped) for stragglers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the backup service against the Synology share over SMB/CIFS (the NAS
authenticates with a user/password; NFS is IP-allowlist only). Also fixes
three defects found bringing the service up live:
- entrypoint crash-loop: dcron's crond fails "setpgid: Operation not
permitted" in this runtime -> use busybox crond; make repo-init idempotent
(check via `restic cat config`, tolerate init-on-existing) so a transient
CIFS/lock hiccup can't kill PID 1.
- OOM: pg_dump of a ~276MB DB + tar + restic exceeded the 256M cap -> 1G.
- live tar abort: GNU tar exits 1 when TDLib files change mid-read (worker is
live); per design this is best-effort, so tolerate exit 1, fatal only >=2.
Kuma push is now optional (empty URL disables alerting) since it's deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Multipart ZIP fingerprint reads now use per-part sizes instead of the
whole-archive total, so the tail download offset stays within the last
part's bounds on both the scanned side and the destination-copy side
(scannedFileId replaced with an ordered scannedParts list). A fingerprint
comparison is now only treated as a real mismatch when both sides have
complete CRCs and differ; incomplete comparisons (e.g. empty files) fall
back to name+size confidence instead of silently refusing to backfill.
Name+size-confidence backfills now also create an INFO
INTEGRITY_AUDIT systemNotification for later review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When multiple placeholder packages share the same name+size, try to
disambiguate via ZIP fingerprint; if that can't uniquely resolve a
single match, emit a SystemNotification and skip the backfill rather
than attributing provenance to the wrong package.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild-created placeholder candidates have no PackageFile CRCs, so
name-side fingerprinting can't confirm them. When the stored candidate
fingerprint is incomplete, read the candidate's own copy from its
destination message and fingerprint against that instead of falling
straight to name+size confidence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire tryProvenanceBackfill into processOneArchiveSet: before downloading
a scanned ZIP/RAR/7Z, check whether it's the true origin of a
placeholder-provenance package in the destination channel and backfill
in place, skipping the download. Add the zipsBackfilled counter through
PipelineContext, updateRunActivity, and completeIngestionRun.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive migration (applied on deploy via prisma migrate deploy). Counts
packages whose provenance was backfilled during an ingestion run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements tryProvenanceBackfill() per Task 6 of the provenance-backfill
plan: looks up a placeholder candidate by fileName+fileSize, confirms ZIP
candidates via a ranged central-directory CRC32 fingerprint, and falls
back to name+size confidence for RAR/7z/failed listings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds findPlaceholderCandidate, getPackageFileCrcs, and
backfillProvenance to worker/src/db/queries.ts (Task 5). Candidate
predicate matches placeholder packages by source==dest or the
sourceMessageId==0 rebuild sentinel; backfillProvenance re-checks
placeholder status inside the transaction before overwriting fields.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements Task 4 Step 2 of the provenance-backfill plan. The live
spike (Step 1) and manual verification (Step 3) were not run in this
environment because a second TDLib client would corrupt the running
worker's authenticated session; the absolute-offset assumption is
noted as pending live verification on deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild records use sourceMessageId=0 + synthetic 'rebuild:' contentHash and an
arbitrary fallback sourceChannelId, so the original sourceChannelId==destChannelId
candidate definition missed them. Predicate is now
(sourceChannelId==destChannelId OR sourceMessageId==0), verified against 59,893
live rebuilt records.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A JWT session pointing at a user no longer in the DB (e.g. after a DB reset)
made getUserSettings create settings for a non-existent user -> FK violation
(P2003) -> Server Component render crash. getUserSettings now returns defaults
on P2003; the (app) layout detects the missing user and redirects to a new
server-side /logout route that clears the cookie, avoiding the middleware
redirect loop that otherwise blocks reaching /login.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Render the STL view as a flat per-package list (listDisplayItems no longer
collapses packages into group rows) and hide the Ungrouped tab, now that the
creator filter organizes the list. Remove the worker's heuristic auto-grouping
passes (rule/time/pattern/creator/zip-path/reply-chain/caption); album grouping
is kept. Existing groups and manual grouping actions are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adminandClaude Sonnet 5