readSevenZListingRanged only ever inspected parts[0]. A `.7z.001`/`.7z.002`
set is a raw byte split of one logical 7z file, and a 7z file keeps its next
header (the archive index) at the *end* of the stream — i.e. in the last
volume. So the bounds guard `endStart + nextHeaderSize > size`, with `size`
being parts[0].fileSize, rejected every multipart set before a single byte of
the index was fetched.
Measured on the live DB: forwarding channels + SEVEN_Z + partCount >= 2 listed
1 of 151, while single-volume forwards listed 2301 of 2306 and the
full-download path listed 132 of 132. Those 150 packages were forwarded with an
empty file list, invisible to content and keyword search.
Treat the set as one logical byte stream: mapRangeToVolumes() maps a
whole-archive range onto per-volume reads, splitting it when it straddles a
volume boundary, and every header region (signature, next header, and an
encoded header's packed bytes) is fetched through it. All volumes are
reconstructed sparsely, matching what the full-download path already does
successfully — it hands `7z l` the first part's path with the rest of the set
beside it on disk.
Also log every bail-out. The function had six silent `return null` points and
an outer catch that only fires on thrown exceptions, so the production failure
produced no log line at all — the same gap e123a5c closed for the RAR reader.
Byte-split volume sets and single-volume `.7z` are covered by tests; archives
with encrypted headers still cannot be listed by any header-only reader and
return null (now with a logged reason). Fixtures are built byte-by-byte
because no `7z` binary is installed here, so the tests assert the ranged-read
offsets and the sparse reconstruction rather than `7z l` output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A `.z01`/`.z02`/…/`.zip` set is a spanned (multi-disk) archive, not one ZIP
file cut into chunks. Both ZIP listing paths assumed the latter:
- readZipCentralDirectory fed the parts to yauzl through a concatenating
random-access reader; yauzl hard-refuses nonzero disk numbers
("multi-disk zip files are not supported"), and the EOCD's directory
pointer is volume-relative anyway.
- readScannedZipListing (the ranged, no-download path) passed a
whole-archive tailStart, making the computed directory offset wildly
negative so the parser threw RangeError on every tail size.
Both failures were swallowed upstream, so every spanned set was ingested and
uploaded with an empty file list — invisible to content and keyword search.
Add a volume-aware central-directory reader: locate the EOCD in the final
volume, resolve the directory's (volume, offset) via the disk fields, and read
just the directory bytes, spilling across volumes if it straddles a boundary.
ZIP64 spanned archives are handled through the ZIP64 EOCD locator. The
existing hand-rolled walker in central-directory.ts is reused rather than
adding a third parser.
The two shapes are told apart by filename shape, not by the detector's
multipart `pattern`, so this stays independent of how detect.ts labels them.
A set named like volumes whose EOCD reports a single disk really is a byte
split, and falls back to the concatenating reader.
Byte-split (.zip.001) and single-.zip reading are unchanged; both are now
covered by regression tests. Tests build spanned archives byte-by-byte and
cross-check against real Info-ZIP `zip -s` output (including `-fz` ZIP64)
where the CLI is available.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The .7z.001 gap survived for months because an unmatched attachment left
zero trace anywhere. All three isArchiveAttachment call sites now emit a
debug line with the chat, message id and filename when a document is
skipped, so future gaps of this class are greppable instead of invisible.
Debug level because every non-archive message in a channel hits this.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Files whose names match no pattern in detect.ts are dropped with no log,
no packages row and no skipped_packages row. Five gaps of that class:
- Numbered volumes were implemented format-by-format (ZIP_NUMBERED,
SEVENZ_NUMBERED) with no RAR equivalent, so Pack.rar.001 vanished.
Replaced both with a single ARCHIVE_NUMBERED pattern over zip|7z|rar
that derives the format from the match, so adding a format can no
longer be forgotten. \d{2,} also picks up hand-renamed .rar.01 sets.
- RAR legacy sets were ordered wrong: singles were always sorted last,
which is right for .zip/.z01 (the bare .zip is the final disk) but
wrong for .rar/.r00 (the bare .rar is volume 1). parts[0] became a
headerless continuation volume, so listing failed and the package was
labelled from the wrong message. Corrected the misleading comment too.
- Trailing/leading whitespace and trailing dots survive TDLib verbatim
and every pattern is $-anchored, so "Pack.zip " was dropped. The
filename is now normalized before matching and for baseName.
- RAR_PART now accepts .partN.exe, the self-extracting first volume;
previously the set was grouped starting at part 2.
- DOCUMENT_EXTENSIONS gained the slicer-project, 3D-model and CAD
formats present in this corpus (lys, chitubox, ctb, fbx, ztl, ... plus
the blend1 autosave sibling). Image formats stay excluded on purpose.
Tests written first; 31 new cases including pattern-order safety
(ZIP_LEGACY must not swallow .7z.001 and ARCHIVE_NUMBERED must not
swallow .z01/.r00) and both legacy part-ordering directions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Telegram soft-throttles sustained sequential getMessages calls from a
user account with growing per-call latency (no FLOOD_WAIT, so our retry
wrapper never sees it). On a large package count this made the
once-per-startup destination-message verification pass take 90+ minutes
and block the scheduler/fetch-listener from starting. Running up to 3
batches concurrently cuts wall-clock time well under real per-account
rate limits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP
detectArchive() had multipart patterns for ZIP and RAR but none for 7z's
own volume-split naming, so files like "Name.7z.001" matched nothing and
were dropped before ever reaching grouping or the skipped-package
bookkeeping — no Package row, no SkippedPackage row, no log line.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP
files/temp is TDLib's own redundant download cache — the worker
already prunes it after every ingestion run (see the recent
optimizeTdlibStorage fix), and its content still lives in the
source/destination Telegram chats regardless. With the cache grown
back to ~59GB between prune cycles, tarring it made today's backup
run for 4+ hours straight, fighting the actively-ingesting worker for
disk I/O and degrading the whole host. Excluding it keeps the backup
to what's actually irreplaceable: the DB dump and the TDLib session
state itself.
tryForwardArchiveSet only attempted the no-download forward path for
ZIP/RAR/7z archives, and bailed to download+reupload whenever the
ranged listing failed — even though the channel already grants
forwarding permission and forwarding a message costs nothing
regardless of what's inside it. Standalone DOCUMENT/STL/3MF
attachments never got a chance at the forward path at all.
Now any file on a forwarding-enabled channel is forwarded directly;
when there's no listing to derive a content hash from (non-archive
types, or ranged-listing failures), dedup falls back to
remote.unique_id identity — deriveForwardContentHash, crcFingerprint,
and compareFingerprints already degrade to this safely for
empty/incomplete entries, so there's no risk of unrelated files
colliding as false duplicates. Inner-file indexing is simply skipped
for these cases, matching the existing accepted limitation for
RAR/7z rebuild placeholders.
Several channels with allowsForwarding=true were still falling back to
download+reupload for specific RAR archives after the signature-bytes
fix, with zero log output to explain why — walkRarVolume and
readRarListingRanged return null silently on signature-detection
failure, the MAX_RAR_BLOCKS/MAX_RAR_HEADER_BYTES guards, and a
non-positive advance. Log each case so the next occurrence identifies
the actual cause instead of requiring another guess-and-redeploy cycle.
readRarListingRanged read the archive signature only to detect the RAR
version, then walkRarVolume began harvesting header regions at
pos = sigLen — the signature bytes themselves were never captured.
The reconstructed sparse file therefore started with zero bytes
instead of "Rar!", so every real unrar invocation rejected it as
"not RAR archive". This silently forced every RAR archive through the
expensive download+reupload fallback regardless of the source
channel's forwarding permission — defeating the forward-priority path
for the RAR-heavy channels it matters most for. The existing tests
didn't catch this because their assertions accepted either a null or
non-null result as passing, deferring real verification to production.
After the bot's TDLib session was rebuilt from scratch (following a
disk-full corruption on 2026-08-05), sendMessage started failing with
"Chat not found" for every previously-known user — a fresh TDLib
database has no cached chat/peer info until createPrivateChat
explicitly resolves it. Call it before every send.
TDLib keeps a permanent local copy of every file it downloads or
uploads (via inputFileLocal) with no automatic cleanup. Across the two
worker accounts this had grown to ~270GB, filling the host disk to 91%
and triggering a cascading disk-full failure in the bot's TDLib
session on 2026-08-05. Call optimizeStorage after every ingestion run
to clear it; a 5-minute immunity_delay protects files an in-flight
operation might still reference.
Task-by-task plan for the design in
docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md,
grounded against current worker.ts/provenance-backfill.ts/schema
signatures so a fresh agent can execute it without prior context.
Prioritize native Telegram forwarding over download+reupload for
source channels that allow it, reusing the ranged archive-listing
readers to keep indexing complete without a local download. Falls
back to the existing download+reupload pipeline per-channel (when
forwarding is blocked) and per-archive (when ranged listing fails).
Add MAX_RAR_HEADER_BYTES constant to prevent unbounded ranged reads when
a RAR block's HeaderSize is bogus. Real RAR block headers are far smaller;
this guards against amplification attacks on corrupt/desynced archives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prevent silent masking of short reads by validating buffer bounds before
accessing the first byte. Continuation-byte overflow was already caught,
but a short read that leaves pos at/past buffer.length would return {0, pos+1}
instead of throwing, masking the error from callers' try/catch handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live spike showed start+end sparse reconstruction is insufficient for
encoded-header 7z; fetch the mid-file packed header region as a 3rd
region (parse PackInfo). Adds Task 4b.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cheap listing without full download for RAR/7z placeholders: harvest header
regions via ranged reads, sparse-reconstruct, list with native 7z/unrar CLIs.
Full-download fallback (size-capped) for stragglers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the backup service against the Synology share over SMB/CIFS (the NAS
authenticates with a user/password; NFS is IP-allowlist only). Also fixes
three defects found bringing the service up live:
- entrypoint crash-loop: dcron's crond fails "setpgid: Operation not
permitted" in this runtime -> use busybox crond; make repo-init idempotent
(check via `restic cat config`, tolerate init-on-existing) so a transient
CIFS/lock hiccup can't kill PID 1.
- OOM: pg_dump of a ~276MB DB + tar + restic exceeded the 256M cap -> 1G.
- live tar abort: GNU tar exits 1 when TDLib files change mid-read (worker is
live); per design this is best-effort, so tolerate exit 1, fatal only >=2.
Kuma push is now optional (empty URL disables alerting) since it's deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Multipart ZIP fingerprint reads now use per-part sizes instead of the
whole-archive total, so the tail download offset stays within the last
part's bounds on both the scanned side and the destination-copy side
(scannedFileId replaced with an ordered scannedParts list). A fingerprint
comparison is now only treated as a real mismatch when both sides have
complete CRCs and differ; incomplete comparisons (e.g. empty files) fall
back to name+size confidence instead of silently refusing to backfill.
Name+size-confidence backfills now also create an INFO
INTEGRITY_AUDIT systemNotification for later review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When multiple placeholder packages share the same name+size, try to
disambiguate via ZIP fingerprint; if that can't uniquely resolve a
single match, emit a SystemNotification and skip the backfill rather
than attributing provenance to the wrong package.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild-created placeholder candidates have no PackageFile CRCs, so
name-side fingerprinting can't confirm them. When the stored candidate
fingerprint is incomplete, read the candidate's own copy from its
destination message and fingerprint against that instead of falling
straight to name+size confidence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire tryProvenanceBackfill into processOneArchiveSet: before downloading
a scanned ZIP/RAR/7Z, check whether it's the true origin of a
placeholder-provenance package in the destination channel and backfill
in place, skipping the download. Add the zipsBackfilled counter through
PipelineContext, updateRunActivity, and completeIngestionRun.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive migration (applied on deploy via prisma migrate deploy). Counts
packages whose provenance was backfilled during an ingestion run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements tryProvenanceBackfill() per Task 6 of the provenance-backfill
plan: looks up a placeholder candidate by fileName+fileSize, confirms ZIP
candidates via a ranged central-directory CRC32 fingerprint, and falls
back to name+size confidence for RAR/7z/failed listings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds findPlaceholderCandidate, getPackageFileCrcs, and
backfillProvenance to worker/src/db/queries.ts (Task 5). Candidate
predicate matches placeholder packages by source==dest or the
sourceMessageId==0 rebuild sentinel; backfillProvenance re-checks
placeholder status inside the transaction before overwriting fields.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements Task 4 Step 2 of the provenance-backfill plan. The live
spike (Step 1) and manual verification (Step 3) were not run in this
environment because a second TDLib client would corrupt the running
worker's authenticated session; the absolute-offset assumption is
noted as pending live verification on deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adminandClaude Sonnet 5