The 402c317 reader fix stops new spanned sets (.z01 … .zip) being indexed
with an empty file list, but leaves 194 pre-existing packages with
fileCount = 0. Nothing could repair them: the backfill re-downloaded full
archive bytes (~944GB for these), could only be scoped to "every empty
package of type X" (4,330 packages / 5.4TB for ZIP), and 154 of the 194
have an empty destMessageIds array — so it fell back to [destMessageId],
which is the first uploaded part. A lone .z01 has no central directory,
so those could never list no matter how much was downloaded.
Three changes address that:
Scoping (backfill-scope.ts). The payload now takes packageIds and/or a
restricted fileNameLike pattern, and a request with no narrowing selector
is rejected rather than defaulted into a full sweep — omitting a field can
only narrow the job or fail it. The unscoped sweep still exists but has to
ask for itself via allowBroadSweep. Unknown fields are an error too, so a
typo'd selector can't silently widen the scope.
Ranged-first reading (archive/listing-plan.ts). A file list lives in tens
of kilobytes of an archive's header or tail, so the repair reads it with
readScannedListingRanged and only falls back to downloadFile when ranged
reading genuinely cannot work — never, when rangedOnly is set. The route
taken is logged per package so the cost is visible rather than inferred.
Ranged reads go through downloadFileRange, which is already FLOOD_WAIT
aware, and the batch still runs under the account's TDLib mutex.
The planner also refuses the cases no reader can serve. When a source
volume exceeded the upload cap, worker.ts concatenated every volume and
re-split it into <base>.concat.NNN. For a byte split that round-trips
losslessly, but a concatenation of spanned ZIP or RAR volumes is not a
valid archive in any format — such a destination copy is permanently
unlistable, and it is skipped with that reason instead of spending API
calls failing.
destMessageIds recovery (dest-index.ts, tdlib/chat-documents.ts). The
destination-channel paging is lifted out of rebuild.ts and shared, so
there is one scanner rather than a third variant. It now returns every
document and leaves filtering to callers, because a .concat.NNN chunk
matches no archive pattern — with the old filter a repacked package was
indistinguishable from one whose messages had been deleted. One scan per
batch recovers the complete ordered part set for every candidate, and its
fileIds and sizes remove the per-part getMessage as a side effect. A
recovered set is persisted only when its part count matches the package:
the channel can hold two uploads sharing a base name, which groupArchiveSets
merges, and writing that back would hand the bot a mix of two archives. A
package whose volumes cannot be corroborated is left untouched and logged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A `.z01`/`.z02`/…/`.zip` set is a spanned (multi-disk) archive, not one ZIP
file cut into chunks. Both ZIP listing paths assumed the latter:
- readZipCentralDirectory fed the parts to yauzl through a concatenating
random-access reader; yauzl hard-refuses nonzero disk numbers
("multi-disk zip files are not supported"), and the EOCD's directory
pointer is volume-relative anyway.
- readScannedZipListing (the ranged, no-download path) passed a
whole-archive tailStart, making the computed directory offset wildly
negative so the parser threw RangeError on every tail size.
Both failures were swallowed upstream, so every spanned set was ingested and
uploaded with an empty file list — invisible to content and keyword search.
Add a volume-aware central-directory reader: locate the EOCD in the final
volume, resolve the directory's (volume, offset) via the disk fields, and read
just the directory bytes, spilling across volumes if it straddles a boundary.
ZIP64 spanned archives are handled through the ZIP64 EOCD locator. The
existing hand-rolled walker in central-directory.ts is reused rather than
adding a third parser.
The two shapes are told apart by filename shape, not by the detector's
multipart `pattern`, so this stays independent of how detect.ts labels them.
A set named like volumes whose EOCD reports a single disk really is a byte
split, and falls back to the concatenating reader.
Byte-split (.zip.001) and single-.zip reading are unchanged; both are now
covered by regression tests. Tests build spanned archives byte-by-byte and
cross-check against real Info-ZIP `zip -s` output (including `-fz` ZIP64)
where the CLI is available.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The .7z.001 gap survived for months because an unmatched attachment left
zero trace anywhere. All three isArchiveAttachment call sites now emit a
debug line with the chat, message id and filename when a document is
skipped, so future gaps of this class are greppable instead of invisible.
Debug level because every non-archive message in a channel hits this.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Files whose names match no pattern in detect.ts are dropped with no log,
no packages row and no skipped_packages row. Five gaps of that class:
- Numbered volumes were implemented format-by-format (ZIP_NUMBERED,
SEVENZ_NUMBERED) with no RAR equivalent, so Pack.rar.001 vanished.
Replaced both with a single ARCHIVE_NUMBERED pattern over zip|7z|rar
that derives the format from the match, so adding a format can no
longer be forgotten. \d{2,} also picks up hand-renamed .rar.01 sets.
- RAR legacy sets were ordered wrong: singles were always sorted last,
which is right for .zip/.z01 (the bare .zip is the final disk) but
wrong for .rar/.r00 (the bare .rar is volume 1). parts[0] became a
headerless continuation volume, so listing failed and the package was
labelled from the wrong message. Corrected the misleading comment too.
- Trailing/leading whitespace and trailing dots survive TDLib verbatim
and every pattern is $-anchored, so "Pack.zip " was dropped. The
filename is now normalized before matching and for baseName.
- RAR_PART now accepts .partN.exe, the self-extracting first volume;
previously the set was grouped starting at part 2.
- DOCUMENT_EXTENSIONS gained the slicer-project, 3D-model and CAD
formats present in this corpus (lys, chitubox, ctb, fbx, ztl, ... plus
the blend1 autosave sibling). Image formats stay excluded on purpose.
Tests written first; 31 new cases including pattern-order safety
(ZIP_LEGACY must not swallow .7z.001 and ARCHIVE_NUMBERED must not
swallow .z01/.r00) and both legacy part-ordering directions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Telegram soft-throttles sustained sequential getMessages calls from a
user account with growing per-call latency (no FLOOD_WAIT, so our retry
wrapper never sees it). On a large package count this made the
once-per-startup destination-message verification pass take 90+ minutes
and block the scheduler/fetch-listener from starting. Running up to 3
batches concurrently cuts wall-clock time well under real per-account
rate limits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP
detectArchive() had multipart patterns for ZIP and RAR but none for 7z's
own volume-split naming, so files like "Name.7z.001" matched nothing and
were dropped before ever reaching grouping or the skipped-package
bookkeeping — no Package row, no SkippedPackage row, no log line.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP
tryForwardArchiveSet only attempted the no-download forward path for
ZIP/RAR/7z archives, and bailed to download+reupload whenever the
ranged listing failed — even though the channel already grants
forwarding permission and forwarding a message costs nothing
regardless of what's inside it. Standalone DOCUMENT/STL/3MF
attachments never got a chance at the forward path at all.
Now any file on a forwarding-enabled channel is forwarded directly;
when there's no listing to derive a content hash from (non-archive
types, or ranged-listing failures), dedup falls back to
remote.unique_id identity — deriveForwardContentHash, crcFingerprint,
and compareFingerprints already degrade to this safely for
empty/incomplete entries, so there's no risk of unrelated files
colliding as false duplicates. Inner-file indexing is simply skipped
for these cases, matching the existing accepted limitation for
RAR/7z rebuild placeholders.
Several channels with allowsForwarding=true were still falling back to
download+reupload for specific RAR archives after the signature-bytes
fix, with zero log output to explain why — walkRarVolume and
readRarListingRanged return null silently on signature-detection
failure, the MAX_RAR_BLOCKS/MAX_RAR_HEADER_BYTES guards, and a
non-positive advance. Log each case so the next occurrence identifies
the actual cause instead of requiring another guess-and-redeploy cycle.
readRarListingRanged read the archive signature only to detect the RAR
version, then walkRarVolume began harvesting header regions at
pos = sigLen — the signature bytes themselves were never captured.
The reconstructed sparse file therefore started with zero bytes
instead of "Rar!", so every real unrar invocation rejected it as
"not RAR archive". This silently forced every RAR archive through the
expensive download+reupload fallback regardless of the source
channel's forwarding permission — defeating the forward-priority path
for the RAR-heavy channels it matters most for. The existing tests
didn't catch this because their assertions accepted either a null or
non-null result as passing, deferring real verification to production.
TDLib keeps a permanent local copy of every file it downloads or
uploads (via inputFileLocal) with no automatic cleanup. Across the two
worker accounts this had grown to ~270GB, filling the host disk to 91%
and triggering a cascading disk-full failure in the bot's TDLib
session on 2026-08-05. Call optimizeStorage after every ingestion run
to clear it; a 5-minute immunity_delay protects files an in-flight
operation might still reference.
Add MAX_RAR_HEADER_BYTES constant to prevent unbounded ranged reads when
a RAR block's HeaderSize is bogus. Real RAR block headers are far smaller;
this guards against amplification attacks on corrupt/desynced archives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prevent silent masking of short reads by validating buffer bounds before
accessing the first byte. Continuation-byte overflow was already caught,
but a short read that leaves pos at/past buffer.length would return {0, pos+1}
instead of throwing, masking the error from callers' try/catch handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Multipart ZIP fingerprint reads now use per-part sizes instead of the
whole-archive total, so the tail download offset stays within the last
part's bounds on both the scanned side and the destination-copy side
(scannedFileId replaced with an ordered scannedParts list). A fingerprint
comparison is now only treated as a real mismatch when both sides have
complete CRCs and differ; incomplete comparisons (e.g. empty files) fall
back to name+size confidence instead of silently refusing to backfill.
Name+size-confidence backfills now also create an INFO
INTEGRITY_AUDIT systemNotification for later review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When multiple placeholder packages share the same name+size, try to
disambiguate via ZIP fingerprint; if that can't uniquely resolve a
single match, emit a SystemNotification and skip the backfill rather
than attributing provenance to the wrong package.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild-created placeholder candidates have no PackageFile CRCs, so
name-side fingerprinting can't confirm them. When the stored candidate
fingerprint is incomplete, read the candidate's own copy from its
destination message and fingerprint against that instead of falling
straight to name+size confidence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire tryProvenanceBackfill into processOneArchiveSet: before downloading
a scanned ZIP/RAR/7Z, check whether it's the true origin of a
placeholder-provenance package in the destination channel and backfill
in place, skipping the download. Add the zipsBackfilled counter through
PipelineContext, updateRunActivity, and completeIngestionRun.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements tryProvenanceBackfill() per Task 6 of the provenance-backfill
plan: looks up a placeholder candidate by fileName+fileSize, confirms ZIP
candidates via a ranged central-directory CRC32 fingerprint, and falls
back to name+size confidence for RAR/7z/failed listings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds findPlaceholderCandidate, getPackageFileCrcs, and
backfillProvenance to worker/src/db/queries.ts (Task 5). Candidate
predicate matches placeholder packages by source==dest or the
sourceMessageId==0 rebuild sentinel; backfillProvenance re-checks
placeholder status inside the transaction before overwriting fields.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements Task 4 Step 2 of the provenance-backfill plan. The live
spike (Step 1) and manual verification (Step 3) were not run in this
environment because a second TDLib client would corrupt the running
worker's authenticated session; the absolute-offset assumption is
noted as pending live verification on deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Render the STL view as a flat per-package list (listDisplayItems no longer
collapses packages into group rows) and hide the Ungrouped tab, now that the
creator filter organizes the list. Remove the worker's heuristic auto-grouping
passes (rule/time/pattern/creator/zip-path/reply-chain/caption); album grouping
is kept. Existing groups and manual grouping actions are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Track the forum topic currently being processed on IngestionRun
(currentTopicId + currentAccountChannelMapId; additive migration) and
expose it on the live status. processArchiveSets gains an optional
shouldStop callback polled before each archive set; the forum branch
passes a live isTopicFetchEnabled check, so disabling a topic mid-run
lets the in-flight file finish, then skips the rest of that topic.
A new disableActiveTopic server action sets the topic's fetchEnabled
false (upsert), and the worker status panel shows a "Skip & disable
this topic" button while a topic is being processed. Future runs skip
the topic via the existing live per-topic read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pre-upload integrity test is advisory (never blocks upload), but it
raised a WARNING SystemNotification whenever `7z t` failed — most often
because large 7z archives OOM-kill the test process (SIGKILL / exit 137)
in the memory-limited worker container, which is a tool limitation, not
corruption. Classify failures as encrypted | corrupt | inconclusive;
suppress notifications for inconclusive (debug log only) while still
proceeding with the upload as before. Genuine corruption now uses the
INTEGRITY_AUDIT notification type instead of the misleading HASH_MISMATCH;
encrypted archives still notify.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Eager-persist a TopicProgress row for every discovered topic at run
start, and read the live fetchEnabled flag per topic so a mid-run
disable skips topics not yet started. Disabled topics are skipped
before any TDLib scan or fetch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ensureTopicProgressRows inserts missing topic rows only (skipDuplicates),
leaving watermarks/scan-state/fetchEnabled untouched. isTopicFetchEnabled
reads the live per-topic flag for mid-run skip decisions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror of the non-forum guards from 1a4bc6f, scoped to forum topics
inside the topic loop:
- Top-of-topic-loop recency/backoff skip
- getForumTopic short-circuit after the SkippedPackage retry pass
- upsertTopicScanState for end-of-scan persistence (both the
archives-found path and the no-archives path)
Same trulyIdle definition throughout: no archives this scan, no
failures this scan, no retryable SkippedPackage rows pending. Topics
with chronic failures stay out of backoff because their counter
never increments.
For MPE specifically (1,086 forum topics), per-cycle searchChatMessages
calls drop from ~1,086 to roughly the count of topics with new
activity in the last 5 minutes — typically <50.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
For non-forum channels in runWorkerForAccount, three guards:
1. Top-of-loop recency/backoff skip — if recently scanned with no
pending work, or in backoff and not its turn, skip entirely.
Bypassed when retryable SkippedPackages exist.
2. After the SkippedPackage retry pass, a getChat short-circuit —
if TDLib's local cache says the channel's last_message.id <= our
effective watermark, skip the paginated searchChatMessages.
3. End-of-scan persists lastScannedAt + lastScanFoundArchives +
consecutiveEmptyScans via the new upsertChannelScanState helper.
trulyIdle requires: no archives, no failures, no retryable pending.
scheduler.ts exposes getCurrentCycle() so the backoff "every Nth cycle"
modulo can be applied.
Forum-topic branch lands in the next commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both read the server-side last message ID from TDLib's local cache.
Used by the channel-scan-skip guard to short-circuit a paginated
searchChatMessages when last_message.id <= our watermark.
getForumTopic uses forum_topic_id (renamed from message_thread_id in
TDLib 1.8.64, same pattern as searchChatMessages / getForumTopics).
Returns null on any failure so the caller can fall back to scanning —
we'd rather waste a scan than miss new content.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps the existing watermark write with the three new scan-state
columns from the previous commit. Single transaction, sets
lastScannedAt=NOW() server-side. Caller is responsible for computing
the trulyIdle bool and the new consecutiveEmptyScans value
(pre-increment vs reset).
Existing updateLastProcessedMessage / upsertTopicProgress are kept for
callers that don't need the new fields (the SkippedPackage retry pass,
which only adjusts the watermark).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WORKER_SKIP_RECENT_SCAN_WINDOW_MS (default 300000 = 5 min)
WORKER_EMPTY_SCAN_BACKOFF_THRESHOLD (default 5 cycles)
WORKER_EMPTY_SCAN_BACKOFF_EVERY_NTH (default 5)
All optional with safe defaults. Not yet read by any code — the worker
integration lands in follow-up commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diagnosed from production: main was rejecting almost every 7z file with
exit code 137 — kernel OOM-killing 7z t mid-test. p7zip needs to
decompress into memory to verify CRCs; ~1.5GB+ 7z archives with solid
compression exhaust the container's RAM and get SIGKILL'd.
Plus the multipart ZIP false-positive from yesterday (unzip -t can't
span .zip.001 chunks).
Both failure modes are tool limitations, not actual corruption. But
the integrity test in 04effed was a hard gate that THREW on any
non-success, blocking the upload. Result: dozens of valid archives
downloaded then thrown away over the past 6 hours.
This commit demotes the test from gate → advisory:
- Failures get logged at warn level with the actual reason
- A SystemNotification is emitted so the admin sees them in the UI
- Encrypted archives get a clearer notification title but STILL
proceed (the existing UI gives the user a way to see what's
encrypted and decide what to do)
- Upload proceeds normally — we have hash verification + archive
metadata parse for the structural integrity signals we actually
need
Multipart ZIPs are still skipped entirely (they can't be tested at
all without concatenation).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
adminandClaude Sonnet 5