Add MAX_RAR_HEADER_BYTES constant to prevent unbounded ranged reads when
a RAR block's HeaderSize is bogus. Real RAR block headers are far smaller;
this guards against amplification attacks on corrupt/desynced archives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prevent silent masking of short reads by validating buffer bounds before
accessing the first byte. Continuation-byte overflow was already caught,
but a short read that leaves pos at/past buffer.length would return {0, pos+1}
instead of throwing, masking the error from callers' try/catch handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Multipart ZIP fingerprint reads now use per-part sizes instead of the
whole-archive total, so the tail download offset stays within the last
part's bounds on both the scanned side and the destination-copy side
(scannedFileId replaced with an ordered scannedParts list). A fingerprint
comparison is now only treated as a real mismatch when both sides have
complete CRCs and differ; incomplete comparisons (e.g. empty files) fall
back to name+size confidence instead of silently refusing to backfill.
Name+size-confidence backfills now also create an INFO
INTEGRITY_AUDIT systemNotification for later review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When multiple placeholder packages share the same name+size, try to
disambiguate via ZIP fingerprint; if that can't uniquely resolve a
single match, emit a SystemNotification and skip the backfill rather
than attributing provenance to the wrong package.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild-created placeholder candidates have no PackageFile CRCs, so
name-side fingerprinting can't confirm them. When the stored candidate
fingerprint is incomplete, read the candidate's own copy from its
destination message and fingerprint against that instead of falling
straight to name+size confidence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire tryProvenanceBackfill into processOneArchiveSet: before downloading
a scanned ZIP/RAR/7Z, check whether it's the true origin of a
placeholder-provenance package in the destination channel and backfill
in place, skipping the download. Add the zipsBackfilled counter through
PipelineContext, updateRunActivity, and completeIngestionRun.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements tryProvenanceBackfill() per Task 6 of the provenance-backfill
plan: looks up a placeholder candidate by fileName+fileSize, confirms ZIP
candidates via a ranged central-directory CRC32 fingerprint, and falls
back to name+size confidence for RAR/7z/failed listings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds findPlaceholderCandidate, getPackageFileCrcs, and
backfillProvenance to worker/src/db/queries.ts (Task 5). Candidate
predicate matches placeholder packages by source==dest or the
sourceMessageId==0 rebuild sentinel; backfillProvenance re-checks
placeholder status inside the transaction before overwriting fields.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements Task 4 Step 2 of the provenance-backfill plan. The live
spike (Step 1) and manual verification (Step 3) were not run in this
environment because a second TDLib client would corrupt the running
worker's authenticated session; the absolute-offset assumption is
noted as pending live verification on deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Render the STL view as a flat per-package list (listDisplayItems no longer
collapses packages into group rows) and hide the Ungrouped tab, now that the
creator filter organizes the list. Remove the worker's heuristic auto-grouping
passes (rule/time/pattern/creator/zip-path/reply-chain/caption); album grouping
is kept. Existing groups and manual grouping actions are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Track the forum topic currently being processed on IngestionRun
(currentTopicId + currentAccountChannelMapId; additive migration) and
expose it on the live status. processArchiveSets gains an optional
shouldStop callback polled before each archive set; the forum branch
passes a live isTopicFetchEnabled check, so disabling a topic mid-run
lets the in-flight file finish, then skips the rest of that topic.
A new disableActiveTopic server action sets the topic's fetchEnabled
false (upsert), and the worker status panel shows a "Skip & disable
this topic" button while a topic is being processed. Future runs skip
the topic via the existing live per-topic read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pre-upload integrity test is advisory (never blocks upload), but it
raised a WARNING SystemNotification whenever `7z t` failed — most often
because large 7z archives OOM-kill the test process (SIGKILL / exit 137)
in the memory-limited worker container, which is a tool limitation, not
corruption. Classify failures as encrypted | corrupt | inconclusive;
suppress notifications for inconclusive (debug log only) while still
proceeding with the upload as before. Genuine corruption now uses the
INTEGRITY_AUDIT notification type instead of the misleading HASH_MISMATCH;
encrypted archives still notify.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Eager-persist a TopicProgress row for every discovered topic at run
start, and read the live fetchEnabled flag per topic so a mid-run
disable skips topics not yet started. Disabled topics are skipped
before any TDLib scan or fetch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ensureTopicProgressRows inserts missing topic rows only (skipDuplicates),
leaving watermarks/scan-state/fetchEnabled untouched. isTopicFetchEnabled
reads the live per-topic flag for mid-run skip decisions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror of the non-forum guards from 1a4bc6f, scoped to forum topics
inside the topic loop:
- Top-of-topic-loop recency/backoff skip
- getForumTopic short-circuit after the SkippedPackage retry pass
- upsertTopicScanState for end-of-scan persistence (both the
archives-found path and the no-archives path)
Same trulyIdle definition throughout: no archives this scan, no
failures this scan, no retryable SkippedPackage rows pending. Topics
with chronic failures stay out of backoff because their counter
never increments.
For MPE specifically (1,086 forum topics), per-cycle searchChatMessages
calls drop from ~1,086 to roughly the count of topics with new
activity in the last 5 minutes — typically <50.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
For non-forum channels in runWorkerForAccount, three guards:
1. Top-of-loop recency/backoff skip — if recently scanned with no
pending work, or in backoff and not its turn, skip entirely.
Bypassed when retryable SkippedPackages exist.
2. After the SkippedPackage retry pass, a getChat short-circuit —
if TDLib's local cache says the channel's last_message.id <= our
effective watermark, skip the paginated searchChatMessages.
3. End-of-scan persists lastScannedAt + lastScanFoundArchives +
consecutiveEmptyScans via the new upsertChannelScanState helper.
trulyIdle requires: no archives, no failures, no retryable pending.
scheduler.ts exposes getCurrentCycle() so the backoff "every Nth cycle"
modulo can be applied.
Forum-topic branch lands in the next commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both read the server-side last message ID from TDLib's local cache.
Used by the channel-scan-skip guard to short-circuit a paginated
searchChatMessages when last_message.id <= our watermark.
getForumTopic uses forum_topic_id (renamed from message_thread_id in
TDLib 1.8.64, same pattern as searchChatMessages / getForumTopics).
Returns null on any failure so the caller can fall back to scanning —
we'd rather waste a scan than miss new content.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps the existing watermark write with the three new scan-state
columns from the previous commit. Single transaction, sets
lastScannedAt=NOW() server-side. Caller is responsible for computing
the trulyIdle bool and the new consecutiveEmptyScans value
(pre-increment vs reset).
Existing updateLastProcessedMessage / upsertTopicProgress are kept for
callers that don't need the new fields (the SkippedPackage retry pass,
which only adjusts the watermark).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WORKER_SKIP_RECENT_SCAN_WINDOW_MS (default 300000 = 5 min)
WORKER_EMPTY_SCAN_BACKOFF_THRESHOLD (default 5 cycles)
WORKER_EMPTY_SCAN_BACKOFF_EVERY_NTH (default 5)
All optional with safe defaults. Not yet read by any code — the worker
integration lands in follow-up commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diagnosed from production: main was rejecting almost every 7z file with
exit code 137 — kernel OOM-killing 7z t mid-test. p7zip needs to
decompress into memory to verify CRCs; ~1.5GB+ 7z archives with solid
compression exhaust the container's RAM and get SIGKILL'd.
Plus the multipart ZIP false-positive from yesterday (unzip -t can't
span .zip.001 chunks).
Both failure modes are tool limitations, not actual corruption. But
the integrity test in 04effed was a hard gate that THREW on any
non-success, blocking the upload. Result: dozens of valid archives
downloaded then thrown away over the past 6 hours.
This commit demotes the test from gate → advisory:
- Failures get logged at warn level with the actual reason
- A SystemNotification is emitted so the admin sees them in the UI
- Encrypted archives get a clearer notification title but STILL
proceed (the existing UI gives the user a way to see what's
encrypted and decide what to do)
- Upload proceeds normally — we have hash verification + archive
metadata parse for the structural integrity signals we actually
need
Multipart ZIPs are still skipped entirely (they can't be tested at
all without concatenation).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diagnosed from production: main downloaded several 28 GB ZIP sets
(CA 3D STUDIOS 2023-07.zip.001..007, 2023-08.zip.001..006, ...) and
rejected every one of them with:
"Archive integrity check failed: Command failed:
unzip -tqq /tmp/zips/.../CA 3D STUDIOS 2023-07.zip.001"
Root cause: the integrity test I added in 04effed passed `uploadPaths[0]`
to the archive tester. For byte-split multipart ZIPs (`.zip.001`,
`.zip.002`, ...), the first chunk isn't a valid ZIP on its own — the
central directory only exists at the END of the assembled archive.
unzip's spanned-ZIP support uses `.z01/.z02/.../.zip` naming, not
`.zip.001/.002`, so even pointing at the assembled-form parts wouldn't
help.
Three correctness changes:
1. Test runs on `tempPaths[0]` (the original downloaded file) instead
of `uploadPaths[0]` (which may be byte-split chunks we created).
For single-file ZIPs we re-split, this still tests the unsplit
original.
2. Skip the test entirely when archiveType=ZIP AND tempPaths.length>1
— these are source multipart ZIPs we can't validate without
concatenating, and the hash check + central-directory parse we
already do are sufficient structural signals.
3. RAR and 7Z multipart still ARE tested — `unrar t` and `7z t` both
auto-discover sibling parts when pointed at the first one.
This unblocks all multipart-ZIP ingestion for the main account. Hours
of downloaded archives that were being rejected will now pass through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After the TDLib 1.8.50 → 1.8.64 upgrade, the worker now correctly
enumerates all forum topics in MPE (1,086 of them) — a huge win. But a
data-shape mismatch was about to bite us: TDLib changed how the
General topic is identified.
TDLib 1.8.50: info.message_thread_id = 1048576 (magic constant)
TDLib 1.8.64: info.forum_topic_id = 1
Existing topic_progress rows for General carry topicId=1048576. The
worker looks up progress via `topicProgressList.find(tp => tp.topicId === topic.topicId)`,
which fails for General under the new TDLib → progress becomes null →
the scan starts from message 0.
For MPE specifically, that means re-scanning all ~378k General-topic
messages. Dedup catches the previously-ingested ones (no double upload),
but it burns hours of bandwidth before the watermark catches up.
Fix: when topicId lookup misses for a topic named "General", fall back
to a name match. The first watermark write after that saves under the
new ID (1), so future runs hit the topicId match directly without the
fallback. The orphaned 1048576 row stays as harmless dead data — we
don't delete it in case a TDLib downgrade or revert ever happens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Audit of every TDLib call site against the live 1.8.64 schema in
node_modules/@prebuilt-tdlib/types/tdlib-types.d.ts surfaced three
additional silent breakages beyond the getForumTopics fix in 106700b.
1. searchChatMessages parameter restructure
The top-level `message_thread_id` and `saved_messages_topic_id`
request fields were collapsed into a single tagged-union
`topic_id: MessageTopic$Input`. Three call sites affected:
- topics.ts getTopicMessages — was passing message_thread_id, now
sends topic_id with the messageTopicForum variant carrying
forum_topic_id. Without this the topic scan returns the whole
channel (or nothing) instead of just the topic.
- download.ts getChannelMessages — used to pass message_thread_id: 0;
just omit the topic_id field entirely for a flat scan.
- rebuild.ts — same treatment.
2. message.reply_to_message_id replaced with reply_to tagged union
On incoming messages, the flat `reply_to_message_id` field was
replaced with `reply_to: MessageReplyTo` (messageReplyToMessage or
messageReplyToStory). Our reply-chain grouping needs the message-ID
case.
Added extractReplyToMessageId() that reads both old and new shapes
so a transition build or future downgrade still works.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After the TDLib upgrade in 18a0efb, getForumTopicList returned 0 topics
for every forum channel. Confirmed in production logs:
"title":"Model Printing Emporium","topicCount":0
"title":"GB_Butler_Bot2","topicCount":0
"title":"Darnascus 2 : Flamigos Miniatures","topicCount":0
Cycle results: messagesScanned=0, zipsFound=0 — main account's entire
ingestion pipeline was a no-op because all source channels are forums.
Root cause: TDLib 1.8.64 renamed three fields without bumping the
breaking-change indicator we'd notice:
Request offset_message_thread_id → offset_forum_topic_id
Response next_offset_message_thread_id → next_offset_forum_topic_id
Response topics[].info.message_thread_id → topics[].info.forum_topic_id
The old field names became no-ops in the new TDLib, so every request
came back with an empty topic list and the "stuck pagination" detection
correctly bailed out.
Fix: send the new field name on the request side, read both old and
new names on the response side (so a future TDLib version change in
either direction stays handled).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three independent verification improvements landing together.
1. Pre-upload archive integrity test (testArchiveIntegrity)
Before sending an archive to the destination channel, runs the
appropriate CLI test:
- unzip -t for ZIP
- unrar t for RAR
- 7z t for SEVEN_Z
Catches truncated downloads, internal CRC errors, bad central
directories, and password-protected archives BEFORE we burn upload
bandwidth on a file that can't be extracted. Encrypted archives are
specifically flagged so the SkippedPackage error message is clear.
2. Post-upload destination read-back
updateMessageSendSucceeded tells us Telegram accepted the upload,
but says nothing about whether the destination message actually
contains the file we sent. After each successful upload, getMessage
each destMessageId and confirm document.size matches uploadPaths[i]'s
on-disk size.
Mismatches don't abort ingestion — they surface as
HASH_MISMATCH / UPLOAD_FAILED SystemNotifications so the admin can
see them in the UI and decide whether to recover.
3. Batched recovery (verifyMessagesBatch)
recoverIncompleteUploads previously called getMessage (singular)
per Package — at 20k packages that's 20k round-trips. Switched to
TDLib's getMessages (plural) with batch size 100 → 200 round-trips.
On 20k packages this is ~100x faster.
Per-message fallback if a whole batch errors out, so one bad batch
never loses all verification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Indexes 86k+ Lychee Slicer (.lys/.lyt), 23k+ ChituBox (.chitubox/.ctb/
.cbddlp), 1k+ Anycubic (.photon/.pwmo/.pwmx), and Bambu (.3mf)
slicer-specific files. Until now they were just generic extensions in
PackageFile.
After this commit:
- Newly-ingested packages get tags derived from their file list
("lychee", "chitubox", "anycubic", "bambu", "fdm", "mango")
- The `backfill_filelists` listener also applies tags to re-indexed
packages
- A new pure-DB listener `backfill_slicer_tags` walks existing
Packages with file lists and applies tags retroactively — no
downloads, no TDLib, takes seconds for thousands of rows.
Trigger the one-shot retroactive backfill with:
SELECT pg_notify('backfill_slicer_tags', '{"limit":5000}');
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fileName + size repost detection from ff4e150 works but has a
theoretical false-positive: two unrelated files in the same channel
with identical names and identical total sizes get treated as duplicates.
TDLib's document.remote.unique_id is a stable identifier per file
content — every repost of the exact same file across messages keeps
the same unique_id. Using it as the first dedup check eliminates the
false-positive risk entirely.
Schema:
- Package.remoteUniqueId (nullable, since existing rows lack it)
- Index on (sourceChannelId, remoteUniqueId)
Pipeline:
1. Capture remoteUniqueId in getChannelMessages + getTopicMessages
2. Pass through TelegramMessage type
3. processOneArchiveSet checks findPackageByRemoteUniqueId FIRST
(before packageExistsBySourceMessage / findRepostedPackage)
4. createPackageStub stores it on the new Package row
Existing 19,952 Packages have remoteUniqueId = NULL — they fall through
to the existing checks (source-msg-id, name+size, content-hash). New
ingestions populate it and benefit from the strong signal immediately.
Old Packages get backfilled organically when their content is
re-encountered and a new Package would otherwise be created.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12 versions of TDLib bug fixes, performance improvements, and stricter
type definitions in @prebuilt-tdlib/types.
Two API breakages handled:
1. `getChatFolders` (plural) was removed — folder IDs now arrive via
the `updateChatFolders` update event. Replaced the synchronous call
with a 200ms event listener; if no folders arrive, we proceed with
just main + archive lists. Chats inside folders are still reachable
from chatListMain so this isn't a functional regression.
2. The new tdl `Client.invoke` signature requires a literal `_` field
and rejects `Record<string, any>` shapes. Our `invokeWithTimeout`
wrapper is intentionally generic — cast through `any` at the call
site with a comment explaining why.
Both worker and bot type-check + build cleanly with the new versions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The old verifyMessageExists returned a bare boolean. Any error other
than HTTP 404 was treated as "exists" — meaning a TDLib connection
problem or transient TG hiccup at recovery time caused the worker to
declare "all destination messages verified" when it had actually
verified nothing.
Replaced with a discriminated VerifyResult:
- exists — message present and is a document, keep Package
- deleted — TG confirms it's gone (404 / MESSAGE_ID_INVALID /
"Message not found"), reset Package for re-upload
- wrong-content — message exists but isn't messageDocument, reset
- unknown — TDLib threw a non-404 error; do NOT reset, retry
next startup
Recovery summary now reports all four counts and switches to a
non-success message when unknownCount > 0, so a degraded TDLib run
doesn't hide behind a green log line.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to 0bdd4ba (RAR parser fix). 4,380 RAR packages and ~450
ZIP/7Z packages in the DB have fileCount=0 because of the old broken
parser (and a handful of edge cases). This adds an on-demand backfill
that re-indexes their file lists.
Triggered by:
SELECT pg_notify('backfill_filelists', '{"limit":50,"archiveType":"RAR"}');
Both payload fields are optional. archiveType filters to ZIP/RAR/SEVEN_Z;
default limit is 100. Multiple notifications queue sequentially so
TDLib downloads don't compete for the per-account mutex.
For each candidate:
1. Resolve destChannel.telegramId from the Package
2. getMessage for each destMessageId in destMessageIds[] (handles
multipart) to recover the file_id from Telegram
3. downloadFile (uses TDLib cache when available — most are fast)
4. Run readZipCentralDirectory / readRarContents / read7zContents
5. Transactionally replace PackageFile rows + update fileCount
Re-check of fileCount inside the transaction ensures a concurrent
backfill from another worker (or a fresh ingestion of the same archive)
doesn't get clobbered.
Prefers the Premium account when both are linked, for faster downloads
and to avoid the speed-limit throttling on the secondary.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diagnosed from production: all 4,380 RAR packages in the database have
fileCount = 0. The old parser used \`unrar l -v\` and a regex that
expected an 8-column \`Attributes Size Packed Ratio% Date Time CRC32 Name\`
output. unrar 6.21's actual \`l -v\` output is 5 columns: \`Attributes
Size Date Time Name\` — no Packed, no Ratio, no CRC32. So every RAR
silently parsed to zero entries.
Switch to \`unrar lt\` (list technical), which emits one block per file
with key:value lines:
Name: Lost Kingdom 2023 01 January/Nagas/NagaCaptainBody.stl
Type: File
Size: 22503584
Packed size: 21430123
CRC32: A1B2C3D4
...
The new parser tokenizes blocks on blank lines and matches "key: value"
lines per block. Handles multi-word keys ("Packed size", "Host OS") and
gracefully skips Directory entries and the archive header block. Also
tolerates BLAKE2sp checksums for newer RAR archives.
Verified against a live 644MB RAR with 201 entries (194 files, 7 dirs);
parser returns 194 entries with correct paths, sizes, and CRC32s.
Future RAR ingestions will populate fileCount and PackageFile rows
correctly. Backfilling existing 4,380 packages requires a separate
pass — added in a follow-up commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three connected safeguards driven by user feedback after deploying the
incremental watermark and repost-detection fixes.
1. SkippedPackage retry pass (watermark pull-back)
The auto-retry chain (d99a506 + watermark cap) only works for failures
that occur AFTER the fix is deployed. Pre-existing SkippedPackages may
sit below the current watermark — example from prod: secondary's
"Turnbase Delivery Folder.7z" at msgId 37,109,104,640 vs watermark
37,111,201,792. The auto-retry never sees it.
Before scanning each channel/topic, we now query SkippedPackages with
attemptCount < cap for that scope and pull the watermark back to
(lowestSkippedMsgId - 1n) when needed. Both forum and non-forum
branches handle this.
2. Topic scan order: specific topics first, General last
In forum channels, files often appear in both a specific topic (e.g.,
"Artisan Guild January 2022") AND in General. The first encounter
created the Package and locked in the topic context. If we happened
to scan General first, the Package recorded the less-informative
topic.
We now sort topics so General is processed last. New Packages get
the more specific topic name as their context by default.
3. Backfill specific topic on existing Packages
For Packages that were already created with General topic context,
when findRepostedPackage matches and the current scan is in a more
specific topic, update the existing Package's sourceTopicId (and
creator, if it was derived from "General") to the more specific one.
Audit log shows both old and new topic IDs.
The findRepostedPackage query also got an ORDER BY so it returns the
most-specific existing match (non-null sourceTopicId first) when
multiple Packages share the same filename + size in a channel — giving
the audit log richer context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
admin
