The .7z.001 gap survived for months because an unmatched attachment left
zero trace anywhere. All three isArchiveAttachment call sites now emit a
debug line with the chat, message id and filename when a document is
skipped, so future gaps of this class are greppable instead of invisible.
Debug level because every non-archive message in a channel hits this.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Files whose names match no pattern in detect.ts are dropped with no log,
no packages row and no skipped_packages row. Five gaps of that class:
- Numbered volumes were implemented format-by-format (ZIP_NUMBERED,
SEVENZ_NUMBERED) with no RAR equivalent, so Pack.rar.001 vanished.
Replaced both with a single ARCHIVE_NUMBERED pattern over zip|7z|rar
that derives the format from the match, so adding a format can no
longer be forgotten. \d{2,} also picks up hand-renamed .rar.01 sets.
- RAR legacy sets were ordered wrong: singles were always sorted last,
which is right for .zip/.z01 (the bare .zip is the final disk) but
wrong for .rar/.r00 (the bare .rar is volume 1). parts[0] became a
headerless continuation volume, so listing failed and the package was
labelled from the wrong message. Corrected the misleading comment too.
- Trailing/leading whitespace and trailing dots survive TDLib verbatim
and every pattern is $-anchored, so "Pack.zip " was dropped. The
filename is now normalized before matching and for baseName.
- RAR_PART now accepts .partN.exe, the self-extracting first volume;
previously the set was grouped starting at part 2.
- DOCUMENT_EXTENSIONS gained the slicer-project, 3D-model and CAD
formats present in this corpus (lys, chitubox, ctb, fbx, ztl, ... plus
the blend1 autosave sibling). Image formats stay excluded on purpose.
Tests written first; 31 new cases including pattern-order safety
(ZIP_LEGACY must not swallow .7z.001 and ARCHIVE_NUMBERED must not
swallow .z01/.r00) and both legacy part-ordering directions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Telegram soft-throttles sustained sequential getMessages calls from a
user account with growing per-call latency (no FLOOD_WAIT, so our retry
wrapper never sees it). On a large package count this made the
once-per-startup destination-message verification pass take 90+ minutes
and block the scheduler/fetch-listener from starting. Running up to 3
batches concurrently cuts wall-clock time well under real per-account
rate limits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP
detectArchive() had multipart patterns for ZIP and RAR but none for 7z's
own volume-split naming, so files like "Name.7z.001" matched nothing and
were dropped before ever reaching grouping or the skipped-package
bookkeeping — no Package row, no SkippedPackage row, no log line.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP
files/temp is TDLib's own redundant download cache — the worker
already prunes it after every ingestion run (see the recent
optimizeTdlibStorage fix), and its content still lives in the
source/destination Telegram chats regardless. With the cache grown
back to ~59GB between prune cycles, tarring it made today's backup
run for 4+ hours straight, fighting the actively-ingesting worker for
disk I/O and degrading the whole host. Excluding it keeps the backup
to what's actually irreplaceable: the DB dump and the TDLib session
state itself.
tryForwardArchiveSet only attempted the no-download forward path for
ZIP/RAR/7z archives, and bailed to download+reupload whenever the
ranged listing failed — even though the channel already grants
forwarding permission and forwarding a message costs nothing
regardless of what's inside it. Standalone DOCUMENT/STL/3MF
attachments never got a chance at the forward path at all.
Now any file on a forwarding-enabled channel is forwarded directly;
when there's no listing to derive a content hash from (non-archive
types, or ranged-listing failures), dedup falls back to
remote.unique_id identity — deriveForwardContentHash, crcFingerprint,
and compareFingerprints already degrade to this safely for
empty/incomplete entries, so there's no risk of unrelated files
colliding as false duplicates. Inner-file indexing is simply skipped
for these cases, matching the existing accepted limitation for
RAR/7z rebuild placeholders.
Several channels with allowsForwarding=true were still falling back to
download+reupload for specific RAR archives after the signature-bytes
fix, with zero log output to explain why — walkRarVolume and
readRarListingRanged return null silently on signature-detection
failure, the MAX_RAR_BLOCKS/MAX_RAR_HEADER_BYTES guards, and a
non-positive advance. Log each case so the next occurrence identifies
the actual cause instead of requiring another guess-and-redeploy cycle.
readRarListingRanged read the archive signature only to detect the RAR
version, then walkRarVolume began harvesting header regions at
pos = sigLen — the signature bytes themselves were never captured.
The reconstructed sparse file therefore started with zero bytes
instead of "Rar!", so every real unrar invocation rejected it as
"not RAR archive". This silently forced every RAR archive through the
expensive download+reupload fallback regardless of the source
channel's forwarding permission — defeating the forward-priority path
for the RAR-heavy channels it matters most for. The existing tests
didn't catch this because their assertions accepted either a null or
non-null result as passing, deferring real verification to production.
After the bot's TDLib session was rebuilt from scratch (following a
disk-full corruption on 2026-08-05), sendMessage started failing with
"Chat not found" for every previously-known user — a fresh TDLib
database has no cached chat/peer info until createPrivateChat
explicitly resolves it. Call it before every send.
TDLib keeps a permanent local copy of every file it downloads or
uploads (via inputFileLocal) with no automatic cleanup. Across the two
worker accounts this had grown to ~270GB, filling the host disk to 91%
and triggering a cascading disk-full failure in the bot's TDLib
session on 2026-08-05. Call optimizeStorage after every ingestion run
to clear it; a 5-minute immunity_delay protects files an in-flight
operation might still reference.
Task-by-task plan for the design in
docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md,
grounded against current worker.ts/provenance-backfill.ts/schema
signatures so a fresh agent can execute it without prior context.
Prioritize native Telegram forwarding over download+reupload for
source channels that allow it, reusing the ranged archive-listing
readers to keep indexing complete without a local download. Falls
back to the existing download+reupload pipeline per-channel (when
forwarding is blocked) and per-archive (when ranged listing fails).
Add MAX_RAR_HEADER_BYTES constant to prevent unbounded ranged reads when
a RAR block's HeaderSize is bogus. Real RAR block headers are far smaller;
this guards against amplification attacks on corrupt/desynced archives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prevent silent masking of short reads by validating buffer bounds before
accessing the first byte. Continuation-byte overflow was already caught,
but a short read that leaves pos at/past buffer.length would return {0, pos+1}
instead of throwing, masking the error from callers' try/catch handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live spike showed start+end sparse reconstruction is insufficient for
encoded-header 7z; fetch the mid-file packed header region as a 3rd
region (parse PackInfo). Adds Task 4b.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cheap listing without full download for RAR/7z placeholders: harvest header
regions via ranged reads, sparse-reconstruct, list with native 7z/unrar CLIs.
Full-download fallback (size-capped) for stragglers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the backup service against the Synology share over SMB/CIFS (the NAS
authenticates with a user/password; NFS is IP-allowlist only). Also fixes
three defects found bringing the service up live:
- entrypoint crash-loop: dcron's crond fails "setpgid: Operation not
permitted" in this runtime -> use busybox crond; make repo-init idempotent
(check via `restic cat config`, tolerate init-on-existing) so a transient
CIFS/lock hiccup can't kill PID 1.
- OOM: pg_dump of a ~276MB DB + tar + restic exceeded the 256M cap -> 1G.
- live tar abort: GNU tar exits 1 when TDLib files change mid-read (worker is
live); per design this is best-effort, so tolerate exit 1, fatal only >=2.
Kuma push is now optional (empty URL disables alerting) since it's deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Multipart ZIP fingerprint reads now use per-part sizes instead of the
whole-archive total, so the tail download offset stays within the last
part's bounds on both the scanned side and the destination-copy side
(scannedFileId replaced with an ordered scannedParts list). A fingerprint
comparison is now only treated as a real mismatch when both sides have
complete CRCs and differ; incomplete comparisons (e.g. empty files) fall
back to name+size confidence instead of silently refusing to backfill.
Name+size-confidence backfills now also create an INFO
INTEGRITY_AUDIT systemNotification for later review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When multiple placeholder packages share the same name+size, try to
disambiguate via ZIP fingerprint; if that can't uniquely resolve a
single match, emit a SystemNotification and skip the backfill rather
than attributing provenance to the wrong package.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild-created placeholder candidates have no PackageFile CRCs, so
name-side fingerprinting can't confirm them. When the stored candidate
fingerprint is incomplete, read the candidate's own copy from its
destination message and fingerprint against that instead of falling
straight to name+size confidence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire tryProvenanceBackfill into processOneArchiveSet: before downloading
a scanned ZIP/RAR/7Z, check whether it's the true origin of a
placeholder-provenance package in the destination channel and backfill
in place, skipping the download. Add the zipsBackfilled counter through
PipelineContext, updateRunActivity, and completeIngestionRun.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive migration (applied on deploy via prisma migrate deploy). Counts
packages whose provenance was backfilled during an ingestion run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements tryProvenanceBackfill() per Task 6 of the provenance-backfill
plan: looks up a placeholder candidate by fileName+fileSize, confirms ZIP
candidates via a ranged central-directory CRC32 fingerprint, and falls
back to name+size confidence for RAR/7z/failed listings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds findPlaceholderCandidate, getPackageFileCrcs, and
backfillProvenance to worker/src/db/queries.ts (Task 5). Candidate
predicate matches placeholder packages by source==dest or the
sourceMessageId==0 rebuild sentinel; backfillProvenance re-checks
placeholder status inside the transaction before overwriting fields.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements Task 4 Step 2 of the provenance-backfill plan. The live
spike (Step 1) and manual verification (Step 3) were not run in this
environment because a second TDLib client would corrupt the running
worker's authenticated session; the absolute-offset assumption is
noted as pending live verification on deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild records use sourceMessageId=0 + synthetic 'rebuild:' contentHash and an
arbitrary fallback sourceChannelId, so the original sourceChannelId==destChannelId
candidate definition missed them. Predicate is now
(sourceChannelId==destChannelId OR sourceMessageId==0), verified against 59,893
live rebuilt records.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A JWT session pointing at a user no longer in the DB (e.g. after a DB reset)
made getUserSettings create settings for a non-existent user -> FK violation
(P2003) -> Server Component render crash. getUserSettings now returns defaults
on P2003; the (app) layout detects the missing user and redirects to a new
server-side /logout route that clears the cookie, avoiding the middleware
redirect loop that otherwise blocks reaching /login.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Render the STL view as a flat per-package list (listDisplayItems no longer
collapses packages into group rows) and hide the Ungrouped tab, now that the
creator filter organizes the list. Remove the worker's heuristic auto-grouping
passes (rule/time/pattern/creator/zip-path/reply-chain/caption); album grouping
is kept. Existing groups and manual grouping actions are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The backup.sh script hardcodes 'pg_dump -h dragonsstash-db' (matching
production), but the db service lacked a network alias for that hostname.
Added network alias to the backend network so the backup service can
successfully connect using the hardcoded hostname.
Verified: both 'db' and 'dragonsstash-db' resolve from the backend network.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wires the backup service (from Tasks 1-2) into the production compose
configuration, adds NFS-backed nas_backups volume, and documents the four
required env vars (NAS_HOST, NAS_EXPORT_PATH, RESTIC_PASSWORD, KUMA_PUSH_URL, TZ)
in .env.example.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously the dump/tar cleanup only ran after restic forget succeeded,
so a plaintext Postgres dump could be left behind in /tmp if tar or
restic failed partway through. Add an EXIT trap that unconditionally
removes both temp files on any exit path, and drop the now-redundant
explicit rm -f from the success path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Six tasks: backup image, backup.sh, repo compose wiring, CI build
step, production compose wiring (gated on NAS/Kuma details from the
user), deploy + restore-drill verification.
Align the approved backup design and implementation plan on the monthly full Restic integrity check and disposable restore rehearsal without adding production code.
Constraint: PostgreSQL logical dump plus tdlib_state and tdlib_bot_state only
Rejected: Add a second production timer or backup script changes | outside requested docs-only scope
Confidence: high
Scope-risk: narrow
The STLs list queries selected the full previewData JPEG bytes (avg
~700KB, up to 2MB, and every group member's preview) only to compute a
hasPreview boolean. Under concurrent page loads this piled blobs into
the Node heap, exceeded the 512MB container limit, crashed the process,
and surfaced as repeated connection errors while browsing.
Replace the byte-loading select with a fetchPreviewFlags() helper that
checks `previewData IS NOT NULL` in SQL (IDs only, no bytes). Applied to
listPackages, listDisplayItems, searchPackages, and listUngroupedPackages.
Verified: same concurrent load that drove memory to 508/512MB and forced
a restart now peaks at 156MB (30%) with zero failed requests and no restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BigInt("") / BigInt("0") don't throw; guard against them since the
action is exported and admin-callable independent of the UI button.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Track the forum topic currently being processed on IngestionRun
(currentTopicId + currentAccountChannelMapId; additive migration) and
expose it on the live status. processArchiveSets gains an optional
shouldStop callback polled before each archive set; the forum branch
passes a live isTopicFetchEnabled check, so disabling a topic mid-run
lets the in-flight file finish, then skips the rest of that topic.
A new disableActiveTopic server action sets the topic's fetchEnabled
false (upsert), and the worker status panel shows a "Skip & disable
this topic" button while a topic is being processed. Future runs skip
the topic via the existing live per-topic read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pre-upload integrity test is advisory (never blocks upload), but it
raised a WARNING SystemNotification whenever `7z t` failed — most often
because large 7z archives OOM-kill the test process (SIGKILL / exit 137)
in the memory-limited worker container, which is a tool limitation, not
corruption. Classify failures as encrypted | corrupt | inconclusive;
suppress notifications for inconclusive (debug log only) while still
proceeding with the upload as before. Genuine corruption now uses the
INTEGRITY_AUDIT notification type instead of the misleading HASH_MISMATCH;
encrypted archives still notify.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Radix ScrollArea viewport did not get a bounded height inside the
flex-column, max-h, vertically-centred dialog, so a long topic list
overflowed the dialog instead of scrolling (last topics unreachable).
Switch to a native flex-1 min-h-0 overflow-y-auto container, which is
the canonical touch-friendly flex-scroll pattern.
Also add a "Filter topics..." input (matches by topic title) so any
topic can be reached regardless of list length; reset the filter on
close; and add a toast fallback message on toggle failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each loaded candidate gets an enlarge button that opens the ImageLightbox
without changing selection, so the right preview is easier to choose.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The drawer preview image opens the ImageLightbox; replacing the image
stays available via the existing Upload/Pick Preview buttons. No-preview
upload affordance unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Package preview thumbnails open the ImageLightbox on click (hover shows a
maximize affordance). No-preview cells unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Radix Dialog-based lightbox (Esc / overlay / close button to dismiss),
image shown object-contain capped to the viewport. No new dependencies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forum source channels get a Topics menu item that opens the TopicsDrawer.
Non-forum channels are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lists a channel's topics with an enable/disable Switch each (optimistic,
revert on failure). Empty state explains topics appear after the next scan.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Admin-guarded toggle of TopicProgress.fetchEnabled by row id. Persists
immediately; the worker honours it on its next live per-topic read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Returns per-topic fetch state for a channel, mirroring the existing
account-links route auth pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
adminandClaude Sonnet 5