TDLib keeps a permanent local copy of every file it downloads or
uploads (via inputFileLocal) with no automatic cleanup. Across the two
worker accounts this had grown to ~270GB, filling the host disk to 91%
and triggering a cascading disk-full failure in the bot's TDLib
session on 2026-08-05. Call optimizeStorage after every ingestion run
to clear it; a 5-minute immunity_delay protects files an in-flight
operation might still reference.
Task-by-task plan for the design in
docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md,
grounded against current worker.ts/provenance-backfill.ts/schema
signatures so a fresh agent can execute it without prior context.
Prioritize native Telegram forwarding over download+reupload for
source channels that allow it, reusing the ranged archive-listing
readers to keep indexing complete without a local download. Falls
back to the existing download+reupload pipeline per-channel (when
forwarding is blocked) and per-archive (when ranged listing fails).
Add MAX_RAR_HEADER_BYTES constant to prevent unbounded ranged reads when
a RAR block's HeaderSize is bogus. Real RAR block headers are far smaller;
this guards against amplification attacks on corrupt/desynced archives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prevent silent masking of short reads by validating buffer bounds before
accessing the first byte. Continuation-byte overflow was already caught,
but a short read that leaves pos at/past buffer.length would return {0, pos+1}
instead of throwing, masking the error from callers' try/catch handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live spike showed start+end sparse reconstruction is insufficient for
encoded-header 7z; fetch the mid-file packed header region as a 3rd
region (parse PackInfo). Adds Task 4b.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cheap listing without full download for RAR/7z placeholders: harvest header
regions via ranged reads, sparse-reconstruct, list with native 7z/unrar CLIs.
Full-download fallback (size-capped) for stragglers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the backup service against the Synology share over SMB/CIFS (the NAS
authenticates with a user/password; NFS is IP-allowlist only). Also fixes
three defects found bringing the service up live:
- entrypoint crash-loop: dcron's crond fails "setpgid: Operation not
permitted" in this runtime -> use busybox crond; make repo-init idempotent
(check via `restic cat config`, tolerate init-on-existing) so a transient
CIFS/lock hiccup can't kill PID 1.
- OOM: pg_dump of a ~276MB DB + tar + restic exceeded the 256M cap -> 1G.
- live tar abort: GNU tar exits 1 when TDLib files change mid-read (worker is
live); per design this is best-effort, so tolerate exit 1, fatal only >=2.
Kuma push is now optional (empty URL disables alerting) since it's deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Multipart ZIP fingerprint reads now use per-part sizes instead of the
whole-archive total, so the tail download offset stays within the last
part's bounds on both the scanned side and the destination-copy side
(scannedFileId replaced with an ordered scannedParts list). A fingerprint
comparison is now only treated as a real mismatch when both sides have
complete CRCs and differ; incomplete comparisons (e.g. empty files) fall
back to name+size confidence instead of silently refusing to backfill.
Name+size-confidence backfills now also create an INFO
INTEGRITY_AUDIT systemNotification for later review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When multiple placeholder packages share the same name+size, try to
disambiguate via ZIP fingerprint; if that can't uniquely resolve a
single match, emit a SystemNotification and skip the backfill rather
than attributing provenance to the wrong package.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild-created placeholder candidates have no PackageFile CRCs, so
name-side fingerprinting can't confirm them. When the stored candidate
fingerprint is incomplete, read the candidate's own copy from its
destination message and fingerprint against that instead of falling
straight to name+size confidence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire tryProvenanceBackfill into processOneArchiveSet: before downloading
a scanned ZIP/RAR/7Z, check whether it's the true origin of a
placeholder-provenance package in the destination channel and backfill
in place, skipping the download. Add the zipsBackfilled counter through
PipelineContext, updateRunActivity, and completeIngestionRun.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive migration (applied on deploy via prisma migrate deploy). Counts
packages whose provenance was backfilled during an ingestion run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements tryProvenanceBackfill() per Task 6 of the provenance-backfill
plan: looks up a placeholder candidate by fileName+fileSize, confirms ZIP
candidates via a ranged central-directory CRC32 fingerprint, and falls
back to name+size confidence for RAR/7z/failed listings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds findPlaceholderCandidate, getPackageFileCrcs, and
backfillProvenance to worker/src/db/queries.ts (Task 5). Candidate
predicate matches placeholder packages by source==dest or the
sourceMessageId==0 rebuild sentinel; backfillProvenance re-checks
placeholder status inside the transaction before overwriting fields.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements Task 4 Step 2 of the provenance-backfill plan. The live
spike (Step 1) and manual verification (Step 3) were not run in this
environment because a second TDLib client would corrupt the running
worker's authenticated session; the absolute-offset assumption is
noted as pending live verification on deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild records use sourceMessageId=0 + synthetic 'rebuild:' contentHash and an
arbitrary fallback sourceChannelId, so the original sourceChannelId==destChannelId
candidate definition missed them. Predicate is now
(sourceChannelId==destChannelId OR sourceMessageId==0), verified against 59,893
live rebuilt records.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A JWT session pointing at a user no longer in the DB (e.g. after a DB reset)
made getUserSettings create settings for a non-existent user -> FK violation
(P2003) -> Server Component render crash. getUserSettings now returns defaults
on P2003; the (app) layout detects the missing user and redirects to a new
server-side /logout route that clears the cookie, avoiding the middleware
redirect loop that otherwise blocks reaching /login.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Render the STL view as a flat per-package list (listDisplayItems no longer
collapses packages into group rows) and hide the Ungrouped tab, now that the
creator filter organizes the list. Remove the worker's heuristic auto-grouping
passes (rule/time/pattern/creator/zip-path/reply-chain/caption); album grouping
is kept. Existing groups and manual grouping actions are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The backup.sh script hardcodes 'pg_dump -h dragonsstash-db' (matching
production), but the db service lacked a network alias for that hostname.
Added network alias to the backend network so the backup service can
successfully connect using the hardcoded hostname.
Verified: both 'db' and 'dragonsstash-db' resolve from the backend network.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wires the backup service (from Tasks 1-2) into the production compose
configuration, adds NFS-backed nas_backups volume, and documents the four
required env vars (NAS_HOST, NAS_EXPORT_PATH, RESTIC_PASSWORD, KUMA_PUSH_URL, TZ)
in .env.example.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously the dump/tar cleanup only ran after restic forget succeeded,
so a plaintext Postgres dump could be left behind in /tmp if tar or
restic failed partway through. Add an EXIT trap that unconditionally
removes both temp files on any exit path, and drop the now-redundant
explicit rm -f from the success path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Six tasks: backup image, backup.sh, repo compose wiring, CI build
step, production compose wiring (gated on NAS/Kuma details from the
user), deploy + restore-drill verification.
admin
