Adds findPlaceholderCandidate, getPackageFileCrcs, and
backfillProvenance to worker/src/db/queries.ts (Task 5). Candidate
predicate matches placeholder packages by source==dest or the
sourceMessageId==0 rebuild sentinel; backfillProvenance re-checks
placeholder status inside the transaction before overwriting fields.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements Task 4 Step 2 of the provenance-backfill plan. The live
spike (Step 1) and manual verification (Step 3) were not run in this
environment because a second TDLib client would corrupt the running
worker's authenticated session; the absolute-offset assumption is
noted as pending live verification on deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild records use sourceMessageId=0 + synthetic 'rebuild:' contentHash and an
arbitrary fallback sourceChannelId, so the original sourceChannelId==destChannelId
candidate definition missed them. Predicate is now
(sourceChannelId==destChannelId OR sourceMessageId==0), verified against 59,893
live rebuilt records.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A JWT session pointing at a user no longer in the DB (e.g. after a DB reset)
made getUserSettings create settings for a non-existent user -> FK violation
(P2003) -> Server Component render crash. getUserSettings now returns defaults
on P2003; the (app) layout detects the missing user and redirects to a new
server-side /logout route that clears the cookie, avoiding the middleware
redirect loop that otherwise blocks reaching /login.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Render the STL view as a flat per-package list (listDisplayItems no longer
collapses packages into group rows) and hide the Ungrouped tab, now that the
creator filter organizes the list. Remove the worker's heuristic auto-grouping
passes (rule/time/pattern/creator/zip-path/reply-chain/caption); album grouping
is kept. Existing groups and manual grouping actions are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The backup.sh script hardcodes 'pg_dump -h dragonsstash-db' (matching
production), but the db service lacked a network alias for that hostname.
Added network alias to the backend network so the backup service can
successfully connect using the hardcoded hostname.
Verified: both 'db' and 'dragonsstash-db' resolve from the backend network.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wires the backup service (from Tasks 1-2) into the production compose
configuration, adds NFS-backed nas_backups volume, and documents the four
required env vars (NAS_HOST, NAS_EXPORT_PATH, RESTIC_PASSWORD, KUMA_PUSH_URL, TZ)
in .env.example.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously the dump/tar cleanup only ran after restic forget succeeded,
so a plaintext Postgres dump could be left behind in /tmp if tar or
restic failed partway through. Add an EXIT trap that unconditionally
removes both temp files on any exit path, and drop the now-redundant
explicit rm -f from the success path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Six tasks: backup image, backup.sh, repo compose wiring, CI build
step, production compose wiring (gated on NAS/Kuma details from the
user), deploy + restore-drill verification.
The STLs list queries selected the full previewData JPEG bytes (avg
~700KB, up to 2MB, and every group member's preview) only to compute a
hasPreview boolean. Under concurrent page loads this piled blobs into
the Node heap, exceeded the 512MB container limit, crashed the process,
and surfaced as repeated connection errors while browsing.
Replace the byte-loading select with a fetchPreviewFlags() helper that
checks `previewData IS NOT NULL` in SQL (IDs only, no bytes). Applied to
listPackages, listDisplayItems, searchPackages, and listUngroupedPackages.
Verified: same concurrent load that drove memory to 508/512MB and forced
a restart now peaks at 156MB (30%) with zero failed requests and no restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BigInt("") / BigInt("0") don't throw; guard against them since the
action is exported and admin-callable independent of the UI button.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Track the forum topic currently being processed on IngestionRun
(currentTopicId + currentAccountChannelMapId; additive migration) and
expose it on the live status. processArchiveSets gains an optional
shouldStop callback polled before each archive set; the forum branch
passes a live isTopicFetchEnabled check, so disabling a topic mid-run
lets the in-flight file finish, then skips the rest of that topic.
A new disableActiveTopic server action sets the topic's fetchEnabled
false (upsert), and the worker status panel shows a "Skip & disable
this topic" button while a topic is being processed. Future runs skip
the topic via the existing live per-topic read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pre-upload integrity test is advisory (never blocks upload), but it
raised a WARNING SystemNotification whenever `7z t` failed — most often
because large 7z archives OOM-kill the test process (SIGKILL / exit 137)
in the memory-limited worker container, which is a tool limitation, not
corruption. Classify failures as encrypted | corrupt | inconclusive;
suppress notifications for inconclusive (debug log only) while still
proceeding with the upload as before. Genuine corruption now uses the
INTEGRITY_AUDIT notification type instead of the misleading HASH_MISMATCH;
encrypted archives still notify.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Radix ScrollArea viewport did not get a bounded height inside the
flex-column, max-h, vertically-centred dialog, so a long topic list
overflowed the dialog instead of scrolling (last topics unreachable).
Switch to a native flex-1 min-h-0 overflow-y-auto container, which is
the canonical touch-friendly flex-scroll pattern.
Also add a "Filter topics..." input (matches by topic title) so any
topic can be reached regardless of list length; reset the filter on
close; and add a toast fallback message on toggle failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each loaded candidate gets an enlarge button that opens the ImageLightbox
without changing selection, so the right preview is easier to choose.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The drawer preview image opens the ImageLightbox; replacing the image
stays available via the existing Upload/Pick Preview buttons. No-preview
upload affordance unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Package preview thumbnails open the ImageLightbox on click (hover shows a
maximize affordance). No-preview cells unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Radix Dialog-based lightbox (Esc / overlay / close button to dismiss),
image shown object-contain capped to the viewport. No new dependencies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forum source channels get a Topics menu item that opens the TopicsDrawer.
Non-forum channels are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lists a channel's topics with an enable/disable Switch each (optimistic,
revert on failure). Empty state explains topics appear after the next scan.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Admin-guarded toggle of TopicProgress.fetchEnabled by row id. Persists
immediately; the worker honours it on its next live per-topic read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Returns per-topic fetch state for a channel, mirroring the existing
account-links route auth pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ChannelRow now carries isForum so the UI can show the Topics action for
forum channels only. listChannelTopics returns per-topic fetch state for
a channel from TopicProgress.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Eager-persist a TopicProgress row for every discovered topic at run
start, and read the live fetchEnabled flag per topic so a mid-run
disable skips topics not yet started. Disabled topics are skipped
before any TDLib scan or fetch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ensureTopicProgressRows inserts missing topic rows only (skipDuplicates),
leaving watermarks/scan-state/fetchEnabled untouched. isTopicFetchEnabled
reads the live per-topic flag for mid-run skip decisions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Additive Postgres migration (default true) that backs the per-topic
fetch enable/disable toggle. Existing rows backfill to enabled, so
current behaviour is unchanged. Migration applied to the live DB via
CI prisma migrate deploy (deferred, per environment).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror of the non-forum guards from 1a4bc6f, scoped to forum topics
inside the topic loop:
- Top-of-topic-loop recency/backoff skip
- getForumTopic short-circuit after the SkippedPackage retry pass
- upsertTopicScanState for end-of-scan persistence (both the
archives-found path and the no-archives path)
Same trulyIdle definition throughout: no archives this scan, no
failures this scan, no retryable SkippedPackage rows pending. Topics
with chronic failures stay out of backoff because their counter
never increments.
For MPE specifically (1,086 forum topics), per-cycle searchChatMessages
calls drop from ~1,086 to roughly the count of topics with new
activity in the last 5 minutes — typically <50.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
For non-forum channels in runWorkerForAccount, three guards:
1. Top-of-loop recency/backoff skip — if recently scanned with no
pending work, or in backoff and not its turn, skip entirely.
Bypassed when retryable SkippedPackages exist.
2. After the SkippedPackage retry pass, a getChat short-circuit —
if TDLib's local cache says the channel's last_message.id <= our
effective watermark, skip the paginated searchChatMessages.
3. End-of-scan persists lastScannedAt + lastScanFoundArchives +
consecutiveEmptyScans via the new upsertChannelScanState helper.
trulyIdle requires: no archives, no failures, no retryable pending.
scheduler.ts exposes getCurrentCycle() so the backoff "every Nth cycle"
modulo can be applied.
Forum-topic branch lands in the next commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both read the server-side last message ID from TDLib's local cache.
Used by the channel-scan-skip guard to short-circuit a paginated
searchChatMessages when last_message.id <= our watermark.
getForumTopic uses forum_topic_id (renamed from message_thread_id in
TDLib 1.8.64, same pattern as searchChatMessages / getForumTopics).
Returns null on any failure so the caller can fall back to scanning —
we'd rather waste a scan than miss new content.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps the existing watermark write with the three new scan-state
columns from the previous commit. Single transaction, sets
lastScannedAt=NOW() server-side. Caller is responsible for computing
the trulyIdle bool and the new consecutiveEmptyScans value
(pre-increment vs reset).
Existing updateLastProcessedMessage / upsertTopicProgress are kept for
callers that don't need the new fields (the SkippedPackage retry pass,
which only adjusts the watermark).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WORKER_SKIP_RECENT_SCAN_WINDOW_MS (default 300000 = 5 min)
WORKER_EMPTY_SCAN_BACKOFF_THRESHOLD (default 5 cycles)
WORKER_EMPTY_SCAN_BACKOFF_EVERY_NTH (default 5)
All optional with safe defaults. Not yet read by any code — the worker
integration lands in follow-up commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new fields on each table:
- lastScannedAt — when the worker last touched this scope
- lastScanFoundArchives — true if last scan had archives OR pending
retryables; tracks "work might need revisit"
- consecutiveEmptyScans — counter for cold-channel backoff
Schema change only. Worker logic in follow-up commits. Migration is a
metadata-only ALTER (NOT NULL with default) so it runs in ms even on
21k+ Package rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7-task plan covering schema migration, config knobs, DB + TDLib helpers,
and wiring the skip guards + getChat/getForumTopic short-circuits into
both the forum and non-forum branches of runWorkerForAccount.
Each task ends with a type-check step before its commit so the tree
compiles after every step. Task 7 is manual verification covering
restart safety, failure-retry preservation, and backoff behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Design for adding lastScannedAt + lastScanFoundArchives +
consecutiveEmptyScans columns to AccountChannelMap and TopicProgress,
plus a getChat / getForumTopicInfo short-circuit before
searchChatMessages.
Goal: on restart and during cold-channel cycles, skip scanning channels
and forum topics that have nothing new. For MPE specifically, drops
the per-cycle API call count from ~1,086 to ~50.
Key safety rule: "truly idle" requires both no new archives AND no
retryable SkippedPackage rows pending. The 901f32f retry pass continues
to run unchanged. Failure retries are never skipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
adminandClaude Opus 4.8