perf(worker): run startup upload-recovery batches with bounded concurrency

Telegram soft-throttles sustained sequential getMessages calls from a user account with growing per-call latency (no FLOOD_WAIT, so our retry wrapper never sees it). On a large package count this made the once-per-startup destination-message verification pass take 90+ minutes and block the scheduler/fetch-listener from starting. Running up to 3 batches concurrently cuts wall-clock time well under real per-account rate limits. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP
fix(worker): recognize native 7z multipart volumes (.7z.001, .7z.002, ...)
2026-09-21 13:31:42 +00:00 · 2026-08-21 19:00:21 +02:00 · 2026-08-21 16:50:41 +02:00 · 2026-08-14 11:40:03 +02:00 · 2026-08-13 18:52:34 +02:00 · 2026-08-13 12:10:50 +02:00
36 changed files with 3508 additions and 100 deletions
@@ -89,7 +89,14 @@
      "Bash(wait:*)",
      "WebSearch",
      "Bash(SKILL_CREATOR_PATH=\"C:\\\\Users\\\\A00963355\\\\.claude\\\\plugins\\\\cache\\\\claude-plugins-official\\\\skill-creator\\\\d5c15b861cd2\\\\skills\\\\skill-creator\" && WORKSPACE=\"C:\\\\Users\\\\A00963355\\\\OneDrive - Amaris Zorggroep\\\\Documents\\\\VScodeProjects\\\\DragonsStash\\\\.claude\\\\skills\\\\tdlib-telegram-workspace\\\\iteration-1\" && python \"$SKILL_CREATOR_PATH/eval-viewer/generate_review.py\" \"$WORKSPACE\" --skill-name \"tdlib-telegram\" --benchmark \"$WORKSPACE/benchmark.json\" --static \"$WORKSPACE/review.html\" 2>&1)",
-      "Bash(start:*)"
+      "Bash(start:*)",
      "Bash(npm run:*)",
      "Bash(DATABASE_URL=\"postgresql://dragons:stash@localhost:5432/dragonsstash\" npx prisma migrate dev --name add-skipped-packages)",
      "Bash(git checkout:*)",
      "Bash(DATABASE_URL=\"postgresql://dragons:stash@localhost:5432/dragonsstash?schema=public\" npx prisma migrate dev --name add_package_groups 2>&1)",
      "Bash(psql:*)",
      "Bash(git log:*)",
      "Bash(git merge:*)"
    ]
  }
 }
@@ -55,3 +55,4 @@ src/generated
 nul
 tmpclaude-*
 .worktrees/
 worktrees/
@@ -0,0 +1,29 @@
 # Disposable restore documentation fix report
 ## Scope
 Updated `scripts/backup/README.md` only for the documentation change. This
 report is the requested verification artifact.
 ## Change
 The monthly recovery rehearsal now documents a unique disposable Compose
 project with project-labeled volumes, snapshot selection, staging restore,
 PostgreSQL import, restoration of uploads and both TDLib volumes, disposable
 service startup, health and log checks, retained-file validation, known STL
 checksum and metadata comparison, narrowly scoped cleanup, and an evidence
 template.
 The guide explicitly warns operators not to use production project names or
 volumes, and states that this documentation update did not run the rehearsal.
 ## Verification
 - Focused text check: passed. Confirmed the guide contains the unique project
  warning, snapshot selection, staging restore, PostgreSQL import, all three
  protected volume restores, health check, retained-file validation, checksum
  comparison, scoped cleanup, evidence template, and the statement that the
  rehearsal was not run.
 - `git diff --check`: passed.
 - No recovery, Docker, Restic, PostgreSQL, or health-check commands were run;
  this change documents the operator procedure only.
@@ -0,0 +1,37 @@
 # Backup Final Review Fix Report
 ## Implemented findings
 - The backup container now validates that `RESTIC_REPOSITORY` resolves strictly
  below `/backup`, verifies that the Restic repository already has a readable
  configuration before a scheduled backup, and gives the explicit first-run
  initialization command when that preflight fails.
 - The backup manifest now records every successfully applied Prisma migration
  as a JSON object with its name and UTC completion timestamp. The `psql`
  query uses the existing `DATABASE_URL` connection configuration and stops on
  query errors.
 - Restore now requires `BACKUP_REPOSITORY`, validates that it resolves strictly
  below `/backup`, and validates the configured backup mount (including the
  existing writable probe) before `restore-live` can stop services or replace
  live data.
 - The backup runbook now documents explicit repository initialization, monthly
  full-read Restic checks, a disposable restore/checksum rehearsal, and
  post-restore Compose status/log checks.
 ## Verification
 - `bash -n scripts/backup/container-entrypoint.sh`
 - `bash -n scripts/backup/restore.sh`
 - Focused `rg` assertions for repository validation/preflight, migration
  timestamp metadata, restore mount validation, initialization, maintenance,
  and post-restore runbook commands.
 - `npx prisma validate`
 - `git diff --check`
 ## Scope and concerns
 - No Docker, NAS, systemd, Restic repository, or live restore was run, per the
  bounded review scope. The command-level behavior is therefore statically
  validated only.
 - Existing durable STL handling, the exact live-restore confirmation flag, and
  rollback/safety-artifact behavior were retained.
@@ -0,0 +1,71 @@
 # Backup Scope-Correction Documentation Report
 **Date:** 2026-07-22
 ## Changed files
 - `docs/superpowers/specs/2026-07-21-database-and-file-backups-design.md`
 - `docs/superpowers/plans/2026-07-21-database-and-file-backups.md`
 - `.superpowers/sdd/scope-correction-docs-report.md`
 ## Rationale
 The backup design and implementation plan now define the protected data set as a PostgreSQL logical dump plus the `tdlib_state` and `tdlib_bot_state` session volumes. They continue to require a host-restricted Synology NFS repository, Restic encryption, 30 daily snapshots, service quiescing, guarded restore, and session persistence.
 `manual_uploads` and `tmp_zips` are explicitly excluded. The documents no longer require local retention of completed STL binaries, worker cleanup changes, database lifecycle fields for retained uploads, restored local STL files, or file-path validation. They state that STL binaries remain in Telegram and that the restored database preserves the metadata and mappings required to locate and send them.
 Future Telegram channel-forwarding behavior and archive/STL-content integrity validation are explicitly identified as out of scope and future work.
 ## Checks run
 - `git diff --check`
 - Scope scan of both documents for `manual_uploads`, `tmp_zips`, local STL retention, file-path validation, channel forwarding, and integrity language.
 - Reviewed the final diff to confirm the removed implementation work is limited to the specified backup-scope correction.
 - `git status --short` to confirm the commit stages only the two requested documents and this required report.
 ## Concerns
 - This change intentionally updates documentation only. It does not modify backup scripts, Docker Compose, database schema, worker cleanup, or Telegram behavior.
 - A future implementation should validate its actual backup manifests and Compose mounts against this corrected plan before deployment.
 ---
 # Monthly Backup Verification Documentation Follow-up
 **Date:** 2026-07-22
 ## Fix
 Aligned the approved backup design and implementation plan on the missing recurring operational work. The deployment operator now owns a documented monthly manual runbook task to run full `restic check --read-data`, perform a disposable restore rehearsal, and record the date, snapshot ID, integrity-check result, restore/health result, and cleanup result.
 The plan verifies the first full-read check and rehearsal during acceptance, then carries the same procedure into the monthly runbook without adding a second systemd timer, script, or other production implementation. The Synology wording now precisely identifies the dedicated shared folder's NFS export as restricted to the Docker host's fixed IP.
 ## Scope preserved
 The recovery set remains the PostgreSQL logical dump plus `tdlib_state` and `tdlib_bot_state` only. `manual_uploads`, STL binaries, archive/STL-content integrity, and future channel-forwarding checks remain outside this work.
 ## Checks completed
 - `git diff --check` completed with no whitespace errors.
 - Focused assertions passed: `restic check --read-data` (4 matches), `deployment operator` (3), `disposable restore rehearsal` (6), `manual_uploads` (8), and `archive/STL-content integrity` (4) across the two approved documents.
 - Final diff review confirmed that this follow-up changes documentation only; no backup scripts, Compose configuration, or other production implementation files were modified.
 ---
 # Monthly Backup Verification Review-Finding Fix
 **Date:** 2026-07-22
 ## Fix
 Confirmed and kept the approved design and implementation plan aligned on the reviewer finding: monthly recovery verification is an operator-owned operational task, consisting of a full `restic check --read-data` and a disposable restore rehearsal.
 ## Scope
 The approved docs continue to limit the protected recovery set to the PostgreSQL logical dump, `tdlib_state`, and `tdlib_bot_state`. They do not add `manual_uploads`, STL-binary restore/checks, archive-content integrity checks, future channel-forwarding checks, or any new production timer/script.
 ## Checks
 - `git diff --check`
 - Focused scope assertions over the two approved docs for monthly full Restic check/rehearsal wording and exclusions.
 - Staged-file review before commit to confirm the commit contains documentation/report files only.
@@ -0,0 +1,240 @@
 # Scope correction implementation report
 ## Summary
 Implemented the approved backup scope correction for Dragon's Stash. Backups and restores now cover only:
 - PostgreSQL logical custom-format dump plus manifest/migration metadata.
 - `tdlib_state` worker Telegram session volume.
 - `tdlib_bot_state` bot Telegram session volume.
 The implementation no longer treats `manual_uploads`, completed local STL binaries, or `tmp_zips` as protected backup data. Future channel forwarding and archive/STL-content integrity auditing remain out of scope.
 ## Changed files
 - `docker-compose.yml`
  - Removed the backup service's read-only `manual_uploads:/data/uploads` mount.
  - Kept the normal operational app/worker `manual_uploads` mounts.
  - Kept both TDLib backup mounts.
 - `scripts/backup/container-entrypoint.sh`
  - Removed `/data/uploads` as a required mounted directory.
  - Removed uploads from the manifest `volumePaths`.
  - Removed the `source:uploads` Restic tag.
  - Removed uploads from the Restic source list.
  - Kept database dump, manifest, worker TDLib, and bot TDLib sources/tags.
 - `scripts/backup/restore.sh`
  - Removed restored uploads variables.
  - Removed uploads staging validation.
  - Removed manual uploads volume discovery, safety archive, replacement, and rollback.
  - Removed retained/manual upload file-path verification.
  - Removed temporary database verification that existed only for local upload file references.
  - Preserved guarded `restore-live` confirmation, backup mount/repository checks, service stop/start handling, safety PostgreSQL dump, TDLib volume safety archives, TDLib volume replacement/rollback, and `pg_restore --list`/`pg_restore --exit-on-error` validation.
 - `prisma/schema.prisma`
  - Removed `ManualUploadFile.retainedAt`.
 - `prisma/migrations/20260722100000_remove_retained_manual_files/migration.sql`
  - Added forward migration: `ALTER TABLE "manual_upload_files" DROP COLUMN IF EXISTS "retainedAt";`
  - Preserved the existing committed migration that added `retainedAt`.
 - `src/app/api/uploads/route.ts`
  - Removed `retainedAt: new Date()` from manual upload file creation.
 - `worker/src/manual-upload.ts`
  - Restored final best-effort cleanup of `/data/uploads/<uploadId>` using the older `path.join("/data/uploads", uploadId)` behavior.
 - `scripts/backup/README.md`
  - Rewrote backup set and restore rehearsal docs around PostgreSQL plus both TDLib volumes only.
  - Removed local STL file, retainedAt, upload path, retained file reference, and restored checksum checks.
  - Clarified that STL binaries remain in Telegram and recovery preserves database mappings/Telegram IDs.
  - Kept monthly `restic check --read-data` and disposable restore rehearsal runbook.
  - Explicitly left future channel forwarding and archive/STL-content integrity auditing out of scope.
 - `README.md`
  - Updated the production backup summary to name PostgreSQL logical dump plus Telegram session volumes as the protected set.
  - Clarified that `manual_uploads` and temporary ZIPs are excluded and STL binaries remain in Telegram.
 ## Verification
 - `git diff --check`
  - Passed.
 - `bash -n scripts/backup/container-entrypoint.sh scripts/backup/run-backup.sh scripts/backup/restore.sh`
  - Local `bash` failed because Windows only had the WSL shim and no installed WSL distribution.
  - Passed via Docker fallback:
    `docker run --rm --entrypoint bash -v E:\Projects\DragonsStash:/work:ro -w /work postgres:16-alpine -n scripts/backup/container-entrypoint.sh scripts/backup/run-backup.sh scripts/backup/restore.sh`
 - `npx prisma validate`
  - Passed.
 - `npm run build`
  - Passed.
 - `cd worker && npm run build`
  - Passed.
 - Focused backup/restore scope assertions
  - Backup shell paths assertion passed: no `manual_uploads`, `/data/uploads`, `retainedAt`, upload source tag, or upload-restore helper references in `scripts/backup/*.sh`.
  - Compose backup service assertion passed: no `manual_uploads`, `/data/uploads`, `retainedAt`, or `source:uploads` in the `backup` service block.
  - Active retainedAt assertion passed: no `retainedAt` in active Prisma schema, upload API, worker source, or backup shell scripts.
  - Active backup/restore upload-source assertion passed: no `manual_uploads`, `/data/uploads`, or `data/uploads` in backup/restore shell scripts.
  - Remaining expected matches are limited to normal operational app/worker upload mounts and paths, docs stating exclusions, and the historical add/drop migrations.
 ## Concerns
 - None for implementation scope.
 - Environment note: local Bash is unavailable because WSL has no installed distribution; Bash syntax was verified inside Docker instead.
 ---
 # Restore Path Scope Review-Finding Fix
 **Date:** 2026-07-22
 ## Fix
 Addressed the Important restore finding by replacing both unrestricted
 `restic restore` calls in `scripts/backup/restore.sh` with a shared filtered
 restore wrapper. The wrapper restores only the backup source paths actually
 written by `scripts/backup/container-entrypoint.sh`:
 - `/staging/backup-*/database.dump`
 - `/staging/backup-*/manifest` and `/staging/backup-*/manifest/**`
 - `/data/tdlib-worker` and `/data/tdlib-worker/**`
 - `/data/tdlib-bot` and `/data/tdlib-bot/**`
 Added an explicit restored-tree guard that refuses unexpected restored content:
 top-level restored directories other than `staging` and `data`, direct
 `data/*` entries other than `tdlib-worker` and `tdlib-bot`, and direct
 `staging/backup-*/*` entries other than `database.dump` and `manifest`. This
 rejects old/broad snapshots that would otherwise restore `data/uploads`,
 temporary ZIP or database volume trees, or other unexpected volume content.
 Preserved the existing guarded live-restore confirmation, staging-directory
 validation, mount/repository checks, snapshot verification, custom
 PostgreSQL-dump validation, service stop/start lifecycle, health check, safety
 database dump, TDLib safety archives, and rollback of exactly the PostgreSQL
 database plus the two TDLib volumes.
 Added `scripts/backup/restore-path-assertions.sh`, a focused shell assertion
 harness that stubs Docker/Restic and verifies both staging and live restore use
 the expected include filters and that unexpected restored data-volume content is
 rejected explicitly.
 Addressed the Minor documentation gap in the root README backup section by
 stating that forwarding behavior and archive/STL-content integrity auditing are
 future work outside the backup scope.
 ## Verification
 - Red check before implementation:
  ```text
  & 'C:\Program Files\Git\bin\bash.exe' -lc 'scripts/backup/restore-path-assertions.sh'
  ASSERTION FAILED: database dump include filter missing
  ```
 - Bash syntax check:
  ```text
  & 'C:\Program Files\Git\bin\bash.exe' -lc 'bash -n scripts/backup/container-entrypoint.sh scripts/backup/run-backup.sh scripts/backup/restore.sh scripts/backup/restore-path-assertions.sh'
  [passed with no output]
  ```
 - Whitespace check:
  ```text
  git diff --check
  warning: in the working copy of '.superpowers/sdd/scope-correction-implementation-report.md', LF will be replaced by CRLF the next time Git touches it
  warning: in the working copy of 'README.md', LF will be replaced by CRLF the next time Git touches it
  warning: in the working copy of 'scripts/backup/restore.sh', LF will be replaced by CRLF the next time Git touches it
  [exit 0]
  ```
 - Focused restore-path assertions:
  ```text
  & 'C:\Program Files\Git\bin\bash.exe' -lc 'scripts/backup/restore-path-assertions.sh'
  restore-path assertions passed
  ```
 ## Concerns
 - None for implementation scope.
 - Git Bash was available and used for shell syntax/assertion checks, so Docker
  fallback was not needed for the final syntax verification.
 ---
 # Important Operational Findings Fix
 **Date:** 2026-07-22
 ## Fix
 Addressed the two Important operational findings from final review:
 - `scripts/backup/run-backup.sh`
  - Backup wrapper restart failures now make an otherwise successful backup
    exit non-zero.
  - Existing non-zero backup failures remain preserved if service restart also
    fails.
  - Added `scripts/backup/run-backup-assertions.sh` to assert both exit-code
    cases with a fake Docker/Compose environment.
 - `scripts/backup/restore.sh`
  - `restore-live` now captures the managed services that were running before
    live restore using `docker compose --profile full ps --status running`.
  - Live restore stops only those previously running managed services.
  - Successful live restore starts only those previously running services, so
    the profile-gated optional `bot` is not started if it was not running.
  - Failure handling leaves services stopped and still rolls back only the
    PostgreSQL database plus both TDLib volumes.
  - App health wait now runs only when `app` was previously running.
  - Extended `scripts/backup/restore-path-assertions.sh` to assert subset
    stop/start behavior and skipped health checks when `app` was not running.
 Addressed the Minor staging-path documentation mismatch by aligning
 `.env.example` with the backup README example:
 `/var/lib/dragons-stash/backup-staging`.
 ## Verification
 - Red checks before implementation:
  - `docker run --rm -v "${PWD}:/work" -w /work ubuntu:24.04 bash scripts/backup/run-backup-assertions.sh`
    - Failed as expected: backup reported success when restart failed.
  - `docker run --rm -v "${PWD}:/work" -w /work ubuntu:24.04 bash scripts/backup/restore-path-assertions.sh`
    - Failed as expected: restore-live stopped the fixed `app worker bot`
      service set instead of the running subset.
 - Focused assertions after implementation:
  - `docker run --rm -v "${PWD}:/work" -w /work ubuntu:24.04 bash scripts/backup/run-backup-assertions.sh`
    - Passed.
  - `docker run --rm -v "${PWD}:/work" -w /work ubuntu:24.04 bash scripts/backup/restore-path-assertions.sh`
    - Passed.
 - `git diff --check`
  - Passed.
 - Docker Bash syntax check:
  - `docker run --rm -v "${PWD}:/work" -w /work ubuntu:24.04 bash -n scripts/backup/run-backup.sh scripts/backup/restore.sh scripts/backup/container-entrypoint.sh scripts/backup/restore-path-assertions.sh scripts/backup/run-backup-assertions.sh`
  - Passed.
 - `npx prisma validate`
  - Passed.
 - `npm run build`
  - Passed.
 - `npm run lint`
  - Failed on unrelated existing React lint issues in `src/` and mirrored
    `.worktrees/worker-improvements` files; no failures were in touched backup
    files.
 ## Concerns
 - Local `bash` is unavailable because the Windows `bash` command resolves to a
  WSL shim with no installed distribution; shell checks used Docker fallback.
 - Full `npm run lint` remains blocked by pre-existing unrelated lint errors.
@@ -140,6 +140,18 @@ docker compose --profile bot up -d
 > **Tip:** Create a bot token via [@BotFather](https://t.me/BotFather) on Telegram and set `BOT_TOKEN` in `.env`.
 > Get Telegram API credentials from [my.telegram.org/apps](https://my.telegram.org/apps).
 ### Production Backups
 Docker volumes are not backups. Production backups protect a PostgreSQL
 logical dump plus the worker and bot Telegram session volumes in an encrypted
 Restic repository on a Synology NFS share. `manual_uploads` and temporary ZIP
 processing data are excluded; STL binaries remain in Telegram, while the
 database mappings and Telegram IDs are what recovery preserves for lookup and
 delivery. Forwarding behavior and archive/STL-content integrity auditing are
 future work outside this backup scope. See the [backup and recovery guide](scripts/backup/README.md)
 for Synology setup, secrets, systemd installation, monitoring, retention, and
 guarded restore procedures.
 ### Seeding the Database
 To seed the database with sample data on first run:
@@ -19,7 +19,13 @@ pg_dump -h dragonsstash-db -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc -f "$DUMP_F
 # TDLib volumes are tarred live (best-effort, per design). A file changing
 # mid-read makes GNU tar exit 1 (warning) — that is expected here and must not
 # abort the backup. Only a genuine error (exit >= 2) is fatal.
-tar --warning=no-file-changed -czf "$TAR_FILE" -C /data tdlib-worker tdlib-bot \
+#
 # files/temp is TDLib's own disposable download cache (redundant with the
 # source/destination Telegram chats, pruned by the worker itself after each
 # ingestion run) — it has no business in a backup and its size is what made
 # this tar take 4+ hours once the cache grew back to tens of GB.
 tar --warning=no-file-changed --exclude='tdlib-worker/*/files/temp' \
  -czf "$TAR_FILE" -C /data tdlib-worker tdlib-bot \
  || { rc=$?; [ "$rc" -le 1 ] || exit "$rc"; }
 restic backup "$DUMP_FILE" "$TAR_FILE"
@@ -53,6 +53,21 @@ export async function closeBotClient(): Promise<void> {
  }
 }
 /**
 * Make sure TDLib has resolved this user into a private chat before sending.
 * A bot's local TDLib database only knows about chats it has seen since its
 * last (re)authentication — after a state-directory reset it has no chat
 * history cached, so `sendMessage` fails with "Chat not found" even for
 * users who messaged the bot long ago. `createPrivateChat` forces TDLib to
 * resolve/fetch the chat first.
 */
 async function ensurePrivateChat(c: tdl.Client, userId: number): Promise<void> {
  await withFloodWait(
    () => c.invoke({ _: "createPrivateChat", user_id: userId, force: false }),
    "createPrivateChat"
  );
 }
 /**
 * Send a document from a channel to a user's DM.
 *
@@ -71,6 +86,8 @@ export async function copyMessageToUser(
  if (!client) throw new Error("Bot client not initialized");
  const c = client;
  await ensurePrivateChat(c, Number(toUserId));
  log.info(
    { fromChatId: fromChatId.toString(), messageId: messageId.toString(), toUserId: toUserId.toString() },
    "Sending file to user"
@@ -233,6 +250,8 @@ export async function sendTextMessage(
  if (!client) throw new Error("Bot client not initialized");
  const c = client;
  await ensurePrivateChat(c, Number(chatId));
  // Parse the text first
  const parsed = await withFloodWait(
    () =>
@@ -269,6 +288,8 @@ export async function sendPhotoMessage(
  if (!client) throw new Error("Bot client not initialized");
  const c = client;
  await ensurePrivateChat(c, Number(chatId));
  // Write the photo to a temp file
  const { writeFile, unlink } = await import("fs/promises");
  const path = await import("path");
@@ -0,0 +1,13 @@
 [Unit]
 Description=Dragon's Stash off-host backup
 After=network-online.target docker.service
 Requires=docker.service
 Wants=network-online.target
 [Service]
 Type=oneshot
 User=root
 EnvironmentFile=-/etc/dragons-stash/backup.env
 WorkingDirectory=/opt/stacks/DragonsStash
 ExecStart=/opt/stacks/DragonsStash/scripts/backup/run-backup.sh
 TimeoutStartSec=infinity
@@ -0,0 +1,11 @@
 [Unit]
 Description=Nightly Dragon's Stash off-host backup
 [Timer]
 OnCalendar=*-*-* 03:00:00
 Persistent=true
 RandomizedDelaySec=15m
 Unit=dragons-stash-backup.service
 [Install]
 WantedBy=timers.target
@@ -0,0 +1,450 @@
 # Database and File Backups Implementation Plan
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Add a nightly, encrypted disaster-recovery backup for PostgreSQL and Telegram session volumes, stored on a Synology NAS.
 **Architecture:** A Linux-host systemd timer invokes a host orchestration script. The script verifies the mounted Synology NFS share, stops the app/worker/bot services for consistency, and runs a one-shot Docker Compose backup service. The backup service creates a PostgreSQL custom-format dump and stores it with the two TDLib session volumes in a Restic repository on the NAS. A guarded restore command reconstructs the database and sessions, and documentation describes setup and testing. STL binaries remain in Telegram; restored database metadata and mappings continue to identify the Telegram content used for lookup and delivery.
 **Tech Stack:** Docker Compose, PostgreSQL 16 `pg_dump`/`pg_restore`, Restic repository encryption and retention, Synology NFS, Linux systemd service/timer, Bash.
 ## Global Constraints
 - PostgreSQL data must be backed up as a logical custom-format dump; the raw `postgres_data` volume is not the primary backup.
 - The `tdlib_state` and `tdlib_bot_state` volumes are included in every successful snapshot.
 - The `manual_uploads` and `tmp_zips` volumes are excluded.
 - Completed STL binaries are not retained locally for backup; existing worker cleanup behavior remains unchanged. Telegram remains the binary store, while PostgreSQL retains the metadata and mappings needed to locate and send the files after restore.
 - The Docker host must stop `app`, `worker`, and `bot` while session volumes are captured; PostgreSQL remains running for `pg_dump`.
 - The backup repository is encrypted and stored on a Synology NFS share restricted to the Docker host.
 - Retention is 30 daily snapshots; pruning is allowed only after a verified successful backup.
 - A failed run must restart services and preserve the last known-good snapshot.
 - A guarded restore must require explicit confirmation before replacing live database or session-volume data.
 - No in-app backup UI is part of this implementation.
 - Future Telegram channel-forwarding behavior and archive/STL-content integrity validation are explicitly out of scope.
 - The repository has no automated test framework; verification uses Bash syntax checks, Docker Compose validation, logs, Restic checks, and a disposable restore rehearsal.
 ---
 ## File and Responsibility Map
 Create or modify only these focused units:
 - Create `backup/Dockerfile`: build the one-shot image containing PostgreSQL client tools, Restic, Bash, and the backup entrypoint.
 - Create `scripts/backup/container-entrypoint.sh`: run the backup inside Compose, including dump creation, manifest creation, Restic snapshot, verification, and retention.
 - Create `scripts/backup/run-backup.sh`: host-level lock, NFS mount validation, service stop/start, and invocation of the one-shot Compose service.
 - Create `scripts/backup/restore.sh`: guarded restore orchestration for a selected Restic snapshot.
 - Create `deploy/systemd/dragons-stash-backup.service`: systemd unit invoking the host backup script.
 - Create `deploy/systemd/dragons-stash-backup.timer`: nightly schedule.
 - Create `scripts/backup/README.md`: Synology setup, host mount, secrets, first backup, restore, and operational troubleshooting.
 - Modify `docker-compose.yml`: add the profile-gated one-shot `backup` service and its read-only session-volume mounts.
 - Modify `.env.example`: document backup mount, staging, repository, and secret-file configuration without committing secrets.
 - Modify `README.md`: add the production backup setup and restore entry points, linking to the detailed backup guide.
 ---
 ### Task 1: Add the backup service and configuration contract
 **Files:**
 - Modify: `docker-compose.yml`
 - Modify: `.env.example`
 - Create: `backup/Dockerfile`
 **Interfaces:**
 - Consumes: existing `db`, `tdlib_state`, and `tdlib_bot_state` Compose resources.
 - Produces: a profile-gated Compose service named `backup` that mounts the two session volumes read-only, connects to the `backend` network, and exposes `/backup` and `/staging` to the container entrypoint.
 - [ ] **Step 1: Add explicit backup environment variables to `.env.example`**
 Add this block without real credentials:
 ```dotenv
 # Disaster recovery backups
 BACKUP_MOUNT_PATH="/mnt/dragonsstash-backups"
 BACKUP_STAGING_PATH="/var/lib/dragons-stash-backup/staging"
 BACKUP_REPOSITORY="/backup/restic"
 BACKUP_RESTIC_PASSWORD_FILE="/etc/dragons-stash/restic-password"
 BACKUP_RETENTION_DAYS=30
 BACKUP_APP_VERSION="unknown"
 ```
 - [ ] **Step 2: Add the profile-gated `backup` service to `docker-compose.yml`**
 Add a service with these properties:
 ```yaml
  backup:
    profiles: ["backup"]
    build:
      context: .
      dockerfile: backup/Dockerfile
    environment:
      DATABASE_URL: postgresql://${POSTGRES_USER:-dragons}:${POSTGRES_PASSWORD:-stash}@db:5432/${POSTGRES_DB:-dragonsstash}
      RESTIC_REPOSITORY: ${BACKUP_REPOSITORY:-/backup/restic}
      RESTIC_PASSWORD_FILE: /run/secrets/restic-password
      BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-30}
      BACKUP_APP_VERSION: ${BACKUP_APP_VERSION:-unknown}
    user: "0:0"
    volumes:
      - tdlib_state:/data/tdlib-worker:ro
      - tdlib_bot_state:/data/tdlib-bot:ro
      - ${BACKUP_MOUNT_PATH:?Set BACKUP_MOUNT_PATH to the mounted Synology share}:/backup:rw
      - ${BACKUP_STAGING_PATH:?Set BACKUP_STAGING_PATH to a local staging directory}:/staging:rw
      - ${BACKUP_RESTIC_PASSWORD_FILE:?Set BACKUP_RESTIC_PASSWORD_FILE to a root-readable secret file}:/run/secrets/restic-password:ro
    depends_on:
      db:
        condition: service_healthy
    networks:
      - backend
 ```
 Do not mount `manual_uploads` or `tmp_zips`. Ensure the new service does not have `restart: always` and is not started by the normal production `docker compose up -d` command unless the `backup` profile is explicitly requested.
 - [ ] **Step 3: Create the backup image definition**
 Create `backup/Dockerfile`:
 ```dockerfile
 FROM postgres:16-alpine
 RUN apk add --no-cache bash restic coreutils
 COPY scripts/backup/container-entrypoint.sh /usr/local/bin/dragons-stash-backup
 RUN chmod 0755 /usr/local/bin/dragons-stash-backup
 ENTRYPOINT ["/usr/local/bin/dragons-stash-backup"]
 ```
 - [ ] **Step 4: Validate the Compose contract**
 Run on a Linux host with the required variables available:
 ```bash
 docker compose --profile backup config --quiet
 ```
 Expected: exit code `0` and no Compose validation errors. If the required NAS/secret paths are absent, the command must fail with the explicit variable-name error rather than silently using a host path.
 - [ ] **Step 5: Commit the service boundary**
 ```bash
 git add backup/Dockerfile docker-compose.yml .env.example
 git commit -m "feat: add backup compose service"
 ```
 ### Task 2: Implement the one-shot backup container
 **Files:**
 - Create: `scripts/backup/container-entrypoint.sh`
 **Interfaces:**
 - Consumes: `DATABASE_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD_FILE`, `BACKUP_RETENTION_DAYS`, `/data/tdlib-worker`, `/data/tdlib-bot`, `/backup`, and `/staging`.
 - Produces: exit `0` only after a verified Restic snapshot and successful retention pruning; non-zero on any failed dump, snapshot, verification, or prune step.
 - [ ] **Step 1: Define strict shell behavior and required inputs**
 The script must begin with:
 ```bash
 #!/usr/bin/env bash
 set -Eeuo pipefail
 ```
 Validate that `DATABASE_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD_FILE`, and `BACKUP_RETENTION_DAYS` are set, that the password file is readable, and that `/backup` and `/staging` are mounted directories.
 - [ ] **Step 2: Create a per-run staging directory and cleanup trap**
 Use a directory below `/staging` named with UTC timestamp and process ID. Register an `EXIT` trap that removes only that directory. Never remove `/staging` itself or any directory under `/backup`.
 - [ ] **Step 3: Create the PostgreSQL dump**
 Run `pg_dump` using the connection URL and custom format:
 ```bash
 pg_dump --format=custom --file="$RUN_DIR/database.dump" "$DATABASE_URL"
 ```
 After the command succeeds, require the dump to be a non-empty regular file. Generate a SHA-256 checksum for the dump in the manifest directory.
 - [ ] **Step 4: Create the manifest**
 Write a JSON manifest containing the UTC backup timestamp, repository path, retention value, dump filename, dump checksum, and the two TDLib volume paths captured. Obtain the application image/version from an explicit `BACKUP_APP_VERSION` environment value when supplied; otherwise record `unknown` rather than guessing from mutable container state.
 - [ ] **Step 5: Create one Restic snapshot**
 Run one `restic backup` command against the staged database dump, manifest, and the two mounted persistent session volumes. Use stable source labels so the snapshot can be recognized during restore. Do not mount or include `manual_uploads` or `tmp_zips`.
 - [ ] **Step 6: Verify and apply retention**
 After `restic backup` succeeds:
 ```bash
 restic snapshots --latest 1
 restic check
 restic forget --keep-daily "$BACKUP_RETENTION_DAYS" --prune
 ```
 If any command fails, exit non-zero and do not run `forget --prune`. The host wrapper will restart the stopped services. When invoked with an unrecognized first argument, the entrypoint must pass the remaining arguments to the `restic` binary so operators can inspect the repository through the Compose image without installing Restic on the host:
 ```bash
 case "${1:-backup}" in
  backup) run_backup ;;
  restore) run_restore "$@" ;;
  *) exec restic "$@" ;;
 esac
 ```
 - [ ] **Step 7: Build and run a container-only smoke test**
 Run:
 ```bash
 docker build -f backup/Dockerfile -t dragons-stash-backup:smoke .
 bash -n scripts/backup/container-entrypoint.sh
 ```
 Expected: image build succeeds and Bash reports no syntax errors. The full snapshot test waits until the host wrapper and a real PostgreSQL/session-volume environment exist.
 - [ ] **Step 8: Commit the backup container**
 ```bash
 git add backup/Dockerfile scripts/backup/container-entrypoint.sh
 git commit -m "feat: implement encrypted database and session snapshots"
 ```
 ### Task 3: Add the host orchestration script and nightly systemd timer
 **Files:**
 - Create: `scripts/backup/run-backup.sh`
 - Create: `deploy/systemd/dragons-stash-backup.service`
 - Create: `deploy/systemd/dragons-stash-backup.timer`
 **Interfaces:**
 - Consumes: `.env`/deployment environment, the mounted `BACKUP_MOUNT_PATH`, Docker Compose project, and the `backup` service from Task 1.
 - Produces: one host command that safely stops and restarts services and returns the backup container's exit status; systemd runs it nightly.
 - [ ] **Step 1: Implement lock and mount validation**
 The host script must use `flock` on `/run/lock/dragons-stash-backup.lock`, reject a concurrent run, and validate the NAS mount with both `mountpoint --q "$BACKUP_MOUNT_PATH"` and a writable probe file that is immediately removed. A local directory at the same path must not pass validation. The script reads `BACKUP_MOUNT_PATH`, `BACKUP_STAGING_PATH`, `BACKUP_RESTIC_PASSWORD_FILE`, and `BACKUP_RETENTION_DAYS` from the systemd environment file.
 - [ ] **Step 2: Capture service state and define guaranteed restart**
 Before stopping services, record which of `app`, `worker`, and `bot` are running with `docker compose ps --status running -q SERVICE`. Stop only the services that were running. Register an `EXIT` trap that starts exactly those services and preserves the backup command's original exit code.
 - [ ] **Step 3: Invoke the profile-gated backup service**
 After the services stop, run:
 ```bash
 docker compose --profile backup run --rm backup backup
 ```
 Pass through the container exit code. The wrapper must not call the Restic retention command itself; that responsibility stays inside the backup container.
 - [ ] **Step 4: Add the systemd service**
 Create a unit with `Type=oneshot`, `User=root`, `EnvironmentFile=-/etc/dragons-stash/backup.env`, `WorkingDirectory` set to the production Compose directory, `ExecStart` pointing to the absolute `run-backup.sh` path, and `TimeoutStartSec=infinity`. Configure `After=network-online.target docker.service` and `Requires=docker.service`. Do not put the Restic password or database password in the unit file.
 - [ ] **Step 5: Add the nightly timer**
 Create a timer using `OnCalendar=*-*-* 03:00:00`, `Persistent=true`, and `RandomizedDelaySec=15m`. Set `Unit=dragons-stash-backup.service` and `WantedBy=timers.target`.
 - [ ] **Step 6: Validate shell and systemd files**
 Run:
 ```bash
 bash -n scripts/backup/run-backup.sh
 systemd-analyze verify deploy/systemd/dragons-stash-backup.service deploy/systemd/dragons-stash-backup.timer
 ```
 Expected: both commands exit `0`. Run `systemctl list-timers dragons-stash-backup.timer` after installation and confirm the next run is scheduled.
 - [ ] **Step 7: Commit scheduling and orchestration**
 ```bash
 git add scripts/backup/run-backup.sh deploy/systemd/dragons-stash-backup.service deploy/systemd/dragons-stash-backup.timer
 git commit -m "feat: schedule nightly off-host backups"
 ```
 ### Task 4: Implement guarded restore tooling
 **Files:**
 - Create: `scripts/backup/restore.sh`
 **Interfaces:**
 - Consumes: a Restic snapshot ID, the same repository/password configuration, the backup Compose service, and the live Compose project.
 - Produces: restored PostgreSQL data and Telegram session volumes only after explicit confirmation for live replacement; a non-destructive staging restore by default.
 - [ ] **Step 1: Define restore modes and destructive guard**
 Support these commands:
 ```bash
 ./scripts/backup/restore.sh list
 ./scripts/backup/restore.sh verify SNAPSHOT_ID
 ./scripts/backup/restore.sh restore-to-staging SNAPSHOT_ID STAGING_DIR
 ./scripts/backup/restore.sh restore-live SNAPSHOT_ID --confirm-replace-live-data
 ```
 Reject `restore-live` unless the exact confirmation flag is present. `list`, `verify`, and `restore-to-staging` must not stop services or modify live volumes.
 - [ ] **Step 2: Implement snapshot verification and staging restore**
 Use `restic snapshots`, `restic check`, and `restic restore SNAPSHOT_ID --target STAGING_DIR`. Verify that the restored staging tree contains a non-empty custom-format dump, a manifest, `tdlib-worker`, and `tdlib-bot` before reporting success. Do not add file-path checks, binary checksums, archive/STL-content validation, or channel-forwarding behavior.
 - [ ] **Step 3: Implement live restore sequencing**
 For `restore-live`:
 1. Confirm the Compose project and target repository.
 2. Stop `app`, `worker`, and `bot`.
 3. Create a safety PostgreSQL dump of the current database into local staging.
 4. Restore the selected snapshot to a separate staging directory.
 5. Replace the two Docker session volumes only after the restored tree passes validation.
 6. Recreate the configured database from the restored custom-format dump using `pg_restore --no-owner`.
 7. Start services and run the health endpoint plus worker/bot startup and authentication checks.
 If any step fails, leave the services stopped, print the exact staging path and failure, and do not delete the safety dump.
 - [ ] **Step 4: Validate the restore command without touching live data**
 Run:
 ```bash
 bash -n scripts/backup/restore.sh
 ./scripts/backup/restore.sh list
 ```
 Expected: syntax passes and `list` prints available snapshot IDs without stopping any service or modifying a volume.
 - [ ] **Step 5: Commit guarded restore tooling**
 ```bash
 git add scripts/backup/restore.sh
 git commit -m "feat: add guarded database and session restore"
 ```
 ### Task 5: Document Synology setup, operations, and recovery
 **Files:**
 - Create: `scripts/backup/README.md`
 - Modify: `README.md`
 **Interfaces:**
 - Consumes: the exact environment variables, systemd units, and restore commands from Tasks 1-4.
 - Produces: operator-facing instructions that do not require reading implementation files.
 - [ ] **Step 1: Document Synology configuration**
 Document creating the `dragonsstash-backups` shared folder, enabling NFS, and configuring that shared folder's NFS export to allow only the Docker host's fixed IP. Document mounting it at `/mnt/dragonsstash-backups`. Include commands for checking the mount:
 ```bash
 mountpoint /mnt/dragonsstash-backups
 touch /mnt/dragonsstash-backups/.write-test
 rm /mnt/dragonsstash-backups/.write-test
 ```
 Do not document exposing NFS to the Internet.
 - [ ] **Step 2: Document secret and staging setup**
 Document creating the root-readable Restic password file at `/etc/dragons-stash/restic-password`, creating the local staging directory, setting ownership/permissions, and adding the backup variables to the production environment without committing secrets.
 - [ ] **Step 3: Document installation and first-run commands**
 Include:
 ```bash
 sudo install -m 0644 deploy/systemd/dragons-stash-backup.service /etc/systemd/system/
 sudo install -m 0644 deploy/systemd/dragons-stash-backup.timer /etc/systemd/system/
 sudo systemctl daemon-reload
 sudo systemctl enable --now dragons-stash-backup.timer
 sudo systemctl start dragons-stash-backup.service
 sudo journalctl -u dragons-stash-backup.service -n 100 --no-pager
 ```
 Explain that the first run captures PostgreSQL and TDLib session state. State clearly that STL binaries stay in Telegram, and that restored PostgreSQL metadata and mappings are what allow normal lookup and delivery after restore.
 - [ ] **Step 4: Document monitoring, retention, restore, and the monthly recovery check**
 Document how to inspect timer status, service failures, Restic snapshots, repository checks, and the four restore modes. Explicitly state that `restore-live` is destructive and requires the confirmation flag. Assign the deployment operator a recurring monthly runbook task: run `docker compose --profile backup run --rm backup check --read-data`, then perform the documented disposable restore rehearsal using a selected snapshot. Record the date, snapshot ID, full-check result, restore/health result, and cleanup result. This is an operator-owned manual procedure, not a second production timer or a change to the nightly backup service. Limit the rehearsal to the PostgreSQL logical dump, `tdlib_state`, and `tdlib_bot_state`; do not add `manual_uploads`, STL-binary, archive-content, or channel-forwarding checks. Explain that channel-forwarding behavior and archive/STL-content integrity validation are future work, not restore checks.
 - [ ] **Step 5: Add a concise production-backup section to the root README**
 Add a link from the deployment/operations section to `scripts/backup/README.md`, state that Docker volumes are not backups, and identify the PostgreSQL logical dump and Telegram session volumes as the protected data set. State that manual uploads and temporary ZIPs are excluded and STL binaries remain in Telegram.
 - [ ] **Step 6: Commit documentation**
 ```bash
 git add scripts/backup/README.md README.md
 git commit -m "docs: document Synology backup and recovery"
 ```
 ### Task 6: Verify backup, failure recovery, retention, and restore
 **Files:**
 - Modify: `scripts/backup/README.md` only if verification commands need correction.
 **Interfaces:**
 - Consumes: the complete backup stack from Tasks 1-5.
 - Produces: evidence that the acceptance criteria are met, including a full `restic check --read-data`, a disposable restore rehearsal, and a failure-path result. After deployment, the same full-check and rehearsal are an operator-owned monthly runbook task documented in Task 5.
 - [ ] **Step 1: Validate configuration and scripts**
 Run:
 ```bash
 docker compose --profile backup config --quiet
 bash -n scripts/backup/container-entrypoint.sh scripts/backup/run-backup.sh scripts/backup/restore.sh
 systemd-analyze verify deploy/systemd/dragons-stash-backup.service deploy/systemd/dragons-stash-backup.timer
 ```
 Expected: all commands exit `0`.
 - [ ] **Step 2: Seed recognizable database metadata**
 Using the existing app/database workflow, identify a record whose Telegram archive, message, package, and file metadata can be recognized after restore. Record the expected database identifiers before backup. Do not create or retain a local STL binary for this verification.
 - [ ] **Step 3: Run a real backup and inspect the snapshot**
 Run the systemd service manually, then inspect:
 ```bash
 sudo systemctl start dragons-stash-backup.service
 sudo journalctl -u dragons-stash-backup.service --since "10 minutes ago" --no-pager
 docker compose --profile backup run --rm backup snapshots
 docker compose --profile backup run --rm backup check
 ```
 Expected: the service succeeds, the snapshot exists, the repository check succeeds, and all services are running again.
 - [ ] **Step 4: Test the failure path with the NAS unavailable**
 Temporarily unmount the Synology share in a controlled maintenance session, run the systemd service, and confirm it fails before creating a new snapshot. Remount the share and confirm the previously successful snapshot remains listed. Verify that services are running after the failed attempt.
 - [ ] **Step 5: Run the full-read integrity check and rehearse a disposable restore**
 Run `docker compose --profile backup run --rm backup check --read-data` against the selected repository, then restore the selected snapshot to a disposable Compose project or isolated Docker volumes. Import the database dump, restore the two TDLib session trees, start the disposable app/worker/bot services, and call `/api/health`. Confirm the recognizable database metadata and Telegram mappings match the pre-backup record. Record the check and rehearsal evidence as the initial monthly-runbook baseline. Do not assert the presence, checksum, content, or forwarding behavior of STL binaries.
 - [ ] **Step 6: Verify retention behavior**
 Use a disposable repository or controlled test timestamps to create more than 30 daily snapshots, run the retention command after a successful backup, and confirm that the latest 30 daily snapshots remain. Confirm a failed backup does not invoke pruning.
 - [ ] **Step 7: Record verification evidence**
 Add the actual commands, dates, snapshot ID, restore result, and any environment-specific caveats to the operational notes. Do not commit passwords, session contents, database dumps, or NAS addresses that are intended to remain private.
 - [ ] **Step 8: Commit any documentation corrections**
 ```bash
 git add scripts/backup/README.md
 git commit -m "test: document verified backup and restore procedure"
 ```
 ## Plan Self-Review
 - **Spec coverage:** PostgreSQL logical dump, both Telegram session volumes, Synology NFS, Restic encryption, 30-day retention, maintenance window, service restart on failure, guarded restore, and an explicitly deployment-operator-owned monthly `restic check --read-data` plus disposable restore rehearsal are covered by Tasks 1-6. The initial run is verified in Task 6 and the recurring runbook is documented in Task 5; neither adds a second production timer.
 - **Exclusions:** `manual_uploads` and `tmp_zips` are excluded; local STL retention, restored STL binaries, file-path/checksum validation, channel forwarding, and archive/STL-content integrity checks are not implementation requirements.
 - **Placeholder scan:** No `TBD`, `TODO`, or unspecified implementation task remains. Environment-dependent values are explicit configuration variables or operator-supplied paths.
 - **Type/interface consistency:** The Compose service name is consistently `backup`; the container command modes are `backup` and `restore`; the host wrapper owns service lifecycle; the restore script owns destructive confirmation; Restic owns snapshots and pruning.
 - **Scope check:** The plan contains one operational subsystem with separate backup, restore, scheduling, and documentation units that can each be reviewed and tested independently.
@@ -0,0 +1,125 @@
 # Database and File Backup Design
 **Date:** 2026-07-21
 **Status:** Design approved for written-spec review
 **Scope:** Disaster recovery for the Docker Compose deployment
 ## Problem
 Dragon's Stash currently persists PostgreSQL in a Docker named volume. The Telegram worker and bot also persist authentication and session state in the `tdlib_state` and `tdlib_bot_state` volumes. Docker volumes protect against container recreation, but they are not off-host backups. A host disk failure, accidental deletion, corruption, or ransomware event could destroy the database and require both Telegram clients to authenticate again.
 STL binaries are intentionally not retained as a local recovery set. They remain in Telegram. The PostgreSQL database preserves the archive, message, package, and file metadata needed to locate and send those Telegram-hosted binaries after the application database is restored. Existing worker cleanup behavior is unchanged.
 ## Goals
 - Protect PostgreSQL data against loss of the application host with a logical backup.
 - Preserve Telegram worker and bot session state so a host restore does not normally require re-authentication.
 - Store backups on a Synology NAS over an authenticated, host-restricted NFS share.
 - Create one recoverable snapshot containing related database and session state.
 - Retain 30 daily recovery points.
 - Provide a documented, repeatable, guarded restore process.
 - Detect failed or corrupt backups instead of silently pruning the last good copy.
 ## Non-goals
 - Building an in-app backup-management UI.
 - Backing up `manual_uploads`, retaining completed STL binaries locally, or changing worker cleanup behavior.
 - Backing up temporary ZIP processing data in `tmp_zips`.
 - Copying the raw `postgres_data` volume as the primary database backup.
 - Implementing future Telegram channel-forwarding behavior.
 - Validating the content or binary integrity of Telegram archives or STL files. This is future work and is outside the backup/restore feature.
 - Providing protection against loss of the NAS itself. A later Synology Hyper Backup task can replicate this repository to another device or cloud destination.
 ## Selected approach
 Use a Linux-host backup script, scheduled by a systemd timer, with Restic writing to an encrypted repository on a Synology NFS share.
 This approach keeps backup and restore explicit, avoids tying recovery to PostgreSQL's internal data-directory layout, and captures the sensitive TDLib state alongside the database. Restic repository encryption protects database contents and Telegram session state if the NAS share is accessed directly.
 ## Storage layout
 ### Synology
 Create a dedicated shared folder, for example `dragonsstash-backups`, with:
 - The dedicated backup shared folder is exported through NFS only to the Docker host's fixed IP address.
 - No Internet exposure.
 - Sufficient capacity for the repository plus growth and safety margin.
 The Linux host mounts the share at a stable path such as `/mnt/dragonsstash-backups` using systemd-aware network mount options so a NAS outage does not block normal boot indefinitely.
 ### Restic repository
 The repository lives below the mounted share. The Restic password is stored separately from the repository in a root-readable host secret file and must also be recorded in the operator's offline password-management system. Losing both the NAS and the only copy of the Restic password makes encrypted backups unrecoverable.
 Each successful Restic snapshot contains:
 - A PostgreSQL custom-format dump generated for that run.
 - The contents of the `tdlib_state` Docker volume.
 - The contents of the `tdlib_bot_state` Docker volume.
 - A small manifest with the backup timestamp, application image/version, database migration state, and captured volume paths.
 The `manual_uploads` and `tmp_zips` volumes are excluded. STL binaries continue to live in Telegram; the restored database supplies the metadata and mappings required for the worker and bot to locate and send them.
 ## Backup flow
 The systemd timer invokes one backup command at the chosen nightly time. The command:
 1. Acquires an exclusive lock and refuses to run if another backup is active.
 2. Verifies that the NFS mount is present, writable, and points to the expected backup directory.
 3. Stops the `app`, `worker`, and `bot` services while leaving PostgreSQL running.
 4. Creates a PostgreSQL custom-format dump from the running database.
 5. Creates a manifest for the backup.
 6. Runs one Restic backup over the dump and the read-only mounted TDLib session volumes.
 7. Verifies that Restic created the snapshot successfully.
 8. Applies the retention policy: keep the latest 30 daily snapshots, then prune unreferenced data.
 9. Restarts all stopped services, whether the backup succeeded or failed.
 The service-stop window ensures that application writes and TDLib session updates do not occur while the corresponding data is captured. A failed run must never trigger retention pruning.
 ## Restore flow
 The restore tooling and documentation will support this sequence:
 1. Stop `app`, `worker`, and `bot` for a live restore.
 2. Select and inspect a Restic snapshot.
 3. Restore the PostgreSQL dump and TDLib session contents to a staging location.
 4. Preserve the current database and volumes or confirm that the operator intends to replace them.
 5. Restore `tdlib_state` and `tdlib_bot_state` into their Docker volumes with the expected ownership and paths.
 6. Restore the database from the custom-format dump into the configured PostgreSQL database.
 7. Verify the dump, session-volume layout, database connectivity, and worker/bot authentication startup state.
 8. Start the services and inspect logs for startup, migration, and worker/bot authentication errors.
 After restore, existing database metadata and mappings allow normal Telegram-based STL lookup and delivery. Restoring STL binaries, forwarding Telegram content, and checking archive/STL binary integrity are outside this restore flow.
 The restore process must be safe to rehearse against a disposable Compose project without modifying the live deployment.
 ## Failure handling and verification
 - A missing or read-only NAS mount fails the backup before services are stopped where possible.
 - A lock prevents overlapping backups.
 - A cleanup trap or equivalent guarantees service restart after errors.
 - Backup failure produces a non-zero systemd result and a clear log entry.
 - Retention pruning runs only after a verified successful snapshot.
 - A snapshot listing and repository metadata check run after each backup.
 - The deployment operator completes and records a monthly operational check: `restic check --read-data` followed by a disposable restore rehearsal. This verifies only the PostgreSQL logical dump and TDLib session-state recovery set; archive/STL-content integrity and future forwarding checks remain out of scope.
 - The project documentation describes how to inspect the last successful snapshot and how to recover when the NAS is unavailable.
 ## Security considerations
 - The NFS share is limited to the Docker host and is not exposed to the Internet.
 - Restic encryption protects the repository at rest.
 - PostgreSQL credentials, NAS credentials/rules, and the Restic password are never committed to the repository.
 - Restore commands must avoid printing database passwords or the Restic password in logs.
 - Telegram session volumes are included because they are operationally valuable, but they must be treated as secrets.
 ## Acceptance criteria
 - A nightly systemd timer creates a Restic snapshot on the Synology share.
 - A snapshot includes a PostgreSQL logical dump and both Telegram session volumes, while excluding `manual_uploads` and `tmp_zips`.
 - At least 30 daily recovery points are retained.
 - Each month, the deployment operator runs and records a full `restic check --read-data` and a disposable restore rehearsal of the PostgreSQL dump plus both TDLib session volumes.
 - A simulated host-loss restore reconstructs the database and Telegram session state in a disposable Compose environment.
 - The restored database retains the Telegram metadata and mappings the worker and bot use to locate and send STL binaries that remain in Telegram.
 - A failed backup leaves services running and preserves the last known-good snapshot.
 - The restore procedure is documented well enough for an operator to execute without reading the implementation.
@@ -0,0 +1,211 @@
 # Forward-priority ingestion — design
 **Date:** 2026-07-30
 **Status:** Approved (design), pending spec review → implementation plan
 ## Problem
 The worker ingests every archive the same way regardless of whether it needs to: download the
 full file from the source channel, then re-upload the full file to the destination (archive)
 channel. That download+reupload round-trip was originally necessary because some source channels
 have "restrict saving content" (protected content) enabled, which blocks Telegram-native
 forwarding — for those channels there is no alternative to moving the bytes through the worker.
 But most source channels do NOT restrict forwarding. For those, the round-trip is pure waste:
 Telegram can copy the message from source chat to destination chat server-side, with no bytes
 ever passing through the worker. The worker still needs to end up with the same outcome it has
 today — a destination-channel copy, a dedup-safe identity, and a full inner-file listing — just
 without paying for a download and re-upload to get there.
 Separately, `feat/ranged-archive-listing` (merged to master ahead of this feature) already built
 exactly the missing piece: reading a ZIP/RAR/7z archive's inner-file listing via small ranged
 reads against the file wherever it currently lives (source channel, destination channel — doesn't
 matter), with no full download. It was built for backfilling listings onto already-deduped
 placeholder packages. This feature generalizes that same capability to fresh ingestion, and pairs
 it with a new native-forward upload path.
 ## Goals
 - For channels that allow forwarding: skip download and re-upload entirely for new archives. Use
  Telegram-native forwarding from source chat to destination chat, and the existing ranged-listing
  readers to index inner files, with no full download in the common case.
 - For channels that block forwarding (or when forwarding isn't yet known): keep today's
  download+reupload pipeline exactly as-is.
 - Every ingested package — regardless of path — ends up with the same outcome as today: a
  `Package` row with a valid dedup identity, `destMessageId`/`destMessageIds`, creator, tags, and a
  full inner-file listing (`PackageFile` rows). Indexing completeness must not regress.
 - If the cheap ranged listing fails for a specific archive (bad/unsupported header, CLI error,
  etc.) in an otherwise-forwarding-eligible channel, fall back to today's full download+reupload
  pipeline for that one archive — never forward with an empty or partial listing.
 ## Non-goals
 - No ranged single-entry preview extraction. Forward-path packages still get a preview when a
  channel photo message matches (cheap, unrelated to archive bytes); when there's no matching
  photo, forward-path packages simply have no preview, same as any package where preview
  extraction fails today. In-archive preview extraction (unzip/unrar/7z against a local file) stays
  as a download-path-only feature. May be revisited as a follow-up if it turns out to matter.
 - No reprocessing of already-ingested packages. This only changes behavior for newly-scanned
  archives going forward.
 - No change to the bot's user-delivery leg (`bot/src/tdlib/client.ts` `copyMessageToUser`) — it
  already sends via `inputFileRemote` with no download, and is unaffected by this feature.
 - No change to `config.maxZipSizeMB` or the multipart byte-level split/repack logic. The existing
  size guard runs before either path is chosen, so nothing above the cap reaches the forward path's
  fallback-to-download step either. Splitting simply never engages on the forward path — a
  forwarded message is already within whatever size Telegram accepted when it was first uploaded.
 ## Approaches considered
 **A — Branch inside the existing pipeline (chosen).** Add one fork point in
 `processOneArchiveSet`, immediately after the existing pre-download dedup checks: if the channel
 allows forwarding, attempt the ranged-listing + forward path; on any failure, fall through into
 today's download-based code for that one archive, unchanged. Smallest diff; reuses the existing
 dedup/retry/watermark machinery as-is; matches the file's existing forum-vs-non-forum branching
 style.
 **B — Separate pipeline per channel.** Decide once per channel and route the whole channel through
 either a "forward module" or the existing "download module." Cleaner separation on paper, but
 duplicates the SkippedPackage/stall/watermark bookkeeping that currently lives once in
 `processArchiveSets`/`processOneArchiveSet` — higher regression risk in a large orchestration file
 with no tests at that level. Rejected.
 **C — Strategy-object refactor.** Extract an `IngestStrategy` interface (`download` / `forward`)
 and slim `processOneArchiveSet` to delegate to it. The more "proper" abstraction, but it's a
 structural refactor of already-battle-tested code that doesn't need it for this feature to work.
 Rejected — can revisit later if a third strategy ever appears.
 ## Sequencing
 `feat/ranged-archive-listing` merges to master first, as-is (it's complete and serves a different
 purpose already). This feature is built on a fresh branch off master afterward.
 ## Components
 ### 1. `TelegramChannel.allowsForwarding` (new column, new migration)
 `Boolean?` — nullable, `null` means "not yet checked". Refreshed from TDLib's chat
 protected-content flag (exact field name to be confirmed against the pinned `tdl`/TDLib version
 via docs lookup during implementation — expected to be `chat.has_protected_content`) at the same
 point the worker already calls `getChat` per channel per cycle, mirroring the existing
 `isForum`/`setChannelForum` read-and-persist pattern precisely. `null` or `false` both route to the
 download path — a channel never uses the forward path on unverified permission.
 ### 2. Shared ranged-listing dispatcher
 `readScannedListingRanged` (plus `RangedPart`, `tdlibRangeReader`, and the format-specific
 ZIP/RAR/7z readers) currently live inside `provenance-backfill.ts`. Promote the dispatcher (and
 whatever it depends on) into a shared module (e.g. `worker/src/archive/ranged/dispatch.ts`) so
 `worker.ts` can call the same no-download listing logic for fresh ingestion without a circular
 import. `provenance-backfill.ts` switches to importing from the new shared location; behavior
 unchanged for the existing backfill path.
 ### 3. `forwardArchiveToChannel` (new, `worker/src/upload/forward.ts`)
 Mirrors `uploadToChannel`'s shape and return type (`{ messageId, messageIds }`). Uses TDLib
 `forwardMessages` to copy all parts of an archive set from the source chat to the destination chat
 in one batch call (message IDs in original order), wrapped in the same flood-wait/retry handling
 style as `uploadToChannel`. Followed by the same destination read-back verification style as
 today's post-upload check (`getMessage` on each new destination message ID, confirm a document is
 present).
 ### 4. Dedup identity for forward-path packages
 `Package.contentHash` stays a required unique string, but forward-path packages can't hash real
 bytes. Derivation order:
 1. If the ranged listing's CRC32s are complete (ZIP/RAR today) — hash the sorted CRC32 list into a
   synthetic `fingerprint:<hash>` value, reusing `archive/fingerprint.ts`'s existing
   `crcFingerprint`.
 2. Otherwise (7z, or any incomplete-CRC case) — synthesize `forward:<remoteUniqueId>`, following
   the existing `rebuild:`-prefixed placeholder-hash precedent in `rebuild.ts`.
 Additionally, extend repost detection: before committing to the forward path, compare the new
 listing's CRC fingerprint (via the existing `compareFingerprints`/`fingerprintsMatch` logic already
 used in `provenance-backfill.ts`'s ambiguous-candidate disambiguation) against recent Packages
 sharing the same file name + size. A fingerprint match is treated as a duplicate and skipped, same
 as today's `findRepostedPackage` handling — this is what lets a forwarded copy and a previously
 fully-downloaded copy of the same archive still dedupe against each other, despite never sharing a
 byte-hash-derived `contentHash`.
 ### 5. Fork point in `processOneArchiveSet`
 All existing pre-download checks run first, completely unchanged, in the same order:
 `remote.unique_id` match → `packageExistsBySourceMessage` → `findRepostedPackage` (name+size) →
 cross-channel provenance backfill → size guard (`maxZipSizeMB`).
 Then:
 ```
 if channel.allowsForwarding === true:
    entries = readScannedListingRanged(archiveType, client, scannedParts)
    if entries is not null:
        contentHash = deriveForwardContentHash(entries, remoteUniqueId)
        if fingerprintRepostCheck(entries, fileName, fileSize) finds a match:
            → treat as duplicate, skip (same bookkeeping as today's dup path)
        destResult = forwardArchiveToChannel(client, sourceChatId, partMessageIds, destChatId)
        creator, tags ← derived from entries/filename/channel/topic, same as today
        preview ← channel-photo match only (no in-archive extraction)
        createPackageStub(...) + updatePackageWithMetadata(...), same as today
        counters.zipsForwarded++
        → done
    else:
        → fall through into the existing download/hash/split/upload flow below, unchanged
        (log the fallback for observability)
 else:
    → existing download/hash/split/upload flow, completely unchanged
 ```
 ### 6. Observability
 New `zipsForwarded` counter alongside the existing `zipsFound`/`zipsDuplicate`/`zipsIngested`/
 `zipsBackfilled` counters, surfaced the same way (run activity, ingestion run summary). A WARN-level
 log line when a forwarding-eligible archive falls back to download (mirrors the existing
 `confidence: "ranged" | "full-download-fallback"` logging convention from the ranged-listing
 backfill work), so the fallback rate is visible without digging through debug logs.
 ## Data flow
 ```
 scan → pre-download dedup + size guard (unchanged)
     → channel.allowsForwarding?
         true  → ranged listing
                   ok   → fingerprint dedup check → forward → stub + entries + tags (no in-archive preview) → done
                   null → [fall through] existing download pipeline
         false/unknown → existing download pipeline (unchanged)
 ```
 ## Error handling
 - `forwardMessages` failure (permission revoked mid-run, rate limit, transient Telegram error) —
  same `SkippedPackage`/`SystemNotification` bookkeeping as today's upload failures. Extend
  `inferSkipReason` to recognize forward-specific error text the same way it already recognizes
  upload errors.
 - Fingerprint-repost check finds multiple ambiguous same-name/size candidates that can't be
  uniquely disambiguated — same `INTEGRITY_AUDIT` notification pattern already used in
  `provenance-backfill.ts`: don't guess, surface for manual triage.
 - `allowsForwarding` unknown (channel just linked, not yet scanned by the refresh point) — treated
  as `false`; the download path runs. No channel uses an unverified forwarding permission.
 - Ranged listing throwing instead of returning `null` — treated identically to returning `null`
  (fall through to download), consistent with how the existing ranged readers already treat
  internal errors (they catch and return `null` themselves).
 ## Testing
 - Unit tests (vitest, alongside the existing `archive/*.test.ts` and `archive/ranged/*.test.ts`
  files): the dedup-identity derivation function (fingerprint-hash vs remoteUniqueId-fallback
  branches), the extended fingerprint-based repost check, and `forwardArchiveToChannel`'s
  request-building logic against a mocked TDLib client — same style as the existing ranged-reader
  tests (pure logic, no live TDLib).
 - Live verification (manual — matches this repo's existing convention that the large
  `worker.ts`/`worker.py`-equivalent orchestration function has no automated test coverage and is
  verified live post-deploy): one forwarding-enabled test channel and one protected-content test
  channel. Confirm forward-path packages land with correct entries/tags/dedup identity and
  `destMessageIds`; confirm the protected channel still goes through the unchanged full pipeline;
  confirm a deliberately-unparseable archive in a forwarding-enabled channel correctly falls back
  to download+reupload and still ends up fully indexed.
 ## Rollout
 Local build + deploy, following the same recipe as the ranged-archive-listing work: build
 `worker/Dockerfile` locally, recreate the `dragonsstash-worker` container from the local image (no
 `pull`, no GitHub push required). New DB migration for `TelegramChannel.allowsForwarding`. No
 changes required to the bot or app services.
@@ -0,0 +1 @@
 ALTER TABLE "manual_upload_files" ADD COLUMN "retainedAt" TIMESTAMP(3);
@@ -0,0 +1 @@
 ALTER TABLE "manual_upload_files" DROP COLUMN IF EXISTS "retainedAt";
@@ -0,0 +1,10 @@
 -- AlterTable: forward-priority ingestion support.
 -- allowsForwarding is nullable — null means "not yet checked", treated the
 -- same as false until confirmed true (safe default: download+reupload path).
 ALTER TABLE "telegram_channels"
  ADD COLUMN "allowsForwarding" BOOLEAN;
 -- AlterTable: count of packages forwarded (no download) during a run.
 -- Additive, non-null with a default of 0 — no data change for existing rows.
 ALTER TABLE "ingestion_runs"
  ADD COLUMN "zipsForwarded" INTEGER NOT NULL DEFAULT 0;
@@ -428,6 +428,11 @@ model TelegramChannel {
  isForum    Boolean     @default(false)
  isActive   Boolean     @default(false)
  category   String?     @db.VarChar(64)
  /// Whether this chat currently allows forwarding/saving (the inverse of
  /// TDLib's chat.has_protected_content). Null = not yet checked; treated the
  /// same as false everywhere in the worker (safe default: use the
  /// download+reupload path until this is confirmed true).
  allowsForwarding Boolean?
  createdAt  DateTime    @default(now())
  updatedAt  DateTime    @updatedAt
@@ -570,6 +575,7 @@ model IngestionRun {
  zipsDuplicate   Int             @default(0)
  zipsIngested    Int             @default(0)
  zipsBackfilled  Int             @default(0)
  zipsForwarded   Int             @default(0)
  errorMessage    String?
  // Live activity tracking — written by worker in real-time
@@ -0,0 +1,39 @@
 import { describe, it, expect } from "vitest";
 import { detectArchive, isArchiveAttachment } from "./detect.js";
 describe("detectArchive — 7z numbered multipart (pack.7z.001, pack.7z.002, ...)", () => {
  it("recognizes a 7z multipart part as an archive attachment", () => {
    expect(isArchiveAttachment("Lost Adventures Vol2.7z.001")).toBe(true);
  });
  it("extracts format, baseName, and partNumber", () => {
    const info = detectArchive("Lost Adventures Vol2.7z.001");
    expect(info).toEqual({
      baseName: "Lost Adventures Vol2.7z",
      partNumber: 1,
      format: "7Z",
      pattern: "SEVENZ_NUMBERED",
    });
  });
  it("groups multiple parts under the same baseName + format key regardless of part number", () => {
    const part1 = detectArchive("Lost Adventures Vol2.7z.001");
    const part2 = detectArchive("Lost Adventures Vol2.7z.010");
    expect(part1?.baseName).toBe(part2?.baseName);
    expect(part1?.format).toBe(part2?.format);
    expect(part2?.partNumber).toBe(10);
  });
  it("is case-insensitive on the .7z extension", () => {
    expect(detectArchive("Archive.7Z.002")?.format).toBe("7Z");
  });
  it("still recognizes a standalone single .7z file", () => {
    expect(detectArchive("Single Pack.7z")).toEqual({
      baseName: "Single Pack",
      partNumber: -1,
      format: "7Z",
      pattern: "SINGLE",
    });
  });
 });
@@ -4,7 +4,7 @@ export interface MultipartInfo {
  baseName: string;
  partNumber: number;
  format: ArchiveFormat;
-  pattern: "ZIP_NUMBERED" | "ZIP_LEGACY" | "RAR_PART" | "RAR_LEGACY" | "SINGLE";
+  pattern: "ZIP_NUMBERED" | "ZIP_LEGACY" | "RAR_PART" | "RAR_LEGACY" | "SEVENZ_NUMBERED" | "SINGLE";
 }
 const patterns: {
@@ -46,6 +46,14 @@ const patterns: {
    getBaseName: (m) => m[1],
    getPartNumber: (m) => parseInt(m[2], 10),
  },
  // pack.7z.001, pack.7z.002 (native 7z volume split)
  {
    regex: /^(.+\.7z)\.(\d{3,})$/i,
    format: "7Z",
    pattern: "SEVENZ_NUMBERED",
    getBaseName: (m) => m[1],
    getPartNumber: (m) => parseInt(m[2], 10),
  },
 ];
 /** Extensions we recognize as fetchable documents (archives + standalone files) */
@@ -0,0 +1,30 @@
 import { describe, it, expect } from "vitest";
 import { createHash } from "crypto";
 import { deriveForwardContentHash } from "./forward-identity.js";
 import type { FileEntry } from "./zip-reader.js";
 function entry(crc32: string | null): FileEntry {
  return { path: "a", fileName: "a", extension: null, compressedSize: 1n, uncompressedSize: 1n, crc32 };
 }
 describe("deriveForwardContentHash", () => {
  it("hashes the sorted CRC list when all entries have a CRC32 (ZIP/RAR)", () => {
    const entries = [entry("BBBB"), entry("AAAA")];
    const expectedHash = createHash("sha256").update(["aaaa", "bbbb"].join(",")).digest("hex");
    expect(deriveForwardContentHash(entries, "unique-1", "chan-1", 42n)).toBe(`fingerprint:${expectedHash}`);
  });
  it("falls back to remoteUniqueId when CRCs are incomplete (7z today)", () => {
    const entries = [entry(null), entry("AAAA")];
    expect(deriveForwardContentHash(entries, "unique-42", "chan-1", 42n)).toBe("forward:unique-42");
  });
  it("falls back to sourceChannelId+sourceMessageId when there's no CRC and no remoteUniqueId", () => {
    const entries = [entry(null)];
    expect(deriveForwardContentHash(entries, null, "chan-1", 42n)).toBe("forward:chan-1:42");
  });
  it("falls back past an empty entries list the same way", () => {
    expect(deriveForwardContentHash([], null, "chan-1", 7n)).toBe("forward:chan-1:7");
  });
 });
@@ -0,0 +1,33 @@
 import { createHash } from "crypto";
 import { crcFingerprint } from "./fingerprint.js";
 import type { FileEntry } from "./zip-reader.js";
 /**
 * Derive a Package.contentHash-compatible identity string for a forward-path
 * package (no downloaded bytes exist to hash directly). Priority order:
 *   1. A CRC32-fingerprint hash, when the ranged listing's CRCs are complete
 *      (ZIP/RAR today) — the strongest available signal, since it lets
 *      forward-path and download-path copies of the same archive still
 *      collide/dedupe on identical content.
 *   2. TDLib's remote.unique_id, when CRCs are incomplete (7z today has none).
 *   3. sourceChannelId+sourceMessageId, as a last-resort unique value so the
 *      required-unique Package.contentHash column is always satisfiable.
 * Follows the same `<prefix>:<value>` synthetic-hash convention already used
 * by `rebuild.ts`'s `rebuild:${destChannelId}:${destMessageId}` placeholder.
 */
 export function deriveForwardContentHash(
  entries: FileEntry[],
  remoteUniqueId: string | null,
  sourceChannelId: string,
  sourceMessageId: bigint,
 ): string {
  const fp = crcFingerprint(entries);
  if (fp.complete && fp.crcs.length > 0) {
    const hash = createHash("sha256").update(fp.crcs.join(",")).digest("hex");
    return `fingerprint:${hash}`;
  }
  if (remoteUniqueId) {
    return `forward:${remoteUniqueId}`;
  }
  return `forward:${sourceChannelId}:${sourceMessageId}`;
 }
@@ -0,0 +1,35 @@
 import { describe, it, expect, vi } from "vitest";
 const candidate = {
  id: "pkg-1", archiveType: "ZIP", fileName: "a.zip", fileCount: 3, fileSize: 100n,
  destMessageId: 1n, destMessageIds: [1n], destChannel: { telegramId: 999n },
 };
 vi.mock("../db/queries.js", () => ({
  findFingerprintDedupCandidates: vi.fn(async () => [candidate]),
 }));
 const resolveMock = vi.fn(async (..._args: unknown[]) => [{ path: "x", fileName: "x", extension: null, compressedSize: 1n, uncompressedSize: 1n, crc32: "AAAA" }]);
 const compareMock = vi.fn();
 vi.mock("../provenance-backfill.js", () => ({
  resolveCandidateFingerprintEntries: (...args: unknown[]) => resolveMock(...args),
  compareFingerprints: (...args: unknown[]) => compareMock(...args),
 }));
 import { checkFingerprintRepost } from "./forward-repost-check.js";
 import type { FileEntry } from "./zip-reader.js";
 const newEntries: FileEntry[] = [{ path: "x", fileName: "x", extension: null, compressedSize: 1n, uncompressedSize: 1n, crc32: "AAAA" }];
 describe("checkFingerprintRepost", () => {
  it("reports a duplicate when a candidate's fingerprint matches", async () => {
    compareMock.mockReturnValueOnce("match");
    const result = await checkFingerprintRepost({} as never, newEntries, "a.zip", 100n);
    expect(result).toEqual({ isDuplicate: true, matchedPackageId: "pkg-1" });
  });
  it("reports no duplicate when no candidate matches", async () => {
    compareMock.mockReturnValueOnce("mismatch");
    const result = await checkFingerprintRepost({} as never, newEntries, "a.zip", 100n);
    expect(result).toEqual({ isDuplicate: false, matchedPackageId: null });
  });
 });
@@ -0,0 +1,33 @@
 import type { Client } from "tdl";
 import type { FileEntry } from "./zip-reader.js";
 import { compareFingerprints, resolveCandidateFingerprintEntries } from "../provenance-backfill.js";
 import { findFingerprintDedupCandidates } from "../db/queries.js";
 export interface FingerprintRepostResult {
  isDuplicate: boolean;
  matchedPackageId: string | null;
 }
 /**
 * Cross-channel duplicate check for the forward-priority path: compare the
 * new archive's CRC fingerprint against every existing Package sharing its
 * name+size, regardless of which channel or ingestion path produced them.
 * This is what lets a forwarded copy dedupe against a previously
 * fully-downloaded copy of the same archive, despite never sharing a
 * byte-hash-derived contentHash.
 */
 export async function checkFingerprintRepost(
  client: Client,
  entries: FileEntry[],
  fileName: string,
  fileSize: bigint,
 ): Promise<FingerprintRepostResult> {
  const candidates = await findFingerprintDedupCandidates(fileName, fileSize);
  for (const candidate of candidates) {
    const candidateEntries = await resolveCandidateFingerprintEntries(client, candidate);
    if (compareFingerprints(entries, candidateEntries) === "match") {
      return { isDuplicate: true, matchedPackageId: candidate.id };
    }
  }
  return { isDuplicate: false, matchedPackageId: null };
 }
@@ -0,0 +1,15 @@
 import { describe, it, expect } from "vitest";
 import { readScannedListingRanged } from "./dispatch.js";
 describe("readScannedListingRanged", () => {
  it("returns null for an unknown archive type without calling the reader", async () => {
    const read = async () => Buffer.alloc(0);
    const result = await readScannedListingRanged(
      "DOCUMENT",
      { invoke: async () => ({}) } as never,
      [{ fileId: "1", fileSize: 100n, fileName: "a.pdf" }],
    );
    expect(result).toBeNull();
    void read; // unused placeholder kept out of the dispatch call — DOCUMENT never reaches a reader
  });
 });
@@ -0,0 +1,62 @@
 import type { Client } from "tdl";
 import { downloadFileRange } from "../../tdlib/range-download.js";
 import { parseZipCentralDirectoryFromTail, MIN_ZIP_TAIL_BYTES } from "../central-directory.js";
 import { childLogger } from "../../util/logger.js";
 import type { FileEntry } from "../zip-reader.js";
 import { readSevenZListingRanged, type RangedPart } from "./sevenz-ranged.js";
 import { readRarListingRanged } from "./rar-ranged.js";
 import { tdlibRangeReader } from "./range-reader.js";
 const log = childLogger("ranged-dispatch");
 /**
 * Read a ZIP central directory from the tail of a (possibly multipart)
 * archive. `parts` is ordered; only the LAST part carries the EOCD record.
 * `fileSize` on each part is that part's own size (NOT the whole-archive
 * total) so the download offset stays within that part's bounds, while
 * `tailStart` passed to the parser is the logical whole-archive offset
 * (preceding parts' sizes + the offset within the last part).
 */
 export async function readScannedZipListing(
  client: Client,
  parts: { fileId: string; fileSize: bigint }[],
 ): Promise<FileEntry[] | null> {
  if (parts.length === 0) return null;
  const lastPart = parts[parts.length - 1];
  const precedingSize = parts.slice(0, -1).reduce((sum, p) => sum + Number(p.fileSize), 0);
  const lastSize = Number(lastPart.fileSize);
  for (const tailBytes of [MIN_ZIP_TAIL_BYTES, MIN_ZIP_TAIL_BYTES * 4]) {
    const partOffset = Math.max(0, lastSize - tailBytes);
    const downloadLen = Math.min(tailBytes, lastSize);
    try {
      const buf = await downloadFileRange(client, lastPart.fileId, partOffset, downloadLen, lastPart.fileSize);
      const tailStart = precedingSize + partOffset;
      return parseZipCentralDirectoryFromTail(buf, tailStart);
    } catch (err) {
      if (err instanceof RangeError) continue; // try a larger tail
      log.warn({ err, fileId: lastPart.fileId }, "ranged ZIP listing failed");
      return null;
    }
  }
  return null;
 }
 /**
 * Dispatch a (no-download) inner-file listing read by archive type. Used both
 * by the provenance-backfill path (reading an already-uploaded copy) and the
 * forward-priority ingestion path (reading the source channel's copy before
 * any download/forward decision is made) — the read itself only needs
 * {fileId, fileSize, fileName}, so it doesn't matter which channel the file
 * currently lives in.
 */
 export async function readScannedListingRanged(
  archiveType: string,
  client: Client,
  parts: RangedPart[],
 ): Promise<FileEntry[] | null> {
  const read = tdlibRangeReader(client);
  if (archiveType === "ZIP") return readScannedZipListing(client, parts);
  if (archiveType === "SEVEN_Z") return readSevenZListingRanged(parts, read);
  if (archiveType === "RAR") return readRarListingRanged(parts, read);
  return null;
 }
@@ -1,7 +1,19 @@
-import { describe, it, expect } from "vitest";
+import { describe, it, expect, vi } from "vitest";
 import { readVint, detectRarSignature, parseRar5BlockExtent, parseRar4BlockExtent, walkRarVolume, readRarListingRanged } from "./rar-ranged.js";
 import type { RangeReader } from "./range-reader.js";
 let capturedFirstBytes: Buffer | null = null;
 vi.mock("../rar-reader.js", () => ({
  readRarContents: async (firstPartPath: string) => {
    const { readFile } = await import("fs/promises");
    const reconstructed = await readFile(firstPartPath);
    capturedFirstBytes = reconstructed.subarray(0, 8);
    return [{ name: "dummy", size: 0 }]; // non-empty so listFromSparse returns it
  },
 }));
 const { readVint, detectRarSignature, parseRar5BlockExtent, parseRar4BlockExtent, walkRarVolume, readRarListingRanged } =
  await import("./rar-ranged.js");
 describe("readVint", () => {
  it("reads single-byte and multi-byte values (base-128 LE)", () => {
    expect(readVint(Buffer.from([0x08]), 0)).toEqual({ value: 8, bytes: 1 });
@@ -123,6 +135,23 @@ describe("readRarListingRanged (single part)", () => {
    const res = await readRarListingRanged([{ fileId: "1", fileSize: BigInt(vol.length), fileName: "a.rar" }], read);
    expect(res === null || Array.isArray(res)).toBe(true); // real unrar parse covered live
  });
  it("preserves the RAR signature bytes in the reconstructed sparse file", async () => {
    // Regression test: walkRarVolume starts at pos = sigLen and never
    // harvests the signature itself. If readRarListingRanged forgets to add
    // it as its own region, the reconstructed file starts with zero bytes
    // instead of "Rar!\x1a\x07\x01\x00", and every real unrar invocation
    // rejects it as "not RAR archive" — silently forcing every RAR archive
    // through the expensive download+reupload fallback regardless of the
    // channel's forwarding permission.
    const vol = buildRar5Volume();
    const sig = vol.subarray(0, 8);
    const read: RangeReader = async (_id, offset, length) => vol.subarray(offset, offset + length);
    capturedFirstBytes = null;
    await readRarListingRanged([{ fileId: "1", fileSize: BigInt(vol.length), fileName: "a.rar" }], read);
    expect(capturedFirstBytes).not.toBeNull();
    expect(capturedFirstBytes).toEqual(sig);
  });
 });
 describe("readRarListingRanged (multipart)", () => {
@@ -69,11 +69,17 @@ export async function walkRarVolume(
  let blocks = 0;
  try {
    while (pos < size) {
-      if (++blocks > MAX_RAR_BLOCKS) return null;
+      if (++blocks > MAX_RAR_BLOCKS) {
        rlog.warn({ fileId: part.fileId, fileName: part.fileName, blocks }, "RAR walk aborted — exceeded MAX_RAR_BLOCKS");
        return null;
      }
      const chunkLen = Math.min(HEADER_CHUNK, size - pos);
      let chunk = await read(part.fileId, pos, chunkLen, part.fileSize);
      const ext = version === 5 ? parseRar5BlockExtent(chunk, 0) : parseRar4BlockExtent(chunk, 0);
-      if (ext.headerBytes > MAX_RAR_HEADER_BYTES) return null;
+      if (ext.headerBytes > MAX_RAR_HEADER_BYTES) {
        rlog.warn({ fileId: part.fileId, fileName: part.fileName, pos, headerBytes: ext.headerBytes }, "RAR walk aborted — headerBytes exceeded MAX_RAR_HEADER_BYTES");
        return null;
      }
      // Ensure we have the full header bytes to harvest (long filenames).
      let headerBuf = chunk;
      if (ext.headerBytes > chunk.length) {
@@ -82,13 +88,16 @@ export async function walkRarVolume(
      regions.push({ offset: pos, bytes: headerBuf.subarray(0, Math.min(ext.headerBytes, size - pos)) });
      if (ext.isEnd) break;
      const advance = ext.headerBytes + ext.dataSize;
-      if (advance <= 0) return null;
+      if (advance <= 0) {
        rlog.warn({ fileId: part.fileId, fileName: part.fileName, pos, headerBytes: ext.headerBytes, dataSize: ext.dataSize }, "RAR walk aborted — non-positive advance");
        return null;
      }
      if (pos + advance > size) break; // data clamped at the volume boundary (multipart continuation)
      pos += advance;
    }
    return regions;
  } catch (err) {
-    rlog.warn({ err, fileId: part.fileId }, "RAR volume walk failed");
+    rlog.warn({ err, fileId: part.fileId, fileName: part.fileName, pos }, "RAR volume walk failed");
    return null;
  }
 }
@@ -101,10 +110,18 @@ export async function readRarListingRanged(
  for (const part of parts) {
    const head = await read(part.fileId, 0, 16, part.fileSize);
    const sig = detectRarSignature(head);
-    if (!sig) return null;
+    if (!sig) {
      rlog.warn({ fileId: part.fileId, fileName: part.fileName, head: head.toString("hex") }, "RAR signature not detected at offset 0");
      return null;
    }
    const regions = await walkRarVolume(read, part, sig.version, sig.sigLen);
    if (!regions) return null;
-    sparseParts.push({ fileName: part.fileName, size: Number(part.fileSize), regions });
+    // walkRarVolume starts at pos = sigLen and never harvests the signature
    // itself, so it must be added as its own region — otherwise the
    // reconstructed sparse file starts with zero bytes instead of the "Rar!"
    // magic, and unrar rejects it outright as "not RAR archive".
    const sigRegion = { offset: 0, bytes: head.subarray(0, sig.sigLen) };
    sparseParts.push({ fileName: part.fileName, size: Number(part.fileSize), regions: [sigRegion, ...regions] });
  }
  return listFromSparse(sparseParts, readRarContents);
 }
@@ -377,6 +377,7 @@ export interface ActivityUpdate {
  zipsDuplicate?: number;
  zipsIngested?: number;
  zipsBackfilled?: number;
  zipsForwarded?: number;
 }
 export async function updateRunActivity(
@@ -401,6 +402,7 @@ export async function updateRunActivity(
      ...(activity.zipsDuplicate !== undefined && { zipsDuplicate: activity.zipsDuplicate }),
      ...(activity.zipsIngested !== undefined && { zipsIngested: activity.zipsIngested }),
      ...(activity.zipsBackfilled !== undefined && { zipsBackfilled: activity.zipsBackfilled }),
      ...(activity.zipsForwarded !== undefined && { zipsForwarded: activity.zipsForwarded }),
      ...(activity.currentTopicId !== undefined && { currentTopicId: activity.currentTopicId }),
      ...(activity.currentAccountChannelMapId !== undefined && {
        currentAccountChannelMapId: activity.currentAccountChannelMapId,
@@ -432,6 +434,7 @@ export async function completeIngestionRun(
    zipsDuplicate: number;
    zipsIngested: number;
    zipsBackfilled: number;
    zipsForwarded: number;
  }
 ) {
  return db.ingestionRun.update({
@@ -639,6 +642,13 @@ export async function setChannelForum(channelId: string, isForum: boolean) {
  });
 }
 export async function setChannelAllowsForwarding(channelId: string, allowsForwarding: boolean) {
  return db.telegramChannel.update({
    where: { id: channelId },
    data: { allowsForwarding },
  });
 }
 export async function getTopicProgress(mappingId: string) {
  return db.topicProgress.findMany({
    where: { accountChannelMapId: mappingId },
@@ -1023,6 +1033,30 @@ export interface PlaceholderCandidate {
  destChannel: { telegramId: bigint } | null;
 }
 type PlaceholderRow = {
  id: string; archiveType: string; fileName: string; fileCount: number; fileSize: bigint;
  destMessageId: bigint | null; destMessageIds: bigint[]; destChannelId: string | null;
 };
 async function enrichWithDestChannel(rows: PlaceholderRow[]): Promise<PlaceholderCandidate[]> {
  if (rows.length === 0) return [];
  const destChannelIds = [...new Set(rows.map((r) => r.destChannelId).filter((id): id is string => !!id))];
  const channels = destChannelIds.length
    ? await db.telegramChannel.findMany({
        where: { id: { in: destChannelIds } },
        select: { id: true, telegramId: true },
      })
    : [];
  const telegramIdById = new Map(channels.map((c) => [c.id, c.telegramId]));
  return rows.map((row) => ({
    id: row.id, archiveType: row.archiveType, fileName: row.fileName, fileCount: row.fileCount, fileSize: row.fileSize,
    destMessageId: row.destMessageId, destMessageIds: row.destMessageIds,
    destChannel: row.destChannelId && telegramIdById.has(row.destChannelId)
      ? { telegramId: telegramIdById.get(row.destChannelId)! }
      : null,
  }));
 }
 /**
 * Find every placeholder Package matching name+size (oldest first). Package
 * has no direct `destChannel` relation (only the scalar `destChannelId`), so
@@ -1052,29 +1086,30 @@ export async function findPlaceholderCandidates(
    },
    orderBy: { indexedAt: "asc" },
  });
-  if (rows.length === 0) return [];
+  return enrichWithDestChannel(rows);
 }
-  const destChannelIds = [...new Set(rows.map((r) => r.destChannelId).filter((id): id is string => !!id))];
+/**
-  const channels = destChannelIds.length
+ * Find every uploaded Package (any provenance, any channel) matching
-    ? await db.telegramChannel.findMany({
+ * name+size, for the forward-priority path's cross-channel CRC-fingerprint
-        where: { id: { in: destChannelIds } },
+ * dedup check. Unlike findPlaceholderCandidates, this is NOT restricted to
-        select: { id: true, telegramId: true },
+ * placeholder rows — it exists to catch the case where the exact same
-      })
+ * archive was independently uploaded (not reposted/forwarded) to two
-    : [];
+ * different source channels.
-  const telegramIdById = new Map(channels.map((c) => [c.id, c.telegramId]));
+ */
-
+export async function findFingerprintDedupCandidates(
-  return rows.map((row) => ({
+  fileName: string,
-    id: row.id,
+  fileSize: bigint,
-    archiveType: row.archiveType,
+): Promise<PlaceholderCandidate[]> {
-    fileName: row.fileName,
+  const rows = await db.package.findMany({
-    fileCount: row.fileCount,
+    where: { fileName, fileSize, destMessageId: { not: null } },
-    fileSize: row.fileSize,
+    select: {
-    destMessageId: row.destMessageId,
+      id: true, archiveType: true, fileName: true, fileCount: true, fileSize: true,
-    destMessageIds: row.destMessageIds,
+      destMessageId: true, destMessageIds: true, destChannelId: true,
-    destChannel: row.destChannelId && telegramIdById.has(row.destChannelId)
+    },
-      ? { telegramId: telegramIdById.get(row.destChannelId)! }
+    orderBy: { indexedAt: "asc" },
-      : null,
+  });
-  }));
+  return enrichWithDestChannel(rows);
 }
 export async function findPlaceholderCandidate(
@@ -1,8 +1,6 @@
 import { db } from "./db/client.js";
 import { childLogger } from "./util/logger.js";
 import { downloadFileRange } from "./tdlib/range-download.js";
 import { invokeWithTimeout } from "./tdlib/download.js";
 import { parseZipCentralDirectoryFromTail, MIN_ZIP_TAIL_BYTES } from "./archive/central-directory.js";
 import { fingerprintsMatch, crcFingerprint } from "./archive/fingerprint.js";
 import {
  findPlaceholderCandidates,
@@ -15,6 +13,7 @@ import { readSevenZListingRanged, type RangedPart } from "./archive/ranged/seven
 import { readRarListingRanged } from "./archive/ranged/rar-ranged.js";
 import { tdlibRangeReader } from "./archive/ranged/range-reader.js";
 import { fullDownloadListing } from "./archive/ranged/fallback.js";
 import { readScannedZipListing, readScannedListingRanged } from "./archive/ranged/dispatch.js";
 import type { Client } from "tdl";
 const log = childLogger("provenance-backfill");
@@ -36,38 +35,6 @@ export interface BackfillArgs {
  previewMsgId?: bigint | null;
 }
 /**
 * Read a ZIP central directory from the tail of a (possibly multipart)
 * archive. `parts` is ordered; only the LAST part carries the EOCD record.
 * `fileSize` on each part is that part's own size (NOT the whole-archive
 * total) so the download offset stays within that part's bounds, while
 * `tailStart` passed to the parser is the logical whole-archive offset
 * (preceding parts' sizes + the offset within the last part).
 */
 async function readScannedZipListing(
  client: Client,
  parts: { fileId: string; fileSize: bigint }[],
 ): Promise<FileEntry[] | null> {
  if (parts.length === 0) return null;
  const lastPart = parts[parts.length - 1];
  const precedingSize = parts.slice(0, -1).reduce((sum, p) => sum + Number(p.fileSize), 0);
  const lastSize = Number(lastPart.fileSize);
  for (const tailBytes of [MIN_ZIP_TAIL_BYTES, MIN_ZIP_TAIL_BYTES * 4]) {
    const partOffset = Math.max(0, lastSize - tailBytes);
    const downloadLen = Math.min(tailBytes, lastSize);
    try {
      const buf = await downloadFileRange(client, lastPart.fileId, partOffset, downloadLen, lastPart.fileSize);
      const tailStart = precedingSize + partOffset;
      return parseZipCentralDirectoryFromTail(buf, tailStart);
    } catch (err) {
      if (err instanceof RangeError) continue; // try a larger tail
      log.warn({ err, fileId: lastPart.fileId }, "ranged ZIP listing failed");
      return null;
    }
  }
  return null;
 }
 /**
 * Resolve the destination copy's message(s) into ranged parts (file id +
 * size + name), in order, so a multipart destination copy is reconstructed
@@ -106,25 +73,13 @@ async function resolveDestParts(
  }
 }
 async function readScannedListingRanged(
  archiveType: string,
  client: Client,
  parts: RangedPart[],
 ): Promise<FileEntry[] | null> {
  const read = tdlibRangeReader(client);
  if (archiveType === "ZIP") return readScannedZipListing(client, parts);
  if (archiveType === "SEVEN_Z") return readSevenZListingRanged(parts, read);
  if (archiveType === "RAR") return readRarListingRanged(parts, read);
  return null;
 }
 /**
 * Build the CRC fingerprint entries for a placeholder candidate: start from
 * its stored PackageFile CRCs, and if those are incomplete (e.g. a rebuild
 * candidate with fileCount === 0), fall back to a fresh ranged read of the
 * candidate's own copy in the destination channel (Task 9).
 */
-async function resolveCandidateFingerprintEntries(
+export async function resolveCandidateFingerprintEntries(
  client: Client,
  candidate: PlaceholderCandidate,
 ): Promise<FileEntry[]> {
@@ -164,7 +119,7 @@ async function resolveCandidateFingerprintEntries(
 * refute a match — callers must fall back to name+size confidence rather
 * than treating this as a mismatch.
 */
-function compareFingerprints(a: FileEntry[], b: FileEntry[]): "match" | "mismatch" | "incomplete" {
+export function compareFingerprints(a: FileEntry[], b: FileEntry[]): "match" | "mismatch" | "incomplete" {
  const fa = crcFingerprint(a);
  const fb = crcFingerprint(b);
  if (!fa.complete || !fb.complete) return "incomplete";
@@ -84,15 +84,26 @@ export async function recoverIncompleteUploads(): Promise<void> {
    // Batch size for getMessages. TDLib accepts up to ~100 IDs per call.
    // Using 100 means 20k packages → ~200 round-trips instead of 20k.
    const BATCH_SIZE = 100;
    // Telegram soft-throttles sustained sequential history reads from user
    // accounts — no FLOOD_WAIT, just growing per-call latency the longer a
    // single in-flight request stream runs. A small number of batches in
    // flight at once cuts wall-clock time substantially without approaching
    // real per-account rate limits.
    const CONCURRENCY = 3;
    const tdlibClient = client;
    const destChannelInfo = destChannel;
    const batches: (typeof packages)[] = [];
    for (const [, channelPackages] of byChannel) {
      // Group packages by destChannelId (already done) — within each group,
      // process in batches via getMessages (plural).
      for (let i = 0; i < channelPackages.length; i += BATCH_SIZE) {
-        const batch = channelPackages.slice(i, i + BATCH_SIZE);
+        batches.push(channelPackages.slice(i, i + BATCH_SIZE));
      }
    }
    async function processBatch(batch: typeof packages) {
      const batchResults = await verifyMessagesBatch(
-          client,
+        tdlibClient,
-          destChannel.telegramId,
+        destChannelInfo.telegramId,
        batch.map((p) => p.destMessageId!)
      );
@@ -143,7 +154,17 @@ export async function recoverIncompleteUploads(): Promise<void> {
        }
      }
    }
    let nextBatchIndex = 0;
    async function poolWorker() {
      while (nextBatchIndex < batches.length) {
        const batch = batches[nextBatchIndex++];
        await processBatch(batch);
      }
    }
    await Promise.all(
      Array.from({ length: Math.min(CONCURRENCY, batches.length) }, () => poolWorker())
    );
    log.info(
      {
@@ -142,3 +142,34 @@ export async function closeTdlibClient(client: Client): Promise<void> {
    log.warn({ err }, "Error closing TDLib client");
  }
 }
 /**
 * Prune TDLib's local file cache (filesDirectory). TDLib keeps a permanent
 * copy of every file it has ever downloaded or uploaded — via inputFileLocal
 * uploads in particular — with no automatic cleanup. That cache is redundant
 * (the content already lives in the source and destination Telegram chats)
 * and grows unbounded, so it's cleared after every ingestion run. A short
 * immunity_delay protects files from an in-flight operation that might still
 * reference them.
 */
 export async function optimizeTdlibStorage(
  client: Client,
  accountId: string
 ): Promise<void> {
  try {
    const result = (await client.invoke({
      _: "optimizeStorage",
      size: 0,
      ttl: 0,
      count: 0,
      immunity_delay: 300,
      return_deleted_file_statistics: true,
    })) as { size?: number; count?: number };
    log.info(
      { accountId, freedBytes: result.size, freedCount: result.count },
      "TDLib local file cache pruned"
    );
  } catch (err) {
    log.warn({ err, accountId }, "TDLib storage optimization failed");
  }
 }
@@ -0,0 +1,39 @@
 import { describe, it, expect, vi } from "vitest";
 import { forwardArchiveToChannel } from "./forward.js";
 function fakeClient(response: unknown) {
  return { invoke: vi.fn(async () => response) } as never;
 }
 describe("forwardArchiveToChannel", () => {
  it("sorts message ids ascending and sends them via forwardMessages", async () => {
    const invoke = vi.fn(async (req: { message_ids: number[] }) => ({
      messages: req.message_ids.map((id) => ({ id: id + 1000 })),
    }));
    const client = { invoke } as never;
    const result = await forwardArchiveToChannel(client, 111n, 222n, [30n, 10n, 20n]);
    expect(invoke).toHaveBeenCalledWith(
      expect.objectContaining({
        _: "forwardMessages",
        chat_id: 222,
        from_chat_id: 111,
        message_ids: [10, 20, 30],
        send_copy: false,
      }),
    );
    expect(result.messageId).toBe(1010n);
    expect(result.messageIds).toEqual([1010n, 1020n, 1030n]);
  });
  it("throws when Telegram returns null for a message (can't be forwarded)", async () => {
    const client = fakeClient({ messages: [{ id: 1001 }, null] });
    await expect(forwardArchiveToChannel(client, 111n, 222n, [10n, 20n])).rejects.toThrow(/could not forward/);
  });
  it("throws when the response has the wrong number of messages", async () => {
    const client = fakeClient({ messages: [{ id: 1001 }] });
    await expect(forwardArchiveToChannel(client, 111n, 222n, [10n, 20n])).rejects.toThrow(/expected 2/);
  });
 });
@@ -0,0 +1,73 @@
 import type { Client } from "tdl";
 import { childLogger } from "../util/logger.js";
 import { withFloodWait } from "../util/retry.js";
 const log = childLogger("forward");
 export interface ForwardResult {
  messageId: bigint;
  messageIds: bigint[];
 }
 /**
 * Forward all parts of an archive set from the source chat directly to the
 * destination chat via TDLib's forwardMessages — no download, no re-upload.
 * Only usable when the source channel allows forwarding
 * (TelegramChannel.allowsForwarding); the caller is responsible for that
 * check. message_ids must be in strictly increasing order per the TDLib API,
 * so this always sorts them regardless of the order they're passed in.
 */
 export async function forwardArchiveToChannel(
  client: Client,
  fromChatId: bigint,
  toChatId: bigint,
  sourceMessageIds: bigint[],
 ): Promise<ForwardResult> {
  const sortedIds = [...sourceMessageIds].sort((a, b) => (a < b ? -1 : a > b ? 1 : 0));
  const numericIds = sortedIds.map((id) => Number(id));
  log.info(
    { fromChatId: Number(fromChatId), toChatId: Number(toChatId), count: numericIds.length },
    "Forwarding archive to destination channel"
  );
  const result = (await withFloodWait(
    () =>
      client.invoke({
        _: "forwardMessages",
        chat_id: Number(toChatId),
        topic_id: null,
        from_chat_id: Number(fromChatId),
        message_ids: numericIds,
        options: null,
        send_copy: false,
        remove_caption: false,
      } as never),
    "forwardMessages"
  )) as { messages: ({ id: number } | null)[] };
  const forwarded = result.messages;
  if (!forwarded || forwarded.length !== numericIds.length) {
    throw new Error(
      `forwardMessages returned ${forwarded?.length ?? 0} messages, expected ${numericIds.length}`
    );
  }
  const messageIds: bigint[] = [];
  for (let i = 0; i < forwarded.length; i++) {
    const msg = forwarded[i];
    if (!msg) {
      throw new Error(
        `forwardMessages could not forward source message ${sortedIds[i]} (Telegram returned null — message may not be forwardable)`
      );
    }
    messageIds.push(BigInt(msg.id));
  }
  log.info(
    { fromChatId: Number(fromChatId), toChatId: Number(toChatId), messageIds: messageIds.map(Number) },
    "Forward confirmed by Telegram"
  );
  return { messageId: messageIds[0], messageIds };
 }
@@ -16,6 +16,7 @@ import {
  updateLastProcessedMessage,
  updateRunActivity,
  setChannelForum,
  setChannelAllowsForwarding,
  getTopicProgress,
  upsertTopicProgress,
  upsertChannel,
@@ -41,7 +42,7 @@ import {
  isTopicFetchEnabled,
 } from "./db/queries.js";
 import type { ActivityUpdate } from "./db/queries.js";
-import { createTdlibClient, closeTdlibClient } from "./tdlib/client.js";
+import { createTdlibClient, closeTdlibClient, optimizeTdlibStorage } from "./tdlib/client.js";
 import {
  getAccountChats,
  joinChatByInviteLink,
@@ -63,6 +64,10 @@ import { hashParts } from "./archive/hash.js";
 import { readZipCentralDirectory } from "./archive/zip-reader.js";
 import { readRarContents } from "./archive/rar-reader.js";
 import { read7zContents } from "./archive/sevenz-reader.js";
 import { readScannedListingRanged } from "./archive/ranged/dispatch.js";
 import { deriveForwardContentHash } from "./archive/forward-identity.js";
 import { checkFingerprintRepost } from "./archive/forward-repost-check.js";
 import { forwardArchiveToChannel } from "./upload/forward.js";
 import { tryProvenanceBackfill } from "./provenance-backfill.js";
 import { byteLevelSplit, concatenateFiles } from "./archive/split.js";
 import { uploadToChannel, UploadStallError } from "./upload/channel.js";
@@ -316,6 +321,7 @@ interface PipelineContext {
    zipsDuplicate: number;
    zipsIngested: number;
    zipsBackfilled: number;
    zipsForwarded: number;
  };
  /** Creator from forum topic name (null for non-forum). */
  topicCreator: string | null;
@@ -425,6 +431,7 @@ export async function runWorkerForAccount(
      zipsDuplicate: 0,
      zipsIngested: 0,
      zipsBackfilled: 0,
      zipsForwarded: 0,
    };
    try {
@@ -493,9 +500,13 @@ export async function runWorkerForAccount(
        try {
        // ── Ensure TDLib knows about this chat ──
        // getChats may not have loaded all channels (pagination, archive folder, etc.)
-        // so we explicitly load each channel before scanning.
+        // so we explicitly load each channel before scanning. The response is
        // also where we read has_protected_content (below) to decide whether
        // this channel is eligible for the forward-priority ingestion path.
        // eslint-disable-next-line @typescript-eslint/no-explicit-any
        let chatInfo: any;
        try {
-          await client.invoke({
+          chatInfo = await client.invoke({
            _: "getChat",
            chat_id: Number(channel.telegramId),
          });
@@ -517,6 +528,28 @@ export async function runWorkerForAccount(
          );
        }
        // ── Check if channel allows forwarding ──
        // TDLib's chat.has_protected_content is documented on the general
        // Chat object (core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1chat.html),
        // but PENDING LIVE VERIFICATION here: confirm on first deploy that a
        // real chatTypeSupergroup/channel response actually populates this
        // field (some TDLib doc pages describe it in the context of basic
        // groups only). If it's ever `undefined` in practice, this block is a
        // no-op and allowsForwarding stays at its last-known/null value —
        // which safely keeps the channel on the download path.
        const hasProtectedContent: boolean | undefined = chatInfo?.has_protected_content;
        if (typeof hasProtectedContent === "boolean") {
          const allowsForwarding = !hasProtectedContent;
          if (allowsForwarding !== channel.allowsForwarding) {
            await setChannelAllowsForwarding(channel.id, allowsForwarding);
            accountLog.info(
              { channelId: channel.id, title: channel.title, allowsForwarding },
              "Updated channel forwarding permission"
            );
          }
          channel.allowsForwarding = allowsForwarding;
        }
        const pipelineCtx: PipelineContext = {
          client,
          runId: activeRunId,
@@ -1170,6 +1203,7 @@ export async function runWorkerForAccount(
      accountLog.info({ counters }, "Ingestion run completed");
    } finally {
      await throttled.flush();
      await optimizeTdlibStorage(client, account.id);
      await closeTdlibClient(client);
    }
  } catch (err) {
@@ -1190,7 +1224,7 @@ export async function runWorkerForAccount(
 */
 function inferSkipReason(errMsg: string): "DOWNLOAD_FAILED" | "UPLOAD_FAILED" | "EXTRACT_FAILED" {
  const lower = errMsg.toLowerCase();
-  if (lower.includes("upload") || lower.includes("too many requests") || lower.includes("retry after") || lower.includes("send")) {
+  if (lower.includes("upload") || lower.includes("forward") || lower.includes("too many requests") || lower.includes("retry after") || lower.includes("send")) {
    return "UPLOAD_FAILED";
  }
  if (lower.includes("extract") || lower.includes("metadata") || lower.includes("central directory") || lower.includes("archive")) {
@@ -1731,6 +1765,31 @@ async function processOneArchiveSet(
    return null;
  }
  // ── Forward-priority path ──
  // If the source channel allows forwarding, try to index + forward without a
  // local download. Any failure (ranged listing miss, blocked/failed forward)
  // falls through into the existing download pipeline below so indexing
  // completeness never regresses.
  if (channel.allowsForwarding === true) {
    try {
      const forwardResult = await tryForwardArchiveSet(
        ctx, archiveSet, setIdx, totalSets, previewMatches, ingestionRunId
      );
      if (forwardResult !== undefined) {
        return forwardResult;
      }
      accountLog.info(
        { fileName: archiveName },
        "Forward path unavailable for this archive — falling back to download+reupload"
      );
    } catch (forwardPathErr) {
      accountLog.warn(
        { err: forwardPathErr, fileName: archiveName },
        "Forward path threw unexpectedly — falling back to download+reupload"
      );
    }
  }
  const tempPaths: string[] = [];
  let splitPaths: string[] = [];
@@ -2297,6 +2356,173 @@ async function processOneArchiveSet(
  }
 }
 /**
 * Attempt the forward-priority path for one archive set: ranged listing (no
 * download) + native Telegram forward to the destination channel.
 *
 * Returns `undefined` when the forward path isn't usable for this specific
 * archive (ranged listing failed, or the forward itself failed) — the caller
 * falls through to the existing download+reupload pipeline in that case, so
 * indexing completeness never regresses.
 *
 * Returns `null` when the archive is a confirmed duplicate (skip, same
 * contract as the pre-download dedup checks earlier in the caller).
 *
 * Returns the new Package id on success.
 */
 async function tryForwardArchiveSet(
  ctx: PipelineContext,
  archiveSet: ArchiveSet,
  setIdx: number,
  totalSets: number,
  previewMatches: Map<string, { id: bigint; fileId: string }>,
  ingestionRunId: string,
 ): Promise<string | null | undefined> {
  const {
    client, channelTitle, channel,
    destChannelTelegramId, destChannelId,
    counters, topicCreator, sourceTopicId, accountLog,
  } = ctx;
  void setIdx;
  void totalSets;
  const archiveName = archiveSet.parts[0].fileName;
  const archType = archiveSet.type === "7Z" ? ("SEVEN_Z" as const) : archiveSet.type;
  const scannedParts = archiveSet.parts.map((p) => ({
    fileId: p.fileId,
    fileSize: p.fileSize,
    fileName: p.fileName,
  }));
  // Only ZIP/RAR/7z have a ranged-listing reader. For anything else (a
  // standalone DOCUMENT/STL/3MF attachment), or when the ranged listing
  // fails for a type that does have one, forward anyway with an empty
  // entries list instead of falling back to download+reupload —
  // deriveForwardContentHash and the repost/dedup checks all degrade
  // gracefully to remote.unique_id-based identity when entries are
  // empty/incomplete (see forward-identity.ts), and the entire point of a
  // forwarding-enabled channel is to avoid the download+reupload cost
  // regardless of whether inner contents can be indexed.
  const entries =
    archType === "ZIP" || archType === "RAR" || archType === "SEVEN_Z"
      ? (await readScannedListingRanged(archType, client, scannedParts)) ?? []
      : [];
  const totalArchiveSize = archiveSet.parts.reduce((sum, p) => sum + p.fileSize, 0n);
  const firstRemoteUniqueId = archiveSet.parts[0].remoteUniqueId ?? null;
  const contentHash = deriveForwardContentHash(
    entries,
    firstRemoteUniqueId,
    channel.id,
    archiveSet.parts[0].id,
  );
  if (await packageExistsByHash(contentHash)) {
    counters.zipsDuplicate++;
    accountLog.debug({ fileName: archiveName, contentHash }, "Forward-path duplicate (hash), skipping");
    return null;
  }
  const repost = await checkFingerprintRepost(client, entries, archiveName, totalArchiveSize);
  if (repost.isDuplicate) {
    counters.zipsDuplicate++;
    accountLog.info(
      { fileName: archiveName, matchedPackageId: repost.matchedPackageId },
      "Forward-path duplicate (CRC fingerprint match against another channel's copy), skipping"
    );
    return null;
  }
  const hashLockAcquired = await tryAcquireHashLock(contentHash);
  if (!hashLockAcquired) {
    counters.zipsDuplicate++;
    accountLog.info(
      { fileName: archiveName, contentHash },
      "Hash lock held by another worker — skipping concurrent duplicate"
    );
    return null;
  }
  try {
    if (await packageExistsByHash(contentHash)) {
      counters.zipsDuplicate++;
      return null;
    }
    let destResult: { messageId: bigint; messageIds: bigint[] };
    try {
      destResult = await forwardArchiveToChannel(
        client,
        channel.telegramId,
        destChannelTelegramId,
        archiveSet.parts.map((p) => p.id),
      );
    } catch (forwardErr) {
      accountLog.warn(
        { err: forwardErr, fileName: archiveName },
        "Forward failed — falling back to download+reupload for this archive"
      );
      return undefined;
    }
    await deleteOrphanedPackageByHash(contentHash);
    const creator =
      topicCreator ??
      extractCreatorFromFileName(archiveName) ??
      extractCreatorFromChannelTitle(channelTitle) ??
      null;
    const tags: string[] = [];
    if (channel.category) tags.push(channel.category);
    for (const tag of extractSlicerTags(entries)) {
      if (!tags.includes(tag)) tags.push(tag);
    }
    const stub = await createPackageStub({
      contentHash,
      fileName: archiveName,
      fileSize: totalArchiveSize,
      archiveType: archType,
      sourceChannelId: channel.id,
      sourceMessageId: archiveSet.parts[0].id,
      sourceTopicId,
      remoteUniqueId: firstRemoteUniqueId,
      destChannelId,
      destMessageId: destResult.messageId,
      destMessageIds: destResult.messageIds,
      isMultipart: archiveSet.parts.length > 1,
      partCount: archiveSet.parts.length,
      ingestionRunId,
      creator,
      tags,
    });
    counters.zipsForwarded++;
    await deleteSkippedPackage(channel.id, archiveSet.parts[0].id);
    let previewData: Buffer | null = null;
    let previewMsgId: bigint | null = null;
    const matchedPhoto = previewMatches.get(archiveSet.baseName);
    if (matchedPhoto) {
      previewData = await downloadPhotoThumbnail(client, matchedPhoto.fileId);
      if (previewData) previewMsgId = matchedPhoto.id;
    }
    await updatePackageWithMetadata(stub.id, { files: entries, previewData, previewMsgId });
    accountLog.info(
      { fileName: archiveName, contentHash, fileCount: entries.length, creator },
      "Archive forwarded (no download)"
    );
    return stub.id;
  } finally {
    await releaseHashLock(contentHash);
  }
 }
 async function deleteFiles(paths: string[]): Promise<void> {
  for (const p of paths) {
    try {
Author	SHA1	Message	Date
adminandClaude Sonnet 5	d786f3f23b	perf(worker): run startup upload-recovery batches with bounded concurrency continuous-integration/drone/push Build is passing Details Telegram soft-throttles sustained sequential getMessages calls from a user account with growing per-call latency (no FLOOD_WAIT, so our retry wrapper never sees it). On a large package count this made the once-per-startup destination-message verification pass take 90+ minutes and block the scheduler/fetch-listener from starting. Running up to 3 batches concurrently cuts wall-clock time well under real per-account rate limits. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP	2026-08-21 19:00:21 +02:00
adminandClaude Sonnet 5	f6381e3178	fix(worker): recognize native 7z multipart volumes (.7z.001, .7z.002, ...) continuous-integration/drone/push Build is passing Details detectArchive() had multipart patterns for ZIP and RAR but none for 7z's own volume-split naming, so files like "Name.7z.001" matched nothing and were dropped before ever reaching grouping or the skipped-package bookkeeping — no Package row, no SkippedPackage row, no log line. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JMm2E4ecmATJo8HuBx92NP	2026-08-21 16:50:41 +02:00
admin	d8e01f3398	fix(backup): exclude TDLib's disposable file cache from the tdlib tar continuous-integration/drone/push Build is passing Details files/temp is TDLib's own redundant download cache — the worker already prunes it after every ingestion run (see the recent optimizeTdlibStorage fix), and its content still lives in the source/destination Telegram chats regardless. With the cache grown back to ~59GB between prune cycles, tarring it made today's backup run for 4+ hours straight, fighting the actively-ingesting worker for disk I/O and degrading the whole host. Excluding it keeps the backup to what's actually irreplaceable: the DB dump and the TDLib session state itself.	2026-08-14 11:40:03 +02:00
admin	b8672a44d0	feat(worker): forward on allowsForwarding channels regardless of file type or listing success continuous-integration/drone/push Build is passing Details tryForwardArchiveSet only attempted the no-download forward path for ZIP/RAR/7z archives, and bailed to download+reupload whenever the ranged listing failed — even though the channel already grants forwarding permission and forwarding a message costs nothing regardless of what's inside it. Standalone DOCUMENT/STL/3MF attachments never got a chance at the forward path at all. Now any file on a forwarding-enabled channel is forwarded directly; when there's no listing to derive a content hash from (non-archive types, or ranged-listing failures), dedup falls back to remote.unique_id identity — deriveForwardContentHash, crcFingerprint, and compareFingerprints already degrade to this safely for empty/incomplete entries, so there's no risk of unrelated files colliding as false duplicates. Inner-file indexing is simply skipped for these cases, matching the existing accepted limitation for RAR/7z rebuild placeholders.	2026-08-13 18:52:34 +02:00
admin	e123a5cc44	diag(worker): log the reason when RAR ranged-listing silently returns null continuous-integration/drone/push Build is passing Details Several channels with allowsForwarding=true were still falling back to download+reupload for specific RAR archives after the signature-bytes fix, with zero log output to explain why — walkRarVolume and readRarListingRanged return null silently on signature-detection failure, the MAX_RAR_BLOCKS/MAX_RAR_HEADER_BYTES guards, and a non-positive advance. Log each case so the next occurrence identifies the actual cause instead of requiring another guess-and-redeploy cycle.	2026-08-13 12:10:50 +02:00
admin	73b2c33305	fix(worker): include the RAR signature in the ranged-listing sparse reconstruction continuous-integration/drone/push Build is passing Details readRarListingRanged read the archive signature only to detect the RAR version, then walkRarVolume began harvesting header regions at pos = sigLen — the signature bytes themselves were never captured. The reconstructed sparse file therefore started with zero bytes instead of "Rar!", so every real unrar invocation rejected it as "not RAR archive". This silently forced every RAR archive through the expensive download+reupload fallback regardless of the source channel's forwarding permission — defeating the forward-priority path for the RAR-heavy channels it matters most for. The existing tests didn't catch this because their assertions accepted either a null or non-null result as passing, deferring real verification to production.	2026-08-11 23:17:22 +02:00
admin	06a48a419b	fix(bot): resolve private chat before sending to a user continuous-integration/drone/push Build is passing Details After the bot's TDLib session was rebuilt from scratch (following a disk-full corruption on 2026-08-05), sendMessage started failing with "Chat not found" for every previously-known user — a fresh TDLib database has no cached chat/peer info until createPrivateChat explicitly resolves it. Call it before every send.	2026-08-10 08:27:41 +02:00
admin	7bc57ec227	fix(worker): prune TDLib's unbounded local file cache after each run continuous-integration/drone/push Build is failing Details TDLib keeps a permanent local copy of every file it downloads or uploads (via inputFileLocal) with no automatic cleanup. Across the two worker accounts this had grown to ~270GB, filling the host disk to 91% and triggering a cascading disk-full failure in the bot's TDLib session on 2026-08-05. Call optimizeStorage after every ingestion run to clear it; a 5-minute immunity_delay protects files an in-flight operation might still reference.	2026-08-10 08:17:51 +02:00
admin	148d688d43	fix(worker): fall through to download pipeline on any forward-path throw continuous-integration/drone/push Build is passing Details	2026-07-31 05:25:16 +02:00
admin	b28e38d233	feat(worker): fork to the forward-priority path in processOneArchiveSet	2026-07-31 05:14:11 +02:00
admin	662f5ac711	feat(worker): native TDLib forward from source to destination channel	2026-07-31 05:06:11 +02:00
admin	960da01ec6	feat(worker): cross-channel CRC-fingerprint repost check for the forward path	2026-07-31 05:00:14 +02:00
admin	a46e746298	feat(worker): derive a dedup identity for forward-path packages without bytes	2026-07-31 04:53:24 +02:00
admin	26be615918	refactor(worker): promote ranged-listing dispatcher to a shared module	2026-07-31 04:46:39 +02:00
admin	eda882dc90	feat(worker): detect + persist per-channel forwarding permission	2026-07-31 04:40:15 +02:00
admin	80d41ac78f	feat(db): add TelegramChannel.allowsForwarding + IngestionRun.zipsForwarded	2026-07-31 04:30:33 +02:00
admin	40894267d4	Merge branch 'feat/ranged-archive-listing' into main # Conflicts: # .env.example # backup/Dockerfile # docker-compose.yml	2026-07-31 04:24:58 +02:00
admin	9a45fdf6d9	docs: implementation plan for forward-priority ingestion pipeline Task-by-task plan for the design in docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md, grounded against current worker.ts/provenance-backfill.ts/schema signatures so a fresh agent can execute it without prior context.	2026-07-30 23:15:57 +02:00
admin	809d72660d	docs: design for forward-priority ingestion pipeline Prioritize native Telegram forwarding over download+reupload for source channels that allow it, reusing the ranged archive-listing readers to keep indexing complete without a local download. Falls back to the existing download+reupload pipeline per-channel (when forwarding is blocked) and per-archive (when ranged listing fails).	2026-07-30 23:02:49 +02:00
admin	6102fd474f	Merge branch 'main' of https://github.com/xCyanGrizzly/DragonsStash	2026-07-30 22:38:41 +02:00
admin	ffe5c920c6	Fix backup service restart handling	2026-07-22 10:04:18 +02:00
admin	c533d034f9	fix: restrict backup restore paths	2026-07-22 09:44:08 +02:00
admin	03822e0763	fix: narrow backup scope to database and telegram sessions	2026-07-22 09:31:07 +02:00
admin	3536089d52	docs: add monthly backup verification owner Align the approved backup design and implementation plan on the monthly full Restic integrity check and disposable restore rehearsal without adding production code. Constraint: PostgreSQL logical dump plus tdlib_state and tdlib_bot_state only Rejected: Add a second production timer or backup script changes \| outside requested docs-only scope Confidence: high Scope-risk: narrow	2026-07-22 09:19:15 +02:00
admin	a8818dcf0c	docs: correct backup scope to database and sessions Clarify that PostgreSQL logical dumps and TDLib session volumes are protected, while STL binaries remain in Telegram and manual uploads stay excluded.	2026-07-22 09:10:19 +02:00
admin	d57ec0458f	docs: harden disposable restore rehearsal	2026-07-22 03:57:39 +02:00
admin	6763af731f	docs: document disposable restore rehearsal	2026-07-22 03:49:02 +02:00
admin	d5ba4fd4fd	fix: close backup review findings	2026-07-22 03:40:35 +02:00
admin	def8edb029	docs: document Synology backup and recovery	2026-07-22 02:54:44 +02:00
admin	68742d682d	fix: harden backup restore safety	2026-07-22 02:49:06 +02:00
admin	f8bc737214	feat: retain uploaded STL files for recovery	2026-07-22 02:24:59 +02:00
admin	238ec17155	docs: amend backup design for durable STL retention	2026-07-22 02:13:11 +02:00
admin	a2b7e77cc4	feat: add guarded backup restore workflow	2026-07-22 00:55:24 +02:00
admin	2beac6d62f	feat: schedule nightly off-host backups	2026-07-22 00:41:32 +02:00
admin	897cd3d95e	feat: implement encrypted database and volume snapshots	2026-07-22 00:30:56 +02:00
admin	34e88a8cf5	feat: add backup compose service	2026-07-22 00:22:56 +02:00
admin	5e7807b056	chore: ignore local worktrees	2026-07-21 23:49:33 +02:00
admin	7693ed6f02	docs: plan database and file backup implementation	2026-07-21 23:32:41 +02:00
admin	75c67b2036	docs: design database and file backups	2026-07-21 23:23:52 +02:00
		`@@ -0,0 +1 @@`
							`ALTER TABLE "manual_upload_files" ADD COLUMN "retainedAt" TIMESTAMP(3);`
		`@@ -0,0 +1 @@`
							`ALTER TABLE "manual_upload_files" DROP COLUMN IF EXISTS "retainedAt";`