admin/dragonsstash
1
0
Fork 0
mirror of https://github.com/xCyanGrizzly/DragonsStash.git synced 2026-09-21 05:21:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'feat/ranged-archive-listing' into main

Browse Source 
# Conflicts:
#	.env.example
#	backup/Dockerfile
#	docker-compose.yml
This commit is contained in:
admin
2026-07-31 04:24:58 +02:00
parent 6102fd474f 9a45fdf6d9
commit 40894267d4
49 changed files with 7768 additions and 1627 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/plans/2026-07-23-nas-backup.md
+555
View File
@@ -0,0 +1,555 @@
# NAS Backup for Postgres + TDLib State Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add a `backup` container to the DragonsStash stack that takes daily, encrypted, deduplicated backups of the Postgres database and both TDLib state volumes, and ships them to a Synology NAS over NFS.
**Architecture:** A small Alpine-based image (restic + postgresql16-client + curl + dcron) runs as its own compose service. A crontab fires `backup.sh` daily at 03:00, which dumps Postgres, tars the TDLib volumes, hands both to `restic backup` against an NFS-backed Docker volume, prunes with `restic forget --keep-daily 14`, and reports success/failure to an Uptime Kuma push monitor. Matches the existing `worker`/`bot` pattern: build context in the repo's `docker-compose.yml`, prebuilt image in `/opt/stacks/DragonsStash/docker-compose.yml`, built and pushed by `.drone.yml`.
**Tech Stack:** Alpine 3.20, restic 0.16, postgresql16-client, dcron, bash, Docker Compose NFS volume driver.
## Global Constraints
- Retention: `restic forget --keep-daily 14 --prune` (14-day window, per approved spec).
- Schedule: daily backup at 03:00, weekly `restic check` at 04:00 Sunday.
- No Docker socket mount, no `privileged: true` — the backup container must not be able to control sibling containers.
- No host-level mount — NFS access only via Docker's native `driver_opts: type: nfs` volume, never `/etc/fstab`.
- Encryption and retention are restic's job — no hand-rolled `age`/`gpg`/`find -mtime` logic.
- TDLib volumes are tarred live (best-effort) — never pause `worker`/`bot` for the backup.
- Restore is a manual, documented procedure only — never scripted/automated.
**Required user input before Task 5 can run:** `NAS_HOST` and `NAS_EXPORT_PATH` (the Synology NFS share details) and a Kuma Push-monitor URL (`KUMA_PUSH_URL`, created manually in the existing Uptime Kuma instance, ~26h expected heartbeat interval). Tasks 1–4 need none of these and can proceed immediately; do not substitute placeholder values for them in Task 5 — stop and ask the user instead.
---
### Task 1: Backup image (Dockerfile + entrypoint)
**Files:**
- Create: `backup/Dockerfile`
- Create: `backup/entrypoint.sh`
- Create: `backup/crontab`
**Interfaces:**
- Produces: a buildable image tagged `dragonsstash-backup:test` locally, with `/entrypoint.sh` as `ENTRYPOINT`, `/backup.sh` present at the image root (written in Task 2 — this task only needs the `COPY` line and a placeholder-free stub isn't acceptable, so create an empty‑body-but-real `backup/backup.sh` here containing just `#!/bin/bash` + `exit 0`, and Task 2 replaces its contents), `restic`, `pg_dump`/`pg_restore`, `curl`, `tar`, `bash`, `dcron` all on `PATH`.
- Consumes: nothing from earlier tasks.
- [ ] **Step 1: Write `backup/backup.sh` stub**
```bash
#!/bin/bash
set -euo pipefail
exit 0
```
- [ ] **Step 2: Write `backup/entrypoint.sh`**
```bash
#!/bin/bash
set -euo pipefail
if ! restic snapshots >/dev/null 2>&1; then
restic init
fi
exec crond -f -l 2
```
- [ ] **Step 3: Write `backup/crontab`**
```
0 3 * * * /backup.sh >> /proc/1/fd/1 2>&1
0 4 * * 0 restic check >> /proc/1/fd/1 2>&1
```
- [ ] **Step 4: Write `backup/Dockerfile`**
```dockerfile
FROM alpine:3.20
RUN apk add --no-cache restic postgresql16-client curl tzdata dcron tar bash
COPY backup/backup.sh /backup.sh
COPY backup/entrypoint.sh /entrypoint.sh
COPY backup/crontab /etc/crontabs/root
RUN chmod +x /backup.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
```
- [ ] **Step 5: Build the image**
Run: `cd /home/sam/Documents/DragonsStash && docker build -t dragonsstash-backup:test -f backup/Dockerfile .`
Expected: build completes with `Successfully tagged dragonsstash-backup:test` (or Buildkit's equivalent final `naming to docker.io/library/dragonsstash-backup:test done`), no errors.
- [ ] **Step 6: Verify the tools are present**
Run: `docker run --rm dragonsstash-backup:test restic version && docker run --rm dragonsstash-backup:test pg_dump --version`
Expected: `restic 0.16.x ...` and `pg_dump (PostgreSQL) 16.x` printed, both commands exit 0.
- [ ] **Step 7: Verify the entrypoint initializes an empty repo and starts cron**
```bash
mkdir -p /tmp/backup-repo-smoke
docker run -d --name backup-smoke \
-e RESTIC_REPOSITORY=/backups/restic-repo -e RESTIC_PASSWORD=smoketest \
-v /tmp/backup-repo-smoke:/backups \
dragonsstash-backup:test
sleep 2
docker logs backup-smoke
docker exec backup-smoke restic snapshots
docker rm -f backup-smoke
rm -rf /tmp/backup-repo-smoke
```
Expected: `docker logs` shows no errors (restic init ran silently); `restic snapshots` prints an empty snapshot list (repo exists, header row only, no error).
- [ ] **Step 8: Commit**
```bash
git add backup/Dockerfile backup/entrypoint.sh backup/backup.sh backup/crontab
git commit -m "Add backup service image (Dockerfile, entrypoint, crontab)"
```
---
### Task 2: `backup.sh` script
**Files:**
- Modify: `backup/backup.sh` (replace Task 1's stub with the real script)
**Interfaces:**
- Consumes: the image built in Task 1 (`dragonsstash-backup:test`), rebuilt after this change.
- Produces: `/backup.sh`, invoked by cron in Task 1's `crontab` and manually in Task 6's verification. Reads env vars `POSTGRES_USER`, `PGPASSWORD`, `POSTGRES_DB`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD`, `KUMA_PUSH_URL`. Assumes network hostname `dragonsstash-db:5432` for Postgres and mounts `/data/tdlib-worker`, `/data/tdlib-bot` (read-only) for TDLib state.
- [ ] **Step 1: Replace `backup/backup.sh` with the real script**
```bash
#!/bin/bash
set -euo pipefail
report_failure() {
curl -fsS "$KUMA_PUSH_URL" --get \
--data-urlencode "status=down" \
--data-urlencode "msg=$BASH_COMMAND failed" || true
}
trap report_failure ERR
DUMP_FILE=/tmp/dragonsstash.dump
TAR_FILE=/tmp/tdlib.tar.gz
pg_dump -h dragonsstash-db -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc -f "$DUMP_FILE"
tar czf "$TAR_FILE" -C /data tdlib-worker tdlib-bot
restic backup "$DUMP_FILE" "$TAR_FILE"
restic forget --keep-daily 14 --prune
rm -f "$DUMP_FILE" "$TAR_FILE"
curl -fsS "$KUMA_PUSH_URL" --get \
--data-urlencode "status=up" \
--data-urlencode "msg=OK"
```
- [ ] **Step 2: Rebuild the image**
Run: `docker build -t dragonsstash-backup:test -f backup/Dockerfile .`
Expected: build succeeds.
- [ ] **Step 3: Stand up a scratch Postgres aliased as `dragonsstash-db`**
```bash
docker network create backup-test-net 2>/dev/null || true
docker run -d --name test-pg --network backup-test-net --network-alias dragonsstash-db \
-e POSTGRES_USER=dragons -e POSTGRES_PASSWORD=stash -e POSTGRES_DB=dragonsstash \
postgres:16-alpine
sleep 5
docker exec test-pg pg_isready -U dragons -d dragonsstash
```
Expected: `accepting connections`.
- [ ] **Step 4: Stand up a mock Kuma push endpoint**
```bash
mkdir -p /tmp/mock-kuma-root && touch /tmp/mock-kuma-root/push
docker run -d --name mock-kuma --network backup-test-net \
-v /tmp/mock-kuma-root:/srv -w /srv python:3-alpine \
python3 -m http.server 8000
sleep 1
```
- [ ] **Step 5: Prepare fake TDLib state and a local restic repo dir**
```bash
mkdir -p /tmp/backup-test/tdlib-worker /tmp/backup-test/tdlib-bot /tmp/backup-test/repo
echo "fake-session" > /tmp/backup-test/tdlib-worker/state.bin
echo "fake-session" > /tmp/backup-test/tdlib-bot/state.bin
```
- [ ] **Step 6: Initialize the test restic repo and run `backup.sh` (success path)**
```bash
docker run --rm --network backup-test-net \
-e RESTIC_REPOSITORY=/backups/restic-repo -e RESTIC_PASSWORD=testpassword \
-v /tmp/backup-test/repo:/backups \
--entrypoint restic dragonsstash-backup:test init
docker run --rm --network backup-test-net \
-e POSTGRES_USER=dragons -e POSTGRES_PASSWORD=stash -e PGPASSWORD=stash -e POSTGRES_DB=dragonsstash \
-e RESTIC_REPOSITORY=/backups/restic-repo -e RESTIC_PASSWORD=testpassword \
-e KUMA_PUSH_URL=http://mock-kuma:8000/push \
-v /tmp/backup-test/tdlib-worker:/data/tdlib-worker:ro \
-v /tmp/backup-test/tdlib-bot:/data/tdlib-bot:ro \
-v /tmp/backup-test/repo:/backups \
--entrypoint /backup.sh dragonsstash-backup:test
echo "exit code: $?"
```
Expected: exit code `0`, restic prints a line like `snapshot xxxxxxxx saved`, no error output.
- [ ] **Step 7: Verify the snapshot landed and contains both files**
```bash
docker run --rm -v /tmp/backup-test/repo:/backups \
-e RESTIC_REPOSITORY=/backups/restic-repo -e RESTIC_PASSWORD=testpassword \
--entrypoint restic dragonsstash-backup:test snapshots
docker run --rm -v /tmp/backup-test/repo:/backups \
-e RESTIC_REPOSITORY=/backups/restic-repo -e RESTIC_PASSWORD=testpassword \
--entrypoint restic dragonsstash-backup:test ls latest
```
Expected: `snapshots` shows exactly one entry; `ls latest` lists `/tmp/dragonsstash.dump` and `/tmp/tdlib.tar.gz`.
- [ ] **Step 8: Verify the failure path reports to Kuma**
```bash
docker run --rm --network backup-test-net \
-e POSTGRES_USER=dragons -e POSTGRES_PASSWORD=wrongpass -e PGPASSWORD=wrongpass -e POSTGRES_DB=dragonsstash \
-e RESTIC_REPOSITORY=/backups/restic-repo -e RESTIC_PASSWORD=testpassword \
-e KUMA_PUSH_URL=http://mock-kuma:8000/push \
-v /tmp/backup-test/tdlib-worker:/data/tdlib-worker:ro \
-v /tmp/backup-test/tdlib-bot:/data/tdlib-bot:ro \
-v /tmp/backup-test/repo:/backups \
--entrypoint /backup.sh dragonsstash-backup:test
echo "exit code: $?"
docker logs mock-kuma | tail -5
```
Expected: exit code nonzero (pg_dump auth failure trips `set -e`); `docker logs mock-kuma` shows a GET request line containing `status=down`.
- [ ] **Step 9: Clean up test resources**
```bash
docker rm -f test-pg mock-kuma
docker network rm backup-test-net
rm -rf /tmp/backup-test /tmp/mock-kuma-root
```
- [ ] **Step 10: Commit**
```bash
git add backup/backup.sh
git commit -m "Implement backup.sh: pg_dump + tdlib tar + restic backup/forget + Kuma reporting"
```
---
### Task 3: Wire the `backup` service into the repo's `docker-compose.yml`
**Files:**
- Modify: `docker-compose.yml`
- Modify: `.env.example`
**Interfaces:**
- Consumes: `backup/Dockerfile` (Task 1), `backup/backup.sh` (Task 2).
- Produces: a `backup` compose service buildable via `docker compose build backup`, and a `nas_backups` named volume other tasks (5) will mirror into the production compose file.
- [ ] **Step 1: Add the `backup` service to `docker-compose.yml`**
Insert after the existing `bot` service (before `db`):
```yaml
backup:
build:
context: .
dockerfile: backup/Dockerfile
pull_policy: never
environment:
- POSTGRES_USER=${POSTGRES_USER:-dragons}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-stash}
- PGPASSWORD=${POSTGRES_PASSWORD:-stash}
- POSTGRES_DB=${POSTGRES_DB:-dragonsstash}
- RESTIC_REPOSITORY=/backups/restic-repo
- RESTIC_PASSWORD=${RESTIC_PASSWORD:?Set RESTIC_PASSWORD in .env}
- KUMA_PUSH_URL=${KUMA_PUSH_URL:?Set KUMA_PUSH_URL in .env}
- TZ=${TZ:-Etc/UTC}
volumes:
- tdlib_state:/data/tdlib-worker:ro
- tdlib_bot_state:/data/tdlib-bot:ro
- nas_backups:/backups
depends_on:
db:
condition: service_healthy
restart: unless-stopped
deploy:
resources:
limits:
memory: 256M
networks:
- backend
```
- [ ] **Step 2: Add the `nas_backups` volume to the `volumes:` block**
```yaml
nas_backups:
driver_opts:
type: nfs
o: "addr=${NAS_HOST},rw,nfsvers=4,soft,timeo=100"
device: ":${NAS_EXPORT_PATH}"
```
- [ ] **Step 3: Document the new env vars in `.env.example`**
Append:
```
# Backup (NAS via NFS + restic)
NAS_HOST="" # Synology NAS IP or hostname reachable from this host
NAS_EXPORT_PATH="" # NFS export path, e.g. /volume1/dragonsstash-backups
RESTIC_PASSWORD="" # generate with: openssl rand -base64 32
KUMA_PUSH_URL="" # Uptime Kuma Push monitor URL (create the monitor first)
TZ="Etc/UTC"
```
- [ ] **Step 4: Validate the compose file parses**
Run (with dummy values so the `:?` guards don't fail parsing — `AUTH_SECRET` is required by the existing `app` service, not by this change, but `config` validates the whole file):
```bash
RESTIC_PASSWORD=dummy KUMA_PUSH_URL=http://dummy NAS_HOST=dummy NAS_EXPORT_PATH=/dummy AUTH_SECRET=dummy \
docker compose -f docker-compose.yml config --quiet
```
Expected: no output, exit code 0 (a syntax/interpolation error would print to stderr and exit nonzero).
- [ ] **Step 5: Validate the service actually builds through Compose**
Run:
```bash
RESTIC_PASSWORD=dummy KUMA_PUSH_URL=http://dummy NAS_HOST=dummy NAS_EXPORT_PATH=/dummy AUTH_SECRET=dummy \
docker compose -f docker-compose.yml build backup
```
Expected: build succeeds (reuses Task 1's image layers).
- [ ] **Step 6: Commit**
```bash
git add docker-compose.yml .env.example
git commit -m "Add backup service and nas_backups volume to docker-compose.yml"
```
---
### Task 4: CI — build and push the backup image
**Files:**
- Modify: `.drone.yml`
**Interfaces:**
- Consumes: `backup/Dockerfile` (Task 1).
- Produces: `git.samagsteribbe.nl/admin/dragonsstash-backup:latest` (and `:<short-sha>`), pushed on every push to `main`. Task 5's production compose file references this image tag.
- [ ] **Step 1: Add a `build-backup` step, mirroring `build-worker`/`build-bot`**
Insert after the existing `build-bot` step in `.drone.yml`:
```yaml
- name: build-backup
image: plugins/docker
depends_on: [clone]
settings:
repo: git.samagsteribbe.nl/admin/dragonsstash-backup
registry: git.samagsteribbe.nl
dockerfile: backup/Dockerfile
tags:
- latest
- "${DRONE_COMMIT_SHA:0:8}"
username:
from_secret: gitea_username
password:
from_secret: gitea_password
```
- [ ] **Step 2: Add `build-backup` to the `deploy` step's `depends_on`**
Change:
```yaml
- name: deploy
image: alpine
depends_on: [build-app, build-worker, build-bot]
```
to:
```yaml
- name: deploy
image: alpine
depends_on: [build-app, build-worker, build-bot, build-backup]
```
- [ ] **Step 3: Validate YAML syntax**
Run: `python3 -c "import yaml; yaml.safe_load(open('.drone.yml')); print('OK')"`
Expected: `OK` printed, no exception.
- [ ] **Step 4: Commit**
```bash
git add .drone.yml
git commit -m "Add build-backup CI step, include it in deploy dependencies"
```
---
### Task 5: Wire production (`/opt/stacks/DragonsStash`) — requires NAS + Kuma details from the user
**Do not start this task until the user has supplied `NAS_HOST` and `NAS_EXPORT_PATH` for the Synology NFS share, and has created an Uptime Kuma Push monitor (name it `dragonsstash-backup`, ~26h expected heartbeat interval) and shared its push URL. If any of these are missing, stop and ask — do not substitute placeholder values here, since this file drives the real deployment.**
**Files:**
- Modify: `/opt/stacks/DragonsStash/docker-compose.yml`
- Modify: `/opt/stacks/DragonsStash/.env`
**Interfaces:**
- Consumes: `git.samagsteribbe.nl/admin/dragonsstash-backup:latest` (published by Task 4's CI step once merged/pushed), the real `NAS_HOST`/`NAS_EXPORT_PATH`/`KUMA_PUSH_URL` values gathered above.
- Produces: a running `dragonsstash-backup` container on the production host, verified in Task 6.
- [ ] **Step 1: Generate `RESTIC_PASSWORD` and add all four new vars to `/opt/stacks/DragonsStash/.env`**
```bash
cd /opt/stacks/DragonsStash
printf '\n# Backup (NAS via NFS + restic)\nNAS_HOST="<value from user>"\nNAS_EXPORT_PATH="<value from user>"\nRESTIC_PASSWORD="%s"\nKUMA_PUSH_URL="<value from user>"\nTZ="Etc/UTC"\n' "$(openssl rand -base64 32)" >> .env
```
Replace the two `<value from user>` placeholders with the real NAS details and Kuma push URL before saving — this step cannot be completed with the literal placeholder text left in place.
- [ ] **Step 2: Add the `backup` service to `/opt/stacks/DragonsStash/docker-compose.yml`**
Insert after the existing `bot` service (before `db`):
```yaml
backup:
image: git.samagsteribbe.nl/admin/dragonsstash-backup:latest
container_name: dragonsstash-backup
restart: unless-stopped
environment:
- POSTGRES_USER=${POSTGRES_USER:-dragons}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-stash}
- PGPASSWORD=${POSTGRES_PASSWORD:-stash}
- POSTGRES_DB=${POSTGRES_DB:-dragonsstash}
- RESTIC_REPOSITORY=/backups/restic-repo
- RESTIC_PASSWORD=${RESTIC_PASSWORD:?Set RESTIC_PASSWORD in .env}
- KUMA_PUSH_URL=${KUMA_PUSH_URL:?Set KUMA_PUSH_URL in .env}
- TZ=${TZ:-Etc/UTC}
volumes:
- tdlib_state:/data/tdlib-worker:ro
- tdlib_bot_state:/data/tdlib-bot:ro
- nas_backups:/backups
depends_on:
db:
condition: service_healthy
deploy:
resources:
limits:
memory: 256M
networks:
- internal
```
- [ ] **Step 3: Add the `nas_backups` volume**
```yaml
nas_backups:
driver_opts:
type: nfs
o: "addr=${NAS_HOST},rw,nfsvers=4,soft,timeo=100"
device: ":${NAS_EXPORT_PATH}"
```
- [ ] **Step 4: Validate the production compose file parses with the real `.env`**
Run: `cd /opt/stacks/DragonsStash && docker compose config --quiet`
Expected: no output, exit code 0.
- [ ] **Step 5: Commit is not applicable here** — `/opt/stacks/DragonsStash` is a deployed copy, not the git repo (confirm with `git -C /opt/stacks/DragonsStash status` — expect "not a git repository"). Skip committing; Task 6 deploys these file changes directly.
---
### Task 6: Deploy and verify
**Files:** none (operational task)
**Interfaces:**
- Consumes: everything from Tasks 1–5.
- Produces: a running, verified backup on the real NAS, and one completed restore drill.
- [ ] **Step 1: Confirm with the user before pushing/deploying**
Pushing to `main` triggers Drone CI to build all four images and deploy to the production host via SSH (`docker compose pull && docker compose up -d`). Confirm the user wants this to happen now before proceeding — this affects the live stack.
- [ ] **Step 2: Push to `main`**
```bash
cd /home/sam/Documents/DragonsStash
git push origin main
```
Expected: Drone pipeline runs `build-app`, `build-worker`, `build-bot`, `build-backup`, then `deploy`, all green. Check via the Drone UI or `drone build info admin/DragonsStash <build-number>` if the `drone` CLI is configured.
- [ ] **Step 3: Confirm the container is up on the production host**
```bash
ssh sam@192.168.68.68 "docker ps --filter name=dragonsstash-backup --format '{{.Names}}\t{{.Status}}'"
```
Expected: `dragonsstash-backup Up ...`.
- [ ] **Step 4: Trigger one manual backup run and confirm a snapshot lands on the NAS**
```bash
ssh sam@192.168.68.68 "docker exec dragonsstash-backup /backup.sh"
ssh sam@192.168.68.68 "docker exec dragonsstash-backup restic snapshots"
```
Expected: `backup.sh` exits 0; `restic snapshots` lists exactly one entry.
- [ ] **Step 5: Confirm the Uptime Kuma push monitor shows green**
Open the Uptime Kuma dashboard and check the `dragonsstash-backup` monitor's status is up with a recent heartbeat.
- [ ] **Step 6: Restore drill — prove the backup is actually restorable**
```bash
ssh sam@192.168.68.68 "docker exec dragonsstash-backup restic restore latest --target /tmp/restore-drill"
ssh sam@192.168.68.68 "docker exec dragonsstash-backup ls -la /tmp/restore-drill/tmp"
```
Expected: `/tmp/restore-drill/tmp/dragonsstash.dump` and `/tmp/restore-drill/tmp/tdlib.tar.gz` both present with nonzero size. Then, on a scratch Postgres (not the live `dragonsstash-db`), confirm the dump restores cleanly:
```bash
ssh sam@192.168.68.68 "docker run -d --rm --name restore-drill-pg --network dragonsstash_internal \
-e POSTGRES_USER=drill -e POSTGRES_PASSWORD=drill -e POSTGRES_DB=drill postgres:16-alpine"
ssh sam@192.168.68.68 "docker cp dragonsstash-backup:/tmp/restore-drill/tmp/dragonsstash.dump /tmp/dragonsstash.dump"
ssh sam@192.168.68.68 "docker cp /tmp/dragonsstash.dump restore-drill-pg:/tmp/dragonsstash.dump"
ssh sam@192.168.68.68 "docker exec -e PGPASSWORD=drill restore-drill-pg pg_restore -U drill -d drill --clean --if-exists /tmp/dragonsstash.dump"
ssh sam@192.168.68.68 "docker exec -e PGPASSWORD=drill restore-drill-pg psql -U drill -d drill -c '\\dt' | head -20"
ssh sam@192.168.68.68 "docker rm -f restore-drill-pg"
```
Expected: `pg_restore` completes without fatal errors; `\dt` lists the app's tables (e.g. `Package`, `User`, `TelegramLink`).
- [ ] **Step 7: Clean up drill artifacts**
```bash
ssh sam@192.168.68.68 "docker exec dragonsstash-backup rm -rf /tmp/restore-drill"
ssh sam@192.168.68.68 "rm -f /tmp/dragonsstash.dump"
```
docs/superpowers/plans/2026-07-23-provenance-backfill.md
+1091
View File
File diff suppressed because it is too large Load Diff
docs/superpowers/plans/2026-07-27-ranged-archive-listing.md
+1073
View File
File diff suppressed because it is too large Load Diff
docs/superpowers/plans/2026-07-30-forward-priority-ingestion.md
+1475
View File
File diff suppressed because it is too large Load Diff
docs/superpowers/specs/2026-07-23-nas-backup-design.md
+205
View File
@@ -0,0 +1,205 @@
# NAS Backup for Postgres + TDLib State — Design
**Date:** 2026-07-23
**Status:** Approved for planning
## Summary
Add a dedicated `backup` container to the DragonsStash stack that takes daily,
encrypted, deduplicated backups of the two things that can't be regenerated —
the Postgres database (inventory/STL metadata, users, everything the app
manages) and the two TDLib state volumes (Telegram session/auth state for the
worker and bot) — and ships them to a Synology NAS over NFS. STL archive
contents themselves are explicitly out of scope: they only live on this host
temporarily and are not backed up.
Backups are stored via [restic](https://restic.net/), which provides
encryption-at-rest, block-level dedup, and retention pruning natively, so no
custom encryption or pruning scripts need to be written or maintained.
## Context
Current state (as of this design):
- Production stack runs from `/opt/stacks/DragonsStash/docker-compose.yml` on
this Dockge-managed host, pulling prebuilt images from
`git.samagsteribbe.nl`. The `docker-compose.yml` in this repo is the
build/dev reference and should be kept in sync.
- Named volumes in use: `postgres_data` (Postgres 16 data directory),
`tdlib_state` (worker's TDLib session), `tdlib_bot_state` (bot's TDLib
session), `tmp_zips` and `manual_uploads` (both transient, explicitly out of
scope here).
- No backup mechanism, NFS mount, or host cron currently exists anywhere in
this deployment.
- The host already runs Uptime Kuma (used here for backup alerting) and Loki
(container logs are presumably already collected there).
## Requirements
1. Daily backup of the Postgres database and both TDLib state volumes.
2. Backups stored on a Synology NAS via NFS, not on local disk.
3. 14-day retention, oldest snapshots pruned automatically.
4. Backups encrypted at rest (Postgres dumps and TDLib session files both
contain sensitive material — password hashes, Telegram API secrets, live
session state).
5. Postgres backups must be transactionally consistent regardless of live app
traffic. TDLib state backups are best-effort (see Decisions below) — this
is an accepted trade-off, not a defect.
6. Alert (via existing Uptime Kuma) if a backup run fails or doesn't happen.
7. No new host-level state (no `/etc/fstab` entries, no host crontab) — the
backup mechanism should be a container, consistent with how everything
else on this host is deployed and versioned.
8. No new privileged access — specifically, the backup container must not
have Docker socket access or any ability to control sibling containers.
## Decisions
- **NFS mounted via Docker's native NFS volume driver** (`driver_opts: type:
nfs`), not a host-level mount. Keeps all backup-related state inside the
compose file instead of split across host config.
- **Restic, not hand-rolled tar+age+find.** Restic already solves encryption,
dedup, and retention correctly; hand-rolled scripts would be reinventing
that logic with more room for bugs.
- **TDLib state is tarred live (best-effort), not paused.** Pausing the
worker/bot for a clean snapshot would require mounting the Docker socket
into the backup container so it could stop/start sibling containers — a
real privilege escalation (a compromised backup container could then
control any container on the host). The downside of a best-effort tar is
bounded: worst case, a bad TDLib restore means redoing the Telegram SMS
auth flow, which is the same outcome as having no backup at all. That
bounded, low-severity downside doesn't justify the privilege escalation.
- **Fixed-time cron (`crond`), not a sleep-loop.** A `sleep 86400` loop drifts
on every container restart; a real crontab entry fires at a fixed time of
day regardless of restarts, for negligible extra complexity.
- **Restore is manual, not automated.** A script capable of restoring can
overwrite live state; that should always require a human deliberately
running it, not run unattended.
## Design
### New service: `backup`
Added to both `/opt/stacks/DragonsStash/docker-compose.yml` (production) and
this repo's `docker-compose.yml` (dev/build reference).
- **Image**: custom, `FROM alpine:3.20`, `apk add --no-cache restic
postgresql16-client curl tzdata dcron tar bash`. No dependency on app
source — independent Dockerfile, e.g. `backup/Dockerfile`.
- **Scheduling**: `crond -f` in the foreground as the container's entrypoint,
with a crontab installed at build time:
```
0 3 * * * /backup.sh >> /proc/1/fd/1 2>&1
0 4 * * 0 restic check >> /proc/1/fd/1 2>&1
```
(daily dump/backup at 03:00, weekly repo integrity check at 04:00 Sunday).
`restic init` runs once at container startup (entrypoint, before `crond`
starts), swallowing the "already initialized" error on subsequent
container (re)starts.
- **Network**: `internal` only — reaches `dragonsstash-db:5432` for
`pg_dump`. No ports exposed.
- **Volumes**:
- `tdlib_state:/data/tdlib-worker:ro`
- `tdlib_bot_state:/data/tdlib-bot:ro`
- `nas_backups:/backups`, a named volume defined with:
```yaml
nas_backups:
driver_opts:
type: nfs
o: "addr=${NAS_HOST},rw,nfsvers=4,soft,timeo=100"
device: ":${NAS_EXPORT_PATH}"
```
- **New `.env` entries**: `NAS_HOST`, `NAS_EXPORT_PATH` (NFS share details),
`RESTIC_PASSWORD` (repo encryption key), `KUMA_PUSH_URL` (Uptime Kuma push
monitor URL). All four are inputs to gather during implementation, not
hardcoded.
- `restart: unless-stopped`, no `privileged`, no Docker socket mount.
### `backup.sh`
```
set -euo pipefail
trap 'curl -fsS "$KUMA_PUSH_URL" --get --data-urlencode "status=down" \
--data-urlencode "msg=$BASH_COMMAND failed"' ERR
pg_dump -h dragonsstash-db -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
-Fc -f /tmp/dragonsstash.dump
tar czf /tmp/tdlib.tar.gz -C /data tdlib-worker tdlib-bot
restic backup /tmp/dragonsstash.dump /tmp/tdlib.tar.gz
restic forget --keep-daily 14 --prune
rm -f /tmp/dragonsstash.dump /tmp/tdlib.tar.gz
curl -fsS "$KUMA_PUSH_URL" --get --data-urlencode "status=up" \
--data-urlencode "msg=OK"
```
`PGPASSWORD` and `RESTIC_REPOSITORY=/backups/restic-repo` are set as
container environment variables (from `.env`), not inline in the script.
### Data flow
```
crond (daily 03:00)
→ pg_dump (consistent snapshot via Postgres MVCC) → /tmp/dragonsstash.dump
→ tar tdlib_state + tdlib_bot_state (best-effort, live) → /tmp/tdlib.tar.gz
→ restic backup (encrypt + dedup) → NFS-mounted repo on Synology NAS
→ restic forget --keep-daily 14 --prune
→ curl Uptime Kuma push monitor (up on success, down + reason on any failure)
```
### Restore (manual, documented procedure — not scripted/automated)
```
restic -r /backups/restic-repo restore latest --target /tmp/restore
pg_restore -h dragonsstash-db -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
--clean --if-exists /tmp/restore/tmp/dragonsstash.dump
# untar /tmp/restore/tmp/tdlib.tar.gz back into the tdlib_state /
# tdlib_bot_state volumes (via a throwaway container mounting both)
```
## Alerting
- One Uptime Kuma **Push** monitor, created manually in the existing Kuma
instance, with an expected heartbeat interval of ~26 hours (slack past the
24h schedule so one slow run doesn't false-positive). Whatever notification
channels are already configured on that monitor fire automatically — no new
alerting integration.
- `backup.sh` pushes `status=up` on success and `status=down` (with the
failing command in `msg`) on any failure, via the `ERR` trap.
- Container logs go to stdout, collected the same way every other container's
logs already are on this host.
## Testing
This repo has no automated test framework (documented convention: manual
testing). For this infra change:
- After deploy: manually run `docker exec dragonsstash-backup /backup.sh`
once, confirm a snapshot appears (`restic snapshots`), confirm the Kuma
monitor goes green.
- **Restore drill** (once, during setup): actually restore the dump into a
scratch Postgres and untar the TDLib archive into scratch volumes, to prove
the backup is really restorable. Not automated or recurring for now.
## Out of scope / non-goals
- Backing up `tmp_zips` or `manual_uploads` — both transient by design.
- Automated/scheduled restore testing.
- Backing up any other stack on this host (this design is DragonsStash-only,
though the pattern — Docker-native NFS volume + restic — could be reused
for other stacks later).
- Pausing worker/bot for a guaranteed-consistent TDLib snapshot (see
Decisions).
## Files touched
- `docker-compose.yml` (this repo) and
`/opt/stacks/DragonsStash/docker-compose.yml` (production) — add `backup`
service, `nas_backups` volume.
- `backup/Dockerfile` — new.
- `backup/backup.sh` — new.
- `backup/crontab` — new.
- `.env.example` / `.env` — add `NAS_HOST`, `NAS_EXPORT_PATH`,
`RESTIC_PASSWORD`, `KUMA_PUSH_URL`.
docs/superpowers/specs/2026-07-23-provenance-backfill-design.md
+271
View File
@@ -0,0 +1,271 @@
# Provenance Backfill on Re-Index — Design
**Date:** 2026-07-23
**Status:** Approved for planning
## Summary
Recover the true origin of packages whose recorded "source" is a placeholder —
specifically the manual-upload and `rebuild.ts`-created records whose
`sourceChannelId` points at the destination (archive) channel itself. When a
real source channel is re-indexed and the worker encounters an archive that
matches such a placeholder package, it backfills the real
`sourceChannelId` / `sourceMessageId` / `sourceTopicId` / `sourceCaption` /
`creator` (and, for records that never had one, the file listing and preview)
onto the existing package — **without downloading the full archive**.
Matching is a two-stage process: a zero-download candidate lookup by
`fileName` + total `fileSize`, confirmed by a CRC32 fingerprint read from just
the archive's central directory via a **ranged (tail) download** of a few
KB–MB, instead of the multi-GB whole file.
This is the first of four linked sub-projects. The others (creator
normalization, provenance display, missing-files) are out of scope here and get
their own spec → plan cycles. "Missing files" is explicitly deferred until this
lands.
## Context
Current state (as of this design):
- `Package` records their origin via `sourceChannelId`, `sourceMessageId`,
`sourceTopicId`, `sourceCaption`, and (for content dedup) `contentHash` and
`remoteUniqueId`. `creator` is a free-text string extracted at ingestion.
- Two ingestion paths create records with **placeholder provenance**, where the
"source" is the destination channel, not a real origin:
- **Manual uploads** (`worker/src/manual-upload.ts`) set
`sourceChannelId = destChannel.id` and `sourceMessageId = destResult.messageId`.
They *do* populate `PackageFile` (with `crc32`) from a local central-directory
read at upload time.
- **`rebuild.ts`** scans the destination channel and creates minimal records
with `fileCount == 0` and **no** `PackageFile` rows.
- The main worker upload path (`worker/src/worker.ts` → `uploadToChannel`) sends
archives to the destination channel with **no caption**, so provenance cannot
be recovered by re-reading destination messages — it has to come from matching
against real source channels.
- The scan/dedup ladder in `processOneArchiveSet` (`worker/src/worker.ts:1521`)
is entirely **source-channel-scoped**:
1. `findPackageByRemoteUniqueId(channel.id, …)` — same channel only
2. `packageExistsBySourceMessage(channel.id, …)` — same channel only
3. `findRepostedPackage(channel.id, fileName, size)` — same channel only
4. …then full download → `packageExistsByHash(contentHash)` — global, but only
*after* downloading the whole archive.
- Because a placeholder package's `sourceChannelId` is the destination, checks
1–3 never match it when a real source channel is scanned. Today the worker
therefore downloads the entire archive, hits check 4, finds it is a duplicate,
and skips — **wasting the download and leaving the wrong provenance in place.**
- There is already an in-production precedent for "match an already-known
archive during scan and enrich its metadata": when `findRepostedPackage`
matches, the worker backfills richer *topic* context onto the existing package
via `updatePackageTopicContext` (`worker/src/worker.ts:1608`+). Provenance
backfill is the same move, widened from same-channel to cross-channel.
- `remote.unique_id` is **not** a reliable cross-channel key: it identifies a
stored file object on Telegram's servers, not the content. The same archive
independently uploaded to two channels gets different `unique_id`s; it only
matches for forwarded messages (same underlying file object). This is why the
existing dedup scopes it to a single channel.
## Goals / Non-Goals
**Goals**
- Attribute true origin to placeholder-provenance packages during normal
re-index scans, opportunistically (no separate pass).
- Do it without downloading whole archives (ranged central-directory read only).
- Be non-destructive: only ever touch packages that currently have placeholder
provenance; never overwrite a real, non-placeholder source.
- Be idempotent: re-running a re-index does not re-mutate or duplicate.
**Non-Goals (separate sub-projects / deferred)**
- Creator name normalization / canonical creator entity.
- UI changes to display provenance.
- Recovering "missing files."
- Choosing a *preferred* origin when an archive genuinely exists in several
source channels (first confirmed source wins).
## Design
### 1. Candidate definition
> A package is a backfill candidate iff **`sourceChannelId == destChannelId`
> OR `sourceMessageId == 0`**.
Two placeholder shapes exist (verified against live data 2026-07-23):
- **Manual uploads** (`manual-upload.ts`): `sourceChannelId == destChannelId`,
real `contentHash`, real `sourceMessageId`, has a `PackageFile` listing.
- **Rebuild records** (`rebuild.ts`): `sourceMessageId == 0n` (deliberate
"unknown" sentinel), synthetic `contentHash = "rebuild:<destChannelId>:<destMessageId>"`,
`fileCount == 0`, and `sourceChannelId` set to an **arbitrary fallback source
channel** (`sourceChannels[0]`) — NOT the destination. (This is the common
case: e.g. 59,893 records after a destination rebuild.)
Normal ingestion always sets a real `sourceMessageId` (> 0) and a real source
channel, so neither marker matches a genuinely-sourced package. Both markers are
overwritten on backfill (source channel + message become real), so a record
stops being a candidate once fixed — this is what makes re-scans idempotent.
**Known limitation:** backfill does NOT rewrite a rebuild record's synthetic
`"rebuild:"` `contentHash` (the true content hash would require a full download,
which this feature avoids). That is acceptable — dedup after backfill relies on
`remoteUniqueId` + name/size within the source channel, not on `contentHash`.
### 2. Where it hooks
A new step is inserted into `processOneArchiveSet` **between check #3
(`findRepostedPackage`) and the full download**. It runs only after the
same-channel checks have missed (so genuine same-channel reposts keep their
existing fast paths).
Flow for the scanned archive set:
1. Stage A — **candidate lookup (zero download).** Query for a package where
`(sourceChannelId == destChannelId OR sourceMessageId == 0)` AND
`fileName == archiveName` AND `fileSize == totalArchiveSize`.
(`Package` has `@@index([fileName])`.)
- No candidate → fall through to normal ingestion unchanged.
2. Stage B — **fingerprint confirmation (tiny download).** See §3.
3. On confirmation → **backfill** (see §4) and return `null` (treated as a
duplicate; no full download, no new package). Increment a `zipsBackfilled`
counter.
4. On mismatch / failure / ambiguity → fall through to normal ingestion (see §6).
### 3. Fingerprint confirmation
The confirmation signal is the **multiset of internal CRC32s** of the archive's
entries (CRC32 of each entry's *uncompressed* data). This value is a property of
the archive contents and is identical regardless of which channel hosts the
file.
- **Candidate side:** for manual uploads, `PackageFile.crc32` is already
populated from the local central-directory read — zero cost. For **rebuild
records** (`fileCount == 0`, no `crc32`), there is nothing stored to compare
against; obtain the candidate's fingerprint with a **second ranged tail read
of its destination copy** (ZIP/7z — still no full download). RAR candidates
cannot be tail-read on either side, so they fall back to name+size (see §5).
- **Scanned-source side:** read via a **ranged (tail) download** of the central
directory:
- **ZIP** (incl. multipart): the End-of-Central-Directory record + central
directory live at the tail of the last part. Download only that tail and
parse entries. (Reuses the lightweight-listing mechanism that is
sub-project 4's core.)
- **7z**: a start header at byte 0 points to an end header at the tail; fetch
both small pieces.
- **RAR**: headers are scattered through the file — no cheap tail read. See §5.
- **Match rule:** confirmed iff the sorted CRC32 multisets are equal **and** file
counts are equal.
The comparison logic (CRC32 multiset equality) and the candidate-match predicate
are implemented as **pure functions** with no TDLib dependency, so they are unit
testable (see §7).
### 4. What gets written
On a confirmed match, update the candidate package in a single transaction,
overwriting only placeholder/empty fields:
| Field | New value | Condition |
|---|---|---|
| `sourceChannelId` | scanned `channel.id` | always |
| `sourceMessageId` | scanned `parts[0].id` | always |
| `sourceTopicId` | `ctx.sourceTopicId` | always |
| `sourceCaption` | scanned message caption | always |
| `remoteUniqueId` | scanned `firstRemoteUniqueId` | always (enables future same-channel dedup via check #1) |
| `creator` | re-derived from source (topic > filename > channel) | **always** — un-normalized; the later creator-normalization sub-project cleans it up |
| `fileCount` + `PackageFile[]` | from the central-directory read | only if candidate had none (`fileCount == 0`) — folds in the sub-project 4 outcome for rebuild records |
| `previewData` / `previewMsgId` | matched preview from scan | only if candidate has none |
**Left untouched:** `contentHash`, `destChannelId`, `destMessageId`,
`destMessageIds` — the bytes physically live in the destination channel; that is
correct and must not change.
**Transaction safety:** re-check `sourceChannelId == destChannelId` *inside* the
transaction before writing, so a concurrent worker that already backfilled the
record causes this one to no-op (mirrors the existing `backfill.ts` guard).
### 5. RAR handling
RAR sources cannot be tail-read, so no cheap CRC fingerprint is available.
Decision: **backfill RAR matches on `fileName` + `fileSize` alone**, flagged as
lower-confidence in logs and via a `SystemNotification`, so they can be audited.
No full download.
This name+size-only fallback applies to any candidate that cannot produce a
CRC32 fingerprint cheaply: a RAR **source**, a RAR **candidate**, or a rebuild
candidate whose destination copy is RAR. ZIP/7z rebuild candidates are still
confirmed by fingerprint via the second tail read described in §3.
### 6. Conflict, mismatch, ambiguity
- **Fingerprint mismatch** → the scanned archive is genuinely different content
that merely shares name+size. Fall through to **normal ingestion** (download +
index) — it is a new package for this source.
- **Ambiguous candidates** (2+ match name+size and the fingerprint cannot
disambiguate) → log a `SystemNotification`, backfill nothing, and fall through
to normal ingestion; the post-download `packageExistsByHash` check still
dedups it safely.
- **Same archive in multiple source channels** → the **first re-indexed source
that confirms wins.** After backfill the package has a real source, so it is no
longer a candidate; later scans treat it as an ordinary duplicate via checks #1
(the `remoteUniqueId` we set) or #3.
### 7. Idempotency
Falls out of the candidate definition. Once backfilled:
- `sourceChannelId` is the real channel → no longer a candidate.
- A re-scan of that source hits check #1 (`remoteUniqueId`, now set) or check #3
(`findRepostedPackage`) → normal dedup skip. No re-mutation, no duplicate.
### 8. Error handling
- **Tail-download failure** (network / `FLOOD_WAIT`) → cannot confirm this round.
Do **not** fall back to a full download and do **not** guess: leave the
candidate untouched and let the next re-index retry. Wrap the ranged read in
`withFloodWait` (per the TDLib skill).
- All new TDLib calls follow the skill's patterns: `withFloodWait`, listener
attached before the async op, client closed in `finally`.
### 9. Visibility
- Add a `zipsBackfilled` counter to `IngestionRun` activity so a re-index run
reports "N provenance backfills" instead of silently mutating records.
- Info log per backfill (candidate id, old vs new source, confidence:
fingerprint | name+size-RAR).
- `SystemNotification` for ambiguous-candidate cases.
## Testing
The repo currently has no test framework (`CLAUDE.md`: "testing is manual"). This
work introduces a **lightweight test harness** for the worker (e.g. `vitest` or
node's built-in `node:test`) and unit-tests the correctness-critical pure logic:
- **Unit (automated):**
- CRC32 multiset fingerprint equality (equal sets, different order, differing
counts, disjoint sets).
- Candidate-match predicate (name+size+placeholder true/false cases).
- Field-merge rules (which fields overwrite, which only fill-if-empty).
- **Manual integration checklist:**
1. Re-index a real source channel containing a known manually-uploaded pack →
verify `sourceChannelId`/`sourceMessageId`/`sourceCaption`/`creator` are
backfilled and no full download occurs.
2. A rebuild-created record (`fileCount == 0`) → verify listing + provenance
are both populated from the single tail read.
3. A RAR pack → verify name+size backfill with the lower-confidence log/notice.
4. Re-run the same re-index → verify it is a no-op (idempotent).
5. A genuine name+size collision (different content) → verify it ingests as a
new package rather than being mis-attributed.
## Open Questions
None blocking. Preferred-origin selection among multiple real sources is
deliberately out of scope (first confirmed wins).
## Affected Code (indicative, for planning)
- `worker/src/worker.ts` — `processOneArchiveSet`: new Stage A/B step; new counter.
- `worker/src/archive/` — ranged central-directory reader (ZIP/7z tail); shared
with sub-project 4. Pure CRC32-fingerprint compare helper.
- `worker/src/tdlib/download.ts` — ranged/partial download support (`offset`/`limit`).
- `worker/src/db/queries.ts` — candidate lookup + transactional backfill update.
- `prisma/schema.prisma` — `IngestionRun.zipsBackfilled` (and run-counter plumbing).
- Worker test harness + first unit tests.
docs/superpowers/specs/2026-07-27-ranged-archive-listing-design.md
+180
View File
@@ -0,0 +1,180 @@
# Ranged inner-file listing for RAR & 7z — design
**Date:** 2026-07-27
**Status:** Approved (design), pending spec review → implementation plan
## Problem
The reindex/provenance-backfill path (`worker/src/provenance-backfill.ts`) can index an
archive's inner files *without* re-downloading it, by reading the file listing from a small
ranged read of the copy already in the source/destination channel. This works **only for
ZIP** today (ZIP keeps its central directory in a tail that `parseZipCentralDirectoryFromTail`
reads). For **RAR and 7z**, `tryProvenanceBackfill` backfills provenance (creator, source
channel, `remoteUniqueId`) so the file is skipped on re-scan and never re-downloaded — but it
leaves the inner listing empty (`fileCount = 0`), because `scannedEntries` is only computed for
ZIP.
Scope of the gap (rebuild placeholders with `fileCount = 0`, as of 2026-07-27):
| Type | Count | Total | Avg | Max | Multipart |
|---|---|---|---|---|---|
| 7z | 19,646 | 14 TB | 0.73 GB | 3.9 GB | 0 |
| RAR | 12,780 | 13 TB | 1.04 GB | 116 GB | 1,016 |
A "just download the whole archive" fallback for all of these means ~27 TB of re-downloads —
the exact cost this path exists to avoid.
## Goals
- Index the inner files (names + sizes; CRCs where cheaply available) of RAR and 7z
placeholders **without** downloading the whole archive in the common case.
- Reuse the existing, battle-tested CLI listing parsers (`parse7zOutput`,
`parseUnrarTechnical`) rather than reimplementing filename/size/CRC extraction.
- Keep cost proportional to **file count**, not archive size (so even the 116 GB RAR is cheap).
- Guarantee a listing for the rare archives the cheap path can't handle, via a full-download
fallback that respects the existing max-size guard.
## Non-goals
- No change to ingestion of genuinely-new files (those are downloaded in full to be re-uploaded
regardless, so a cheap listing does not help there). This feature only affects the
provenance-backfill / skip path.
- No new ZIP behaviour — the existing ZIP tail reader stays as-is.
- Not attempting to list password-encrypted-header archives from ranged reads (no password);
those take the fallback and, if oversized, are flagged.
## Approach (chosen: "harvest header regions → sparse file → native CLI")
Do the *minimum* binary parsing needed to locate an archive's header bytes, fetch only those
via ranged reads, write them into a sparse temp file at their true offsets (data regions left
as unwritten zero holes → ~no disk use), then run the real `7z l` / `unrar lt` and reuse the
existing parsers. The native tools do the hard parsing (7z's LZMA-encoded headers, RAR's two
format versions, Unicode names) — we only compute where the headers are.
Rejected alternatives: full native TS parsers (most custom binary code, highest risk);
RAR5-quick-open-only (most community RARs lack it → collapses to ~13 TB of RAR downloads).
## Components
New directory `worker/src/archive/ranged/`, one focused module per concern, all returning the
existing `FileEntry[]` type from `zip-reader.ts`:
- `sparse-list.ts` — `listFromSparse(parts, runner, parse) → FileEntry[] | null`, where each
`part` is `{ fileName, size, regions: {offset, bytes}[] }`. For each part it writes a sparse
temp file (`truncate` to `size`, then write only the header `regions` at their offsets),
co-locates all parts in one temp dir under their real names, invokes the supplied CLI runner
(`7z l` / `unrar lt`) on the first part, feeds stdout to the supplied `parse` fn
(`parse7zOutput` / `parseUnrarTechnical`), and cleans up. Single-part archives are just the
one-element case. Returns `null` on CLI error / empty parse.
- `sevenz-ranged.ts` — `readSevenZListingRanged(client, parts) → FileEntry[] | null`.
- `rar-ranged.ts` — `readRarListingRanged(client, parts) → FileEntry[] | null`.
- Dispatcher in `provenance-backfill.ts`: `readScannedListingRanged(archiveType, client, parts)`
replacing the current `if (archiveType === "ZIP")` branch; the destination-copy read in
`resolveCandidateFingerprintEntries` gets the same dispatch.
`FileEntry` shape (unchanged): `{ path, fileName, extension, compressedSize, uncompressedSize, crc32 }`.
### 7z ranged listing
7z layout: 32-byte signature header at offset 0 → packed streams → end header (lists files) at
the end; the signature header stores the end header's location.
1. Ranged-read `[0, 32)`; validate magic `37 7A BC AF 27 1C`. Read LE `uint64`
`NextHeaderOffset` (byte 12) and `NextHeaderSize` (byte 20). End header is at absolute offset
`32 + NextHeaderOffset`, length `NextHeaderSize`.
2. Ranged-read `[32 + NextHeaderOffset, NextHeaderSize)` — the "next header".
3. Branch on the next header's first byte (a 7z property id):
- **`0x01` (kHeader, plain/uncompressed header):** two regions suffice —
`{0: sigHeader}` and `{32+NextHeaderOffset: endHeader}`.
- **`0x17` (kEncodedHeader, LZMA-compressed header):** the next header is only a *descriptor*
whose `PackInfo` points at a packed header stream stored **in the middle** of the file (not
at EOF). Parse the descriptor's `StreamsInfo → kPackInfo (0x06)` to read `PackPos` and the
`PackSize`s (7z variable-length "numbers"; sum them). Ranged-read the contiguous packed
region `[32 + PackPos, Σ PackSize)` and add it as a **third** sparse region. `7z l` then
decodes the header from that region.
- **anything else:** return `null` (→ fallback).
4. `listFromSparse` with the 2 or 3 regions, runner = `7z l`. `parse7zOutput` yields
names+sizes (`crc32: null`, as today). Return `null` on bad magic / read failure / CLI error.
**Why the third region is required (spike finding, 2026-07-27):** the original two-region
(start+end) reconstruction was proven insufficient in a live test — `7z l` rejected it with
"Cannot open the file as [7z] archive" because these archives use an *encoded* header whose
compressed bytes live in a packed stream in the file body (a sparse hole), not at EOF. The
`0x17` branch fetches exactly that packed region. `7z l` still never touches the file-data
packed streams (it only lists), so those gaps stay sparse. The `read7zNumber` reader (7z's
base-128-ish variable-length integer with a first-byte length mask) and the minimal
`kPackInfo` walk are the only new 7z binary parsing; `7z l` still does the actual file listing.
All 7z placeholders are single-part.
### RAR ranged listing
RAR has no index; walk the block chain, parsing only each block's **size fields** to step
forward and harvest header bytes.
1. Read first ~16 bytes; detect **RAR4** (`52 61 72 21 1A 07 00`) vs **RAR5** (`…07 01 00`) and
the signature length.
2. From just after the signature, loop:
- Ranged-read a header chunk (start 8 KB; if parsed `HeaderSize` exceeds it — long filenames
— re-read exactly).
- Minimal block-extent parse:
- RAR5: `CRC32(4)` + vint `HeaderSize` + vint `HeaderType` + vint `HeaderFlags`; if the
"extra area" flag (`0x0001`) → vint `ExtraAreaSize`; if the "data present" flag
(`0x0002`) → vint `DataSize`. Next block = `pos + 4 + len(HeaderSize vint) + HeaderSize
+ DataSize`.
- RAR4: `HEAD_CRC(2)` + `HEAD_TYPE(1)` + `HEAD_FLAGS(2)` + `HEAD_SIZE(2)`; if flag `0x8000`
→ `ADD_SIZE(4)`. Next block = `pos + HEAD_SIZE + ADD_SIZE`.
- Harvest `[blockOffset, blockOffset + HeaderSize)` into the regions list.
- Stop at the end-of-archive block or EOF.
3. `listFromSparse` (headers present, data sparse) → `unrar lt` → `parseUnrarTechnical`. RAR
headers carry CRC32, so RAR contributes CRCs (fingerprint disambiguation keeps working).
**Multipart RAR** (1,016): each volume starts with its own signature + headers. Walk **each
part from its own signature**, reconstruct one sparse temp file per part with correct names
(`name.part1.rar`, `.part2.rar`, …) co-located in a temp dir, and run `unrar lt` on part 1 —
`unrar` auto-discovers co-located siblings (per the existing reader's note). The global-offset →
`(part, offsetInPart)` mapping reuses the multipart size math the ZIP path already uses.
## Fallback & integration
- A ranged reader returning `null` = cheap read failed (bad magic, read error, walk gave up, or
CLI error on the sparse file) → **full-download fallback**: download the whole archive, run the
existing `readRarContents` / `read7zContents`, backfill.
- The fallback is gated by `config.maxZipSizeMB` (the same guard used at ingest). Over the cap →
no download; write a `SystemNotification` (`INTEGRITY_AUDIT`, WARNING) and leave the listing
empty for manual review. This ensures nothing pathological (e.g. the 116 GB RAR) is pulled.
- Downstream is unchanged: `compareFingerprints` already treats null/incomplete CRCs as
"incomplete" (name-size path), and `backfillProvenance` writes entries when the candidate's
`fileCount === 0`.
- Observability: reuse the `zipsBackfilled` counter; add structured logs with
`confidence: "ranged" | "full-download-fallback"` and a WARN on fallback so miss-rate is
visible.
## Risks & de-risking spike (do before the full build)
On 3–4 real placeholder archives per format:
1. Confirm `downloadFileRange` returns correct bytes at **arbitrary (non-tail) offsets** —
currently only tail-verified in production. Underpins everything; if it fails, stop and
rethink. (Note: `range-download.ts` flags absolute-offset behaviour as pending live
verification; tail reads are proven by the 43 ZIP backfills done 2026-07-26.)
2. Confirm `7z l` and `unrar lt` list correctly from a **sparse reconstructed file** — single
part first, then multipart RAR (the highest-risk case).
If multipart-RAR sparse reconstruction proves unreliable in the spike, multipart RAR uses the
full-download fallback (respecting the size cap → oversized ones flagged, not downloaded).
## Testing
- **Unit (vitest, alongside `central-directory.test.ts`):** 7z signature-header parse; RAR4 &
RAR5 block-extent walk against committed small fixtures; `sparse-list` writes the correct
regions. Pure logic, no TDLib.
- **Live post-deploy:** watch `zipsBackfilled` climb for RAR/7z via the ranged path; spot-check
a handful of backfilled packages' `package_files` against a real `unrar lt` / `7z l` on a full
download of the same file; confirm the fallback/flag path fires on a deliberately-broken case.
## Rollout
Local build + deploy (no GitHub push required), per the established recipe: build
`worker/Dockerfile` locally, recreate the `dragonsstash-worker` container from the local image
(no `pull`). No new DB migration. The scheduler re-runs hourly and will backfill RAR/7z
placeholders on subsequent cycles.
docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md
+211
View File
@@ -0,0 +1,211 @@
# Forward-priority ingestion — design
**Date:** 2026-07-30
**Status:** Approved (design), pending spec review → implementation plan
## Problem
The worker ingests every archive the same way regardless of whether it needs to: download the
full file from the source channel, then re-upload the full file to the destination (archive)
channel. That download+reupload round-trip was originally necessary because some source channels
have "restrict saving content" (protected content) enabled, which blocks Telegram-native
forwarding — for those channels there is no alternative to moving the bytes through the worker.
But most source channels do NOT restrict forwarding. For those, the round-trip is pure waste:
Telegram can copy the message from source chat to destination chat server-side, with no bytes
ever passing through the worker. The worker still needs to end up with the same outcome it has
today — a destination-channel copy, a dedup-safe identity, and a full inner-file listing — just
without paying for a download and re-upload to get there.
Separately, `feat/ranged-archive-listing` (merged to master ahead of this feature) already built
exactly the missing piece: reading a ZIP/RAR/7z archive's inner-file listing via small ranged
reads against the file wherever it currently lives (source channel, destination channel — doesn't
matter), with no full download. It was built for backfilling listings onto already-deduped
placeholder packages. This feature generalizes that same capability to fresh ingestion, and pairs
it with a new native-forward upload path.
## Goals
- For channels that allow forwarding: skip download and re-upload entirely for new archives. Use
Telegram-native forwarding from source chat to destination chat, and the existing ranged-listing
readers to index inner files, with no full download in the common case.
- For channels that block forwarding (or when forwarding isn't yet known): keep today's
download+reupload pipeline exactly as-is.
- Every ingested package — regardless of path — ends up with the same outcome as today: a
`Package` row with a valid dedup identity, `destMessageId`/`destMessageIds`, creator, tags, and a
full inner-file listing (`PackageFile` rows). Indexing completeness must not regress.
- If the cheap ranged listing fails for a specific archive (bad/unsupported header, CLI error,
etc.) in an otherwise-forwarding-eligible channel, fall back to today's full download+reupload
pipeline for that one archive — never forward with an empty or partial listing.
## Non-goals
- No ranged single-entry preview extraction. Forward-path packages still get a preview when a
channel photo message matches (cheap, unrelated to archive bytes); when there's no matching
photo, forward-path packages simply have no preview, same as any package where preview
extraction fails today. In-archive preview extraction (unzip/unrar/7z against a local file) stays
as a download-path-only feature. May be revisited as a follow-up if it turns out to matter.
- No reprocessing of already-ingested packages. This only changes behavior for newly-scanned
archives going forward.
- No change to the bot's user-delivery leg (`bot/src/tdlib/client.ts` `copyMessageToUser`) — it
already sends via `inputFileRemote` with no download, and is unaffected by this feature.
- No change to `config.maxZipSizeMB` or the multipart byte-level split/repack logic. The existing
size guard runs before either path is chosen, so nothing above the cap reaches the forward path's
fallback-to-download step either. Splitting simply never engages on the forward path — a
forwarded message is already within whatever size Telegram accepted when it was first uploaded.
## Approaches considered
**A — Branch inside the existing pipeline (chosen).** Add one fork point in
`processOneArchiveSet`, immediately after the existing pre-download dedup checks: if the channel
allows forwarding, attempt the ranged-listing + forward path; on any failure, fall through into
today's download-based code for that one archive, unchanged. Smallest diff; reuses the existing
dedup/retry/watermark machinery as-is; matches the file's existing forum-vs-non-forum branching
style.
**B — Separate pipeline per channel.** Decide once per channel and route the whole channel through
either a "forward module" or the existing "download module." Cleaner separation on paper, but
duplicates the SkippedPackage/stall/watermark bookkeeping that currently lives once in
`processArchiveSets`/`processOneArchiveSet` — higher regression risk in a large orchestration file
with no tests at that level. Rejected.
**C — Strategy-object refactor.** Extract an `IngestStrategy` interface (`download` / `forward`)
and slim `processOneArchiveSet` to delegate to it. The more "proper" abstraction, but it's a
structural refactor of already-battle-tested code that doesn't need it for this feature to work.
Rejected — can revisit later if a third strategy ever appears.
## Sequencing
`feat/ranged-archive-listing` merges to master first, as-is (it's complete and serves a different
purpose already). This feature is built on a fresh branch off master afterward.
## Components
### 1. `TelegramChannel.allowsForwarding` (new column, new migration)
`Boolean?` — nullable, `null` means "not yet checked". Refreshed from TDLib's chat
protected-content flag (exact field name to be confirmed against the pinned `tdl`/TDLib version
via docs lookup during implementation — expected to be `chat.has_protected_content`) at the same
point the worker already calls `getChat` per channel per cycle, mirroring the existing
`isForum`/`setChannelForum` read-and-persist pattern precisely. `null` or `false` both route to the
download path — a channel never uses the forward path on unverified permission.
### 2. Shared ranged-listing dispatcher
`readScannedListingRanged` (plus `RangedPart`, `tdlibRangeReader`, and the format-specific
ZIP/RAR/7z readers) currently live inside `provenance-backfill.ts`. Promote the dispatcher (and
whatever it depends on) into a shared module (e.g. `worker/src/archive/ranged/dispatch.ts`) so
`worker.ts` can call the same no-download listing logic for fresh ingestion without a circular
import. `provenance-backfill.ts` switches to importing from the new shared location; behavior
unchanged for the existing backfill path.
### 3. `forwardArchiveToChannel` (new, `worker/src/upload/forward.ts`)
Mirrors `uploadToChannel`'s shape and return type (`{ messageId, messageIds }`). Uses TDLib
`forwardMessages` to copy all parts of an archive set from the source chat to the destination chat
in one batch call (message IDs in original order), wrapped in the same flood-wait/retry handling
style as `uploadToChannel`. Followed by the same destination read-back verification style as
today's post-upload check (`getMessage` on each new destination message ID, confirm a document is
present).
### 4. Dedup identity for forward-path packages
`Package.contentHash` stays a required unique string, but forward-path packages can't hash real
bytes. Derivation order:
1. If the ranged listing's CRC32s are complete (ZIP/RAR today) — hash the sorted CRC32 list into a
synthetic `fingerprint:<hash>` value, reusing `archive/fingerprint.ts`'s existing
`crcFingerprint`.
2. Otherwise (7z, or any incomplete-CRC case) — synthesize `forward:<remoteUniqueId>`, following
the existing `rebuild:`-prefixed placeholder-hash precedent in `rebuild.ts`.
Additionally, extend repost detection: before committing to the forward path, compare the new
listing's CRC fingerprint (via the existing `compareFingerprints`/`fingerprintsMatch` logic already
used in `provenance-backfill.ts`'s ambiguous-candidate disambiguation) against recent Packages
sharing the same file name + size. A fingerprint match is treated as a duplicate and skipped, same
as today's `findRepostedPackage` handling — this is what lets a forwarded copy and a previously
fully-downloaded copy of the same archive still dedupe against each other, despite never sharing a
byte-hash-derived `contentHash`.
### 5. Fork point in `processOneArchiveSet`
All existing pre-download checks run first, completely unchanged, in the same order:
`remote.unique_id` match → `packageExistsBySourceMessage` → `findRepostedPackage` (name+size) →
cross-channel provenance backfill → size guard (`maxZipSizeMB`).
Then:
```
if channel.allowsForwarding === true:
entries = readScannedListingRanged(archiveType, client, scannedParts)
if entries is not null:
contentHash = deriveForwardContentHash(entries, remoteUniqueId)
if fingerprintRepostCheck(entries, fileName, fileSize) finds a match:
→ treat as duplicate, skip (same bookkeeping as today's dup path)
destResult = forwardArchiveToChannel(client, sourceChatId, partMessageIds, destChatId)
creator, tags ← derived from entries/filename/channel/topic, same as today
preview ← channel-photo match only (no in-archive extraction)
createPackageStub(...) + updatePackageWithMetadata(...), same as today
counters.zipsForwarded++
→ done
else:
→ fall through into the existing download/hash/split/upload flow below, unchanged
(log the fallback for observability)
else:
→ existing download/hash/split/upload flow, completely unchanged
```
### 6. Observability
New `zipsForwarded` counter alongside the existing `zipsFound`/`zipsDuplicate`/`zipsIngested`/
`zipsBackfilled` counters, surfaced the same way (run activity, ingestion run summary). A WARN-level
log line when a forwarding-eligible archive falls back to download (mirrors the existing
`confidence: "ranged" | "full-download-fallback"` logging convention from the ranged-listing
backfill work), so the fallback rate is visible without digging through debug logs.
## Data flow
```
scan → pre-download dedup + size guard (unchanged)
→ channel.allowsForwarding?
true → ranged listing
ok → fingerprint dedup check → forward → stub + entries + tags (no in-archive preview) → done
null → [fall through] existing download pipeline
false/unknown → existing download pipeline (unchanged)
```
## Error handling
- `forwardMessages` failure (permission revoked mid-run, rate limit, transient Telegram error) —
same `SkippedPackage`/`SystemNotification` bookkeeping as today's upload failures. Extend
`inferSkipReason` to recognize forward-specific error text the same way it already recognizes
upload errors.
- Fingerprint-repost check finds multiple ambiguous same-name/size candidates that can't be
uniquely disambiguated — same `INTEGRITY_AUDIT` notification pattern already used in
`provenance-backfill.ts`: don't guess, surface for manual triage.
- `allowsForwarding` unknown (channel just linked, not yet scanned by the refresh point) — treated
as `false`; the download path runs. No channel uses an unverified forwarding permission.
- Ranged listing throwing instead of returning `null` — treated identically to returning `null`
(fall through to download), consistent with how the existing ranged readers already treat
internal errors (they catch and return `null` themselves).
## Testing
- Unit tests (vitest, alongside the existing `archive/*.test.ts` and `archive/ranged/*.test.ts`
files): the dedup-identity derivation function (fingerprint-hash vs remoteUniqueId-fallback
branches), the extended fingerprint-based repost check, and `forwardArchiveToChannel`'s
request-building logic against a mocked TDLib client — same style as the existing ranged-reader
tests (pure logic, no live TDLib).
- Live verification (manual — matches this repo's existing convention that the large
`worker.ts`/`worker.py`-equivalent orchestration function has no automated test coverage and is
verified live post-deploy): one forwarding-enabled test channel and one protected-content test
channel. Confirm forward-path packages land with correct entries/tags/dedup identity and
`destMessageIds`; confirm the protected channel still goes through the unchanged full pipeline;
confirm a deliberately-unparseable archive in a forwarding-enabled channel correctly falls back
to download+reupload and still ends up fully indexed.
## Rollout
Local build + deploy, following the same recipe as the ranged-archive-listing work: build
`worker/Dockerfile` locally, recreate the `dragonsstash-worker` container from the local image (no
`pull`, no GitHub push required). New DB migration for `TelegramChannel.allowsForwarding`. No
changes required to the bot or app services.