mirror of
https://github.com/xCyanGrizzly/DragonsStash.git
synced 2026-09-21 13:31:42 +00:00
docs: correct backup scope to database and sessions
Clarify that PostgreSQL logical dumps and TDLib session volumes are protected, while STL binaries remain in Telegram and manual uploads stay excluded.
This commit is contained in:
@@ -2,22 +2,25 @@
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add a nightly, encrypted, deduplicated disaster-recovery backup for PostgreSQL, uploaded STL files, and Telegram session volumes, stored on a Synology NAS.
|
||||
**Goal:** Add a nightly, encrypted disaster-recovery backup for PostgreSQL and Telegram session volumes, stored on a Synology NAS.
|
||||
|
||||
**Architecture:** A Linux-host systemd timer invokes a host orchestration script. The script verifies the mounted Synology NFS share, stops the app/worker/bot services for consistency, and runs a one-shot Docker Compose backup service. The backup service creates a PostgreSQL custom-format dump and stores it together with the three persistent file volumes in a Restic repository on the NAS. A guarded restore command reconstructs the database and volumes, and documentation describes setup and testing.
|
||||
**Architecture:** A Linux-host systemd timer invokes a host orchestration script. The script verifies the mounted Synology NFS share, stops the app/worker/bot services for consistency, and runs a one-shot Docker Compose backup service. The backup service creates a PostgreSQL custom-format dump and stores it with the two TDLib session volumes in a Restic repository on the NAS. A guarded restore command reconstructs the database and sessions, and documentation describes setup and testing. STL binaries remain in Telegram; restored database metadata and mappings continue to identify the Telegram content used for lookup and delivery.
|
||||
|
||||
**Tech Stack:** Docker Compose, PostgreSQL 16 `pg_dump`/`pg_restore`, Restic repository encryption and retention, Synology NFS, Linux systemd service/timer, Bash.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- PostgreSQL data must be backed up as a logical custom-format dump; the raw `postgres_data` volume is not the primary backup.
|
||||
- The `manual_uploads`, `tdlib_state`, and `tdlib_bot_state` volumes are included in every successful snapshot.
|
||||
- The `tmp_zips` scratch volume is excluded.
|
||||
- The Docker host must stop `app`, `worker`, and `bot` while volumes are captured; PostgreSQL remains running for `pg_dump`.
|
||||
- The `tdlib_state` and `tdlib_bot_state` volumes are included in every successful snapshot.
|
||||
- The `manual_uploads` and `tmp_zips` volumes are excluded.
|
||||
- Completed STL binaries are not retained locally for backup; existing worker cleanup behavior remains unchanged. Telegram remains the binary store, while PostgreSQL retains the metadata and mappings needed to locate and send the files after restore.
|
||||
- The Docker host must stop `app`, `worker`, and `bot` while session volumes are captured; PostgreSQL remains running for `pg_dump`.
|
||||
- The backup repository is encrypted and stored on a Synology NFS share restricted to the Docker host.
|
||||
- Retention is 30 daily snapshots; pruning is allowed only after a verified successful backup.
|
||||
- A failed run must restart services and preserve the last known-good snapshot.
|
||||
- A guarded restore must require explicit confirmation before replacing live database or session-volume data.
|
||||
- No in-app backup UI is part of this implementation.
|
||||
- Future Telegram channel-forwarding behavior and archive/STL-content integrity validation are explicitly out of scope.
|
||||
- The repository has no automated test framework; verification uses Bash syntax checks, Docker Compose validation, logs, Restic checks, and a disposable restore rehearsal.
|
||||
|
||||
---
|
||||
@@ -33,7 +36,7 @@ Create or modify only these focused units:
|
||||
- Create `deploy/systemd/dragons-stash-backup.service`: systemd unit invoking the host backup script.
|
||||
- Create `deploy/systemd/dragons-stash-backup.timer`: nightly schedule.
|
||||
- Create `scripts/backup/README.md`: Synology setup, host mount, secrets, first backup, restore, and operational troubleshooting.
|
||||
- Modify `docker-compose.yml`: add the profile-gated one-shot `backup` service and its read-only volume mounts.
|
||||
- Modify `docker-compose.yml`: add the profile-gated one-shot `backup` service and its read-only session-volume mounts.
|
||||
- Modify `.env.example`: document backup mount, staging, repository, and secret-file configuration without committing secrets.
|
||||
- Modify `README.md`: add the production backup setup and restore entry points, linking to the detailed backup guide.
|
||||
|
||||
@@ -47,8 +50,8 @@ Create or modify only these focused units:
|
||||
- Create: `backup/Dockerfile`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: existing `db`, `manual_uploads`, `tdlib_state`, and `tdlib_bot_state` Compose resources.
|
||||
- Produces: a profile-gated Compose service named `backup` that mounts the three file volumes read-only, connects to the `backend` network, and exposes `/backup` and `/staging` to the container entrypoint.
|
||||
- Consumes: existing `db`, `tdlib_state`, and `tdlib_bot_state` Compose resources.
|
||||
- Produces: a profile-gated Compose service named `backup` that mounts the two session volumes read-only, connects to the `backend` network, and exposes `/backup` and `/staging` to the container entrypoint.
|
||||
|
||||
- [ ] **Step 1: Add explicit backup environment variables to `.env.example`**
|
||||
|
||||
@@ -82,7 +85,6 @@ Add a service with these properties:
|
||||
BACKUP_APP_VERSION: ${BACKUP_APP_VERSION:-unknown}
|
||||
user: "0:0"
|
||||
volumes:
|
||||
- manual_uploads:/data/uploads:ro
|
||||
- tdlib_state:/data/tdlib-worker:ro
|
||||
- tdlib_bot_state:/data/tdlib-bot:ro
|
||||
- ${BACKUP_MOUNT_PATH:?Set BACKUP_MOUNT_PATH to the mounted Synology share}:/backup:rw
|
||||
@@ -95,7 +97,7 @@ Add a service with these properties:
|
||||
- backend
|
||||
```
|
||||
|
||||
Ensure the new service does not have `restart: always` and is not started by the normal production `docker compose up -d` command unless the `backup` profile is explicitly requested.
|
||||
Do not mount `manual_uploads` or `tmp_zips`. Ensure the new service does not have `restart: always` and is not started by the normal production `docker compose up -d` command unless the `backup` profile is explicitly requested.
|
||||
|
||||
- [ ] **Step 3: Create the backup image definition**
|
||||
|
||||
@@ -135,7 +137,7 @@ git commit -m "feat: add backup compose service"
|
||||
- Create: `scripts/backup/container-entrypoint.sh`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `DATABASE_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD_FILE`, `BACKUP_RETENTION_DAYS`, `/data/uploads`, `/data/tdlib-worker`, `/data/tdlib-bot`, `/backup`, and `/staging`.
|
||||
- Consumes: `DATABASE_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD_FILE`, `BACKUP_RETENTION_DAYS`, `/data/tdlib-worker`, `/data/tdlib-bot`, `/backup`, and `/staging`.
|
||||
- Produces: exit `0` only after a verified Restic snapshot and successful retention pruning; non-zero on any failed dump, snapshot, verification, or prune step.
|
||||
|
||||
- [ ] **Step 1: Define strict shell behavior and required inputs**
|
||||
@@ -165,11 +167,11 @@ After the command succeeds, require the dump to be a non-empty regular file. Gen
|
||||
|
||||
- [ ] **Step 4: Create the manifest**
|
||||
|
||||
Write a JSON manifest containing the UTC backup timestamp, repository path, retention value, dump filename, dump checksum, and the three volume paths captured. Obtain the application image/version from an explicit `BACKUP_APP_VERSION` environment value when supplied; otherwise record `unknown` rather than guessing from mutable container state.
|
||||
Write a JSON manifest containing the UTC backup timestamp, repository path, retention value, dump filename, dump checksum, and the two TDLib volume paths captured. Obtain the application image/version from an explicit `BACKUP_APP_VERSION` environment value when supplied; otherwise record `unknown` rather than guessing from mutable container state.
|
||||
|
||||
- [ ] **Step 5: Create one Restic snapshot**
|
||||
|
||||
Run one `restic backup` command against the staged database dump, manifest, and the three mounted persistent volumes. Use stable source labels so the snapshot can be recognized during restore. Do not include `/data/uploads` more than once and do not pass `/data/tdlib` or `/tmp/zips` paths that are not mounted.
|
||||
Run one `restic backup` command against the staged database dump, manifest, and the two mounted persistent session volumes. Use stable source labels so the snapshot can be recognized during restore. Do not mount or include `manual_uploads` or `tmp_zips`.
|
||||
|
||||
- [ ] **Step 6: Verify and apply retention**
|
||||
|
||||
@@ -200,13 +202,13 @@ docker build -f backup/Dockerfile -t dragons-stash-backup:smoke .
|
||||
bash -n scripts/backup/container-entrypoint.sh
|
||||
```
|
||||
|
||||
Expected: image build succeeds and Bash reports no syntax errors. The full snapshot test waits until the host wrapper and a real PostgreSQL/volume environment exist.
|
||||
Expected: image build succeeds and Bash reports no syntax errors. The full snapshot test waits until the host wrapper and a real PostgreSQL/session-volume environment exist.
|
||||
|
||||
- [ ] **Step 8: Commit the backup container**
|
||||
|
||||
```bash
|
||||
git add backup/Dockerfile scripts/backup/container-entrypoint.sh
|
||||
git commit -m "feat: implement encrypted database and volume snapshots"
|
||||
git commit -m "feat: implement encrypted database and session snapshots"
|
||||
```
|
||||
|
||||
### Task 3: Add the host orchestration script and nightly systemd timer
|
||||
@@ -264,26 +266,16 @@ git add scripts/backup/run-backup.sh deploy/systemd/dragons-stash-backup.service
|
||||
git commit -m "feat: schedule nightly off-host backups"
|
||||
```
|
||||
|
||||
### Task 4: Make completed STL uploads durable and implement guarded restore tooling
|
||||
### Task 4: Implement guarded restore tooling
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/backup/restore.sh`
|
||||
- Create: `prisma/migrations/20260722000000_mark_retained_manual_files/migration.sql`
|
||||
- Modify: `prisma/schema.prisma`
|
||||
- Modify: `src/app/api/uploads/route.ts`
|
||||
- Modify: `worker/src/manual-upload.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: a Restic snapshot ID, the same repository/password configuration, the backup Compose service, and the live Compose project.
|
||||
- Produces: durable completed manual-upload files in the existing `manual_uploads` volume; restored PostgreSQL data and persistent volumes only after explicit confirmation for live replacement; a non-destructive staging restore by default.
|
||||
- Produces: restored PostgreSQL data and Telegram session volumes only after explicit confirmation for live replacement; a non-destructive staging restore by default.
|
||||
|
||||
- [ ] **Step 1: Make new manual-upload files durable**
|
||||
|
||||
Add nullable `retainedAt DateTime?` to `ManualUploadFile`. Create an additive migration that leaves existing rows `NULL`. Set `retainedAt: new Date()` when `src/app/api/uploads/route.ts` creates each new file record. Remove the worker's final recursive deletion of `/data/uploads/<uploadId>` while retaining split-file cleanup. This makes the existing `manual_uploads` volume the durable STL archive for new uploads.
|
||||
|
||||
Document and report that files already deleted by older worker runs cannot be reconstructed by this feature. The restore verifier must treat `retainedAt IS NULL` rows as legacy warnings and must require every `retainedAt IS NOT NULL` path to exist.
|
||||
|
||||
- [ ] **Step 2: Define restore modes and destructive guard**
|
||||
- [ ] **Step 1: Define restore modes and destructive guard**
|
||||
|
||||
Support these commands:
|
||||
|
||||
@@ -296,11 +288,11 @@ Support these commands:
|
||||
|
||||
Reject `restore-live` unless the exact confirmation flag is present. `list`, `verify`, and `restore-to-staging` must not stop services or modify live volumes.
|
||||
|
||||
- [ ] **Step 3: Implement snapshot verification and staging restore**
|
||||
- [ ] **Step 2: Implement snapshot verification and staging restore**
|
||||
|
||||
Use `restic snapshots`, `restic check`, and `restic restore SNAPSHOT_ID --target STAGING_DIR`. Verify that the restored staging tree contains a non-empty custom-format dump, a manifest, `manual_uploads`, `tdlib-worker`, and `tdlib-bot` before reporting success.
|
||||
Use `restic snapshots`, `restic check`, and `restic restore SNAPSHOT_ID --target STAGING_DIR`. Verify that the restored staging tree contains a non-empty custom-format dump, a manifest, `tdlib-worker`, and `tdlib-bot` before reporting success. Do not add file-path checks, binary checksums, archive/STL-content validation, or channel-forwarding behavior.
|
||||
|
||||
- [ ] **Step 4: Implement live restore sequencing**
|
||||
- [ ] **Step 3: Implement live restore sequencing**
|
||||
|
||||
For `restore-live`:
|
||||
|
||||
@@ -308,17 +300,13 @@ For `restore-live`:
|
||||
2. Stop `app`, `worker`, and `bot`.
|
||||
3. Create a safety PostgreSQL dump of the current database into local staging.
|
||||
4. Restore the selected snapshot to a separate staging directory.
|
||||
5. Replace the three Docker volumes only after the restored tree passes validation, including retained STL files in `manual_uploads`.
|
||||
5. Replace the two Docker session volumes only after the restored tree passes validation.
|
||||
6. Recreate the configured database from the restored custom-format dump using `pg_restore --no-owner`.
|
||||
7. Start services and run the health endpoint plus file-reference verification.
|
||||
7. Start services and run the health endpoint plus worker/bot startup and authentication checks.
|
||||
|
||||
If any step fails, leave the services stopped, print the exact staging path and failure, and do not delete the safety dump.
|
||||
|
||||
- [ ] **Step 5: Add file-reference verification**
|
||||
|
||||
Run a small SQL query against `manual_upload_files` to enumerate retained `filePath` values and check each path inside the restored `/data/uploads` tree. Report legacy rows with `retainedAt IS NULL` as warnings. Return non-zero if any retained database reference is broken.
|
||||
|
||||
- [ ] **Step 6: Validate the restore command without touching live data**
|
||||
- [ ] **Step 4: Validate the restore command without touching live data**
|
||||
|
||||
Run:
|
||||
|
||||
@@ -329,11 +317,11 @@ bash -n scripts/backup/restore.sh
|
||||
|
||||
Expected: syntax passes and `list` prints available snapshot IDs without stopping any service or modifying a volume.
|
||||
|
||||
- [ ] **Step 7: Commit durable STL retention and restore tooling**
|
||||
- [ ] **Step 5: Commit guarded restore tooling**
|
||||
|
||||
```bash
|
||||
git add scripts/backup/restore.sh prisma/schema.prisma prisma/migrations/20260722000000_mark_retained_manual_files/migration.sql src/app/api/uploads/route.ts worker/src/manual-upload.ts
|
||||
git commit -m "feat: retain uploaded STL files for recovery"
|
||||
git add scripts/backup/restore.sh
|
||||
git commit -m "feat: add guarded database and session restore"
|
||||
```
|
||||
|
||||
### Task 5: Document Synology setup, operations, and recovery
|
||||
@@ -343,7 +331,7 @@ git commit -m "feat: retain uploaded STL files for recovery"
|
||||
- Modify: `README.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the exact environment variables, systemd units, and restore commands from Tasks 1–4.
|
||||
- Consumes: the exact environment variables, systemd units, and restore commands from Tasks 1-4.
|
||||
- Produces: operator-facing instructions that do not require reading implementation files.
|
||||
|
||||
- [ ] **Step 1: Document Synology configuration**
|
||||
@@ -375,15 +363,15 @@ sudo systemctl start dragons-stash-backup.service
|
||||
sudo journalctl -u dragons-stash-backup.service -n 100 --no-pager
|
||||
```
|
||||
|
||||
Explain that the first run may be long because it uploads all existing STL and session data; later Restic snapshots deduplicate unchanged data.
|
||||
Explain that the first run captures PostgreSQL and TDLib session state. State clearly that STL binaries stay in Telegram, and that restored PostgreSQL metadata and mappings are what allow normal lookup and delivery after restore.
|
||||
|
||||
- [ ] **Step 4: Document monitoring, retention, and restore**
|
||||
|
||||
Document how to inspect timer status, service failures, Restic snapshots, repository checks, and the four restore modes. Explicitly state that `restore-live` is destructive and requires the confirmation flag.
|
||||
Document how to inspect timer status, service failures, Restic snapshots, repository checks, and the four restore modes. Explicitly state that `restore-live` is destructive and requires the confirmation flag. Explain that channel-forwarding behavior and archive/STL-content integrity validation are future work, not restore checks.
|
||||
|
||||
- [ ] **Step 5: Add a concise production-backup section to the root README**
|
||||
|
||||
Add a link from the deployment/operations section to `scripts/backup/README.md`, state that Docker volumes are not backups, and identify PostgreSQL, STL uploads, and Telegram session volumes as the protected data set.
|
||||
Add a link from the deployment/operations section to `scripts/backup/README.md`, state that Docker volumes are not backups, and identify the PostgreSQL logical dump and Telegram session volumes as the protected data set. State that manual uploads and temporary ZIPs are excluded and STL binaries remain in Telegram.
|
||||
|
||||
- [ ] **Step 6: Commit documentation**
|
||||
|
||||
@@ -398,7 +386,7 @@ git commit -m "docs: document Synology backup and recovery"
|
||||
- Modify: `scripts/backup/README.md` only if verification commands need correction.
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the complete backup stack from Tasks 1–5.
|
||||
- Consumes: the complete backup stack from Tasks 1-5.
|
||||
- Produces: evidence that the acceptance criteria are met, including a disposable restore rehearsal and a failure-path result.
|
||||
|
||||
- [ ] **Step 1: Validate configuration and scripts**
|
||||
@@ -413,9 +401,9 @@ systemd-analyze verify deploy/systemd/dragons-stash-backup.service deploy/system
|
||||
|
||||
Expected: all commands exit `0`.
|
||||
|
||||
- [ ] **Step 2: Seed a recognizable test record and STL file**
|
||||
- [ ] **Step 2: Seed recognizable database metadata**
|
||||
|
||||
Using the existing app/database workflow, create one test upload whose database metadata and file can be identified after restore. Record the expected filename, upload ID, and file checksum before backup.
|
||||
Using the existing app/database workflow, identify a record whose Telegram archive, message, package, and file metadata can be recognized after restore. Record the expected database identifiers before backup. Do not create or retain a local STL binary for this verification.
|
||||
|
||||
- [ ] **Step 3: Run a real backup and inspect the snapshot**
|
||||
|
||||
@@ -428,7 +416,7 @@ docker compose --profile backup run --rm backup snapshots
|
||||
docker compose --profile backup run --rm backup check
|
||||
```
|
||||
|
||||
Expected: the service succeeds, the snapshot exists, the repository check succeeds, all services are running again, and the test file checksum is represented in the backed-up volume.
|
||||
Expected: the service succeeds, the snapshot exists, the repository check succeeds, and all services are running again.
|
||||
|
||||
- [ ] **Step 4: Test the failure path with the NAS unavailable**
|
||||
|
||||
@@ -436,7 +424,7 @@ Temporarily unmount the Synology share in a controlled maintenance session, run
|
||||
|
||||
- [ ] **Step 5: Rehearse a disposable restore**
|
||||
|
||||
Restore the selected snapshot to a disposable Compose project or isolated Docker volumes. Import the database dump, restore the three file trees, start the disposable app/worker/bot services, call `/api/health`, and run the file-reference verification. Confirm the test upload metadata and STL checksum match the pre-backup record.
|
||||
Restore the selected snapshot to a disposable Compose project or isolated Docker volumes. Import the database dump, restore the two TDLib session trees, start the disposable app/worker/bot services, and call `/api/health`. Confirm the recognizable database metadata and Telegram mappings match the pre-backup record. Do not assert the presence, checksum, content, or forwarding behavior of STL binaries.
|
||||
|
||||
- [ ] **Step 6: Verify retention behavior**
|
||||
|
||||
@@ -455,7 +443,8 @@ git commit -m "test: document verified backup and restore procedure"
|
||||
|
||||
## Plan Self-Review
|
||||
|
||||
- **Spec coverage:** PostgreSQL dump, STL volume, both Telegram volumes, Synology NFS, Restic encryption/deduplication, 30-day retention, maintenance window, service restart on failure, guarded restore, file-reference validation, monthly repository/restore checks, and NAS-loss caveat are covered by Tasks 1–6.
|
||||
- **Spec coverage:** PostgreSQL logical dump, both Telegram session volumes, Synology NFS, Restic encryption, 30-day retention, maintenance window, service restart on failure, guarded restore, monthly repository/restore checks, database metadata recovery, and NAS-loss caveat are covered by Tasks 1-6.
|
||||
- **Exclusions:** `manual_uploads` and `tmp_zips` are excluded; local STL retention, restored STL binaries, file-path/checksum validation, channel forwarding, and archive/STL-content integrity checks are not implementation requirements.
|
||||
- **Placeholder scan:** No `TBD`, `TODO`, or unspecified implementation task remains. Environment-dependent values are explicit configuration variables or operator-supplied paths.
|
||||
- **Type/interface consistency:** The Compose service name is consistently `backup`; the container command modes are `backup` and `restore`; the host wrapper owns service lifecycle; the restore script owns destructive confirmation; Restic owns snapshots and pruning.
|
||||
- **Scope check:** The plan contains one operational subsystem with separate backup, restore, scheduling, and documentation units that can each be reviewed and tested independently.
|
||||
|
||||
Reference in New Issue
Block a user