admin/dragonsstash
1
0
Fork 0
mirror of https://github.com/xCyanGrizzly/DragonsStash.git synced 2026-09-21 05:21:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: amend backup design for durable STL retention

Browse Source
This commit is contained in:
admin
2026-07-22 02:13:11 +02:00
parent a2b7e77cc4
commit 238ec17155
2 changed files with 29 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/plans/2026-07-21-database-and-file-backups.md
+22 -12
View File
@@ -264,16 +264,26 @@ git add scripts/backup/run-backup.sh deploy/systemd/dragons-stash-backup.service
git commit -m "feat: schedule nightly off-host backups" git commit -m "feat: schedule nightly off-host backups"
``` ```
### Task 4: Implement guarded restore tooling ### Task 4: Make completed STL uploads durable and implement guarded restore tooling
**Files:** **Files:**
- Create: `scripts/backup/restore.sh` - Create: `scripts/backup/restore.sh`
- Create: `prisma/migrations/20260722000000_mark_retained_manual_files/migration.sql`
- Modify: `prisma/schema.prisma`
- Modify: `src/app/api/uploads/route.ts`
- Modify: `worker/src/manual-upload.ts`
**Interfaces:** **Interfaces:**
- Consumes: a Restic snapshot ID, the same repository/password configuration, the backup Compose service, and the live Compose project. - Consumes: a Restic snapshot ID, the same repository/password configuration, the backup Compose service, and the live Compose project.
- Produces: restored PostgreSQL data and persistent volumes only after explicit confirmation for live replacement; a non-destructive staging restore by default. - Produces: durable completed manual-upload files in the existing `manual_uploads` volume; restored PostgreSQL data and persistent volumes only after explicit confirmation for live replacement; a non-destructive staging restore by default.
- [ ] **Step 1: Define restore modes and destructive guard** - [ ] **Step 1: Make new manual-upload files durable**
Add nullable `retainedAt DateTime?` to `ManualUploadFile`. Create an additive migration that leaves existing rows `NULL`. Set `retainedAt: new Date()` when `src/app/api/uploads/route.ts` creates each new file record. Remove the worker's final recursive deletion of `/data/uploads/<uploadId>` while retaining split-file cleanup. This makes the existing `manual_uploads` volume the durable STL archive for new uploads.
Document and report that files already deleted by older worker runs cannot be reconstructed by this feature. The restore verifier must treat `retainedAt IS NULL` rows as legacy warnings and must require every `retainedAt IS NOT NULL` path to exist.
- [ ] **Step 2: Define restore modes and destructive guard**
Support these commands: Support these commands:
@@ -286,11 +296,11 @@ Support these commands:
Reject `restore-live` unless the exact confirmation flag is present. `list`, `verify`, and `restore-to-staging` must not stop services or modify live volumes. Reject `restore-live` unless the exact confirmation flag is present. `list`, `verify`, and `restore-to-staging` must not stop services or modify live volumes.
- [ ] **Step 2: Implement snapshot verification and staging restore** - [ ] **Step 3: Implement snapshot verification and staging restore**
Use `restic snapshots`, `restic check`, and `restic restore SNAPSHOT_ID --target STAGING_DIR`. Verify that the restored staging tree contains a non-empty custom-format dump, a manifest, `manual_uploads`, `tdlib-worker`, and `tdlib-bot` before reporting success. Use `restic snapshots`, `restic check`, and `restic restore SNAPSHOT_ID --target STAGING_DIR`. Verify that the restored staging tree contains a non-empty custom-format dump, a manifest, `manual_uploads`, `tdlib-worker`, and `tdlib-bot` before reporting success.
- [ ] **Step 3: Implement live restore sequencing** - [ ] **Step 4: Implement live restore sequencing**
For `restore-live`: For `restore-live`:
@@ -298,17 +308,17 @@ For `restore-live`:
2. Stop `app`, `worker`, and `bot`. 2. Stop `app`, `worker`, and `bot`.
3. Create a safety PostgreSQL dump of the current database into local staging. 3. Create a safety PostgreSQL dump of the current database into local staging.
4. Restore the selected snapshot to a separate staging directory. 4. Restore the selected snapshot to a separate staging directory.
5. Replace the three Docker volumes only after the restored tree passes validation. 5. Replace the three Docker volumes only after the restored tree passes validation, including retained STL files in `manual_uploads`.
6. Recreate the configured database from the restored custom-format dump using `pg_restore --no-owner`. 6. Recreate the configured database from the restored custom-format dump using `pg_restore --no-owner`.
7. Start services and run the health endpoint plus file-reference verification. 7. Start services and run the health endpoint plus file-reference verification.
If any step fails, leave the services stopped, print the exact staging path and failure, and do not delete the safety dump. If any step fails, leave the services stopped, print the exact staging path and failure, and do not delete the safety dump.
- [ ] **Step 4: Add file-reference verification** - [ ] **Step 5: Add file-reference verification**
Run a small SQL query against `manual_upload_files` to enumerate `filePath` values and check each path inside the restored `/data/uploads` tree. Report missing paths and return non-zero if any database reference is broken. Run a small SQL query against `manual_upload_files` to enumerate retained `filePath` values and check each path inside the restored `/data/uploads` tree. Report legacy rows with `retainedAt IS NULL` as warnings. Return non-zero if any retained database reference is broken.
- [ ] **Step 5: Validate the restore command without touching live data** - [ ] **Step 6: Validate the restore command without touching live data**
Run: Run:
@@ -319,11 +329,11 @@ bash -n scripts/backup/restore.sh
Expected: syntax passes and `list` prints available snapshot IDs without stopping any service or modifying a volume. Expected: syntax passes and `list` prints available snapshot IDs without stopping any service or modifying a volume.
- [ ] **Step 6: Commit restore tooling** - [ ] **Step 7: Commit durable STL retention and restore tooling**
```bash ```bash
git add scripts/backup/restore.sh git add scripts/backup/restore.sh prisma/schema.prisma prisma/migrations/20260722000000_mark_retained_manual_files/migration.sql src/app/api/uploads/route.ts worker/src/manual-upload.ts
git commit -m "feat: add guarded backup restore workflow" git commit -m "feat: retain uploaded STL files for recovery"
``` ```
### Task 5: Document Synology setup, operations, and recovery ### Task 5: Document Synology setup, operations, and recovery
docs/superpowers/specs/2026-07-21-database-and-file-backups-design.md
+7
View File
@@ -8,11 +8,14 @@
Dragon's Stash currently persists PostgreSQL in a Docker named volume and stores uploaded STL files in the `manual_uploads` volume. A Docker volume protects data from container recreation, but it is not an off-host backup. A host disk failure, accidental deletion, corruption, or ransomware event could destroy both the database and the files it references. Dragon's Stash currently persists PostgreSQL in a Docker named volume and stores uploaded STL files in the `manual_uploads` volume. A Docker volume protects data from container recreation, but it is not an off-host backup. A host disk failure, accidental deletion, corruption, or ransomware event could destroy both the database and the files it references.
The worker currently deletes manual-upload directories after processing. The backup feature will change that behavior for new uploads so the existing `manual_uploads` volume becomes a durable STL archive. Files already deleted before this change cannot be reconstructed by the backup feature.
The Telegram worker and bot also persist authentication/session state in `tdlib_state` and `tdlib_bot_state`. These files are sensitive and losing them requires Telegram re-authentication. The Telegram worker and bot also persist authentication/session state in `tdlib_state` and `tdlib_bot_state`. These files are sensitive and losing them requires Telegram re-authentication.
## Goals ## Goals
- Protect PostgreSQL data and uploaded STL files against loss of the application host. - Protect PostgreSQL data and uploaded STL files against loss of the application host.
- Preserve newly uploaded STL files after worker processing so the backup contains completed uploads, not only in-flight uploads.
- Store backups on a Synology NAS over an authenticated, host-restricted NFS share. - Store backups on a Synology NAS over an authenticated, host-restricted NFS share.
- Create one recoverable snapshot containing related database and file state. - Create one recoverable snapshot containing related database and file state.
- Retain 30 daily recovery points. - Retain 30 daily recovery points.
@@ -25,6 +28,7 @@ The Telegram worker and bot also persist authentication/session state in `tdlib_
- Building an in-app backup-management UI. - Building an in-app backup-management UI.
- Backing up temporary ZIP processing data in `tmp_zips`. - Backing up temporary ZIP processing data in `tmp_zips`.
- Copying the raw `postgres_data` volume as the primary database backup. - Copying the raw `postgres_data` volume as the primary database backup.
- Recovering STL binaries that were deleted by older worker runs before durable retention was enabled.
- Providing protection against loss of the NAS itself. A later Synology Hyper Backup task can replicate this repository to another device or cloud destination. - Providing protection against loss of the NAS itself. A later Synology Hyper Backup task can replicate this repository to another device or cloud destination.
## Selected approach ## Selected approach
@@ -76,6 +80,8 @@ The systemd timer invokes one backup command at the chosen nightly time. The com
The service-stop window ensures that application writes and TDLib session updates do not occur while the corresponding file volumes are being captured. A failed run must never trigger retention pruning. The service-stop window ensures that application writes and TDLib session updates do not occur while the corresponding file volumes are being captured. A failed run must never trigger retention pruning.
New `ManualUploadFile` rows carry a retention timestamp. Restore verification requires retained file paths to exist and reports older rows without a retention timestamp as legacy warnings, because those binaries may already have been deleted before this feature was enabled.
## Restore flow ## Restore flow
The restore tooling and documentation will support this sequence: The restore tooling and documentation will support this sequence:
@@ -115,6 +121,7 @@ The restore process must be safe to rehearse against a disposable Compose projec
- A nightly systemd timer creates a Restic snapshot on the Synology share. - A nightly systemd timer creates a Restic snapshot on the Synology share.
- A snapshot includes PostgreSQL, all uploaded STL files, and both Telegram session volumes. - A snapshot includes PostgreSQL, all uploaded STL files, and both Telegram session volumes.
- At least 30 daily recovery points are retained without duplicating unchanged STL content unnecessarily. - At least 30 daily recovery points are retained without duplicating unchanged STL content unnecessarily.
- Newly uploaded STL files remain in `manual_uploads` after worker processing and are included in subsequent backup snapshots.
- A simulated host-loss restore reconstructs the database and uploaded files in a disposable Compose environment. - A simulated host-loss restore reconstructs the database and uploaded files in a disposable Compose environment.
- A failed backup leaves services running and preserves the last known-good snapshot. - A failed backup leaves services running and preserves the last known-good snapshot.
- The restore procedure is documented well enough for an operator to execute without reading the implementation. - The restore procedure is documented well enough for an operator to execute without reading the implementation.