dragonsstash/scripts/backup/README.md

Production backup and recovery

Dragon's Stash backs up its PostgreSQL database and the worker/bot Telegram (TDLib) session volumes to an encrypted Restic repository on a Synology NFS share. Docker volumes are local runtime storage; they are not, by themselves, backups.

The backup job runs as root because it reads the Restic password file and mounts host paths into the backup container. Keep the Synology share on the private network. Do not expose NFS to the Internet.

Protected data and limits

Each snapshot contains:

• a custom-format PostgreSQL dump;
• a backup manifest with dump checksum, migration metadata, version metadata, and captured source paths;
• the worker TDLib session volume; and
• the bot TDLib session volume.

The manual_uploads and tmp_zips volumes are excluded. Completed STL binaries are not retained as a local recovery set by this backup feature; they remain in Telegram. Recovery preserves the PostgreSQL archive, message, package, file, Telegram channel, and Telegram message ID mappings needed for normal lookup and delivery after restore.

Future Telegram channel-forwarding behavior and archive/STL-content integrity auditing are intentionally outside this backup and restore procedure.

1. Configure the Synology NFS share

On the Synology DSM host:

1. In Control Panel > Shared Folder, create a shared folder named dragonsstash-backups.
2. In Control Panel > File Services > NFS, enable NFS.
3. Edit the new shared folder and add an NFS Permissions entry for the Docker host's fixed private-network IP address. Grant read/write access. Use the least permissive squash and authentication settings that work for the root-run backup job, and do not use a broad subnet or public address.
4. Record the NFS export path shown by DSM, for example /volume1/dragonsstash-backups.

On the Docker host, install the NFS client package for its distribution, create the mountpoint, and mount the export. Replace NAS_IP and the export path with the values for the Synology:

sudo mkdir -p /mnt/dragonsstash-backups
sudo mount -t nfs -o nfsvers=4 NAS_IP:/volume1/dragonsstash-backups /mnt/dragonsstash-backups

For a persistent mount, add an /etc/fstab entry using the DSM export path:

NAS_IP:/volume1/dragonsstash-backups /mnt/dragonsstash-backups nfs nfsvers=4,_netdev,nofail,x-systemd.automount 0 0

Then verify that the host sees an actual writable mount rather than an empty local directory:

mountpoint /mnt/dragonsstash-backups
touch /mnt/dragonsstash-backups/.write-test
rm /mnt/dragonsstash-backups/.write-test

The scheduled backup refuses to run if BACKUP_MOUNT_PATH is not a mountpoint or is not writable.

2. Create secrets and host configuration

Create a root-readable Restic password file. Generate and store a strong, unique repository password by your approved secret-management process; never commit it, place it in the Compose file, or paste it into shell history.

sudo install -d -m 0700 -o root -g root /etc/dragons-stash
sudo install -m 0600 -o root -g root /dev/null /etc/dragons-stash/restic-password
sudoedit /etc/dragons-stash/restic-password

Create a local staging directory. It must be on host-local storage, not the NFS share, and must have enough room for a PostgreSQL dump and restore staging data.

sudo install -d -m 0700 -o root -g root /var/lib/dragons-stash/backup-staging

Create the systemd environment file at /etc/dragons-stash/backup.env and keep it root-only. It supplies both the host wrapper and the profile-gated backup container:

sudo install -m 0600 -o root -g root /dev/null /etc/dragons-stash/backup.env
sudoedit /etc/dragons-stash/backup.env

Set these production values, replacing paths and retention as needed:

BACKUP_MOUNT_PATH=/mnt/dragonsstash-backups
BACKUP_STAGING_PATH=/var/lib/dragons-stash/backup-staging
BACKUP_RESTIC_PASSWORD_FILE=/etc/dragons-stash/restic-password
BACKUP_REPOSITORY=/backup/restic
BACKUP_RETENTION_DAYS=30
BACKUP_APP_VERSION=unknown

BACKUP_REPOSITORY is evaluated inside the backup container, where the NFS share is mounted at /backup; /backup/restic keeps the Restic repository in the Synology share. BACKUP_APP_VERSION is optional metadata. The production Compose environment must also retain its existing database and application secrets; do not add secrets to Git.

3. Initialize the Restic repository once

Before enabling the scheduled job, initialize a new repository explicitly:

docker compose --profile backup run --rm backup init

The backup container preflight refuses to create a repository implicitly. If a scheduled backup reports that the repository is unavailable or uninitialized, verify the NFS mount, BACKUP_REPOSITORY, and password file before running the explicit initialization command for an intended new repository.

4. Install the nightly systemd job

The supplied unit assumes the production Compose checkout is /opt/stacks/DragonsStash. If your deployment lives elsewhere, update the installed service's WorkingDirectory and ExecStart to the corresponding absolute paths before enabling it.

Install and start the units:

sudo install -m 0644 deploy/systemd/dragons-stash-backup.service /etc/systemd/system/
sudo install -m 0644 deploy/systemd/dragons-stash-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now dragons-stash-backup.timer
sudo systemctl start dragons-stash-backup.service
sudo journalctl -u dragons-stash-backup.service -n 100 --no-pager

The timer runs nightly at 03:00 with up to 15 minutes of randomized delay and catches up after downtime. The first run captures PostgreSQL and both TDLib session volumes. Later Restic snapshots deduplicate unchanged session state and database dump content.

5. Monitor and maintain backups

Inspect the next scheduled run and the last service result:

systemctl list-timers dragons-stash-backup.timer
systemctl status dragons-stash-backup.timer
systemctl status dragons-stash-backup.service
sudo journalctl -u dragons-stash-backup.service -n 100 --no-pager
sudo systemctl --failed

List snapshots through the configured backup container:

./scripts/backup/restore.sh list

Check a selected repository snapshot before restoring it:

./scripts/backup/restore.sh verify SNAPSHOT_ID

verify runs restic snapshots SNAPSHOT_ID and restic check. Routine backup runs apply retention with restic forget --keep-daily "$BACKUP_RETENTION_DAYS" --prune; with the example configuration, this retains 30 daily snapshots. Choose BACKUP_RETENTION_DAYS based on storage capacity and the recovery window you need. Watch Synology capacity and investigate any failed timer or service promptly.

At least monthly, perform a full repository read check:

docker compose --profile backup run --rm backup check --read-data

6. Monthly disposable recovery rehearsal

Perform the following procedure at least monthly. It restores one snapshot into a unique, disposable Compose project, so the database and Compose volumes are isolated from production. Never use the production Compose project name, production volume names, or restore-live for this rehearsal. Do not run docker compose down -v without the explicit rehearsal --project-name shown below.

Run these commands from the production Compose checkout as an operator allowed to read /etc/dragons-stash/backup.env. They use a separate application stack; do not expose its published port beyond the host. The worker and bot are deliberately replaced with inert processes, so this validates restored images, database state, and TDLib volume layout without executing Telegram clients or using production Telegram credentials.

set -Eeuo pipefail
set -a
. /etc/dragons-stash/backup.env
set +a

./scripts/backup/restore.sh list
SNAPSHOT_ID=SNAPSHOT_ID_FROM_LIST
REHEARSAL_ID="$(date -u +%Y%m%dT%H%M%SZ)-$$"
REHEARSAL_PROJECT="dragonsstash-rehearsal-$REHEARSAL_ID"
REHEARSAL_DIR="$BACKUP_STAGING_PATH/monthly-rehearsal-$REHEARSAL_ID"
REHEARSAL_ENV="$REHEARSAL_DIR/compose.env"
REHEARSAL_OVERRIDE="$REHEARSAL_DIR/compose.rehearsal.override.yml"
REHEARSAL_APP_PORT=13000  # Choose an unused host-local port.

./scripts/backup/restore.sh verify "$SNAPSHOT_ID"
./scripts/backup/restore.sh restore-to-staging "$SNAPSHOT_ID" "$REHEARSAL_DIR"
RESTORE_ROOT="$(printf '%s\n' "$REHEARSAL_DIR"/staging/backup-*)"
sha256sum --check "$RESTORE_ROOT/manifest/database.dump.sha256"

test -s "$RESTORE_ROOT/database.dump"
test -s "$RESTORE_ROOT/manifest/backup-manifest.json"
test -d "$REHEARSAL_DIR/data/tdlib-worker"
test -d "$REHEARSAL_DIR/data/tdlib-bot"

umask 077
cp .env "$REHEARSAL_ENV"
printf '\nAPP_PORT=%s\nNEXT_PUBLIC_APP_URL=http://localhost:%s\n' \
  "$REHEARSAL_APP_PORT" "$REHEARSAL_APP_PORT" >> "$REHEARSAL_ENV"

cat > "$REHEARSAL_OVERRIDE" <<'EOF'
services:
  worker:
    environment:
      TELEGRAM_API_ID: ""
      TELEGRAM_API_HASH: ""
    entrypoint: ["/bin/sh", "-c"]
    command: ["exec sleep infinity"]
  bot:
    environment:
      BOT_TOKEN: ""
      TELEGRAM_API_ID: ""
      TELEGRAM_API_HASH: ""
    entrypoint: ["/bin/sh", "-c"]
    command: ["exec sleep infinity"]
EOF

Confirm that RESTORE_ROOT names exactly one backup-* directory before continuing. The staging restore has already verified the custom PostgreSQL dump and restored data/tdlib-worker and data/tdlib-bot.

Create the isolated project and volumes, start only its database, then import the dump. compose_rehearsal() always applies the disposable override created above: worker and bot retain their restored images and volumes but have their Telegram credential variables blanked and run only sleep infinity, never Telegram clients. The create command makes project-scoped application volumes without starting app, worker, or bot.

compose_rehearsal() {
  docker compose --project-name "$REHEARSAL_PROJECT" --env-file "$REHEARSAL_ENV" \
    -f docker-compose.yml -f "$REHEARSAL_OVERRIDE" "$@"
}
rehearsal_volume() {
  docker volume ls \
    --filter "label=com.docker.compose.project=$REHEARSAL_PROJECT" \
    --filter "label=com.docker.compose.volume=$1" \
    --format '{{.Name}}'
}

compose_rehearsal --profile full create app worker bot
compose_rehearsal --profile full up -d db
compose_rehearsal exec -T db dropdb --if-exists --force \
  --username "${POSTGRES_USER:-dragons}" "${POSTGRES_DB:-dragonsstash}"
compose_rehearsal exec -T db createdb --username "${POSTGRES_USER:-dragons}" \
  "${POSTGRES_DB:-dragonsstash}"
compose_rehearsal exec -T db pg_restore --no-owner --exit-on-error \
  --username "${POSTGRES_USER:-dragons}" --dbname "${POSTGRES_DB:-dragonsstash}" \
  < "$RESTORE_ROOT/database.dump"

Copy each restored TDLib tree into its matching rehearsal volume. Each target is new and empty; the function rejects an ambiguous volume lookup.

restore_rehearsal_volume() {
  local logical_name="$1"
  local source_dir="$2"
  local volume_name
  volume_name="$(rehearsal_volume "$logical_name")"
  test -n "$volume_name"
  test "$(printf '%s\n' "$volume_name" | wc -l)" -eq 1
  compose_rehearsal --profile backup run --rm --no-deps \
    --entrypoint bash \
    -v "$source_dir:/restore-source:ro" \
    -v "$volume_name:/restore-target" \
    backup -ceu 'cp -a /restore-source/. /restore-target/'
}

restore_rehearsal_volume tdlib_state "$REHEARSAL_DIR/data/tdlib-worker"
restore_rehearsal_volume tdlib_bot_state "$REHEARSAL_DIR/data/tdlib-bot"

Start the disposable app, worker, and bot containers. The app and database perform their normal health checks; the worker and bot remain inert sleep infinity processes, so this does not execute Telegram clients or send messages. Check the health endpoint, then retain the ps and log output as rehearsal evidence.

compose_rehearsal --profile full up -d app worker bot
curl --fail --silent --show-error \
  "http://localhost:$REHEARSAL_APP_PORT/api/health"
compose_rehearsal --profile full ps
compose_rehearsal --profile full logs --tail=100 app worker bot

Optionally spot-check the restored database metadata that lets Dragon's Stash locate Telegram-hosted STL binaries after restore. This checks database metadata and Telegram IDs only; it does not assert local STL file presence, STL checksums, Telegram forwarding behavior, or archive/STL binary integrity.

compose_rehearsal exec -T db psql --no-psqlrc --tuples-only --no-align --quiet \
  --username "${POSTGRES_USER:-dragons}" \
  --dbname "${POSTGRES_DB:-dragonsstash}" \
  --command 'SELECT count(*) FROM "packages" WHERE "destChannelId" IS NOT NULL AND "destMessageId" IS NOT NULL;'

After recording the evidence, destroy only the explicitly named disposable project and its project-labeled volumes. Do not remove REHEARSAL_DIR until the evidence is recorded; then remove only that generated child directory by your approved host cleanup procedure. Production services, volumes, and backup snapshots must remain untouched.

compose_rehearsal --profile full down --volumes --remove-orphans
docker volume ls --filter "label=com.docker.compose.project=$REHEARSAL_PROJECT"
test -z "$(docker volume ls --quiet --filter "label=com.docker.compose.project=$REHEARSAL_PROJECT")"

Record this evidence for every rehearsal; do not include secrets, database dumps, Telegram session contents, or private NAS details.

Disposable restore rehearsal
Date/time (UTC):
Operator:
Snapshot ID:
Snapshot backup date:
Disposable Compose project:
Full restic check --read-data result:
Database dump manifest checksum result:
TDLib worker tree restored:
TDLib bot tree restored:
Health endpoint result (HTTP/body):
docker compose ps result:
app/worker/bot log review result:
Database Telegram metadata/mapping spot-check result:
Cleanup result (project and rehearsal volumes absent):
Notes/caveats:

This document describes the procedure only; it has not been run by this documentation update.

7. Restore modes

Run restore commands from the production Compose checkout after loading the same backup environment used by systemd, for example as root with /etc/dragons-stash/backup.env exported. All restore staging directories must be children of BACKUP_STAGING_PATH.

Mode	Command	Effect
List snapshots	./scripts/backup/restore.sh list	Lists available Restic snapshots. Does not stop services or change volumes.
Verify a snapshot	./scripts/backup/restore.sh verify SNAPSHOT_ID	Confirms the snapshot exists and runs a Restic repository check. Does not change live data.
Restore to staging	./scripts/backup/restore.sh restore-to-staging SNAPSHOT_ID STAGING_DIR	Restores and validates a snapshot in a new or empty child directory of BACKUP_STAGING_PATH. Does not stop services or change volumes.
Replace live data	./scripts/backup/restore.sh restore-live SNAPSHOT_ID --confirm-replace-live-data	Stops application services and replaces the PostgreSQL database plus both TDLib session volumes after validation.

restore-live is destructive. It requires the exact --confirm-replace-live-data flag and should be used only after a successful staging restore has been inspected. It creates a safety database dump and archives of the current TDLib volumes in local staging before replacement. If a live restore fails, it leaves the application services stopped, retains the safety artifacts and staging directory, and attempts rollback after replacement has begun. Review the reported paths and service health before manually starting services.

After a successful live restore, confirm the services are running and inspect their recent logs before declaring recovery complete:

docker compose ps
docker compose logs --tail=100 app worker bot

For a non-destructive recovery rehearsal, choose a snapshot from list and restore it to a fresh staging directory, for example:

./scripts/backup/restore.sh restore-to-staging SNAPSHOT_ID /var/lib/dragons-stash/backup-staging/rehearsal-SNAPSHOT_ID

The restored tree must contain a non-empty PostgreSQL dump, its manifest, worker TDLib state, and bot TDLib state. STL binaries remain in Telegram, and the restored database mappings and Telegram IDs are what recovery preserves for normal lookup and delivery.