admin/dragonsstash
1
0
Fork 0
mirror of https://github.com/xCyanGrizzly/DragonsStash.git synced 2026-09-21 13:31:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
6763af731f03429f894f31cdd4b0abd3a694569a
dragonsstash/scripts/backup
T
History

README.md

Production backup and recovery

Dragon's Stash backs up its PostgreSQL database, durable STL uploads, and the worker and bot Telegram (TDLib) session volumes to an encrypted Restic repository on a Synology NFS share. Docker volumes are local runtime storage; they are not, by themselves, backups.

The backup job runs as root because it reads the Restic password file and mounts host paths into the backup container. Keep the Synology share on the private network. Do not expose NFS to the Internet.

Protected data and limits

Each snapshot contains:

  • a custom-format PostgreSQL dump;
  • the manual_uploads volume at /data/uploads (including newly completed manual STL uploads);
  • the worker TDLib session volume; and
  • the bot TDLib session volume.

New manual-upload STL files are retained in manual_uploads and can therefore be included in future backups. STL files that an older worker run already deleted cannot be recovered by this backup feature. Their legacy database rows are reported as warnings during restore validation; only files marked as retained must be present in a restored snapshot.

1. Configure the Synology NFS share

On the Synology DSM host:

  1. In Control Panel > Shared Folder, create a shared folder named dragonsstash-backups.
  2. In Control Panel > File Services > NFS, enable NFS.
  3. Edit the new shared folder and add an NFS Permissions entry for the Docker host's fixed private-network IP address. Grant read/write access. Use the least permissive squash and authentication settings that work for the root-run backup job, and do not use a broad subnet or public address.
  4. Record the NFS export path shown by DSM (for example, /volume1/dragonsstash-backups).

On the Docker host, install the NFS client package for its distribution, create the mountpoint, and mount the export. Replace NAS_IP and the export path with the values for the Synology:

sudo mkdir -p /mnt/dragonsstash-backups
sudo mount -t nfs -o nfsvers=4 NAS_IP:/volume1/dragonsstash-backups /mnt/dragonsstash-backups

For a persistent mount, add an /etc/fstab entry using the DSM export path:

NAS_IP:/volume1/dragonsstash-backups /mnt/dragonsstash-backups nfs nfsvers=4,_netdev,nofail,x-systemd.automount 0 0

Then verify that the host sees an actual writable mount rather than an empty local directory:

mountpoint /mnt/dragonsstash-backups
touch /mnt/dragonsstash-backups/.write-test
rm /mnt/dragonsstash-backups/.write-test

The scheduled backup refuses to run if BACKUP_MOUNT_PATH is not a mountpoint or is not writable.

2. Create secrets and host configuration

Create a root-readable Restic password file. Generate and store a strong, unique repository password by your approved secret-management process; never commit it, place it in the Compose file, or paste it into shell history.

sudo install -d -m 0700 -o root -g root /etc/dragons-stash
sudo install -m 0600 -o root -g root /dev/null /etc/dragons-stash/restic-password
sudoedit /etc/dragons-stash/restic-password

Create a local staging directory. It must be on host-local storage, not the NFS share, and must have enough room for a PostgreSQL dump and restore staging data.

sudo install -d -m 0700 -o root -g root /var/lib/dragons-stash/backup-staging

Create the systemd environment file at /etc/dragons-stash/backup.env and keep it root-only. It supplies both the host wrapper and the profile-gated backup container:

sudo install -m 0600 -o root -g root /dev/null /etc/dragons-stash/backup.env
sudoedit /etc/dragons-stash/backup.env

Set these production values (replace the example paths and retention period as needed):

BACKUP_MOUNT_PATH=/mnt/dragonsstash-backups
BACKUP_STAGING_PATH=/var/lib/dragons-stash/backup-staging
BACKUP_RESTIC_PASSWORD_FILE=/etc/dragons-stash/restic-password
BACKUP_REPOSITORY=/backup/restic
BACKUP_RETENTION_DAYS=30
BACKUP_APP_VERSION=unknown

BACKUP_REPOSITORY is evaluated inside the backup container, where the NFS share is mounted at /backup; /backup/restic keeps the Restic repository in the Synology share. BACKUP_APP_VERSION is optional metadata. The production Compose environment must also retain its existing database and application secrets; do not add any secrets to Git.

3. Initialize the Restic repository once

Before enabling the scheduled job, initialize a new repository explicitly:

docker compose --profile backup run --rm backup init

The backup container preflight refuses to create a repository implicitly. If a scheduled backup reports that the repository is unavailable or uninitialized, verify the NFS mount, BACKUP_REPOSITORY, and password file before running the explicit initialization command for an intended new repository.

4. Install the nightly systemd job

The supplied unit assumes the production Compose checkout is /opt/stacks/DragonsStash. If your deployment lives elsewhere, update the installed service's WorkingDirectory and ExecStart to the corresponding absolute paths before enabling it.

Install and start the units:

sudo install -m 0644 deploy/systemd/dragons-stash-backup.service /etc/systemd/system/
sudo install -m 0644 deploy/systemd/dragons-stash-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now dragons-stash-backup.timer
sudo systemctl start dragons-stash-backup.service
sudo journalctl -u dragons-stash-backup.service -n 100 --no-pager

The timer runs nightly at 03:00 with up to 15 minutes of randomized delay and catches up after downtime. The first run can take a long time because it uploads all existing STL and session data. Later Restic snapshots deduplicate unchanged data.

5. Monitor and maintain backups

Inspect the next scheduled run and the last service result:

systemctl list-timers dragons-stash-backup.timer
systemctl status dragons-stash-backup.timer
systemctl status dragons-stash-backup.service
sudo journalctl -u dragons-stash-backup.service -n 100 --no-pager
sudo systemctl --failed

List the snapshots through the configured backup container:

./scripts/backup/restore.sh list

Check a selected repository snapshot before restoring it:

./scripts/backup/restore.sh verify SNAPSHOT_ID

verify runs restic snapshots SNAPSHOT_ID and restic check. Routine backup runs apply retention with restic forget --keep-daily "$BACKUP_RETENTION_DAYS" --prune; with the example configuration, this retains 30 daily snapshots. Choose BACKUP_RETENTION_DAYS based on storage capacity and the recovery window you need. Watch Synology capacity and investigate any failed timer or service promptly.

At least monthly, perform a full repository read check:

docker compose --profile backup run --rm backup check --read-data

Monthly disposable recovery rehearsal

Perform the following procedure at least monthly. It restores one snapshot into a unique, disposable Compose project, so the database and all Compose volumes are isolated from production. Never use the production Compose project name, production volume names, or restore-live for this rehearsal. In particular, do not run docker compose down -v without the explicit rehearsal --project-name shown below.

Run these commands from the production Compose checkout as an operator allowed to read /etc/dragons-stash/backup.env. They use production secrets only to start a separate application stack; do not expose its published port beyond the host. First select a snapshot and set the expected values for a known retained STL that was recorded when the backup was made. EXPECTED_FILE_PATH must be the database value under /data/uploads, and EXPECTED_FILE_SIZE is bytes.

set -Eeuo pipefail
set -a
. /etc/dragons-stash/backup.env
set +a

./scripts/backup/restore.sh list
SNAPSHOT_ID=SNAPSHOT_ID_FROM_LIST
REHEARSAL_ID="$(date -u +%Y%m%dT%H%M%SZ)-$$"
REHEARSAL_PROJECT="dragonsstash-rehearsal-$REHEARSAL_ID"
REHEARSAL_DIR="$BACKUP_STAGING_PATH/monthly-rehearsal-$REHEARSAL_ID"
REHEARSAL_ENV="$REHEARSAL_DIR/compose.env"
REHEARSAL_APP_PORT=13000  # Choose an unused host-local port.

EXPECTED_UPLOAD_ID=RECORDED_UPLOAD_ID
EXPECTED_UPLOAD_STATUS=COMPLETED
EXPECTED_FILE_NAME=RECORDED_FILENAME.stl
EXPECTED_FILE_PATH=/data/uploads/RECORDED_RELATIVE_PATH.stl
EXPECTED_FILE_SIZE=RECORDED_SIZE_IN_BYTES
EXPECTED_SHA256=RECORDED_SHA256

./scripts/backup/restore.sh verify "$SNAPSHOT_ID"
./scripts/backup/restore.sh restore-to-staging "$SNAPSHOT_ID" "$REHEARSAL_DIR"
RESTORE_ROOT="$(printf '%s\n' "$REHEARSAL_DIR"/staging/backup-*)"
sha256sum --check "$RESTORE_ROOT/manifest/database.dump.sha256"

umask 077
cp .env "$REHEARSAL_ENV"
printf '\nAPP_PORT=%s\nNEXT_PUBLIC_APP_URL=http://localhost:%s\n' \
  "$REHEARSAL_APP_PORT" "$REHEARSAL_APP_PORT" >> "$REHEARSAL_ENV"

Confirm that RESTORE_ROOT names exactly one backup-* directory before continuing. The staging restore has already verified the custom PostgreSQL dump and restored data/uploads, data/tdlib-worker, and data/tdlib-bot.

Create the isolated project and volumes, start only its database, then import the dump. The create command makes the project-scoped application volumes without starting app, worker, or bot.

compose_rehearsal() {
  docker compose --project-name "$REHEARSAL_PROJECT" --env-file "$REHEARSAL_ENV" "$@"
}
rehearsal_volume() {
  docker volume ls \
    --filter "label=com.docker.compose.project=$REHEARSAL_PROJECT" \
    --filter "label=com.docker.compose.volume=$1" \
    --format '{{.Name}}'
}

compose_rehearsal --profile full create app worker bot
compose_rehearsal --profile full up -d db
compose_rehearsal exec -T db dropdb --if-exists --force \
  --username "${POSTGRES_USER:-dragons}" "${POSTGRES_DB:-dragonsstash}"
compose_rehearsal exec -T db createdb --username "${POSTGRES_USER:-dragons}" \
  "${POSTGRES_DB:-dragonsstash}"
compose_rehearsal exec -T db pg_restore --no-owner --exit-on-error \
  --username "${POSTGRES_USER:-dragons}" --dbname "${POSTGRES_DB:-dragonsstash}" \
  < "$RESTORE_ROOT/database.dump"

Copy each restored file tree into its matching rehearsal volume. Each target is new and empty; the function rejects an ambiguous volume lookup.

restore_rehearsal_volume() {
  local logical_name="$1"
  local source_dir="$2"
  local volume_name
  volume_name="$(rehearsal_volume "$logical_name")"
  test -n "$volume_name"
  test "$(printf '%s\n' "$volume_name" | wc -l)" -eq 1
  compose_rehearsal --profile backup run --rm --no-deps \
    --entrypoint bash \
    -v "$source_dir:/restore-source:ro" \
    -v "$volume_name:/restore-target" \
    backup -ceu 'cp -a /restore-source/. /restore-target/'
}

restore_rehearsal_volume manual_uploads "$REHEARSAL_DIR/data/uploads"
restore_rehearsal_volume tdlib_state "$REHEARSAL_DIR/data/tdlib-worker"
restore_rehearsal_volume tdlib_bot_state "$REHEARSAL_DIR/data/tdlib-bot"

Start the disposable app, worker, and bot services. Check the health endpoint, then retain the ps and log output as rehearsal evidence. Do not use the bot to send messages during this check.

compose_rehearsal --profile full up -d app worker bot
curl --fail --silent --show-error \
  "http://localhost:$REHEARSAL_APP_PORT/api/health"
compose_rehearsal --profile full ps
compose_rehearsal --profile full logs --tail=100 app worker bot

Validate that every retained database file reference has a restored file. Rows whose retainedAt is NULL are legacy references and are warnings, not failures. Then compare the recorded STL checksum and metadata with the disposable database and volume; both commands must succeed.

compose_rehearsal exec -T db psql --no-psqlrc --tuples-only --no-align --quiet \
  --field-separator=$'\t' --username "${POSTGRES_USER:-dragons}" \
  --dbname "${POSTGRES_DB:-dragonsstash}" \
  --command "SELECT CASE WHEN \"retainedAt\" IS NULL THEN 'legacy' ELSE 'retained' END, \"filePath\" FROM \"manual_upload_files\" ORDER BY 2" |
  compose_rehearsal --profile backup run --rm --no-deps -T --entrypoint bash backup -ceu '
    missing=0
    while IFS="$(printf "\\t")" read -r retention file_path; do
      if [[ "$retention" == legacy ]]; then
        printf "Warning: legacy reference is not required: %s\\n" "$file_path" >&2
      elif [[ "$retention" != retained || "$file_path" != /data/uploads/* || ! -f "/data/uploads/${file_path#/data/uploads/}" ]]; then
        printf "Missing or invalid retained upload: %s\\n" "$file_path" >&2
        missing=1
      fi
    done
    exit "$missing"
  '

actual_sha256="$(compose_rehearsal --profile backup run --rm --no-deps \
  --entrypoint sha256sum backup "$EXPECTED_FILE_PATH" | awk '{print $1}')"
test "$actual_sha256" = "$EXPECTED_SHA256"

metadata_rows="$(compose_rehearsal exec -T db psql --no-psqlrc --tuples-only \
  --no-align --quiet --username "${POSTGRES_USER:-dragons}" \
  --dbname "${POSTGRES_DB:-dragonsstash}" \
  --set="upload_id=$EXPECTED_UPLOAD_ID" --set="upload_status=$EXPECTED_UPLOAD_STATUS" \
  --set="file_name=$EXPECTED_FILE_NAME" --set="file_path=$EXPECTED_FILE_PATH" \
  --set="file_size=$EXPECTED_FILE_SIZE" --command '
    SELECT count(*) FROM "manual_uploads" u
    JOIN "manual_upload_files" f ON f."uploadId" = u.id
    WHERE u.id = :'upload_id'
      AND u.status::text = :'upload_status'
      AND f."fileName" = :'file_name'
      AND f."filePath" = :'file_path'
      AND f."fileSize" = :'file_size'::bigint
      AND f."retainedAt" IS NOT NULL;
  ')"
test "$metadata_rows" = 1

After recording the evidence, destroy only the explicitly named disposable project and its project-labeled volumes. Do not remove REHEARSAL_DIR until the evidence is recorded; then remove only that generated child directory by your approved host cleanup procedure. Production services, volumes, and backup snapshots must remain untouched.

compose_rehearsal --profile full down --volumes --remove-orphans
docker volume ls --filter "label=com.docker.compose.project=$REHEARSAL_PROJECT"
test -z "$(docker volume ls --quiet --filter "label=com.docker.compose.project=$REHEARSAL_PROJECT")"

Record this evidence for every rehearsal; do not include secrets, database dumps, Telegram session contents, or private NAS details.

Disposable restore rehearsal
Date/time (UTC):
Operator:
Snapshot ID:
Snapshot backup date:
Disposable Compose project:
Health endpoint result (HTTP/body):
docker compose ps result:
app/worker/bot log review result:
Retained-file reference validation result:
Known retained STL upload ID/path:
Expected SHA-256 / restored SHA-256:
Expected metadata (status, filename, size, retainedAt) / restored result:
Database dump manifest checksum result:
Cleanup result (project and rehearsal volumes absent):
Notes/caveats:

This document describes the procedure only; it has not been run by this documentation update.

6. Restore modes

Run restore commands from the production Compose checkout after loading the same backup environment used by systemd (for example, as root with /etc/dragons-stash/backup.env exported). All restore staging directories must be children of BACKUP_STAGING_PATH.

Mode Command Effect
List snapshots ./scripts/backup/restore.sh list Lists available Restic snapshots. Does not stop services or change volumes.
Verify a snapshot ./scripts/backup/restore.sh verify SNAPSHOT_ID Confirms the snapshot exists and runs a Restic repository check. Does not change live data.
Restore to staging ./scripts/backup/restore.sh restore-to-staging SNAPSHOT_ID STAGING_DIR Restores and validates a snapshot in a new or empty child directory of BACKUP_STAGING_PATH. Does not stop services or change volumes.
Replace live data ./scripts/backup/restore.sh restore-live SNAPSHOT_ID --confirm-replace-live-data Stops application services and replaces the PostgreSQL database plus all protected volumes after validation.

restore-live is destructive. It requires the exact --confirm-replace-live-data flag and should be used only after a successful staging restore has been inspected. It creates a safety database dump and archives of the current protected volumes in local staging before replacement. If a live restore fails, it leaves the application services stopped, retains the safety artifacts and staging directory, and attempts rollback after replacement has begun. Review the reported paths and service health before manually starting services.

After a successful live restore, confirm the services are running and inspect their recent logs before declaring recovery complete:

docker compose ps
docker compose logs --tail=100 app worker bot

For a non-destructive recovery rehearsal, choose a snapshot from list and restore it to a fresh staging directory, for example:

./scripts/backup/restore.sh restore-to-staging SNAPSHOT_ID /var/lib/dragons-stash/backup-staging/rehearsal-SNAPSHOT_ID

The restored tree must contain a non-empty PostgreSQL dump, its manifest, uploads, worker TDLib state, and bot TDLib state. Live restore additionally checks every retained manual-upload file reference against the staged uploads before replacing any live volume.

Reference in New Issue View Git Blame Copy Permalink