dragonsstash/docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md

# Forward-priority ingestion — design

**Date:** 2026-07-30
**Status:** Approved (design), pending spec review → implementation plan

## Problem

The worker ingests every archive the same way regardless of whether it needs to: download the
full file from the source channel, then re-upload the full file to the destination (archive)
channel. That download+reupload round-trip was originally necessary because some source channels
have "restrict saving content" (protected content) enabled, which blocks Telegram-native
forwarding — for those channels there is no alternative to moving the bytes through the worker.

But most source channels do NOT restrict forwarding. For those, the round-trip is pure waste:
Telegram can copy the message from source chat to destination chat server-side, with no bytes
ever passing through the worker. The worker still needs to end up with the same outcome it has
today — a destination-channel copy, a dedup-safe identity, and a full inner-file listing — just
without paying for a download and re-upload to get there.

Separately, `feat/ranged-archive-listing` (merged to master ahead of this feature) already built
exactly the missing piece: reading a ZIP/RAR/7z archive's inner-file listing via small ranged
reads against the file wherever it currently lives (source channel, destination channel — doesn't
matter), with no full download. It was built for backfilling listings onto already-deduped
placeholder packages. This feature generalizes that same capability to fresh ingestion, and pairs
it with a new native-forward upload path.

## Goals

- For channels that allow forwarding: skip download and re-upload entirely for new archives. Use
  Telegram-native forwarding from source chat to destination chat, and the existing ranged-listing
  readers to index inner files, with no full download in the common case.
- For channels that block forwarding (or when forwarding isn't yet known): keep today's
  download+reupload pipeline exactly as-is.
- Every ingested package — regardless of path — ends up with the same outcome as today: a
  `Package` row with a valid dedup identity, `destMessageId`/`destMessageIds`, creator, tags, and a
  full inner-file listing (`PackageFile` rows). Indexing completeness must not regress.
- If the cheap ranged listing fails for a specific archive (bad/unsupported header, CLI error,
  etc.) in an otherwise-forwarding-eligible channel, fall back to today's full download+reupload
  pipeline for that one archive — never forward with an empty or partial listing.

## Non-goals

- No ranged single-entry preview extraction. Forward-path packages still get a preview when a
  channel photo message matches (cheap, unrelated to archive bytes); when there's no matching
  photo, forward-path packages simply have no preview, same as any package where preview
  extraction fails today. In-archive preview extraction (unzip/unrar/7z against a local file) stays
  as a download-path-only feature. May be revisited as a follow-up if it turns out to matter.
- No reprocessing of already-ingested packages. This only changes behavior for newly-scanned
  archives going forward.
- No change to the bot's user-delivery leg (`bot/src/tdlib/client.ts` `copyMessageToUser`) — it
  already sends via `inputFileRemote` with no download, and is unaffected by this feature.
- No change to `config.maxZipSizeMB` or the multipart byte-level split/repack logic. The existing
  size guard runs before either path is chosen, so nothing above the cap reaches the forward path's
  fallback-to-download step either. Splitting simply never engages on the forward path — a
  forwarded message is already within whatever size Telegram accepted when it was first uploaded.

## Approaches considered

**A — Branch inside the existing pipeline (chosen).** Add one fork point in
`processOneArchiveSet`, immediately after the existing pre-download dedup checks: if the channel
allows forwarding, attempt the ranged-listing + forward path; on any failure, fall through into
today's download-based code for that one archive, unchanged. Smallest diff; reuses the existing
dedup/retry/watermark machinery as-is; matches the file's existing forum-vs-non-forum branching
style.

**B — Separate pipeline per channel.** Decide once per channel and route the whole channel through
either a "forward module" or the existing "download module." Cleaner separation on paper, but
duplicates the SkippedPackage/stall/watermark bookkeeping that currently lives once in
`processArchiveSets`/`processOneArchiveSet` — higher regression risk in a large orchestration file
with no tests at that level. Rejected.

**C — Strategy-object refactor.** Extract an `IngestStrategy` interface (`download` / `forward`)
and slim `processOneArchiveSet` to delegate to it. The more "proper" abstraction, but it's a
structural refactor of already-battle-tested code that doesn't need it for this feature to work.
Rejected — can revisit later if a third strategy ever appears.

## Sequencing

`feat/ranged-archive-listing` merges to master first, as-is (it's complete and serves a different
purpose already). This feature is built on a fresh branch off master afterward.

## Components

### 1. `TelegramChannel.allowsForwarding` (new column, new migration)

`Boolean?` — nullable, `null` means "not yet checked". Refreshed from TDLib's chat
protected-content flag (exact field name to be confirmed against the pinned `tdl`/TDLib version
via docs lookup during implementation — expected to be `chat.has_protected_content`) at the same
point the worker already calls `getChat` per channel per cycle, mirroring the existing
`isForum`/`setChannelForum` read-and-persist pattern precisely. `null` or `false` both route to the
download path — a channel never uses the forward path on unverified permission.

### 2. Shared ranged-listing dispatcher

`readScannedListingRanged` (plus `RangedPart`, `tdlibRangeReader`, and the format-specific
ZIP/RAR/7z readers) currently live inside `provenance-backfill.ts`. Promote the dispatcher (and
whatever it depends on) into a shared module (e.g. `worker/src/archive/ranged/dispatch.ts`) so
`worker.ts` can call the same no-download listing logic for fresh ingestion without a circular
import. `provenance-backfill.ts` switches to importing from the new shared location; behavior
unchanged for the existing backfill path.

### 3. `forwardArchiveToChannel` (new, `worker/src/upload/forward.ts`)

Mirrors `uploadToChannel`'s shape and return type (`{ messageId, messageIds }`). Uses TDLib
`forwardMessages` to copy all parts of an archive set from the source chat to the destination chat
in one batch call (message IDs in original order), wrapped in the same flood-wait/retry handling
style as `uploadToChannel`. Followed by the same destination read-back verification style as
today's post-upload check (`getMessage` on each new destination message ID, confirm a document is
present).

### 4. Dedup identity for forward-path packages

`Package.contentHash` stays a required unique string, but forward-path packages can't hash real
bytes. Derivation order:
1. If the ranged listing's CRC32s are complete (ZIP/RAR today) — hash the sorted CRC32 list into a
   synthetic `fingerprint:<hash>` value, reusing `archive/fingerprint.ts`'s existing
   `crcFingerprint`.
2. Otherwise (7z, or any incomplete-CRC case) — synthesize `forward:<remoteUniqueId>`, following
   the existing `rebuild:`-prefixed placeholder-hash precedent in `rebuild.ts`.

Additionally, extend repost detection: before committing to the forward path, compare the new
listing's CRC fingerprint (via the existing `compareFingerprints`/`fingerprintsMatch` logic already
used in `provenance-backfill.ts`'s ambiguous-candidate disambiguation) against recent Packages
sharing the same file name + size. A fingerprint match is treated as a duplicate and skipped, same
as today's `findRepostedPackage` handling — this is what lets a forwarded copy and a previously
fully-downloaded copy of the same archive still dedupe against each other, despite never sharing a
byte-hash-derived `contentHash`.

### 5. Fork point in `processOneArchiveSet`

All existing pre-download checks run first, completely unchanged, in the same order:
`remote.unique_id` match → `packageExistsBySourceMessage` → `findRepostedPackage` (name+size) →
cross-channel provenance backfill → size guard (`maxZipSizeMB`).

Then:

```
if channel.allowsForwarding === true:
    entries = readScannedListingRanged(archiveType, client, scannedParts)
    if entries is not null:
        contentHash = deriveForwardContentHash(entries, remoteUniqueId)
        if fingerprintRepostCheck(entries, fileName, fileSize) finds a match:
            → treat as duplicate, skip (same bookkeeping as today's dup path)
        destResult = forwardArchiveToChannel(client, sourceChatId, partMessageIds, destChatId)
        creator, tags ← derived from entries/filename/channel/topic, same as today
        preview ← channel-photo match only (no in-archive extraction)
        createPackageStub(...) + updatePackageWithMetadata(...), same as today
        counters.zipsForwarded++
        → done
    else:
        → fall through into the existing download/hash/split/upload flow below, unchanged
        (log the fallback for observability)
else:
    → existing download/hash/split/upload flow, completely unchanged
```

### 6. Observability

New `zipsForwarded` counter alongside the existing `zipsFound`/`zipsDuplicate`/`zipsIngested`/
`zipsBackfilled` counters, surfaced the same way (run activity, ingestion run summary). A WARN-level
log line when a forwarding-eligible archive falls back to download (mirrors the existing
`confidence: "ranged" | "full-download-fallback"` logging convention from the ranged-listing
backfill work), so the fallback rate is visible without digging through debug logs.

## Data flow

```
scan → pre-download dedup + size guard (unchanged)
     → channel.allowsForwarding?
         true  → ranged listing
                   ok   → fingerprint dedup check → forward → stub + entries + tags (no in-archive preview) → done
                   null → [fall through] existing download pipeline
         false/unknown → existing download pipeline (unchanged)
```

## Error handling

- `forwardMessages` failure (permission revoked mid-run, rate limit, transient Telegram error) —
  same `SkippedPackage`/`SystemNotification` bookkeeping as today's upload failures. Extend
  `inferSkipReason` to recognize forward-specific error text the same way it already recognizes
  upload errors.
- Fingerprint-repost check finds multiple ambiguous same-name/size candidates that can't be
  uniquely disambiguated — same `INTEGRITY_AUDIT` notification pattern already used in
  `provenance-backfill.ts`: don't guess, surface for manual triage.
- `allowsForwarding` unknown (channel just linked, not yet scanned by the refresh point) — treated
  as `false`; the download path runs. No channel uses an unverified forwarding permission.
- Ranged listing throwing instead of returning `null` — treated identically to returning `null`
  (fall through to download), consistent with how the existing ranged readers already treat
  internal errors (they catch and return `null` themselves).

## Testing

- Unit tests (vitest, alongside the existing `archive/*.test.ts` and `archive/ranged/*.test.ts`
  files): the dedup-identity derivation function (fingerprint-hash vs remoteUniqueId-fallback
  branches), the extended fingerprint-based repost check, and `forwardArchiveToChannel`'s
  request-building logic against a mocked TDLib client — same style as the existing ranged-reader
  tests (pure logic, no live TDLib).
- Live verification (manual — matches this repo's existing convention that the large
  `worker.ts`/`worker.py`-equivalent orchestration function has no automated test coverage and is
  verified live post-deploy): one forwarding-enabled test channel and one protected-content test
  channel. Confirm forward-path packages land with correct entries/tags/dedup identity and
  `destMessageIds`; confirm the protected channel still goes through the unchanged full pipeline;
  confirm a deliberately-unparseable archive in a forwarding-enabled channel correctly falls back
  to download+reupload and still ends up fully indexed.

## Rollout

Local build + deploy, following the same recipe as the ranged-archive-listing work: build
`worker/Dockerfile` locally, recreate the `dragonsstash-worker` container from the local image (no
`pull`, no GitHub push required). New DB migration for `TelegramChannel.allowsForwarding`. No
changes required to the bot or app services.