diff --git a/.claude/settings.local.json b/.claude/settings.local.json index 16595d1..2e582aa 100644 --- a/.claude/settings.local.json +++ b/.claude/settings.local.json @@ -89,7 +89,14 @@ "Bash(wait:*)", "WebSearch", "Bash(SKILL_CREATOR_PATH=\"C:\\\\Users\\\\A00963355\\\\.claude\\\\plugins\\\\cache\\\\claude-plugins-official\\\\skill-creator\\\\d5c15b861cd2\\\\skills\\\\skill-creator\" && WORKSPACE=\"C:\\\\Users\\\\A00963355\\\\OneDrive - Amaris Zorggroep\\\\Documents\\\\VScodeProjects\\\\DragonsStash\\\\.claude\\\\skills\\\\tdlib-telegram-workspace\\\\iteration-1\" && python \"$SKILL_CREATOR_PATH/eval-viewer/generate_review.py\" \"$WORKSPACE\" --skill-name \"tdlib-telegram\" --benchmark \"$WORKSPACE/benchmark.json\" --static \"$WORKSPACE/review.html\" 2>&1)", - "Bash(start:*)" + "Bash(start:*)", + "Bash(npm run:*)", + "Bash(DATABASE_URL=\"postgresql://dragons:stash@localhost:5432/dragonsstash\" npx prisma migrate dev --name add-skipped-packages)", + "Bash(git checkout:*)", + "Bash(DATABASE_URL=\"postgresql://dragons:stash@localhost:5432/dragonsstash?schema=public\" npx prisma migrate dev --name add_package_groups 2>&1)", + "Bash(psql:*)", + "Bash(git log:*)", + "Bash(git merge:*)" ] } } diff --git a/docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md b/docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md new file mode 100644 index 0000000..f3d1154 --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md @@ -0,0 +1,211 @@ +# Forward-priority ingestion — design + +**Date:** 2026-07-30 +**Status:** Approved (design), pending spec review → implementation plan + +## Problem + +The worker ingests every archive the same way regardless of whether it needs to: download the +full file from the source channel, then re-upload the full file to the destination (archive) +channel. That download+reupload round-trip was originally necessary because some source channels +have "restrict saving content" (protected content) enabled, which blocks Telegram-native +forwarding — for those channels there is no alternative to moving the bytes through the worker. + +But most source channels do NOT restrict forwarding. For those, the round-trip is pure waste: +Telegram can copy the message from source chat to destination chat server-side, with no bytes +ever passing through the worker. The worker still needs to end up with the same outcome it has +today — a destination-channel copy, a dedup-safe identity, and a full inner-file listing — just +without paying for a download and re-upload to get there. + +Separately, `feat/ranged-archive-listing` (merged to master ahead of this feature) already built +exactly the missing piece: reading a ZIP/RAR/7z archive's inner-file listing via small ranged +reads against the file wherever it currently lives (source channel, destination channel — doesn't +matter), with no full download. It was built for backfilling listings onto already-deduped +placeholder packages. This feature generalizes that same capability to fresh ingestion, and pairs +it with a new native-forward upload path. + +## Goals + +- For channels that allow forwarding: skip download and re-upload entirely for new archives. Use + Telegram-native forwarding from source chat to destination chat, and the existing ranged-listing + readers to index inner files, with no full download in the common case. +- For channels that block forwarding (or when forwarding isn't yet known): keep today's + download+reupload pipeline exactly as-is. +- Every ingested package — regardless of path — ends up with the same outcome as today: a + `Package` row with a valid dedup identity, `destMessageId`/`destMessageIds`, creator, tags, and a + full inner-file listing (`PackageFile` rows). Indexing completeness must not regress. +- If the cheap ranged listing fails for a specific archive (bad/unsupported header, CLI error, + etc.) in an otherwise-forwarding-eligible channel, fall back to today's full download+reupload + pipeline for that one archive — never forward with an empty or partial listing. + +## Non-goals + +- No ranged single-entry preview extraction. Forward-path packages still get a preview when a + channel photo message matches (cheap, unrelated to archive bytes); when there's no matching + photo, forward-path packages simply have no preview, same as any package where preview + extraction fails today. In-archive preview extraction (unzip/unrar/7z against a local file) stays + as a download-path-only feature. May be revisited as a follow-up if it turns out to matter. +- No reprocessing of already-ingested packages. This only changes behavior for newly-scanned + archives going forward. +- No change to the bot's user-delivery leg (`bot/src/tdlib/client.ts` `copyMessageToUser`) — it + already sends via `inputFileRemote` with no download, and is unaffected by this feature. +- No change to `config.maxZipSizeMB` or the multipart byte-level split/repack logic. The existing + size guard runs before either path is chosen, so nothing above the cap reaches the forward path's + fallback-to-download step either. Splitting simply never engages on the forward path — a + forwarded message is already within whatever size Telegram accepted when it was first uploaded. + +## Approaches considered + +**A — Branch inside the existing pipeline (chosen).** Add one fork point in +`processOneArchiveSet`, immediately after the existing pre-download dedup checks: if the channel +allows forwarding, attempt the ranged-listing + forward path; on any failure, fall through into +today's download-based code for that one archive, unchanged. Smallest diff; reuses the existing +dedup/retry/watermark machinery as-is; matches the file's existing forum-vs-non-forum branching +style. + +**B — Separate pipeline per channel.** Decide once per channel and route the whole channel through +either a "forward module" or the existing "download module." Cleaner separation on paper, but +duplicates the SkippedPackage/stall/watermark bookkeeping that currently lives once in +`processArchiveSets`/`processOneArchiveSet` — higher regression risk in a large orchestration file +with no tests at that level. Rejected. + +**C — Strategy-object refactor.** Extract an `IngestStrategy` interface (`download` / `forward`) +and slim `processOneArchiveSet` to delegate to it. The more "proper" abstraction, but it's a +structural refactor of already-battle-tested code that doesn't need it for this feature to work. +Rejected — can revisit later if a third strategy ever appears. + +## Sequencing + +`feat/ranged-archive-listing` merges to master first, as-is (it's complete and serves a different +purpose already). This feature is built on a fresh branch off master afterward. + +## Components + +### 1. `TelegramChannel.allowsForwarding` (new column, new migration) + +`Boolean?` — nullable, `null` means "not yet checked". Refreshed from TDLib's chat +protected-content flag (exact field name to be confirmed against the pinned `tdl`/TDLib version +via docs lookup during implementation — expected to be `chat.has_protected_content`) at the same +point the worker already calls `getChat` per channel per cycle, mirroring the existing +`isForum`/`setChannelForum` read-and-persist pattern precisely. `null` or `false` both route to the +download path — a channel never uses the forward path on unverified permission. + +### 2. Shared ranged-listing dispatcher + +`readScannedListingRanged` (plus `RangedPart`, `tdlibRangeReader`, and the format-specific +ZIP/RAR/7z readers) currently live inside `provenance-backfill.ts`. Promote the dispatcher (and +whatever it depends on) into a shared module (e.g. `worker/src/archive/ranged/dispatch.ts`) so +`worker.ts` can call the same no-download listing logic for fresh ingestion without a circular +import. `provenance-backfill.ts` switches to importing from the new shared location; behavior +unchanged for the existing backfill path. + +### 3. `forwardArchiveToChannel` (new, `worker/src/upload/forward.ts`) + +Mirrors `uploadToChannel`'s shape and return type (`{ messageId, messageIds }`). Uses TDLib +`forwardMessages` to copy all parts of an archive set from the source chat to the destination chat +in one batch call (message IDs in original order), wrapped in the same flood-wait/retry handling +style as `uploadToChannel`. Followed by the same destination read-back verification style as +today's post-upload check (`getMessage` on each new destination message ID, confirm a document is +present). + +### 4. Dedup identity for forward-path packages + +`Package.contentHash` stays a required unique string, but forward-path packages can't hash real +bytes. Derivation order: +1. If the ranged listing's CRC32s are complete (ZIP/RAR today) — hash the sorted CRC32 list into a + synthetic `fingerprint:` value, reusing `archive/fingerprint.ts`'s existing + `crcFingerprint`. +2. Otherwise (7z, or any incomplete-CRC case) — synthesize `forward:`, following + the existing `rebuild:`-prefixed placeholder-hash precedent in `rebuild.ts`. + +Additionally, extend repost detection: before committing to the forward path, compare the new +listing's CRC fingerprint (via the existing `compareFingerprints`/`fingerprintsMatch` logic already +used in `provenance-backfill.ts`'s ambiguous-candidate disambiguation) against recent Packages +sharing the same file name + size. A fingerprint match is treated as a duplicate and skipped, same +as today's `findRepostedPackage` handling — this is what lets a forwarded copy and a previously +fully-downloaded copy of the same archive still dedupe against each other, despite never sharing a +byte-hash-derived `contentHash`. + +### 5. Fork point in `processOneArchiveSet` + +All existing pre-download checks run first, completely unchanged, in the same order: +`remote.unique_id` match → `packageExistsBySourceMessage` → `findRepostedPackage` (name+size) → +cross-channel provenance backfill → size guard (`maxZipSizeMB`). + +Then: + +``` +if channel.allowsForwarding === true: + entries = readScannedListingRanged(archiveType, client, scannedParts) + if entries is not null: + contentHash = deriveForwardContentHash(entries, remoteUniqueId) + if fingerprintRepostCheck(entries, fileName, fileSize) finds a match: + → treat as duplicate, skip (same bookkeeping as today's dup path) + destResult = forwardArchiveToChannel(client, sourceChatId, partMessageIds, destChatId) + creator, tags ← derived from entries/filename/channel/topic, same as today + preview ← channel-photo match only (no in-archive extraction) + createPackageStub(...) + updatePackageWithMetadata(...), same as today + counters.zipsForwarded++ + → done + else: + → fall through into the existing download/hash/split/upload flow below, unchanged + (log the fallback for observability) +else: + → existing download/hash/split/upload flow, completely unchanged +``` + +### 6. Observability + +New `zipsForwarded` counter alongside the existing `zipsFound`/`zipsDuplicate`/`zipsIngested`/ +`zipsBackfilled` counters, surfaced the same way (run activity, ingestion run summary). A WARN-level +log line when a forwarding-eligible archive falls back to download (mirrors the existing +`confidence: "ranged" | "full-download-fallback"` logging convention from the ranged-listing +backfill work), so the fallback rate is visible without digging through debug logs. + +## Data flow + +``` +scan → pre-download dedup + size guard (unchanged) + → channel.allowsForwarding? + true → ranged listing + ok → fingerprint dedup check → forward → stub + entries + tags (no in-archive preview) → done + null → [fall through] existing download pipeline + false/unknown → existing download pipeline (unchanged) +``` + +## Error handling + +- `forwardMessages` failure (permission revoked mid-run, rate limit, transient Telegram error) — + same `SkippedPackage`/`SystemNotification` bookkeeping as today's upload failures. Extend + `inferSkipReason` to recognize forward-specific error text the same way it already recognizes + upload errors. +- Fingerprint-repost check finds multiple ambiguous same-name/size candidates that can't be + uniquely disambiguated — same `INTEGRITY_AUDIT` notification pattern already used in + `provenance-backfill.ts`: don't guess, surface for manual triage. +- `allowsForwarding` unknown (channel just linked, not yet scanned by the refresh point) — treated + as `false`; the download path runs. No channel uses an unverified forwarding permission. +- Ranged listing throwing instead of returning `null` — treated identically to returning `null` + (fall through to download), consistent with how the existing ranged readers already treat + internal errors (they catch and return `null` themselves). + +## Testing + +- Unit tests (vitest, alongside the existing `archive/*.test.ts` and `archive/ranged/*.test.ts` + files): the dedup-identity derivation function (fingerprint-hash vs remoteUniqueId-fallback + branches), the extended fingerprint-based repost check, and `forwardArchiveToChannel`'s + request-building logic against a mocked TDLib client — same style as the existing ranged-reader + tests (pure logic, no live TDLib). +- Live verification (manual — matches this repo's existing convention that the large + `worker.ts`/`worker.py`-equivalent orchestration function has no automated test coverage and is + verified live post-deploy): one forwarding-enabled test channel and one protected-content test + channel. Confirm forward-path packages land with correct entries/tags/dedup identity and + `destMessageIds`; confirm the protected channel still goes through the unchanged full pipeline; + confirm a deliberately-unparseable archive in a forwarding-enabled channel correctly falls back + to download+reupload and still ends up fully indexed. + +## Rollout + +Local build + deploy, following the same recipe as the ranged-archive-listing work: build +`worker/Dockerfile` locally, recreate the `dragonsstash-worker` container from the local image (no +`pull`, no GitHub push required). New DB migration for `TelegramChannel.allowsForwarding`. No +changes required to the bot or app services.