diff --git a/docs/superpowers/plans/2026-07-30-forward-priority-ingestion.md b/docs/superpowers/plans/2026-07-30-forward-priority-ingestion.md new file mode 100644 index 0000000..ba77da7 --- /dev/null +++ b/docs/superpowers/plans/2026-07-30-forward-priority-ingestion.md @@ -0,0 +1,1475 @@ +# Forward-Priority Ingestion Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** For Telegram source channels that allow forwarding, skip the download+reupload round-trip for new archives — use Telegram-native forwarding to move the file to the destination channel, and the existing ranged (no-download) inner-file listing readers to keep every archive fully indexed. Channels that block forwarding, and any specific archive where the cheap listing fails, keep using today's unchanged download+reupload pipeline. + +**Architecture:** Add one fork point inside `processOneArchiveSet` (`worker/src/worker.ts`), right after the existing pre-download dedup checks and size guard. If the source channel's cached `allowsForwarding` flag is true, attempt a new forward path: read the inner-file listing via the already-built ranged readers (`worker/src/archive/ranged/*`), derive a dedup identity from the listing (no downloaded bytes available), forward the message(s) natively via TDLib, and write the same `Package`/`PackageFile` records the download path would have produced. Any failure in that path (ranged listing miss, ambiguous/blocked forward) falls straight through into the existing, unchanged download/hash/split/upload code below for that one archive. + +**Tech Stack:** TypeScript (strict, ESM, `.js` import specifiers), TDLib via `tdl`, vitest, Prisma/Postgres (`@prisma/client` v7). + +**Design doc:** `docs/superpowers/specs/2026-07-30-forward-priority-ingestion-design.md` — read it before starting if anything below is unclear about *why*, not just *what*. + +## Global Constraints + +- TypeScript strict; ESM import specifiers end in `.js`. Copy the surrounding files' style exactly. +- ESLint does NOT cover `worker/` — but keep types clean; no `any` unless mirroring an existing pattern in the same file. +- Tests: vitest, files match `worker/src/**/*.test.ts`; run from `worker/` with `npx vitest run`. +- All TDLib calls must go through the existing FLOOD_WAIT-safe wrappers (`withFloodWait` from `worker/src/util/retry.js`, or `invokeWithTimeout` from `worker/src/tdlib/download.js`). Do not add a second wrapper on top of an already-wrapped call. +- DB migrations run from the **repo root** (the Prisma schema lives at `prisma/schema.prisma`, shared by all three services), not from `worker/`. Command: `npx prisma migrate dev --name ` (ensure `DATABASE_URL` is set — check `.env` at repo root first before passing it inline). +- No new test framework — this repo uses vitest only for pure/isolable logic (archive parsers, fingerprint math, request-shape builders). The large `processOneArchiveSet`/`runWorkerForAccount` orchestration in `worker.ts` has no automated test coverage today and is verified live post-deploy; this plan follows that same convention rather than inventing new orchestration-level tests. +- Deploy is local (no GitHub push): build `worker/Dockerfile` locally, recreate the `dragonsstash-worker` container from the local image WITHOUT `pull` — same recipe used by the `ranged-archive-listing` work. +- **Task 1 touches the shared `master` branch (merge + push). Do not run its merge/push commands unattended — stop and get explicit human confirmation before executing them, even when running under an autonomous execution skill.** + +--- + +## File Structure + +- Create `worker/src/archive/ranged/dispatch.ts` — the shared (no-download) archive-listing dispatcher, promoted out of `provenance-backfill.ts` so both backfill and fresh ingestion can call it. (+ `dispatch.test.ts`) +- Create `worker/src/archive/forward-identity.ts` — derives a `Package.contentHash`-compatible identity string when there are no downloaded bytes to hash. (+ `forward-identity.test.ts`) +- Create `worker/src/archive/forward-repost-check.ts` — cross-channel CRC-fingerprint duplicate check for the forward path. (+ `forward-repost-check.test.ts`) +- Create `worker/src/upload/forward.ts` — native TDLib forward from source chat to destination chat, mirroring `upload/channel.ts`'s shape. (+ `forward.test.ts`) +- Modify `prisma/schema.prisma` — `TelegramChannel.allowsForwarding`, `IngestionRun.zipsForwarded`. +- Modify `worker/src/db/queries.ts` — `setChannelAllowsForwarding`, `findFingerprintDedupCandidates`, counter plumbing (`ActivityUpdate`, `updateRunActivity`, `completeIngestionRun`). +- Modify `worker/src/provenance-backfill.ts` — import the dispatcher from its new shared location instead of defining it locally; export `resolveCandidateFingerprintEntries` and `compareFingerprints` for reuse. +- Modify `worker/src/worker.ts` — persist the per-channel forwarding flag; add the fork point + `tryForwardArchiveSet` helper; wire the new counter. + +--- + +## Task 1: Merge `feat/ranged-archive-listing` to master, branch for this feature + +**This task requires a human to confirm before the merge/push commands run.** `feat/ranged-archive-listing` is already complete and serves a different purpose (backfilling listings onto already-deduped placeholder packages) — it merges independently of this feature. + +- [ ] **Step 1: Confirm the branch is clean and up to date** + +```bash +cd /path/to/DragonsStash +git status +git log --oneline -5 +``` +Expected: working tree clean (or only expected local files), branch `feat/ranged-archive-listing` up to date with its remote. + +- [ ] **Step 2: STOP — get explicit human confirmation** + +Show the human the commit list that will land on `master` (`git log master..feat/ranged-archive-listing --oneline`) and ask them to confirm before proceeding. Do not continue to Step 3 without an explicit yes. + +- [ ] **Step 3: Merge to master and push** + +```bash +git checkout master +git pull +git merge --no-ff feat/ranged-archive-listing +git push +``` +Expected: fast-forward or clean merge commit, push succeeds. + +- [ ] **Step 4: Branch for this feature** + +```bash +git checkout -b feat/forward-priority-ingestion +``` +Expected: new branch created off the just-updated `master`. + +--- + +## Task 2: Schema — `TelegramChannel.allowsForwarding` + `IngestionRun.zipsForwarded` + +**Files:** +- Modify: `prisma/schema.prisma` + +**Interfaces:** +- Produces: `TelegramChannel.allowsForwarding: boolean | null` (Prisma-generated type, consumed by Task 3 and Task 8), `IngestionRun.zipsForwarded: number` (consumed by Task 8). + +- [ ] **Step 1: Edit `TelegramChannel`** + +In `prisma/schema.prisma`, find: + +```prisma +model TelegramChannel { + id String @id @default(cuid()) + telegramId BigInt @unique + title String + type ChannelType + isForum Boolean @default(false) + isActive Boolean @default(false) + category String? @db.VarChar(64) +``` + +Replace with: + +```prisma +model TelegramChannel { + id String @id @default(cuid()) + telegramId BigInt @unique + title String + type ChannelType + isForum Boolean @default(false) + isActive Boolean @default(false) + category String? @db.VarChar(64) + /// Whether this chat currently allows forwarding/saving (the inverse of + /// TDLib's chat.has_protected_content). Null = not yet checked; treated the + /// same as false everywhere in the worker (safe default: use the + /// download+reupload path until this is confirmed true). + allowsForwarding Boolean? +``` + +- [ ] **Step 2: Edit `IngestionRun`** + +Find: + +```prisma + zipsIngested Int @default(0) + zipsBackfilled Int @default(0) + errorMessage String? +``` + +Replace with: + +```prisma + zipsIngested Int @default(0) + zipsBackfilled Int @default(0) + zipsForwarded Int @default(0) + errorMessage String? +``` + +- [ ] **Step 3: Create and apply the migration** + +```bash +cd /path/to/DragonsStash +npx prisma migrate dev --name add_forward_priority_ingestion +``` +Expected: a new folder under `prisma/migrations/`, migration applied to the local dev DB, Prisma client regenerated (no errors). + +- [ ] **Step 4: Commit** + +```bash +git add prisma/schema.prisma prisma/migrations +git commit -m "feat(db): add TelegramChannel.allowsForwarding + IngestionRun.zipsForwarded" +``` + +--- + +## Task 3: Detect + persist per-channel forwarding permission + +**Files:** +- Modify: `worker/src/db/queries.ts` +- Modify: `worker/src/worker.ts:497-518` (the existing per-channel `getChat` + `isForum` check block, inside `runWorkerForAccount`) + +**Interfaces:** +- Produces: `setChannelAllowsForwarding(channelId: string, allowsForwarding: boolean): Promise` (mirrors the existing `setChannelForum`). +- Consumes: nothing new — reuses the `getChat` call already made per channel per cycle. + +- [ ] **Step 1: Add the query function** + +In `worker/src/db/queries.ts`, find: + +```typescript +export async function setChannelForum(channelId: string, isForum: boolean) { + return db.telegramChannel.update({ + where: { id: channelId }, + data: { isForum }, + }); +} +``` + +Add immediately after it: + +```typescript +export async function setChannelAllowsForwarding(channelId: string, allowsForwarding: boolean) { + return db.telegramChannel.update({ + where: { id: channelId }, + data: { allowsForwarding }, + }); +} +``` + +- [ ] **Step 2: Capture the `getChat` response and persist the flag** + +In `worker/src/worker.ts`, find (inside the per-channel loop in `runWorkerForAccount`): + +```typescript + // ── Ensure TDLib knows about this chat ── + // getChats may not have loaded all channels (pagination, archive folder, etc.) + // so we explicitly load each channel before scanning. + try { + await client.invoke({ + _: "getChat", + chat_id: Number(channel.telegramId), + }); + } catch (chatErr) { + accountLog.warn( + { err: chatErr, channelId: channel.id, title: channel.title, telegramId: channel.telegramId.toString() }, + "TDLib does not know about this chat — it may not be accessible to this account. Skipping." + ); + continue; + } + + // ── Check if channel is a forum ── + const forum = await isChatForum(client, channel.telegramId); + if (forum !== channel.isForum) { + await setChannelForum(channel.id, forum); + accountLog.info( + { channelId: channel.id, title: channel.title, isForum: forum }, + "Updated channel forum status" + ); + } +``` + +Replace with: + +```typescript + // ── Ensure TDLib knows about this chat ── + // getChats may not have loaded all channels (pagination, archive folder, etc.) + // so we explicitly load each channel before scanning. The response is + // also where we read has_protected_content (below) to decide whether + // this channel is eligible for the forward-priority ingestion path. + // eslint-disable-next-line @typescript-eslint/no-explicit-any + let chatInfo: any; + try { + chatInfo = await client.invoke({ + _: "getChat", + chat_id: Number(channel.telegramId), + }); + } catch (chatErr) { + accountLog.warn( + { err: chatErr, channelId: channel.id, title: channel.title, telegramId: channel.telegramId.toString() }, + "TDLib does not know about this chat — it may not be accessible to this account. Skipping." + ); + continue; + } + + // ── Check if channel is a forum ── + const forum = await isChatForum(client, channel.telegramId); + if (forum !== channel.isForum) { + await setChannelForum(channel.id, forum); + accountLog.info( + { channelId: channel.id, title: channel.title, isForum: forum }, + "Updated channel forum status" + ); + } + + // ── Check if channel allows forwarding ── + // TDLib's chat.has_protected_content is documented on the general + // Chat object (core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1chat.html), + // but PENDING LIVE VERIFICATION here: confirm on first deploy that a + // real chatTypeSupergroup/channel response actually populates this + // field (some TDLib doc pages describe it in the context of basic + // groups only). If it's ever `undefined` in practice, this block is a + // no-op and allowsForwarding stays at its last-known/null value — + // which safely keeps the channel on the download path. + const hasProtectedContent: boolean | undefined = chatInfo?.has_protected_content; + if (typeof hasProtectedContent === "boolean") { + const allowsForwarding = !hasProtectedContent; + if (allowsForwarding !== channel.allowsForwarding) { + await setChannelAllowsForwarding(channel.id, allowsForwarding); + accountLog.info( + { channelId: channel.id, title: channel.title, allowsForwarding }, + "Updated channel forwarding permission" + ); + } + channel.allowsForwarding = allowsForwarding; + } +``` + +- [ ] **Step 3: Typecheck** + +```bash +cd worker && npx tsc --noEmit +``` +Expected: no errors. (`channel.allowsForwarding` is assignable because `channel` comes from the Prisma-generated `TelegramChannel` type, which now includes the field from Task 2's migration.) + +- [ ] **Step 4: Commit** + +```bash +git add worker/src/db/queries.ts worker/src/worker.ts +git commit -m "feat(worker): detect + persist per-channel forwarding permission" +``` + +--- + +## Task 4: Promote the ranged-listing dispatcher to a shared module + +**Files:** +- Create: `worker/src/archive/ranged/dispatch.ts` +- Test: `worker/src/archive/ranged/dispatch.test.ts` +- Modify: `worker/src/provenance-backfill.ts` + +**Interfaces:** +- Consumes: `parseZipCentralDirectoryFromTail`, `MIN_ZIP_TAIL_BYTES` from `../central-directory.js`; `downloadFileRange` from `../../tdlib/range-download.js`; `readSevenZListingRanged` from `./sevenz-ranged.js`; `readRarListingRanged` from `./rar-ranged.js`; `tdlibRangeReader`, `RangeReader` from `./range-reader.js`; `RangedPart` from `./sevenz-ranged.js`; `FileEntry` from `../zip-reader.js`. +- Produces: `readScannedZipListing(client: Client, parts: { fileId: string; fileSize: bigint }[]): Promise`, `readScannedListingRanged(archiveType: string, client: Client, parts: RangedPart[]): Promise` — both consumed by Task 8 (worker.ts) and already-existing callers in `provenance-backfill.ts`. + +- [ ] **Step 1: Write the failing test** + +```typescript +// worker/src/archive/ranged/dispatch.test.ts +import { describe, it, expect } from "vitest"; +import { readScannedListingRanged } from "./dispatch.js"; + +describe("readScannedListingRanged", () => { + it("returns null for an unknown archive type without calling the reader", async () => { + const read = async () => Buffer.alloc(0); + const result = await readScannedListingRanged( + "DOCUMENT", + { invoke: async () => ({}) } as never, + [{ fileId: "1", fileSize: 100n, fileName: "a.pdf" }], + ); + expect(result).toBeNull(); + void read; // unused placeholder kept out of the dispatch call — DOCUMENT never reaches a reader + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd worker && npx vitest run src/archive/ranged/dispatch.test.ts` +Expected: FAIL — `Cannot find module './dispatch.js'`. + +- [ ] **Step 3: Create `dispatch.ts` — move `readScannedZipListing` and `readScannedListingRanged` out of `provenance-backfill.ts`** + +```typescript +// worker/src/archive/ranged/dispatch.ts +import type { Client } from "tdl"; +import { downloadFileRange } from "../../tdlib/range-download.js"; +import { parseZipCentralDirectoryFromTail, MIN_ZIP_TAIL_BYTES } from "../central-directory.js"; +import { childLogger } from "../../util/logger.js"; +import type { FileEntry } from "../zip-reader.js"; +import { readSevenZListingRanged, type RangedPart } from "./sevenz-ranged.js"; +import { readRarListingRanged } from "./rar-ranged.js"; +import { tdlibRangeReader } from "./range-reader.js"; + +const log = childLogger("ranged-dispatch"); + +/** + * Read a ZIP central directory from the tail of a (possibly multipart) + * archive. `parts` is ordered; only the LAST part carries the EOCD record. + * `fileSize` on each part is that part's own size (NOT the whole-archive + * total) so the download offset stays within that part's bounds, while + * `tailStart` passed to the parser is the logical whole-archive offset + * (preceding parts' sizes + the offset within the last part). + */ +export async function readScannedZipListing( + client: Client, + parts: { fileId: string; fileSize: bigint }[], +): Promise { + if (parts.length === 0) return null; + const lastPart = parts[parts.length - 1]; + const precedingSize = parts.slice(0, -1).reduce((sum, p) => sum + Number(p.fileSize), 0); + const lastSize = Number(lastPart.fileSize); + for (const tailBytes of [MIN_ZIP_TAIL_BYTES, MIN_ZIP_TAIL_BYTES * 4]) { + const partOffset = Math.max(0, lastSize - tailBytes); + const downloadLen = Math.min(tailBytes, lastSize); + try { + const buf = await downloadFileRange(client, lastPart.fileId, partOffset, downloadLen, lastPart.fileSize); + const tailStart = precedingSize + partOffset; + return parseZipCentralDirectoryFromTail(buf, tailStart); + } catch (err) { + if (err instanceof RangeError) continue; // try a larger tail + log.warn({ err, fileId: lastPart.fileId }, "ranged ZIP listing failed"); + return null; + } + } + return null; +} + +/** + * Dispatch a (no-download) inner-file listing read by archive type. Used both + * by the provenance-backfill path (reading an already-uploaded copy) and the + * forward-priority ingestion path (reading the source channel's copy before + * any download/forward decision is made) — the read itself only needs + * {fileId, fileSize, fileName}, so it doesn't matter which channel the file + * currently lives in. + */ +export async function readScannedListingRanged( + archiveType: string, + client: Client, + parts: RangedPart[], +): Promise { + const read = tdlibRangeReader(client); + if (archiveType === "ZIP") return readScannedZipListing(client, parts); + if (archiveType === "SEVEN_Z") return readSevenZListingRanged(parts, read); + if (archiveType === "RAR") return readRarListingRanged(parts, read); + return null; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd worker && npx vitest run src/archive/ranged/dispatch.test.ts` +Expected: PASS (1 passed). + +- [ ] **Step 5: Update `provenance-backfill.ts` to import instead of define** + +In `worker/src/provenance-backfill.ts`, delete the `readScannedZipListing` function body (lines ~47-69) and the `readScannedListingRanged` function body (lines ~109-119). + +Change the top-of-file imports from: + +```typescript +import { downloadFileRange } from "./tdlib/range-download.js"; +import { invokeWithTimeout } from "./tdlib/download.js"; +import { parseZipCentralDirectoryFromTail, MIN_ZIP_TAIL_BYTES } from "./archive/central-directory.js"; +import { fingerprintsMatch, crcFingerprint } from "./archive/fingerprint.js"; +``` + +to: + +```typescript +import { invokeWithTimeout } from "./tdlib/download.js"; +import { fingerprintsMatch, crcFingerprint } from "./archive/fingerprint.js"; +import { readScannedZipListing, readScannedListingRanged } from "./archive/ranged/dispatch.js"; +``` + +(`downloadFileRange`, `parseZipCentralDirectoryFromTail`, and `MIN_ZIP_TAIL_BYTES` are no longer used directly in this file — they're now only used inside `dispatch.ts`. `readSevenZListingRanged`, `readRarListingRanged`, and `tdlibRangeReader` stay imported in `provenance-backfill.ts` as-is; `resolveCandidateFingerprintEntries` still calls them directly for the destination-copy read.) + +- [ ] **Step 6: Typecheck + run the full suite** + +```bash +cd worker && npx tsc --noEmit && npx vitest run +``` +Expected: no TS errors; all existing tests still pass (this is a pure move — behavior is unchanged). + +- [ ] **Step 7: Commit** + +```bash +git add worker/src/archive/ranged/dispatch.ts worker/src/archive/ranged/dispatch.test.ts worker/src/provenance-backfill.ts +git commit -m "refactor(worker): promote ranged-listing dispatcher to a shared module" +``` + +--- + +## Task 5: Dedup-identity derivation for forward-path packages + +**Files:** +- Create: `worker/src/archive/forward-identity.ts` +- Test: `worker/src/archive/forward-identity.test.ts` + +**Interfaces:** +- Consumes: `FileEntry` from `./zip-reader.js`; `crcFingerprint` from `./fingerprint.js`. +- Produces: `deriveForwardContentHash(entries: FileEntry[], remoteUniqueId: string | null, sourceChannelId: string, sourceMessageId: bigint): string` — consumed by Task 8. + +- [ ] **Step 1: Write the failing test** + +```typescript +// worker/src/archive/forward-identity.test.ts +import { describe, it, expect } from "vitest"; +import { createHash } from "crypto"; +import { deriveForwardContentHash } from "./forward-identity.js"; +import type { FileEntry } from "./zip-reader.js"; + +function entry(crc32: string | null): FileEntry { + return { path: "a", fileName: "a", extension: null, compressedSize: 1n, uncompressedSize: 1n, crc32 }; +} + +describe("deriveForwardContentHash", () => { + it("hashes the sorted CRC list when all entries have a CRC32 (ZIP/RAR)", () => { + const entries = [entry("BBBB"), entry("AAAA")]; + const expectedHash = createHash("sha256").update(["aaaa", "bbbb"].join(",")).digest("hex"); + expect(deriveForwardContentHash(entries, "unique-1", "chan-1", 42n)).toBe(`fingerprint:${expectedHash}`); + }); + + it("falls back to remoteUniqueId when CRCs are incomplete (7z today)", () => { + const entries = [entry(null), entry("AAAA")]; + expect(deriveForwardContentHash(entries, "unique-42", "chan-1", 42n)).toBe("forward:unique-42"); + }); + + it("falls back to sourceChannelId+sourceMessageId when there's no CRC and no remoteUniqueId", () => { + const entries = [entry(null)]; + expect(deriveForwardContentHash(entries, null, "chan-1", 42n)).toBe("forward:chan-1:42"); + }); + + it("falls back past an empty entries list the same way", () => { + expect(deriveForwardContentHash([], null, "chan-1", 7n)).toBe("forward:chan-1:7"); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd worker && npx vitest run src/archive/forward-identity.test.ts` +Expected: FAIL — `Cannot find module './forward-identity.js'`. + +- [ ] **Step 3: Write minimal implementation** + +```typescript +// worker/src/archive/forward-identity.ts +import { createHash } from "crypto"; +import { crcFingerprint } from "./fingerprint.js"; +import type { FileEntry } from "./zip-reader.js"; + +/** + * Derive a Package.contentHash-compatible identity string for a forward-path + * package (no downloaded bytes exist to hash directly). Priority order: + * 1. A CRC32-fingerprint hash, when the ranged listing's CRCs are complete + * (ZIP/RAR today) — the strongest available signal, since it lets + * forward-path and download-path copies of the same archive still + * collide/dedupe on identical content. + * 2. TDLib's remote.unique_id, when CRCs are incomplete (7z today has none). + * 3. sourceChannelId+sourceMessageId, as a last-resort unique value so the + * required-unique Package.contentHash column is always satisfiable. + * Follows the same `:` synthetic-hash convention already used + * by `rebuild.ts`'s `rebuild:${destChannelId}:${destMessageId}` placeholder. + */ +export function deriveForwardContentHash( + entries: FileEntry[], + remoteUniqueId: string | null, + sourceChannelId: string, + sourceMessageId: bigint, +): string { + const fp = crcFingerprint(entries); + if (fp.complete && fp.crcs.length > 0) { + const hash = createHash("sha256").update(fp.crcs.join(",")).digest("hex"); + return `fingerprint:${hash}`; + } + if (remoteUniqueId) { + return `forward:${remoteUniqueId}`; + } + return `forward:${sourceChannelId}:${sourceMessageId}`; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd worker && npx vitest run src/archive/forward-identity.test.ts` +Expected: PASS (4 passed). + +- [ ] **Step 5: Commit** + +```bash +git add worker/src/archive/forward-identity.ts worker/src/archive/forward-identity.test.ts +git commit -m "feat(worker): derive a dedup identity for forward-path packages without bytes" +``` + +--- + +## Task 6: Cross-channel CRC-fingerprint repost check + +**Files:** +- Modify: `worker/src/db/queries.ts` +- Modify: `worker/src/provenance-backfill.ts` +- Create: `worker/src/archive/forward-repost-check.ts` +- Test: `worker/src/archive/forward-repost-check.test.ts` + +**Interfaces:** +- Consumes: `PlaceholderCandidate` type from `../db/queries.js`. +- Produces: + - `db/queries.ts`: `findFingerprintDedupCandidates(fileName: string, fileSize: bigint): Promise`. + - `provenance-backfill.ts`: `resolveCandidateFingerprintEntries` and `compareFingerprints` become exported (unchanged behavior, just no longer private). + - `forward-repost-check.ts`: `checkFingerprintRepost(client: Client, entries: FileEntry[], fileName: string, fileSize: bigint): Promise<{ isDuplicate: boolean; matchedPackageId: string | null }>` — consumed by Task 8. + +- [ ] **Step 1: Extract the shared row-enrichment helper and add the new query** + +In `worker/src/db/queries.ts`, find `findPlaceholderCandidates` (around line 1032): + +```typescript +export async function findPlaceholderCandidates( + destChannelId: string, + fileName: string, + fileSize: bigint, +): Promise { + const rows = await db.package.findMany({ + where: { + fileName, + fileSize, + destMessageId: { not: null }, + // Placeholder provenance (spec §1): manual-upload (source == destination) + // OR rebuild record (sourceMessageId == 0 "unknown" sentinel). + OR: [ + { sourceChannelId: destChannelId }, + { sourceMessageId: 0n }, + ], + }, + select: { + id: true, archiveType: true, fileName: true, fileCount: true, fileSize: true, + destMessageId: true, destMessageIds: true, destChannelId: true, + }, + orderBy: { indexedAt: "asc" }, + }); + if (rows.length === 0) return []; + + const destChannelIds = [...new Set(rows.map((r) => r.destChannelId).filter((id): id is string => !!id))]; + const channels = destChannelIds.length + ? await db.telegramChannel.findMany({ + where: { id: { in: destChannelIds } }, + select: { id: true, telegramId: true }, + }) + : []; + const telegramIdById = new Map(channels.map((c) => [c.id, c.telegramId])); + + return rows.map((row) => ({ +``` + +(the function continues with the `.map` return — leave that as-is). Extract the row shape + enrichment into a shared helper by adding this function right ABOVE `findPlaceholderCandidates`: + +```typescript +type PlaceholderRow = { + id: string; archiveType: string; fileName: string; fileCount: number; fileSize: bigint; + destMessageId: bigint | null; destMessageIds: bigint[]; destChannelId: string | null; +}; + +async function enrichWithDestChannel(rows: PlaceholderRow[]): Promise { + if (rows.length === 0) return []; + const destChannelIds = [...new Set(rows.map((r) => r.destChannelId).filter((id): id is string => !!id))]; + const channels = destChannelIds.length + ? await db.telegramChannel.findMany({ + where: { id: { in: destChannelIds } }, + select: { id: true, telegramId: true }, + }) + : []; + const telegramIdById = new Map(channels.map((c) => [c.id, c.telegramId])); + return rows.map((row) => ({ + id: row.id, archiveType: row.archiveType, fileName: row.fileName, fileCount: row.fileCount, fileSize: row.fileSize, + destMessageId: row.destMessageId, destMessageIds: row.destMessageIds, + destChannel: row.destChannelId && telegramIdById.has(row.destChannelId) + ? { telegramId: telegramIdById.get(row.destChannelId)! } + : null, + })); +} +``` + +Then simplify `findPlaceholderCandidates` to use it — replace the whole function body with: + +```typescript +export async function findPlaceholderCandidates( + destChannelId: string, + fileName: string, + fileSize: bigint, +): Promise { + const rows = await db.package.findMany({ + where: { + fileName, + fileSize, + destMessageId: { not: null }, + // Placeholder provenance (spec §1): manual-upload (source == destination) + // OR rebuild record (sourceMessageId == 0 "unknown" sentinel). + OR: [ + { sourceChannelId: destChannelId }, + { sourceMessageId: 0n }, + ], + }, + select: { + id: true, archiveType: true, fileName: true, fileCount: true, fileSize: true, + destMessageId: true, destMessageIds: true, destChannelId: true, + }, + orderBy: { indexedAt: "asc" }, + }); + return enrichWithDestChannel(rows); +} +``` + +Add the new, broader query right after it (no placeholder-only restriction — matches ANY package, any channel, so a forward-path candidate can dedupe against a normal fully-downloaded package elsewhere): + +```typescript +/** + * Find every uploaded Package (any provenance, any channel) matching + * name+size, for the forward-priority path's cross-channel CRC-fingerprint + * dedup check. Unlike findPlaceholderCandidates, this is NOT restricted to + * placeholder rows — it exists to catch the case where the exact same + * archive was independently uploaded (not reposted/forwarded) to two + * different source channels. + */ +export async function findFingerprintDedupCandidates( + fileName: string, + fileSize: bigint, +): Promise { + const rows = await db.package.findMany({ + where: { fileName, fileSize, destMessageId: { not: null } }, + select: { + id: true, archiveType: true, fileName: true, fileCount: true, fileSize: true, + destMessageId: true, destMessageIds: true, destChannelId: true, + }, + orderBy: { indexedAt: "asc" }, + }); + return enrichWithDestChannel(rows); +} +``` + +- [ ] **Step 2: Export the two helpers already defined in `provenance-backfill.ts`** + +In `worker/src/provenance-backfill.ts`, find: + +```typescript +async function resolveCandidateFingerprintEntries( +``` + +Change to: + +```typescript +export async function resolveCandidateFingerprintEntries( +``` + +Find: + +```typescript +function compareFingerprints(a: FileEntry[], b: FileEntry[]): "match" | "mismatch" | "incomplete" { +``` + +Change to: + +```typescript +export function compareFingerprints(a: FileEntry[], b: FileEntry[]): "match" | "mismatch" | "incomplete" { +``` + +No other changes in this file — both functions' bodies and all existing call sites are untouched. + +- [ ] **Step 3: Write the failing test for the new orchestrator** + +```typescript +// worker/src/archive/forward-repost-check.test.ts +import { describe, it, expect, vi } from "vitest"; + +const candidate = { + id: "pkg-1", archiveType: "ZIP", fileName: "a.zip", fileCount: 3, fileSize: 100n, + destMessageId: 1n, destMessageIds: [1n], destChannel: { telegramId: 999n }, +}; + +vi.mock("../db/queries.js", () => ({ + findFingerprintDedupCandidates: vi.fn(async () => [candidate]), +})); +const resolveMock = vi.fn(async () => [{ path: "x", fileName: "x", extension: null, compressedSize: 1n, uncompressedSize: 1n, crc32: "AAAA" }]); +const compareMock = vi.fn(); +vi.mock("../provenance-backfill.js", () => ({ + resolveCandidateFingerprintEntries: (...args: unknown[]) => resolveMock(...args), + compareFingerprints: (...args: unknown[]) => compareMock(...args), +})); + +import { checkFingerprintRepost } from "./forward-repost-check.js"; +import type { FileEntry } from "./zip-reader.js"; + +const newEntries: FileEntry[] = [{ path: "x", fileName: "x", extension: null, compressedSize: 1n, uncompressedSize: 1n, crc32: "AAAA" }]; + +describe("checkFingerprintRepost", () => { + it("reports a duplicate when a candidate's fingerprint matches", async () => { + compareMock.mockReturnValueOnce("match"); + const result = await checkFingerprintRepost({} as never, newEntries, "a.zip", 100n); + expect(result).toEqual({ isDuplicate: true, matchedPackageId: "pkg-1" }); + }); + + it("reports no duplicate when no candidate matches", async () => { + compareMock.mockReturnValueOnce("mismatch"); + const result = await checkFingerprintRepost({} as never, newEntries, "a.zip", 100n); + expect(result).toEqual({ isDuplicate: false, matchedPackageId: null }); + }); +}); +``` + +- [ ] **Step 4: Run test to verify it fails** + +Run: `cd worker && npx vitest run src/archive/forward-repost-check.test.ts` +Expected: FAIL — `Cannot find module './forward-repost-check.js'`. + +- [ ] **Step 5: Write minimal implementation** + +```typescript +// worker/src/archive/forward-repost-check.ts +import type { Client } from "tdl"; +import type { FileEntry } from "./zip-reader.js"; +import { compareFingerprints, resolveCandidateFingerprintEntries } from "../provenance-backfill.js"; +import { findFingerprintDedupCandidates } from "../db/queries.js"; + +export interface FingerprintRepostResult { + isDuplicate: boolean; + matchedPackageId: string | null; +} + +/** + * Cross-channel duplicate check for the forward-priority path: compare the + * new archive's CRC fingerprint against every existing Package sharing its + * name+size, regardless of which channel or ingestion path produced them. + * This is what lets a forwarded copy dedupe against a previously + * fully-downloaded copy of the same archive, despite never sharing a + * byte-hash-derived contentHash. + */ +export async function checkFingerprintRepost( + client: Client, + entries: FileEntry[], + fileName: string, + fileSize: bigint, +): Promise { + const candidates = await findFingerprintDedupCandidates(fileName, fileSize); + for (const candidate of candidates) { + const candidateEntries = await resolveCandidateFingerprintEntries(client, candidate); + if (compareFingerprints(entries, candidateEntries) === "match") { + return { isDuplicate: true, matchedPackageId: candidate.id }; + } + } + return { isDuplicate: false, matchedPackageId: null }; +} +``` + +- [ ] **Step 6: Run test to verify it passes** + +Run: `cd worker && npx vitest run src/archive/forward-repost-check.test.ts` +Expected: PASS (2 passed). + +- [ ] **Step 7: Typecheck the whole worker** + +```bash +cd worker && npx tsc --noEmit +``` +Expected: no errors. + +- [ ] **Step 8: Commit** + +```bash +git add worker/src/db/queries.ts worker/src/provenance-backfill.ts worker/src/archive/forward-repost-check.ts worker/src/archive/forward-repost-check.test.ts +git commit -m "feat(worker): cross-channel CRC-fingerprint repost check for the forward path" +``` + +--- + +## Task 7: `forwardArchiveToChannel` + +**Files:** +- Create: `worker/src/upload/forward.ts` +- Test: `worker/src/upload/forward.test.ts` + +**Interfaces:** +- Consumes: `withFloodWait` from `../util/retry.js`. +- Produces: `interface ForwardResult { messageId: bigint; messageIds: bigint[] }`, `forwardArchiveToChannel(client: Client, fromChatId: bigint, toChatId: bigint, sourceMessageIds: bigint[]): Promise` — consumed by Task 8. + +TDLib reference (confirmed via docs, `forwardMessages`): `chat_id` (destination), `topic_id` (pass `null`), `from_chat_id` (source), `message_ids` (int53 array, **must be in strictly increasing order**, max 100 per call), `options` (pass `null` for defaults), `send_copy` (`false` = plain forward, keeps "Forwarded from" attribution; the destination archive channel is not user-facing so this plan keeps it simple and forwards plainly), `remove_caption` (`false`, only relevant when `send_copy` is true). Response is `{ messages: (Message | null)[] }` in the same order as the request — Telegram returns `null` for any message that couldn't be forwarded (e.g. if `has_protected_content` blocks it), which this function treats as a failure. + +- [ ] **Step 1: Write the failing test** + +```typescript +// worker/src/upload/forward.test.ts +import { describe, it, expect, vi } from "vitest"; +import { forwardArchiveToChannel } from "./forward.js"; + +function fakeClient(response: unknown) { + return { invoke: vi.fn(async () => response) } as never; +} + +describe("forwardArchiveToChannel", () => { + it("sorts message ids ascending and sends them via forwardMessages", async () => { + const invoke = vi.fn(async (req: { message_ids: number[] }) => ({ + messages: req.message_ids.map((id) => ({ id: id + 1000 })), + })); + const client = { invoke } as never; + + const result = await forwardArchiveToChannel(client, 111n, 222n, [30n, 10n, 20n]); + + expect(invoke).toHaveBeenCalledWith( + expect.objectContaining({ + _: "forwardMessages", + chat_id: 222, + from_chat_id: 111, + message_ids: [10, 20, 30], + send_copy: false, + }), + ); + expect(result.messageId).toBe(1010n); + expect(result.messageIds).toEqual([1010n, 1020n, 1030n]); + }); + + it("throws when Telegram returns null for a message (can't be forwarded)", async () => { + const client = fakeClient({ messages: [{ id: 1001 }, null] }); + await expect(forwardArchiveToChannel(client, 111n, 222n, [10n, 20n])).rejects.toThrow(/could not forward/); + }); + + it("throws when the response has the wrong number of messages", async () => { + const client = fakeClient({ messages: [{ id: 1001 }] }); + await expect(forwardArchiveToChannel(client, 111n, 222n, [10n, 20n])).rejects.toThrow(/expected 2/); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd worker && npx vitest run src/upload/forward.test.ts` +Expected: FAIL — `Cannot find module './forward.js'`. + +- [ ] **Step 3: Write minimal implementation** + +```typescript +// worker/src/upload/forward.ts +import type { Client } from "tdl"; +import { childLogger } from "../util/logger.js"; +import { withFloodWait } from "../util/retry.js"; + +const log = childLogger("forward"); + +export interface ForwardResult { + messageId: bigint; + messageIds: bigint[]; +} + +/** + * Forward all parts of an archive set from the source chat directly to the + * destination chat via TDLib's forwardMessages — no download, no re-upload. + * Only usable when the source channel allows forwarding + * (TelegramChannel.allowsForwarding); the caller is responsible for that + * check. message_ids must be in strictly increasing order per the TDLib API, + * so this always sorts them regardless of the order they're passed in. + */ +export async function forwardArchiveToChannel( + client: Client, + fromChatId: bigint, + toChatId: bigint, + sourceMessageIds: bigint[], +): Promise { + const sortedIds = [...sourceMessageIds].sort((a, b) => (a < b ? -1 : a > b ? 1 : 0)); + const numericIds = sortedIds.map((id) => Number(id)); + + log.info( + { fromChatId: Number(fromChatId), toChatId: Number(toChatId), count: numericIds.length }, + "Forwarding archive to destination channel" + ); + + const result = (await withFloodWait( + () => + client.invoke({ + _: "forwardMessages", + chat_id: Number(toChatId), + topic_id: null, + from_chat_id: Number(fromChatId), + message_ids: numericIds, + options: null, + send_copy: false, + remove_caption: false, + } as never), + "forwardMessages" + )) as { messages: ({ id: number } | null)[] }; + + const forwarded = result.messages; + if (!forwarded || forwarded.length !== numericIds.length) { + throw new Error( + `forwardMessages returned ${forwarded?.length ?? 0} messages, expected ${numericIds.length}` + ); + } + + const messageIds: bigint[] = []; + for (let i = 0; i < forwarded.length; i++) { + const msg = forwarded[i]; + if (!msg) { + throw new Error( + `forwardMessages could not forward source message ${sortedIds[i]} (Telegram returned null — message may not be forwardable)` + ); + } + messageIds.push(BigInt(msg.id)); + } + + log.info( + { fromChatId: Number(fromChatId), toChatId: Number(toChatId), messageIds: messageIds.map(Number) }, + "Forward confirmed by Telegram" + ); + + return { messageId: messageIds[0], messageIds }; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd worker && npx vitest run src/upload/forward.test.ts` +Expected: PASS (3 passed). + +- [ ] **Step 5: Commit** + +```bash +git add worker/src/upload/forward.ts worker/src/upload/forward.test.ts +git commit -m "feat(worker): native TDLib forward from source to destination channel" +``` + +--- + +## Task 8: Wire the fork point into `processOneArchiveSet` + +**Files:** +- Modify: `worker/src/db/queries.ts` (counter plumbing) +- Modify: `worker/src/worker.ts` (fork point, new helper function, counter wiring, `inferSkipReason`) + +**Interfaces:** +- Consumes: `readScannedListingRanged` (Task 4), `deriveForwardContentHash` (Task 5), `checkFingerprintRepost` (Task 6), `forwardArchiveToChannel` (Task 7). +- Produces: no new exports — this is the integration task. No isolated unit test (matches this file's existing convention of live-only verification for orchestration code); verified in Tasks 9-11. + +- [ ] **Step 1: Add `zipsForwarded` to the counter plumbing in `db/queries.ts`** + +Find the `ActivityUpdate` interface's `zipsBackfilled` field (around line 379): + +```typescript + zipsBackfilled?: number; +} +``` + +Add a sibling field: + +```typescript + zipsBackfilled?: number; + zipsForwarded?: number; +} +``` + +Find, in `updateRunActivity` (around line 403): + +```typescript + ...(activity.zipsBackfilled !== undefined && { zipsBackfilled: activity.zipsBackfilled }), +``` + +Add immediately after: + +```typescript + ...(activity.zipsBackfilled !== undefined && { zipsBackfilled: activity.zipsBackfilled }), + ...(activity.zipsForwarded !== undefined && { zipsForwarded: activity.zipsForwarded }), +``` + +Find `completeIngestionRun`'s counters parameter type (around line 429): + +```typescript + counters: { + messagesScanned: number; + zipsFound: number; + zipsDuplicate: number; + zipsIngested: number; + zipsBackfilled: number; + } +``` + +Add a sibling field: + +```typescript + counters: { + messagesScanned: number; + zipsFound: number; + zipsDuplicate: number; + zipsIngested: number; + zipsBackfilled: number; + zipsForwarded: number; + } +``` + +- [ ] **Step 2: Add `zipsForwarded` to `PipelineContext.counters` and its initializer in `worker.ts`** + +Find (around line 313-319): + +```typescript + counters: { + messagesScanned: number; + zipsFound: number; + zipsDuplicate: number; + zipsIngested: number; + zipsBackfilled: number; + }; +``` + +Replace with: + +```typescript + counters: { + messagesScanned: number; + zipsFound: number; + zipsDuplicate: number; + zipsIngested: number; + zipsBackfilled: number; + zipsForwarded: number; + }; +``` + +Find (around line 422-428): + +```typescript + const counters = { + messagesScanned: 0, + zipsFound: 0, + zipsDuplicate: 0, + zipsIngested: 0, + zipsBackfilled: 0, + }; +``` + +Replace with: + +```typescript + const counters = { + messagesScanned: 0, + zipsFound: 0, + zipsDuplicate: 0, + zipsIngested: 0, + zipsBackfilled: 0, + zipsForwarded: 0, + }; +``` + +- [ ] **Step 3: Add the new imports** + +Find the import block at the top of `worker.ts` (near the other `archive/*` imports): + +```typescript +import { readZipCentralDirectory } from "./archive/zip-reader.js"; +import { readRarContents } from "./archive/rar-reader.js"; +import { read7zContents } from "./archive/sevenz-reader.js"; +``` + +Add immediately after: + +```typescript +import { readScannedListingRanged } from "./archive/ranged/dispatch.js"; +import { deriveForwardContentHash } from "./archive/forward-identity.js"; +import { checkFingerprintRepost } from "./archive/forward-repost-check.js"; +import { forwardArchiveToChannel } from "./upload/forward.js"; +``` + +- [ ] **Step 4: Extend `inferSkipReason` to recognize forward failures** + +Find: + +```typescript +function inferSkipReason(errMsg: string): "DOWNLOAD_FAILED" | "UPLOAD_FAILED" | "EXTRACT_FAILED" { + const lower = errMsg.toLowerCase(); + if (lower.includes("upload") || lower.includes("too many requests") || lower.includes("retry after") || lower.includes("send")) { + return "UPLOAD_FAILED"; + } +``` + +Replace with: + +```typescript +function inferSkipReason(errMsg: string): "DOWNLOAD_FAILED" | "UPLOAD_FAILED" | "EXTRACT_FAILED" { + const lower = errMsg.toLowerCase(); + if (lower.includes("upload") || lower.includes("forward") || lower.includes("too many requests") || lower.includes("retry after") || lower.includes("send")) { + return "UPLOAD_FAILED"; + } +``` + +- [ ] **Step 5: Add the `tryForwardArchiveSet` helper** + +Find the closing brace of `processOneArchiveSet` (search for `async function deleteFiles(paths: string[])` — the helper goes right BEFORE that function, i.e. right after `processOneArchiveSet` ends). Insert: + +```typescript +/** + * Attempt the forward-priority path for one archive set: ranged listing (no + * download) + native Telegram forward to the destination channel. + * + * Returns `undefined` when the forward path isn't usable for this specific + * archive (ranged listing failed, or the forward itself failed) — the caller + * falls through to the existing download+reupload pipeline in that case, so + * indexing completeness never regresses. + * + * Returns `null` when the archive is a confirmed duplicate (skip, same + * contract as the pre-download dedup checks earlier in the caller). + * + * Returns the new Package id on success. + */ +async function tryForwardArchiveSet( + ctx: PipelineContext, + archiveSet: ArchiveSet, + setIdx: number, + totalSets: number, + previewMatches: Map, + ingestionRunId: string, +): Promise { + const { + client, channelTitle, channel, + destChannelTelegramId, destChannelId, + counters, topicCreator, sourceTopicId, accountLog, + } = ctx; + void setIdx; + void totalSets; + + const archiveName = archiveSet.parts[0].fileName; + const archType = archiveSet.type === "7Z" ? ("SEVEN_Z" as const) : archiveSet.type; + if (archType !== "ZIP" && archType !== "RAR" && archType !== "SEVEN_Z") { + // The ranged listing readers only cover archive formats. Standalone + // DOCUMENT attachments always go through the existing download path, + // which for DOCUMENT is already cheap (no extraction, single entry). + return undefined; + } + + const scannedParts = archiveSet.parts.map((p) => ({ + fileId: p.fileId, + fileSize: p.fileSize, + fileName: p.fileName, + })); + + const entries = await readScannedListingRanged(archType, client, scannedParts); + if (!entries) return undefined; + + const totalArchiveSize = archiveSet.parts.reduce((sum, p) => sum + p.fileSize, 0n); + const firstRemoteUniqueId = archiveSet.parts[0].remoteUniqueId ?? null; + const contentHash = deriveForwardContentHash( + entries, + firstRemoteUniqueId, + channel.id, + archiveSet.parts[0].id, + ); + + if (await packageExistsByHash(contentHash)) { + counters.zipsDuplicate++; + accountLog.debug({ fileName: archiveName, contentHash }, "Forward-path duplicate (hash), skipping"); + return null; + } + + const repost = await checkFingerprintRepost(client, entries, archiveName, totalArchiveSize); + if (repost.isDuplicate) { + counters.zipsDuplicate++; + accountLog.info( + { fileName: archiveName, matchedPackageId: repost.matchedPackageId }, + "Forward-path duplicate (CRC fingerprint match against another channel's copy), skipping" + ); + return null; + } + + const hashLockAcquired = await tryAcquireHashLock(contentHash); + if (!hashLockAcquired) { + counters.zipsDuplicate++; + accountLog.info( + { fileName: archiveName, contentHash }, + "Hash lock held by another worker — skipping concurrent duplicate" + ); + return null; + } + + try { + if (await packageExistsByHash(contentHash)) { + counters.zipsDuplicate++; + return null; + } + + let destResult: { messageId: bigint; messageIds: bigint[] }; + try { + destResult = await forwardArchiveToChannel( + client, + channel.telegramId, + destChannelTelegramId, + archiveSet.parts.map((p) => p.id), + ); + } catch (forwardErr) { + accountLog.warn( + { err: forwardErr, fileName: archiveName }, + "Forward failed — falling back to download+reupload for this archive" + ); + return undefined; + } + + await deleteOrphanedPackageByHash(contentHash); + + const creator = + topicCreator ?? + extractCreatorFromFileName(archiveName) ?? + extractCreatorFromChannelTitle(channelTitle) ?? + null; + + const tags: string[] = []; + if (channel.category) tags.push(channel.category); + for (const tag of extractSlicerTags(entries)) { + if (!tags.includes(tag)) tags.push(tag); + } + + const stub = await createPackageStub({ + contentHash, + fileName: archiveName, + fileSize: totalArchiveSize, + archiveType: archType, + sourceChannelId: channel.id, + sourceMessageId: archiveSet.parts[0].id, + sourceTopicId, + remoteUniqueId: firstRemoteUniqueId, + destChannelId, + destMessageId: destResult.messageId, + destMessageIds: destResult.messageIds, + isMultipart: archiveSet.parts.length > 1, + partCount: archiveSet.parts.length, + ingestionRunId, + creator, + tags, + }); + + counters.zipsForwarded++; + await deleteSkippedPackage(channel.id, archiveSet.parts[0].id); + + let previewData: Buffer | null = null; + let previewMsgId: bigint | null = null; + const matchedPhoto = previewMatches.get(archiveSet.baseName); + if (matchedPhoto) { + previewData = await downloadPhotoThumbnail(client, matchedPhoto.fileId); + if (previewData) previewMsgId = matchedPhoto.id; + } + + await updatePackageWithMetadata(stub.id, { files: entries, previewData, previewMsgId }); + + accountLog.info( + { fileName: archiveName, contentHash, fileCount: entries.length, creator }, + "Archive forwarded (no download)" + ); + + return stub.id; + } finally { + await releaseHashLock(contentHash); + } +} +``` + +- [ ] **Step 6: Insert the fork point in `processOneArchiveSet`** + +Find the end of the size-guard block: + +```typescript + await upsertSkippedPackage({ + fileName: archiveName, + fileSize: totalArchiveSize, + reason: "SIZE_LIMIT", + sourceChannelId: channel.id, + sourceMessageId: archiveSet.parts[0].id, + sourceTopicId: ctx.sourceTopicId, + isMultipart: archiveSet.isMultipart, + partCount: archiveSet.parts.length, + accountId: ctx.accountId, + }); + return null; + } + + const tempPaths: string[] = []; +``` + +Replace with: + +```typescript + await upsertSkippedPackage({ + fileName: archiveName, + fileSize: totalArchiveSize, + reason: "SIZE_LIMIT", + sourceChannelId: channel.id, + sourceMessageId: archiveSet.parts[0].id, + sourceTopicId: ctx.sourceTopicId, + isMultipart: archiveSet.isMultipart, + partCount: archiveSet.parts.length, + accountId: ctx.accountId, + }); + return null; + } + + // ── Forward-priority path ── + // If the source channel allows forwarding, try to index + forward without a + // local download. Any failure (ranged listing miss, blocked/failed forward) + // falls through into the existing download pipeline below so indexing + // completeness never regresses. + if (channel.allowsForwarding === true) { + const forwardResult = await tryForwardArchiveSet( + ctx, archiveSet, setIdx, totalSets, previewMatches, ingestionRunId + ); + if (forwardResult !== undefined) { + return forwardResult; + } + accountLog.info( + { fileName: archiveName }, + "Forward path unavailable for this archive — falling back to download+reupload" + ); + } + + const tempPaths: string[] = []; +``` + +- [ ] **Step 7: Thread `zipsForwarded` through the final run summary** + +Find, near the end of `runWorkerForAccount`: + +```typescript + await throttled.flush(); + await completeIngestionRun(activeRunId, counters); + accountLog.info({ counters }, "Ingestion run completed"); +``` + +This already passes the whole `counters` object, which now includes `zipsForwarded` from Step 2 — no change needed here, just confirming it flows through. (No edit — verification only.) + +- [ ] **Step 8: Typecheck** + +```bash +cd worker && npx tsc --noEmit +``` +Expected: no errors. `archiveSet.parts[0].id` is `bigint` (from `TelegramMessage.id`); `channel.telegramId` is `bigint` (from Prisma) — both line up with `forwardArchiveToChannel`'s signature. + +- [ ] **Step 9: Commit** + +```bash +git add worker/src/db/queries.ts worker/src/worker.ts +git commit -m "feat(worker): fork to the forward-priority path in processOneArchiveSet" +``` + +--- + +## Task 9: Full typecheck + test suite + +- [ ] **Step 1: Run the full worker test suite** + +```bash +cd worker && npx tsc --noEmit && npx vitest run +``` +Expected: no TS errors; all tests pass (existing archive/ranged/*.test.ts tests plus all new tests added in Tasks 4-7). + +- [ ] **Step 2: Fix anything that fails, then re-run until clean.** + +--- + +## Task 10: Local build + deploy (no push) + +Same recipe as the `ranged-archive-listing` work — this repo does not push worker images to a registry; the container is rebuilt and recreated locally. + +- [ ] **Step 1: Build** + +```bash +cd /path/to/DragonsStash +docker build -f worker/Dockerfile -t git.samagsteribbe.nl/admin/dragonsstash-worker:latest . +``` +Expected: image builds successfully. + +- [ ] **Step 2: Recreate the running container** + +```bash +docker compose --project-name dragonsstash --project-directory /opt/stacks/DragonsStash \ + -f /opt/stacks/DragonsStash/docker-compose.yml up -d --no-deps --force-recreate worker +``` +Expected: `dragonsstash-worker` container recreated from the local image, starts cleanly. + +- [ ] **Step 3: Watch startup logs** + +```bash +docker logs -f --since 30s dragonsstash-worker +``` +Expected: no errors on startup; first ingestion cycle begins on schedule. + +--- + +## Task 11: Live verification + +Requires two test source channels already linked to a worker account: one with forwarding allowed, one with "restrict saving content" enabled (create/toggle via `toggleChatHasProtectedContent` or the Telegram client UI if you don't already have one). + +- [ ] **Step 1: Watch for the forwarding-permission detection** + +```bash +docker logs -f --since 30s dragonsstash-worker 2>&1 | grep -iE "Updated channel forwarding permission" +``` +Expected: both test channels get their `allowsForwarding` flag set correctly on the next scan cycle (confirm against the DB: `SELECT title, "allowsForwarding" FROM telegram_channels WHERE title IN ('', '');`). + +- [ ] **Step 2: Post a fresh archive into the forwarding-enabled test channel** + +```bash +docker logs -f --since 30s dragonsstash-worker 2>&1 | grep -iE "Archive forwarded|Forward path unavailable|Forward failed" +``` +Expected: `Archive forwarded (no download)` appears, with no matching `Downloading archive part` log line for that file. + +DB check: +```sql +SELECT "fileName", "contentHash", "fileCount", "destMessageId", "archiveType" +FROM packages ORDER BY "indexedAt" DESC LIMIT 5; +``` +Expected: the new row has `fileCount > 0`, a `contentHash` prefixed `fingerprint:` or `forward:`, and a non-null `destMessageId`. Spot-check `fileCount` against a real `unzip -l` / `unrar lt` / `7z l` on a manually-downloaded copy of the same file. + +- [ ] **Step 3: Post a fresh archive into the protected-content test channel** + +Expected: the existing download+reupload path runs unchanged (`Downloading archive part` appears in logs), and the resulting Package has a normal sha256-style `contentHash` (no `fingerprint:`/`forward:` prefix). + +- [ ] **Step 4: Force a ranged-listing failure in the forwarding-enabled channel** + +Post a deliberately-corrupted or unsupported-format archive (e.g. a password-protected-header 7z) into the forwarding-enabled test channel. + +Expected: `Forward path unavailable for this archive — falling back to download+reupload` appears, followed by the normal download pipeline completing successfully — the package still ends up with `fileCount > 0`. + +- [ ] **Step 5: Confirm bot delivery still works for a forwarded package** + +Use the bot to request the forwarded package from Step 2. Expected: delivery succeeds via the existing `copyMessageToUser` path (unaffected by this feature — it already reads `destMessageId` the same way regardless of how the package was ingested). + +--- + +## Self-Review + +- **Spec coverage:** sequencing/merge-first (Task 1) ✓; `allowsForwarding` schema + detection (Tasks 2-3) ✓; shared ranged-listing dispatcher (Task 4) ✓; dedup identity incl. rebuild:-style fallback chain (Task 5) ✓; cross-channel fingerprint repost check (Task 6) ✓; native forward (Task 7) ✓; fork point + fallback-to-download semantics + zipsForwarded observability (Task 8) ✓; typecheck/tests (Task 9) ✓; local no-push deploy (Task 10) ✓; live verification of both channel types plus the ranged-listing-failure fallback (Task 11) ✓. Non-goals from the spec (no ranged single-entry preview extraction, no reprocessing existing packages, no bot-delivery changes, no size-guard/split changes) require no tasks — confirmed nothing in this plan touches them. +- **Placeholder scan:** no TBD/TODO; every code step has full code; every command has an expected outcome. The one open item (`has_protected_content` field availability on supergroup/channel chats) is explicitly flagged as PENDING LIVE VERIFICATION in Task 3, Step 2's comment and covered by Task 11, Step 1 — not a placeholder, a spike already scheduled for live verification, matching this repo's own established convention for the same kind of TDLib-behavior uncertainty (`range-download.ts`'s absolute-offset note). +- **Type consistency:** `RangedPart` (`{fileId, fileSize, fileName}`), `FileEntry`, `PlaceholderCandidate`, `ForwardResult` (`{messageId, messageIds}`), and `deriveForwardContentHash`/`checkFingerprintRepost`/`forwardArchiveToChannel`/`tryForwardArchiveSet` signatures are consistent everywhere they're referenced across tasks. ✓