dragonsstash/docs/superpowers/specs/2026-07-23-provenance-backfill-design.md

Provenance Backfill on Re-Index — Design

Date: 2026-07-23 Status: Approved for planning

Summary

Recover the true origin of packages whose recorded "source" is a placeholder — specifically the manual-upload and rebuild.ts-created records whose sourceChannelId points at the destination (archive) channel itself. When a real source channel is re-indexed and the worker encounters an archive that matches such a placeholder package, it backfills the real sourceChannelId / sourceMessageId / sourceTopicId / sourceCaption / creator (and, for records that never had one, the file listing and preview) onto the existing package — without downloading the full archive.

Matching is a two-stage process: a zero-download candidate lookup by fileName + total fileSize, confirmed by a CRC32 fingerprint read from just the archive's central directory via a ranged (tail) download of a few KB–MB, instead of the multi-GB whole file.

This is the first of four linked sub-projects. The others (creator normalization, provenance display, missing-files) are out of scope here and get their own spec → plan cycles. "Missing files" is explicitly deferred until this lands.

Context

Current state (as of this design):

• Package records their origin via sourceChannelId, sourceMessageId, sourceTopicId, sourceCaption, and (for content dedup) contentHash and remoteUniqueId. creator is a free-text string extracted at ingestion.
• Two ingestion paths create records with placeholder provenance, where the "source" is the destination channel, not a real origin:
  • Manual uploads (worker/src/manual-upload.ts) set sourceChannelId = destChannel.id and sourceMessageId = destResult.messageId. They do populate PackageFile (with crc32) from a local central-directory read at upload time.
  • rebuild.ts scans the destination channel and creates minimal records with fileCount == 0 and no PackageFile rows.
• The main worker upload path (worker/src/worker.ts → uploadToChannel) sends archives to the destination channel with no caption, so provenance cannot be recovered by re-reading destination messages — it has to come from matching against real source channels.
• The scan/dedup ladder in processOneArchiveSet (worker/src/worker.ts:1521) is entirely source-channel-scoped:
  1. findPackageByRemoteUniqueId(channel.id, …) — same channel only
  2. packageExistsBySourceMessage(channel.id, …) — same channel only
  3. findRepostedPackage(channel.id, fileName, size) — same channel only
  4. …then full download → packageExistsByHash(contentHash) — global, but only after downloading the whole archive.
• Because a placeholder package's sourceChannelId is the destination, checks 1–3 never match it when a real source channel is scanned. Today the worker therefore downloads the entire archive, hits check 4, finds it is a duplicate, and skips — wasting the download and leaving the wrong provenance in place.
• There is already an in-production precedent for "match an already-known archive during scan and enrich its metadata": when findRepostedPackage matches, the worker backfills richer topic context onto the existing package via updatePackageTopicContext (worker/src/worker.ts:1608+). Provenance backfill is the same move, widened from same-channel to cross-channel.
• remote.unique_id is not a reliable cross-channel key: it identifies a stored file object on Telegram's servers, not the content. The same archive independently uploaded to two channels gets different unique_ids; it only matches for forwarded messages (same underlying file object). This is why the existing dedup scopes it to a single channel.

Goals / Non-Goals

Goals

• Attribute true origin to placeholder-provenance packages during normal re-index scans, opportunistically (no separate pass).
• Do it without downloading whole archives (ranged central-directory read only).
• Be non-destructive: only ever touch packages that currently have placeholder provenance; never overwrite a real, non-placeholder source.
• Be idempotent: re-running a re-index does not re-mutate or duplicate.

Non-Goals (separate sub-projects / deferred)

• Creator name normalization / canonical creator entity.
• UI changes to display provenance.
• Recovering "missing files."
• Choosing a preferred origin when an archive genuinely exists in several source channels (first confirmed source wins).

Design

1. Candidate definition

A package is a backfill candidate iff sourceChannelId == destChannelId OR sourceMessageId == 0.

Two placeholder shapes exist (verified against live data 2026-07-23):

• Manual uploads (manual-upload.ts): sourceChannelId == destChannelId, real contentHash, real sourceMessageId, has a PackageFile listing.
• Rebuild records (rebuild.ts): sourceMessageId == 0n (deliberate "unknown" sentinel), synthetic contentHash = "rebuild:<destChannelId>:<destMessageId>", fileCount == 0, and sourceChannelId set to an arbitrary fallback source channel (sourceChannels[0]) — NOT the destination. (This is the common case: e.g. 59,893 records after a destination rebuild.)

Normal ingestion always sets a real sourceMessageId (> 0) and a real source channel, so neither marker matches a genuinely-sourced package. Both markers are overwritten on backfill (source channel + message become real), so a record stops being a candidate once fixed — this is what makes re-scans idempotent.

Known limitation: backfill does NOT rewrite a rebuild record's synthetic "rebuild:" contentHash (the true content hash would require a full download, which this feature avoids). That is acceptable — dedup after backfill relies on remoteUniqueId + name/size within the source channel, not on contentHash.

2. Where it hooks

A new step is inserted into processOneArchiveSet between check #3 (findRepostedPackage) and the full download. It runs only after the same-channel checks have missed (so genuine same-channel reposts keep their existing fast paths).

Flow for the scanned archive set:

1. Stage A — candidate lookup (zero download). Query for a package where (sourceChannelId == destChannelId OR sourceMessageId == 0) AND fileName == archiveName AND fileSize == totalArchiveSize. (Package has @@index([fileName]).)
   • No candidate → fall through to normal ingestion unchanged.
2. Stage B — fingerprint confirmation (tiny download). See §3.
3. On confirmation → backfill (see §4) and return null (treated as a duplicate; no full download, no new package). Increment a zipsBackfilled counter.
4. On mismatch / failure / ambiguity → fall through to normal ingestion (see §6).

3. Fingerprint confirmation

The confirmation signal is the multiset of internal CRC32s of the archive's entries (CRC32 of each entry's uncompressed data). This value is a property of the archive contents and is identical regardless of which channel hosts the file.

• Candidate side: for manual uploads, PackageFile.crc32 is already populated from the local central-directory read — zero cost. For rebuild records (fileCount == 0, no crc32), there is nothing stored to compare against; obtain the candidate's fingerprint with a second ranged tail read of its destination copy (ZIP/7z — still no full download). RAR candidates cannot be tail-read on either side, so they fall back to name+size (see §5).
• Scanned-source side: read via a ranged (tail) download of the central directory:
  • ZIP (incl. multipart): the End-of-Central-Directory record + central directory live at the tail of the last part. Download only that tail and parse entries. (Reuses the lightweight-listing mechanism that is sub-project 4's core.)
  • 7z: a start header at byte 0 points to an end header at the tail; fetch both small pieces.
  • RAR: headers are scattered through the file — no cheap tail read. See §5.
• Match rule: confirmed iff the sorted CRC32 multisets are equal and file counts are equal.

The comparison logic (CRC32 multiset equality) and the candidate-match predicate are implemented as pure functions with no TDLib dependency, so they are unit testable (see §7).

4. What gets written

On a confirmed match, update the candidate package in a single transaction, overwriting only placeholder/empty fields:

Field	New value	Condition
sourceChannelId	scanned channel.id	always
sourceMessageId	scanned parts[0].id	always
sourceTopicId	ctx.sourceTopicId	always
sourceCaption	scanned message caption	always
remoteUniqueId	scanned firstRemoteUniqueId	always (enables future same-channel dedup via check #1)
creator	re-derived from source (topic > filename > channel)	always — un-normalized; the later creator-normalization sub-project cleans it up
fileCount + PackageFile[]	from the central-directory read	only if candidate had none (fileCount == 0) — folds in the sub-project 4 outcome for rebuild records
previewData / previewMsgId	matched preview from scan	only if candidate has none

Left untouched: contentHash, destChannelId, destMessageId, destMessageIds — the bytes physically live in the destination channel; that is correct and must not change.

Transaction safety: re-check sourceChannelId == destChannelId inside the transaction before writing, so a concurrent worker that already backfilled the record causes this one to no-op (mirrors the existing backfill.ts guard).

5. RAR handling

RAR sources cannot be tail-read, so no cheap CRC fingerprint is available. Decision: backfill RAR matches on fileName + fileSize alone, flagged as lower-confidence in logs and via a SystemNotification, so they can be audited. No full download.

This name+size-only fallback applies to any candidate that cannot produce a CRC32 fingerprint cheaply: a RAR source, a RAR candidate, or a rebuild candidate whose destination copy is RAR. ZIP/7z rebuild candidates are still confirmed by fingerprint via the second tail read described in §3.

6. Conflict, mismatch, ambiguity

• Fingerprint mismatch → the scanned archive is genuinely different content that merely shares name+size. Fall through to normal ingestion (download + index) — it is a new package for this source.
• Ambiguous candidates (2+ match name+size and the fingerprint cannot disambiguate) → log a SystemNotification, backfill nothing, and fall through to normal ingestion; the post-download packageExistsByHash check still dedups it safely.
• Same archive in multiple source channels → the first re-indexed source that confirms wins. After backfill the package has a real source, so it is no longer a candidate; later scans treat it as an ordinary duplicate via checks #1 (the remoteUniqueId we set) or #3.

7. Idempotency

Falls out of the candidate definition. Once backfilled:

• sourceChannelId is the real channel → no longer a candidate.
• A re-scan of that source hits check #1 (remoteUniqueId, now set) or check #3 (findRepostedPackage) → normal dedup skip. No re-mutation, no duplicate.

8. Error handling

• Tail-download failure (network / FLOOD_WAIT) → cannot confirm this round. Do not fall back to a full download and do not guess: leave the candidate untouched and let the next re-index retry. Wrap the ranged read in withFloodWait (per the TDLib skill).
• All new TDLib calls follow the skill's patterns: withFloodWait, listener attached before the async op, client closed in finally.

9. Visibility

• Add a zipsBackfilled counter to IngestionRun activity so a re-index run reports "N provenance backfills" instead of silently mutating records.
• Info log per backfill (candidate id, old vs new source, confidence: fingerprint | name+size-RAR).
• SystemNotification for ambiguous-candidate cases.

Testing

The repo currently has no test framework (CLAUDE.md: "testing is manual"). This work introduces a lightweight test harness for the worker (e.g. vitest or node's built-in node:test) and unit-tests the correctness-critical pure logic:

• Unit (automated):
  • CRC32 multiset fingerprint equality (equal sets, different order, differing counts, disjoint sets).
  • Candidate-match predicate (name+size+placeholder true/false cases).
  • Field-merge rules (which fields overwrite, which only fill-if-empty).
• Manual integration checklist:
  1. Re-index a real source channel containing a known manually-uploaded pack → verify sourceChannelId/sourceMessageId/sourceCaption/creator are backfilled and no full download occurs.
  2. A rebuild-created record (fileCount == 0) → verify listing + provenance are both populated from the single tail read.
  3. A RAR pack → verify name+size backfill with the lower-confidence log/notice.
  4. Re-run the same re-index → verify it is a no-op (idempotent).
  5. A genuine name+size collision (different content) → verify it ingests as a new package rather than being mis-attributed.

Open Questions

None blocking. Preferred-origin selection among multiple real sources is deliberately out of scope (first confirmed wins).

Affected Code (indicative, for planning)

• worker/src/worker.ts — processOneArchiveSet: new Stage A/B step; new counter.
• worker/src/archive/ — ranged central-directory reader (ZIP/7z tail); shared with sub-project 4. Pure CRC32-fingerprint compare helper.
• worker/src/tdlib/download.ts — ranged/partial download support (offset/limit).
• worker/src/db/queries.ts — candidate lookup + transactional backfill update.
• prisma/schema.prisma — IngestionRun.zipsBackfilled (and run-counter plumbing).
• Worker test harness + first unit tests.