docs: correct provenance-backfill candidate predicate for rebuild records

Rebuild records use sourceMessageId=0 + synthetic 'rebuild:' contentHash and an
arbitrary fallback sourceChannelId, so the original sourceChannelId==destChannelId
candidate definition missed them. Predicate is now
(sourceChannelId==destChannelId OR sourceMessageId==0), verified against 59,893
live rebuilt records.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-23 13:05:34 +02:00
co-authored by Claude Opus 4.8
parent 018b0f5d74
commit 0bce1168a9
2 changed files with 36 additions and 9 deletions
@@ -493,7 +493,7 @@ git commit -m "feat(worker): ranged TDLib file download (downloadFileRange)"
**Interfaces:** **Interfaces:**
- Produces: - Produces:
- `findPlaceholderCandidate(destChannelId: string, fileName: string, fileSize: bigint): Promise<{ id: string; archiveType: string; fileCount: number } | null>` — a package where `sourceChannelId === destChannelId` (placeholder) AND `fileName` AND `fileSize` match, with a real destination (`destMessageId != null`). - `findPlaceholderCandidate(destChannelId: string, fileName: string, fileSize: bigint): Promise<{ id: string; archiveType: string; fileCount: number } | null>` — a **placeholder** package (`sourceChannelId === destChannelId` OR `sourceMessageId === 0` — see spec §1) AND `fileName` AND `fileSize` match, with a real destination (`destMessageId != null`).
- `getPackageFileCrcs(packageId: string): Promise<(string | null)[]>``PackageFile.crc32` values for the candidate. - `getPackageFileCrcs(packageId: string): Promise<(string | null)[]>``PackageFile.crc32` values for the candidate.
- `backfillProvenance(input: BackfillProvenanceInput): Promise<boolean>` — transactionally overwrite placeholder fields; returns `false` (no-op) if the row is no longer a placeholder. Type: - `backfillProvenance(input: BackfillProvenanceInput): Promise<boolean>` — transactionally overwrite placeholder fields; returns `false` (no-op) if the row is no longer a placeholder. Type:
```ts ```ts
@@ -522,10 +522,15 @@ export async function findPlaceholderCandidate(
): Promise<{ id: string; archiveType: string; fileCount: number } | null> { ): Promise<{ id: string; archiveType: string; fileCount: number } | null> {
return db.package.findFirst({ return db.package.findFirst({
where: { where: {
sourceChannelId: destChannelId, // placeholder: source == destination
fileName, fileName,
fileSize, fileSize,
destMessageId: { not: null }, destMessageId: { not: null },
// Placeholder provenance (spec §1): manual-upload (source == destination)
// OR rebuild record (sourceMessageId == 0 "unknown" sentinel).
OR: [
{ sourceChannelId: destChannelId },
{ sourceMessageId: 0n },
],
}, },
select: { id: true, archiveType: true, fileCount: true }, select: { id: true, archiveType: true, fileCount: true },
orderBy: { indexedAt: "asc" }, orderBy: { indexedAt: "asc" },
@@ -544,10 +549,14 @@ export async function backfillProvenance(input: BackfillProvenanceInput): Promis
return db.$transaction(async (tx) => { return db.$transaction(async (tx) => {
const current = await tx.package.findUnique({ const current = await tx.package.findUnique({
where: { id: input.packageId }, where: { id: input.packageId },
select: { sourceChannelId: true, previewData: true, fileCount: true }, select: { sourceChannelId: true, sourceMessageId: true, previewData: true, fileCount: true },
}); });
// Re-check placeholder status inside the txn (another worker may have won). // Re-check placeholder status inside the txn (another worker may have won).
if (!current || current.sourceChannelId !== input.destChannelId) return false; // Placeholder = manual-upload (source==dest) OR rebuild (sourceMessageId==0).
const stillPlaceholder =
!!current &&
(current.sourceChannelId === input.destChannelId || current.sourceMessageId === 0n);
if (!stillPlaceholder) return false;
// eslint-disable-next-line @typescript-eslint/no-explicit-any // eslint-disable-next-line @typescript-eslint/no-explicit-any
const data: any = { const data: any = {
@@ -88,10 +88,27 @@ Current state (as of this design):
### 1. Candidate definition ### 1. Candidate definition
> A package is a backfill candidate iff `sourceChannelId == destChannelId`. > A package is a backfill candidate iff **`sourceChannelId == destChannelId`
> OR `sourceMessageId == 0`**.
These are exactly the manual-upload and rebuild-created records. Packages with a Two placeholder shapes exist (verified against live data 2026-07-23):
real, non-placeholder source are **never** candidates and are never overwritten. - **Manual uploads** (`manual-upload.ts`): `sourceChannelId == destChannelId`,
real `contentHash`, real `sourceMessageId`, has a `PackageFile` listing.
- **Rebuild records** (`rebuild.ts`): `sourceMessageId == 0n` (deliberate
"unknown" sentinel), synthetic `contentHash = "rebuild:<destChannelId>:<destMessageId>"`,
`fileCount == 0`, and `sourceChannelId` set to an **arbitrary fallback source
channel** (`sourceChannels[0]`) — NOT the destination. (This is the common
case: e.g. 59,893 records after a destination rebuild.)
Normal ingestion always sets a real `sourceMessageId` (> 0) and a real source
channel, so neither marker matches a genuinely-sourced package. Both markers are
overwritten on backfill (source channel + message become real), so a record
stops being a candidate once fixed — this is what makes re-scans idempotent.
**Known limitation:** backfill does NOT rewrite a rebuild record's synthetic
`"rebuild:"` `contentHash` (the true content hash would require a full download,
which this feature avoids). That is acceptable — dedup after backfill relies on
`remoteUniqueId` + name/size within the source channel, not on `contentHash`.
### 2. Where it hooks ### 2. Where it hooks
@@ -103,8 +120,9 @@ existing fast paths).
Flow for the scanned archive set: Flow for the scanned archive set:
1. Stage A — **candidate lookup (zero download).** Query for a package where 1. Stage A — **candidate lookup (zero download).** Query for a package where
`sourceChannelId == destChannelId` AND `fileName == archiveName` AND `(sourceChannelId == destChannelId OR sourceMessageId == 0)` AND
`fileSize == totalArchiveSize`. (`Package` has `@@index([fileName])`.) `fileName == archiveName` AND `fileSize == totalArchiveSize`.
(`Package` has `@@index([fileName])`.)
- No candidate → fall through to normal ingestion unchanged. - No candidate → fall through to normal ingestion unchanged.
2. Stage B — **fingerprint confirmation (tiny download).** See §3. 2. Stage B — **fingerprint confirmation (tiny download).** See §3.
3. On confirmation → **backfill** (see §4) and return `null` (treated as a 3. On confirmation → **backfill** (see §4) and return `null` (treated as a