mirror of
https://github.com/xCyanGrizzly/DragonsStash.git
synced 2026-09-21 13:31:42 +00:00
docs: correct provenance-backfill candidate predicate for rebuild records
Rebuild records use sourceMessageId=0 + synthetic 'rebuild:' contentHash and an arbitrary fallback sourceChannelId, so the original sourceChannelId==destChannelId candidate definition missed them. Predicate is now (sourceChannelId==destChannelId OR sourceMessageId==0), verified against 59,893 live rebuilt records. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -493,7 +493,7 @@ git commit -m "feat(worker): ranged TDLib file download (downloadFileRange)"
|
|||||||
|
|
||||||
**Interfaces:**
|
**Interfaces:**
|
||||||
- Produces:
|
- Produces:
|
||||||
- `findPlaceholderCandidate(destChannelId: string, fileName: string, fileSize: bigint): Promise<{ id: string; archiveType: string; fileCount: number } | null>` — a package where `sourceChannelId === destChannelId` (placeholder) AND `fileName` AND `fileSize` match, with a real destination (`destMessageId != null`).
|
- `findPlaceholderCandidate(destChannelId: string, fileName: string, fileSize: bigint): Promise<{ id: string; archiveType: string; fileCount: number } | null>` — a **placeholder** package (`sourceChannelId === destChannelId` OR `sourceMessageId === 0` — see spec §1) AND `fileName` AND `fileSize` match, with a real destination (`destMessageId != null`).
|
||||||
- `getPackageFileCrcs(packageId: string): Promise<(string | null)[]>` — `PackageFile.crc32` values for the candidate.
|
- `getPackageFileCrcs(packageId: string): Promise<(string | null)[]>` — `PackageFile.crc32` values for the candidate.
|
||||||
- `backfillProvenance(input: BackfillProvenanceInput): Promise<boolean>` — transactionally overwrite placeholder fields; returns `false` (no-op) if the row is no longer a placeholder. Type:
|
- `backfillProvenance(input: BackfillProvenanceInput): Promise<boolean>` — transactionally overwrite placeholder fields; returns `false` (no-op) if the row is no longer a placeholder. Type:
|
||||||
```ts
|
```ts
|
||||||
@@ -522,10 +522,15 @@ export async function findPlaceholderCandidate(
|
|||||||
): Promise<{ id: string; archiveType: string; fileCount: number } | null> {
|
): Promise<{ id: string; archiveType: string; fileCount: number } | null> {
|
||||||
return db.package.findFirst({
|
return db.package.findFirst({
|
||||||
where: {
|
where: {
|
||||||
sourceChannelId: destChannelId, // placeholder: source == destination
|
|
||||||
fileName,
|
fileName,
|
||||||
fileSize,
|
fileSize,
|
||||||
destMessageId: { not: null },
|
destMessageId: { not: null },
|
||||||
|
// Placeholder provenance (spec §1): manual-upload (source == destination)
|
||||||
|
// OR rebuild record (sourceMessageId == 0 "unknown" sentinel).
|
||||||
|
OR: [
|
||||||
|
{ sourceChannelId: destChannelId },
|
||||||
|
{ sourceMessageId: 0n },
|
||||||
|
],
|
||||||
},
|
},
|
||||||
select: { id: true, archiveType: true, fileCount: true },
|
select: { id: true, archiveType: true, fileCount: true },
|
||||||
orderBy: { indexedAt: "asc" },
|
orderBy: { indexedAt: "asc" },
|
||||||
@@ -544,10 +549,14 @@ export async function backfillProvenance(input: BackfillProvenanceInput): Promis
|
|||||||
return db.$transaction(async (tx) => {
|
return db.$transaction(async (tx) => {
|
||||||
const current = await tx.package.findUnique({
|
const current = await tx.package.findUnique({
|
||||||
where: { id: input.packageId },
|
where: { id: input.packageId },
|
||||||
select: { sourceChannelId: true, previewData: true, fileCount: true },
|
select: { sourceChannelId: true, sourceMessageId: true, previewData: true, fileCount: true },
|
||||||
});
|
});
|
||||||
// Re-check placeholder status inside the txn (another worker may have won).
|
// Re-check placeholder status inside the txn (another worker may have won).
|
||||||
if (!current || current.sourceChannelId !== input.destChannelId) return false;
|
// Placeholder = manual-upload (source==dest) OR rebuild (sourceMessageId==0).
|
||||||
|
const stillPlaceholder =
|
||||||
|
!!current &&
|
||||||
|
(current.sourceChannelId === input.destChannelId || current.sourceMessageId === 0n);
|
||||||
|
if (!stillPlaceholder) return false;
|
||||||
|
|
||||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
const data: any = {
|
const data: any = {
|
||||||
|
|||||||
@@ -88,10 +88,27 @@ Current state (as of this design):
|
|||||||
|
|
||||||
### 1. Candidate definition
|
### 1. Candidate definition
|
||||||
|
|
||||||
> A package is a backfill candidate iff `sourceChannelId == destChannelId`.
|
> A package is a backfill candidate iff **`sourceChannelId == destChannelId`
|
||||||
|
> OR `sourceMessageId == 0`**.
|
||||||
|
|
||||||
These are exactly the manual-upload and rebuild-created records. Packages with a
|
Two placeholder shapes exist (verified against live data 2026-07-23):
|
||||||
real, non-placeholder source are **never** candidates and are never overwritten.
|
- **Manual uploads** (`manual-upload.ts`): `sourceChannelId == destChannelId`,
|
||||||
|
real `contentHash`, real `sourceMessageId`, has a `PackageFile` listing.
|
||||||
|
- **Rebuild records** (`rebuild.ts`): `sourceMessageId == 0n` (deliberate
|
||||||
|
"unknown" sentinel), synthetic `contentHash = "rebuild:<destChannelId>:<destMessageId>"`,
|
||||||
|
`fileCount == 0`, and `sourceChannelId` set to an **arbitrary fallback source
|
||||||
|
channel** (`sourceChannels[0]`) — NOT the destination. (This is the common
|
||||||
|
case: e.g. 59,893 records after a destination rebuild.)
|
||||||
|
|
||||||
|
Normal ingestion always sets a real `sourceMessageId` (> 0) and a real source
|
||||||
|
channel, so neither marker matches a genuinely-sourced package. Both markers are
|
||||||
|
overwritten on backfill (source channel + message become real), so a record
|
||||||
|
stops being a candidate once fixed — this is what makes re-scans idempotent.
|
||||||
|
|
||||||
|
**Known limitation:** backfill does NOT rewrite a rebuild record's synthetic
|
||||||
|
`"rebuild:"` `contentHash` (the true content hash would require a full download,
|
||||||
|
which this feature avoids). That is acceptable — dedup after backfill relies on
|
||||||
|
`remoteUniqueId` + name/size within the source channel, not on `contentHash`.
|
||||||
|
|
||||||
### 2. Where it hooks
|
### 2. Where it hooks
|
||||||
|
|
||||||
@@ -103,8 +120,9 @@ existing fast paths).
|
|||||||
Flow for the scanned archive set:
|
Flow for the scanned archive set:
|
||||||
|
|
||||||
1. Stage A — **candidate lookup (zero download).** Query for a package where
|
1. Stage A — **candidate lookup (zero download).** Query for a package where
|
||||||
`sourceChannelId == destChannelId` AND `fileName == archiveName` AND
|
`(sourceChannelId == destChannelId OR sourceMessageId == 0)` AND
|
||||||
`fileSize == totalArchiveSize`. (`Package` has `@@index([fileName])`.)
|
`fileName == archiveName` AND `fileSize == totalArchiveSize`.
|
||||||
|
(`Package` has `@@index([fileName])`.)
|
||||||
- No candidate → fall through to normal ingestion unchanged.
|
- No candidate → fall through to normal ingestion unchanged.
|
||||||
2. Stage B — **fingerprint confirmation (tiny download).** See §3.
|
2. Stage B — **fingerprint confirmation (tiny download).** See §3.
|
||||||
3. On confirmation → **backfill** (see §4) and return `null` (treated as a
|
3. On confirmation → **backfill** (see §4) and return `null` (treated as a
|
||||||
|
|||||||
Reference in New Issue
Block a user