mirror of
https://github.com/xCyanGrizzly/DragonsStash.git
synced 2026-07-24 07:42:47 +00:00
feat(verify): pre-upload integrity test, post-upload read-back, batched recovery
All checks were successful
continuous-integration/drone/push Build is passing
All checks were successful
continuous-integration/drone/push Build is passing
Three independent verification improvements landing together.
1. Pre-upload archive integrity test (testArchiveIntegrity)
Before sending an archive to the destination channel, runs the
appropriate CLI test:
- unzip -t for ZIP
- unrar t for RAR
- 7z t for SEVEN_Z
Catches truncated downloads, internal CRC errors, bad central
directories, and password-protected archives BEFORE we burn upload
bandwidth on a file that can't be extracted. Encrypted archives are
specifically flagged so the SkippedPackage error message is clear.
2. Post-upload destination read-back
updateMessageSendSucceeded tells us Telegram accepted the upload,
but says nothing about whether the destination message actually
contains the file we sent. After each successful upload, getMessage
each destMessageId and confirm document.size matches uploadPaths[i]'s
on-disk size.
Mismatches don't abort ingestion — they surface as
HASH_MISMATCH / UPLOAD_FAILED SystemNotifications so the admin can
see them in the UI and decide whether to recover.
3. Batched recovery (verifyMessagesBatch)
recoverIncompleteUploads previously called getMessage (singular)
per Package — at 20k packages that's 20k round-trips. Switched to
TDLib's getMessages (plural) with batch size 100 → 200 round-trips.
On 20k packages this is ~100x faster.
Per-message fallback if a whole batch errors out, so one bad batch
never loses all verification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -48,6 +48,7 @@ import { groupArchiveSets } from "./archive/multipart.js";
|
||||
import type { ArchiveSet } from "./archive/multipart.js";
|
||||
import { extractCreatorFromFileName, extractCreatorFromChannelTitle } from "./archive/creator.js";
|
||||
import { extractSlicerTags } from "./archive/slicer-tags.js";
|
||||
import { testArchiveIntegrity } from "./archive/integrity.js";
|
||||
import { hashParts } from "./archive/hash.js";
|
||||
import { readZipCentralDirectory } from "./archive/zip-reader.js";
|
||||
import { readRarContents } from "./archive/rar-reader.js";
|
||||
@@ -1653,6 +1654,19 @@ async function processOneArchiveSet(
|
||||
);
|
||||
}
|
||||
|
||||
// ── Pre-upload integrity test ──
|
||||
// Catch broken/encrypted archives before we burn upload bandwidth on
|
||||
// them. Cheap (unzip -t / unrar t / 7z t) compared to a multi-GB upload.
|
||||
// Skipped when we're reusing an existing upload — no point testing the
|
||||
// file again.
|
||||
const integrity = await testArchiveIntegrity(
|
||||
archiveSet.type === "7Z" ? "SEVEN_Z" : archiveSet.type,
|
||||
uploadPaths[0]
|
||||
);
|
||||
if (!integrity.ok) {
|
||||
throw new Error(`Archive integrity check failed: ${integrity.reason}`);
|
||||
}
|
||||
|
||||
// ── Uploading ──
|
||||
// Check if a prior run already uploaded this file (orphaned upload scenario:
|
||||
// file reached Telegram but DB write failed or worker crashed before indexing)
|
||||
@@ -1718,6 +1732,72 @@ async function processOneArchiveSet(
|
||||
}
|
||||
}
|
||||
|
||||
// ── Destination read-back verification ──
|
||||
// Telegram's updateMessageSendSucceeded fires when TG acknowledges the
|
||||
// message, but that's separate from "the message is queryable and
|
||||
// contains the file we sent". Fetch each destination message and
|
||||
// confirm the document's size matches what we uploaded.
|
||||
//
|
||||
// Skipped when reusing an existing upload (we never sent anything).
|
||||
// Failures here surface as a SystemNotification but DO NOT abort the
|
||||
// ingestion — the Package will be created with whatever destMessageIds
|
||||
// Telegram returned, and a future recovery run can reset it if needed.
|
||||
if (!existingUpload && destResult.messageIds.length > 0) {
|
||||
try {
|
||||
const expectedSizes = uploadPaths.length === destResult.messageIds.length
|
||||
? await Promise.all(
|
||||
uploadPaths.map(async (p) => (await import("fs/promises")).stat(p).then((s) => s.size))
|
||||
)
|
||||
: null;
|
||||
|
||||
for (let i = 0; i < destResult.messageIds.length; i++) {
|
||||
const msgId = Number(destResult.messageIds[i]);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const tdMsg = (await client.invoke({
|
||||
_: "getMessage",
|
||||
chat_id: Number(destChannelTelegramId),
|
||||
message_id: msgId,
|
||||
}).catch(() => null)) as any;
|
||||
|
||||
const doc = tdMsg?.content?.document?.document;
|
||||
const actualSize = doc?.size;
|
||||
const expected = expectedSizes?.[i];
|
||||
|
||||
if (!actualSize) {
|
||||
accountLog.warn(
|
||||
{ fileName: archiveName, destMessageId: msgId },
|
||||
"Post-upload read-back: destination message has no document content"
|
||||
);
|
||||
await db.systemNotification.create({
|
||||
data: {
|
||||
type: "UPLOAD_FAILED",
|
||||
severity: "WARNING",
|
||||
title: `Read-back failed: ${archiveName}`,
|
||||
message: `Destination message ${msgId} has no document content after upload. The upload may have failed silently.`,
|
||||
context: { fileName: archiveName, destMessageId: msgId, sourceChannelId: channel.id },
|
||||
},
|
||||
});
|
||||
} else if (expected !== undefined && actualSize !== expected) {
|
||||
accountLog.error(
|
||||
{ fileName: archiveName, destMessageId: msgId, expectedSize: expected, actualSize },
|
||||
"Post-upload read-back: destination file size mismatch"
|
||||
);
|
||||
await db.systemNotification.create({
|
||||
data: {
|
||||
type: "HASH_MISMATCH",
|
||||
severity: "ERROR",
|
||||
title: `Read-back size mismatch: ${archiveName}`,
|
||||
message: `Sent ${expected} bytes but destination message ${msgId} contains a ${actualSize}-byte file.`,
|
||||
context: { fileName: archiveName, destMessageId: msgId, expectedSize: expected, actualSize, sourceChannelId: channel.id },
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch (readBackErr) {
|
||||
accountLog.warn({ err: readBackErr, fileName: archiveName }, "Post-upload read-back failed (non-fatal)");
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase 1: Stub record — persisted immediately after upload ──
|
||||
await deleteOrphanedPackageByHash(contentHash);
|
||||
|
||||
|
||||
Reference in New Issue
Block a user