mirror of
https://github.com/xCyanGrizzly/DragonsStash.git
synced 2026-07-22 23:12:02 +00:00
feat(worker): use TDLib remote.unique_id as zero-false-positive dedup signal
The fileName + size repost detection from ff4e150 works but has a
theoretical false-positive: two unrelated files in the same channel
with identical names and identical total sizes get treated as duplicates.
TDLib's document.remote.unique_id is a stable identifier per file
content — every repost of the exact same file across messages keeps
the same unique_id. Using it as the first dedup check eliminates the
false-positive risk entirely.
Schema:
- Package.remoteUniqueId (nullable, since existing rows lack it)
- Index on (sourceChannelId, remoteUniqueId)
Pipeline:
1. Capture remoteUniqueId in getChannelMessages + getTopicMessages
2. Pass through TelegramMessage type
3. processOneArchiveSet checks findPackageByRemoteUniqueId FIRST
(before packageExistsBySourceMessage / findRepostedPackage)
4. createPackageStub stores it on the new Package row
Existing 19,952 Packages have remoteUniqueId = NULL — they fall through
to the existing checks (source-msg-id, name+size, content-hash). New
ingestions populate it and benefit from the strong signal immediately.
Old Packages get backfilled organically when their content is
re-encountered and a new Package would otherwise be created.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -82,6 +82,8 @@ export interface CreatePackageStubInput {
|
||||
sourceChannelId: string;
|
||||
sourceMessageId: bigint;
|
||||
sourceTopicId?: bigint | null;
|
||||
/** TDLib remote.unique_id of the first part — for future dedup. */
|
||||
remoteUniqueId?: string | null;
|
||||
destChannelId: string;
|
||||
destMessageId: bigint;
|
||||
destMessageIds: bigint[];
|
||||
@@ -111,6 +113,7 @@ export async function createPackageStub(
|
||||
sourceChannelId: input.sourceChannelId,
|
||||
sourceMessageId: input.sourceMessageId,
|
||||
sourceTopicId: input.sourceTopicId ?? undefined,
|
||||
remoteUniqueId: input.remoteUniqueId ?? undefined,
|
||||
destChannelId: input.destChannelId,
|
||||
destMessageId: input.destMessageId,
|
||||
destMessageIds: input.destMessageIds,
|
||||
@@ -189,6 +192,34 @@ export async function packageExistsBySourceMessage(
|
||||
return pkg !== null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Strongest pre-download dedup signal: a Package in this channel already
|
||||
* has a matching TDLib remote.unique_id. The unique_id is stable across
|
||||
* reposts of the exact same file content, so a hit is a guaranteed
|
||||
* (lossless) duplicate. No false positives.
|
||||
*
|
||||
* Falls back to the older findRepostedPackage (name + size) for packages
|
||||
* that were ingested before we started capturing remote.unique_id.
|
||||
*/
|
||||
export async function findPackageByRemoteUniqueId(
|
||||
sourceChannelId: string,
|
||||
remoteUniqueId: string
|
||||
): Promise<{
|
||||
id: string;
|
||||
destMessageId: bigint | null;
|
||||
sourceTopicId: bigint | null;
|
||||
} | null> {
|
||||
return db.package.findFirst({
|
||||
where: {
|
||||
sourceChannelId,
|
||||
remoteUniqueId,
|
||||
destMessageId: { not: null },
|
||||
},
|
||||
orderBy: { sourceTopicId: { sort: "desc", nulls: "last" } },
|
||||
select: { id: true, destMessageId: true, sourceTopicId: true },
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect a likely repost: same source channel + same fileName + same total
|
||||
* fileSize already exists with destMessageId set. Used to skip downloads
|
||||
|
||||
Reference in New Issue
Block a user