mirror of
https://github.com/xCyanGrizzly/DragonsStash.git
synced 2026-06-09 18:51:16 +00:00
xCyanGrizzly
7d39a13310
The fileName + size repost detection from ff4e150 works but has a
theoretical false-positive: two unrelated files in the same channel
with identical names and identical total sizes get treated as duplicates.
TDLib's document.remote.unique_id is a stable identifier per file
content — every repost of the exact same file across messages keeps
the same unique_id. Using it as the first dedup check eliminates the
false-positive risk entirely.
Schema:
- Package.remoteUniqueId (nullable, since existing rows lack it)
- Index on (sourceChannelId, remoteUniqueId)
Pipeline:
1. Capture remoteUniqueId in getChannelMessages + getTopicMessages
2. Pass through TelegramMessage type
3. processOneArchiveSet checks findPackageByRemoteUniqueId FIRST
(before packageExistsBySourceMessage / findRepostedPackage)
4. createPackageStub stores it on the new Package row
Existing 19,952 Packages have remoteUniqueId = NULL — they fall through
to the existing checks (source-msg-id, name+size, content-hash). New
ingestions populate it and benefit from the strong signal immediately.
Old Packages get backfilled organically when their content is
re-encountered and a new Package would otherwise be created.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
11 lines
573 B
SQL
11 lines
573 B
SQL
-- AlterTable: capture TDLib's stable per-content identifier for new packages.
|
|
-- Existing rows are NULL; they fall through to the other dedup checks until
|
|
-- they're re-encountered organically.
|
|
ALTER TABLE "packages" ADD COLUMN "remoteUniqueId" TEXT;
|
|
|
|
-- CreateIndex: scoped to source channel because we want to dedup
|
|
-- per-channel (the same file appearing in two different channels is still
|
|
-- worth indexing twice — they're different ingestion sources).
|
|
CREATE INDEX "packages_sourceChannelId_remoteUniqueId_idx"
|
|
ON "packages"("sourceChannelId", "remoteUniqueId");
|