The pre-upload integrity test is advisory (never blocks upload), but it
raised a WARNING SystemNotification whenever `7z t` failed — most often
because large 7z archives OOM-kill the test process (SIGKILL / exit 137)
in the memory-limited worker container, which is a tool limitation, not
corruption. Classify failures as encrypted | corrupt | inconclusive;
suppress notifications for inconclusive (debug log only) while still
proceeding with the upload as before. Genuine corruption now uses the
INTEGRITY_AUDIT notification type instead of the misleading HASH_MISMATCH;
encrypted archives still notify.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three independent verification improvements landing together.
1. Pre-upload archive integrity test (testArchiveIntegrity)
Before sending an archive to the destination channel, runs the
appropriate CLI test:
- unzip -t for ZIP
- unrar t for RAR
- 7z t for SEVEN_Z
Catches truncated downloads, internal CRC errors, bad central
directories, and password-protected archives BEFORE we burn upload
bandwidth on a file that can't be extracted. Encrypted archives are
specifically flagged so the SkippedPackage error message is clear.
2. Post-upload destination read-back
updateMessageSendSucceeded tells us Telegram accepted the upload,
but says nothing about whether the destination message actually
contains the file we sent. After each successful upload, getMessage
each destMessageId and confirm document.size matches uploadPaths[i]'s
on-disk size.
Mismatches don't abort ingestion — they surface as
HASH_MISMATCH / UPLOAD_FAILED SystemNotifications so the admin can
see them in the UI and decide whether to recover.
3. Batched recovery (verifyMessagesBatch)
recoverIncompleteUploads previously called getMessage (singular)
per Package — at 20k packages that's 20k round-trips. Switched to
TDLib's getMessages (plural) with batch size 100 → 200 round-trips.
On 20k packages this is ~100x faster.
Per-message fallback if a whole batch errors out, so one bad batch
never loses all verification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Indexes 86k+ Lychee Slicer (.lys/.lyt), 23k+ ChituBox (.chitubox/.ctb/
.cbddlp), 1k+ Anycubic (.photon/.pwmo/.pwmx), and Bambu (.3mf)
slicer-specific files. Until now they were just generic extensions in
PackageFile.
After this commit:
- Newly-ingested packages get tags derived from their file list
("lychee", "chitubox", "anycubic", "bambu", "fdm", "mango")
- The `backfill_filelists` listener also applies tags to re-indexed
packages
- A new pure-DB listener `backfill_slicer_tags` walks existing
Packages with file lists and applies tags retroactively — no
downloads, no TDLib, takes seconds for thousands of rows.
Trigger the one-shot retroactive backfill with:
SELECT pg_notify('backfill_slicer_tags', '{"limit":5000}');
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fileName + size repost detection from ff4e150 works but has a
theoretical false-positive: two unrelated files in the same channel
with identical names and identical total sizes get treated as duplicates.
TDLib's document.remote.unique_id is a stable identifier per file
content — every repost of the exact same file across messages keeps
the same unique_id. Using it as the first dedup check eliminates the
false-positive risk entirely.
Schema:
- Package.remoteUniqueId (nullable, since existing rows lack it)
- Index on (sourceChannelId, remoteUniqueId)
Pipeline:
1. Capture remoteUniqueId in getChannelMessages + getTopicMessages
2. Pass through TelegramMessage type
3. processOneArchiveSet checks findPackageByRemoteUniqueId FIRST
(before packageExistsBySourceMessage / findRepostedPackage)
4. createPackageStub stores it on the new Package row
Existing 19,952 Packages have remoteUniqueId = NULL — they fall through
to the existing checks (source-msg-id, name+size, content-hash). New
ingestions populate it and benefit from the strong signal immediately.
Old Packages get backfilled organically when their content is
re-encountered and a new Package would otherwise be created.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diagnosed from production: all 4,380 RAR packages in the database have
fileCount = 0. The old parser used \`unrar l -v\` and a regex that
expected an 8-column \`Attributes Size Packed Ratio% Date Time CRC32 Name\`
output. unrar 6.21's actual \`l -v\` output is 5 columns: \`Attributes
Size Date Time Name\` — no Packed, no Ratio, no CRC32. So every RAR
silently parsed to zero entries.
Switch to \`unrar lt\` (list technical), which emits one block per file
with key:value lines:
Name: Lost Kingdom 2023 01 January/Nagas/NagaCaptainBody.stl
Type: File
Size: 22503584
Packed size: 21430123
CRC32: A1B2C3D4
...
The new parser tokenizes blocks on blank lines and matches "key: value"
lines per block. Handles multi-word keys ("Packed size", "Host OS") and
gracefully skips Directory entries and the archive header block. Also
tolerates BLAKE2sp checksums for newer RAR archives.
Verified against a live 644MB RAR with 201 entries (194 files, 7 dirs);
parser returns 194 entries with correct paths, sizes, and CRC32s.
Future RAR ingestions will populate fileCount and PackageFile rows
correctly. Backfilling existing 4,380 packages requires a separate
pass — added in a follow-up commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Group merge UI:
- Add mergeGroups query and mergeGroupsAction server action
- Add "Start Merge" / "Merge Here" buttons to group row actions
- Two-step UX: click Start on source, click Merge Here on target
ZIP path prefix grouping (Signal 7):
- Compare PackageFile.path root folders across ungrouped packages
- Auto-group if 2+ packages share the same dominant root folder
Reply chain grouping (Signal 6):
- Capture reply_to_message_id during channel scanning
- Group archives that reply to the same root message
- Add replyToMessageId field to Package schema
Caption fuzzy match grouping (Signal 8):
- Capture source caption during channel scanning
- Normalize captions (strip extensions, extract significant words)
- Group packages with matching normalized caption keys
- Add sourceCaption field to Package schema
Periodic integrity audit:
- Check multipart packages for completeness (parts vs destMessageIds)
- Detect orphaned indexes (destChannelId set but no destMessageId)
- Runs after each ingestion cycle, deduplicates notifications
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Raise WORKER_MAX_ZIP_SIZE_MB from 4GB to 200GB (production .env)
- Make MAX_PART_SIZE configurable via MAX_PART_SIZE_MB env var
(default 1950 MiB, set to 3900 for Premium accounts)
- Remove hardcoded 1950 MiB constants in split.ts and worker.ts
- Add grouping system audit report with real-world failure cases
10 archives were blocked by the 4GB limit (up to 70.5GB).
They will be retried on next ingestion cycle.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add downloadStarted flag to prevent false "stopped unexpectedly" errors
when TDLib emits initial updateFile before download is active
- Add 5-minute stall detection for both downloads and uploads
- Reduce max split part size from 2GiB to 1950MiB to stay under
Telegram's internal upload part count limits
- Increase timeouts from max(10min, 15min/GB) to max(15min, 20min/GB)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Auto-extract preview images from ZIP/RAR/7z archives during ingestion
- Upload custom preview images via package drawer
- Select preview from archive contents with on-demand extraction UI
- Manually add Telegram channels by t.me link, username, or invite link
- Invite code UX: bulk create, copy link, usage tracking, delete confirm
- Incomplete upload recovery: verify dest messages on worker startup
- Rebuild package DB by scanning destination channel with live progress
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add p7zip-full to worker Docker image
- New read7zContents() parser using 7z l output
- 7z archives now get full file listings like ZIP/RAR
- Standalone DOCUMENT types still show as single entry
- Add category field to TelegramChannel (filterable tag like STL, PDF, D&D)
- Category column in channels table with edit via dropdown menu
- Improved creator extraction: filename patterns + channel title fallback
- extractCreatorFromChannelTitle strips [Completed], (Paid), emoji, etc.
- Fix ArchiveType in PackageListItem and PackageRow for new types
- Add Prisma migration for category column
Adds full Telegram ZIP ingestion pipeline: TDLib worker service scans source
channels for archive files, deduplicates by content hash, extracts metadata,
uploads to archive channel, and indexes in Postgres. Forum supergroups are
scanned per-topic with topic names used as creator. Filename-based creator
extraction (e.g. "Mammoth Factory - 2026-01.zip") serves as fallback.
Includes admin UI for managing accounts/channels, simplified account setup
(API credentials via env vars), auth code/password submission dialog,
package browser with creator column, and live ingestion activity tracking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
adminandClaude Opus 4.8