Manual file upload:
- Upload dialog in STL page with drag-and-drop file picker
- Files saved to shared Docker volume (/data/uploads)
- Worker processes via pg_notify('manual_upload') channel
- Hashes, reads metadata, splits >2GB, uploads to Telegram
- Multiple files automatically grouped
- Status polling shows upload/processing/complete states
Notification fixes:
- Add dismiss (X) button on each notification
- Add "Clear" button to remove all notifications
- Fix false positive MISSING_PART alerts from legacy packages
(only flag when >1 destMessageIds stored but count wrong,
not when only 1 ID from backfill)
Infrastructure:
- ManualUpload + ManualUploadFile schema + migration
- Shared manual_uploads Docker volume between app and worker
- Upload API routes (POST /api/uploads, GET /api/uploads/[id])
- Worker manual-upload processor with full pipeline
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Manual override training (GroupingRule):
- Learn patterns from manual group creation (common filename prefix or creator)
- Apply learned rules as first auto-grouping pass (highest confidence after albums)
- GroupingRule model stores pattern, channel, signal type, confidence
Hash verification after upload:
- Re-hash upload files on disk before indexing to catch disk corruption
- Creates HASH_MISMATCH notification on discrepancy
Grouping conflict detection:
- After all grouping passes, check if grouped packages match rules from different groups
- Creates GROUPING_CONFLICT notification for manual review
Per-channel grouping flags:
- Add autoGroupEnabled boolean to TelegramChannel (default true)
- Auto-grouping passes (all except album) gated behind this flag
- Album grouping always runs as it reflects Telegram's native behavior
Full-text search (tsvector):
- Add searchVector tsvector column with GIN index and auto-update trigger
- Backfill 1870 existing packages
- FTS with ts_rank for ranked results, ILIKE fallback for short/failed queries
- Applied to both web app and bot search
Bot group awareness:
- /group <query> — view group info or search groups by name
- /sendgroup <id> — send all packages in a group to linked Telegram account
Bulk repair:
- repairPackageAction clears dest info and resets watermark for re-processing
- Repair button in notification bell for MISSING_PART and HASH_MISMATCH alerts
- /api/notifications/repair endpoint
Retroactive category re-tagging:
- When channel category changes, auto-update tags on all existing packages
- Removes old category tag, adds new one
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Group merge UI:
- Add mergeGroups query and mergeGroupsAction server action
- Add "Start Merge" / "Merge Here" buttons to group row actions
- Two-step UX: click Start on source, click Merge Here on target
ZIP path prefix grouping (Signal 7):
- Compare PackageFile.path root folders across ungrouped packages
- Auto-group if 2+ packages share the same dominant root folder
Reply chain grouping (Signal 6):
- Capture reply_to_message_id during channel scanning
- Group archives that reply to the same root message
- Add replyToMessageId field to Package schema
Caption fuzzy match grouping (Signal 8):
- Capture source caption during channel scanning
- Normalize captions (strip extensions, extract significant words)
- Group packages with matching normalized caption keys
- Add sourceCaption field to Package schema
Periodic integrity audit:
- Check multipart packages for completeness (parts vs destMessageIds)
- Detect orphaned indexes (destChannelId set but no destMessageId)
- Runs after each ingestion cycle, deduplicates notifications
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pattern grouping (Signal 3):
- Extract YYYY-MM dates, month names, and project prefixes from filenames
- Auto-group packages sharing the same pattern within a channel
- Groups created with groupingSource=AUTO_PATTERN
Creator grouping (Signal 4):
- Auto-group 3+ ungrouped packages from the same creator within a channel
- Runs after pattern grouping as lowest-priority automatic signal
Notification UI:
- Add NotificationBell component to header with unread badge
- Popover panel shows recent notifications with severity icons
- Mark individual or all notifications as read
- Polls every 30 seconds for updates
Failure notifications:
- Upload/download failures now create SystemNotification records
- Visible in the notification bell alongside hash mismatch alerts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Schema:
- Add GroupingSource enum (ALBUM, MANUAL, AUTO_TIME, AUTO_PATTERN, etc.)
- Add groupingSource field to PackageGroup with backfill
- Add SystemNotification model for persistent alerts
- Add NotificationType and NotificationSeverity enums
Ungrouped staging tab:
- Add listUngroupedPackages/countUngroupedPackages queries
- Add "Ungrouped" tab to STL page showing packages without a group
Time-window auto-grouping:
- After album grouping, cluster ungrouped packages within configurable
time window (default 5 min, AUTO_GROUP_TIME_WINDOW_MINUTES env var)
- Groups named from common filename prefix
- Groups created with groupingSource=AUTO_TIME
Hash verification after split:
- Re-hash split parts and compare to original contentHash
- Log error and create SystemNotification on mismatch
- Prevents silently corrupted split uploads
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Multi-part send fix:
- Add destMessageIds BigInt[] to Package schema with backfill migration
- Worker uploadToChannel now returns all message IDs, stored in DB
- Bot forwards all parts of multi-part archives (not just the first)
- Add retry logic for upload rate limits (429) and download stalls
Kickstarter package linking:
- Add package search/linking queries and API routes
- Add PackageLinkerDialog with search + checkbox selection
- Add "Link Packages" and "Send All" actions to kickstarter table
- Add sendAllKickstarterPackages server action
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- createOrFindPackageGroup: catch unique constraint violation from
concurrent creates and fall back to findFirst
- createManualGroup: guard against empty package results before
accessing first element
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update STL page to use listDisplayItems query for mixed package/group display
- Rewrite package-columns to handle StlTableRow union type (group headers + packages)
- Add group expand/collapse with chevron toggle and indented member rows
- Add checkbox selection with "Group N Selected" toolbar button and dialog
- Add inline group actions: rename, dissolve, send all, remove member
- Add clickable group preview thumbnail with file upload for preview images
- Extend DataTable with optional rowClassName prop for group row styling
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace BigInt literal `1n` with `BigInt(1)` for ES target compatibility
- Add default matchedFileCount/matchedByContent to getPackageById return
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
toggleChannelActive only flipped isActive but never created the
AccountChannelMap READER link needed by the worker. Channels enabled
via the toggle (rather than the channel picker) were invisible to the
scanner. Now auto-creates READER links for all active authenticated
accounts when a SOURCE channel is enabled.
Also ran a one-time DB fix to backfill READER links for the 14 active
channels that were missing them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Channel Discovery:
- Remove channel/supergroup filter from getAccountChats — all chat types
(private, groups, Saved Messages, etc.) are now discoverable as sources
- Detect and label the self-chat as "Saved Messages" via getMe
- Update channel picker dialog to accept any chat type string
Bot Rich Messages:
- Enhance package send preview with creator, file count, tags, and source
channel info in MarkdownV2 caption
- Include tags in new_package subscription notifications
- Expand getPendingSendRequest to fetch richer package data
Performance:
- Reviewed pipeline for many-channel load — getChats pagination fix and
per-channel getChat pre-load from prior commit address the main concerns
- Channels with no new messages skip in 2-3 API calls
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug fixes:
- Fix channels not being scanned by paginating TDLib getChats (was only
loading first batch, additional channels were unknown to TDLib)
- Add per-channel getChat pre-load as safety net before scanning
- Fix preview pictures not loading by checking previewData instead of
previewMsgId for hasPreview flag
- Prevent previewMsgId from being set when preview download fails
Package Tags:
- Add tags Text[] column to Package with migration backfilling from
channel categories
- Worker auto-inherits source channel category as initial tag
- Tag filter dropdown and Tags column in STL Files table
- Server actions for individual and bulk tag editing
Kickstarters Tab:
- New KickstarterHost, Kickstarter, and KickstarterPackage models
- Full CRUD with delivery status, payment status, host management
- Package linking (many-to-many with existing packages)
- Sidebar entry with Gift icon
- Table with search, filters, modal forms
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace 0n literals with BigInt(0) for ES2017 target compatibility
- Parse link code JSON to extract userId and check expiration (was
passing raw JSON string as FK, causing constraint violation)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Auto-extract preview images from ZIP/RAR/7z archives during ingestion
- Upload custom preview images via package drawer
- Select preview from archive contents with on-demand extraction UI
- Manually add Telegram channels by t.me link, username, or invite link
- Invite code UX: bulk create, copy link, usage tracking, delete confirm
- Incomplete upload recovery: verify dest messages on worker startup
- Rebuild package DB by scanning destination channel with live progress
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add category field to TelegramChannel (filterable tag like STL, PDF, D&D)
- Category column in channels table with edit via dropdown menu
- Improved creator extraction: filename patterns + channel title fallback
- extractCreatorFromChannelTitle strips [Completed], (Paid), emoji, etc.
- Fix ArchiveType in PackageListItem and PackageRow for new types
- Add Prisma migration for category column
- Add InviteCode model with code, maxUses, expiry, usage tracking
- Registration now requires a valid invite code
- New users get USER role instead of ADMIN
- Admin-only /invites page to create, manage, and share invite codes
- Invite links auto-fill code via ?code= URL param
- Drone pipeline now builds app, worker, and bot images separately
- Add NEXT_PUBLIC_APP_URL build arg to fix URL redirects
- Add progress callbacks to getChannelMessages and getTopicMessages that
fire after each page of messages is fetched
- Worker now shows channel progress (e.g. "[2/5] Channel Name") when
processing multiple source channels
- Worker now shows topic progress (e.g. "topic 3/12") when scanning forums
- Worker now shows live message scanning count during channel/topic scans
(e.g. "Scanning Channel — 300 messages scanned")
- UI stats line now always shows messagesScanned count
- messagesScanned counter now increments during the scanning phase, not
just during archive processing
Co-authored-by: xCyanGrizzly <53275238+xCyanGrizzly@users.noreply.github.com>
1. Worker trigger: Add ingestion_trigger pg_notify listener so the worker
picks up on-demand triggers from the UI and runs an immediate cycle with
full activity tracking (currentActivity, currentStep, etc).
2. Remove orphaned IngestionRun creation from triggerIngestion server action.
Previously the UI created RUNNING runs without activity fields, causing
the UI to show "Working..." with no details. Now only the worker creates
runs with proper activity tracking.
3. Default channels to disabled (isActive: false) in schema and all creation
paths. Destination channels are explicitly set to active since they must
receive uploads. Includes Prisma migration.
Co-authored-by: xCyanGrizzly <53275238+xCyanGrizzly@users.noreply.github.com>
The createUser event in auth.ts now promotes the first user to ADMIN
if no admin exists yet. The JWT callback also fetches the role from the
database on sign-in to pick up the freshly assigned ADMIN role.
Co-authored-by: xCyanGrizzly <53275238+xCyanGrizzly@users.noreply.github.com>
- Replace next/font/google Geist imports with geist npm package to eliminate
Google Fonts CDN network dependency during Docker image builds (was causing
intermittent build failures → redeployment errors)
- Use ./node_modules/.bin/prisma directly in docker-entrypoint.sh instead of
npx for both migrate deploy and db seed commands
Co-authored-by: xCyanGrizzly <53275238+xCyanGrizzly@users.noreply.github.com>
Adds full Telegram ZIP ingestion pipeline: TDLib worker service scans source
channels for archive files, deduplicates by content hash, extracts metadata,
uploads to archive channel, and indexes in Postgres. Forum supergroups are
scanned per-topic with topic names used as creator. Filename-based creator
extraction (e.g. "Mammoth Factory - 2026-01.zip") serves as fallback.
Includes admin UI for managing accounts/channels, simplified account setup
(API credentials via env vars), auth code/password submission dialog,
package browser with creator column, and live ingestion activity tracking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The custom generator output to src/generated/prisma caused persistent
Turbopack module resolution failures in CI. Switch to the standard
@prisma/client import path which all bundlers resolve correctly.
- Remove custom output from prisma schema generator
- Update all imports from ../generated/prisma to @prisma/client
- Add postinstall script to auto-run prisma generate after npm ci
- Remove generated files from git (no longer needed in source tree)
- Simplify CI workflow (remove verify step and --webpack workaround)
Co-Authored-By: Claude <noreply@anthropic.com>
