mirror of
https://github.com/xCyanGrizzly/DragonsStash.git
synced 2026-05-10 22:01:16 +00:00
Compare commits
14 Commits
fe28c31b9e
...
527aca7c25
| Author | SHA1 | Date | |
|---|---|---|---|
| 527aca7c25 | |||
| a4156b2ac6 | |||
| d50c68f67c | |||
| f6e7f5ed3c | |||
| e7f213eec4 | |||
| 20b7d28fdf | |||
| 21663fc29e | |||
| 218ccb9282 | |||
| b632533f54 | |||
| 4baf5aad83 | |||
| ad7790c07b | |||
| e4398caebe | |||
| 6eb7129637 | |||
| d6386209be |
File diff suppressed because it is too large
Load Diff
1343
docs/superpowers/plans/2026-03-25-package-grouping.md
Normal file
1343
docs/superpowers/plans/2026-03-25-package-grouping.md
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,241 @@
|
|||||||
|
# Design: Search Match Indicators, Size Limit Increase, Skipped/Failed Files Overview
|
||||||
|
|
||||||
|
**Date:** 2026-03-24
|
||||||
|
**Status:** Approved
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Three related improvements to the STL packages system:
|
||||||
|
|
||||||
|
1. **Search match indicators** — Show which internal files matched a search query, with highlighted files in the drawer
|
||||||
|
2. **Size limit increase** — Raise the ingestion limit from 4 GB to 200 GB so large multipart archives aren't skipped
|
||||||
|
3. **Skipped/failed files overview** — Track and display archives that were skipped or failed, with retry capability
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Feature 1: Size Limit Increase
|
||||||
|
|
||||||
|
### Change
|
||||||
|
|
||||||
|
`worker/src/util/config.ts` line 6 — change default from `"4096"` to `"204800"`.
|
||||||
|
|
||||||
|
One-line change. The split/upload pipeline already handles arbitrary sizes. The 2 GB per-part Telegram API limit is a separate hard-coded constant and stays as-is.
|
||||||
|
|
||||||
|
### Impact
|
||||||
|
|
||||||
|
- Archives up to 200 GB will now be attempted
|
||||||
|
- Multipart archives where individual parts are under 2 GB (but total exceeds 4 GB) will no longer be skipped — these upload directly without any splitting
|
||||||
|
- Single files over 2 GB are automatically split into 2 GB parts (existing behavior)
|
||||||
|
- Temp disk usage during processing can now reach up to ~200 GB per archive
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Feature 2: Search Match Indicators
|
||||||
|
|
||||||
|
### Backend Changes
|
||||||
|
|
||||||
|
**File:** `src/lib/telegram/queries.ts` — `searchPackages()`
|
||||||
|
|
||||||
|
When `searchIn` is `"files"` or `"both"`, change the PackageFile query from `distinct` to a **grouped count**:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Current: findMany with select: { packageId }, distinct: ["packageId"]
|
||||||
|
// New: groupBy packageId with _count
|
||||||
|
const fileMatches = await prisma.packageFile.groupBy({
|
||||||
|
by: ["packageId"],
|
||||||
|
where: {
|
||||||
|
OR: [
|
||||||
|
{ fileName: { contains: q, mode: "insensitive" } },
|
||||||
|
{ path: { contains: q, mode: "insensitive" } },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
_count: { _all: true },
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
This returns `{ packageId: string, _count: { _all: number } }[]`.
|
||||||
|
|
||||||
|
Note: `PackageRow` in `package-columns.tsx` mirrors `PackageListItem` and must also receive the two new fields.
|
||||||
|
|
||||||
|
**File:** `src/lib/telegram/types.ts` — `PackageListItem`
|
||||||
|
|
||||||
|
Add two fields:
|
||||||
|
- `matchedFileCount: number` — how many files inside matched (0 if matched by package name only)
|
||||||
|
- `matchedByContent: boolean` — true if any files inside matched
|
||||||
|
|
||||||
|
### Frontend Changes
|
||||||
|
|
||||||
|
**File:** `src/app/(app)/stls/page.tsx`
|
||||||
|
|
||||||
|
Pass the search term to `StlTable` as a new prop.
|
||||||
|
|
||||||
|
**File:** `src/app/(app)/stls/_components/stl-table.tsx`
|
||||||
|
|
||||||
|
Pass search term to columns via TanStack Table column meta.
|
||||||
|
|
||||||
|
**File:** `src/app/(app)/stls/_components/package-columns.tsx`
|
||||||
|
|
||||||
|
When search is active and `matchedByContent` is true, render a clickable badge below the filename: e.g., "3 file matches". Clicking opens the `PackageFilesDrawer` with a `highlightTerm` prop set to the search term.
|
||||||
|
|
||||||
|
**File:** `src/app/(app)/stls/_components/package-files-drawer.tsx`
|
||||||
|
|
||||||
|
- Accept optional `highlightTerm: string` prop
|
||||||
|
- Render full file tree as normal (all files visible)
|
||||||
|
- Files whose `fileName` or `path` case-insensitively contains `highlightTerm` get a subtle highlight (amber/yellow background on the row)
|
||||||
|
- Auto-expand folders that contain highlighted files
|
||||||
|
- The drawer's own search input remains independent
|
||||||
|
|
||||||
|
### Data Flow
|
||||||
|
|
||||||
|
1. User types search term in STL table search input
|
||||||
|
2. URL updates with `?search=value`, page reloads
|
||||||
|
3. `page.tsx` calls `searchPackages()` with `searchIn: "both"`
|
||||||
|
4. Query returns packages with `matchedFileCount` and `matchedByContent`
|
||||||
|
5. Table renders "N file matches" badge on content-matched rows
|
||||||
|
6. User clicks badge -> drawer opens with full tree, matching files highlighted
|
||||||
|
7. Folders containing matches auto-expanded
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Feature 3: Skipped/Failed Files Overview
|
||||||
|
|
||||||
|
### Database Schema
|
||||||
|
|
||||||
|
New model in `prisma/schema.prisma`:
|
||||||
|
|
||||||
|
```prisma
|
||||||
|
enum SkipReason {
|
||||||
|
SIZE_LIMIT
|
||||||
|
DOWNLOAD_FAILED
|
||||||
|
EXTRACT_FAILED
|
||||||
|
UPLOAD_FAILED
|
||||||
|
}
|
||||||
|
|
||||||
|
model SkippedPackage {
|
||||||
|
id String @id @default(cuid())
|
||||||
|
fileName String
|
||||||
|
fileSize BigInt
|
||||||
|
reason SkipReason
|
||||||
|
errorMessage String?
|
||||||
|
sourceChannelId String
|
||||||
|
sourceChannel TelegramChannel @relation(fields: [sourceChannelId], references: [id], onDelete: Cascade)
|
||||||
|
sourceMessageId BigInt
|
||||||
|
sourceTopicId BigInt?
|
||||||
|
isMultipart Boolean @default(false)
|
||||||
|
partCount Int @default(1)
|
||||||
|
accountId String
|
||||||
|
account TelegramAccount @relation(fields: [accountId], references: [id], onDelete: Cascade)
|
||||||
|
createdAt DateTime @default(now())
|
||||||
|
|
||||||
|
@@unique([sourceChannelId, sourceMessageId])
|
||||||
|
@@index([reason])
|
||||||
|
@@index([accountId])
|
||||||
|
@@map("skipped_packages")
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Reverse relations must be added to `TelegramChannel` and `TelegramAccount` models:
|
||||||
|
```prisma
|
||||||
|
// In TelegramChannel:
|
||||||
|
skippedPackages SkippedPackage[]
|
||||||
|
|
||||||
|
// In TelegramAccount:
|
||||||
|
skippedPackages SkippedPackage[]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Worker Changes
|
||||||
|
|
||||||
|
**File:** `worker/src/worker.ts`
|
||||||
|
|
||||||
|
Extend `PipelineContext` interface to include `accountId` (derived from the ingestion run's account).
|
||||||
|
|
||||||
|
At each skip/failure point, upsert a `SkippedPackage` record:
|
||||||
|
|
||||||
|
- **Size limit skip** (line 784): reason `SIZE_LIMIT`, no error message
|
||||||
|
- **Download failure** (catch in download loop): reason `DOWNLOAD_FAILED` + error text
|
||||||
|
- **Extract/metadata failure** (catch in extract): reason `EXTRACT_FAILED` + error text
|
||||||
|
- **Upload failure** (catch in upload): reason `UPLOAD_FAILED` + error text
|
||||||
|
|
||||||
|
On **successful ingestion** of a package, delete any existing `SkippedPackage` with the same `(sourceChannelId, sourceMessageId)` — so successful retries clean up after themselves.
|
||||||
|
|
||||||
|
**File:** `worker/src/db/queries.ts`
|
||||||
|
|
||||||
|
Add functions:
|
||||||
|
- `upsertSkippedPackage(data)` — create or update skip record
|
||||||
|
- `deleteSkippedPackage(sourceChannelId, sourceMessageId)` — remove on success
|
||||||
|
|
||||||
|
### Retry Mechanism
|
||||||
|
|
||||||
|
Retrying a skipped package:
|
||||||
|
1. Delete the `SkippedPackage` record
|
||||||
|
2. Find the `AccountChannelMap` record using both `accountId` and `sourceChannelId`, then reset its `lastProcessedMessageId` to `sourceMessageId - 1` (only if less than current watermark)
|
||||||
|
3. If `sourceTopicId` is non-null, also reset the corresponding `TopicProgress.lastProcessedMessageId` for that topic
|
||||||
|
4. The next ingestion cycle picks up the message and re-attempts processing
|
||||||
|
|
||||||
|
For "Retry All" (e.g., all `SIZE_LIMIT` skips after raising the limit):
|
||||||
|
- Delete all matching `SkippedPackage` records
|
||||||
|
- For each affected (account, channel) pair, reset `AccountChannelMap` watermark to the minimum `sourceMessageId - 1` among deleted records
|
||||||
|
- For each affected (account, channel, topic) triple, reset `TopicProgress` watermark similarly
|
||||||
|
|
||||||
|
**Note on behavioral distinction:** `DOWNLOAD_FAILED`, `EXTRACT_FAILED`, and `UPLOAD_FAILED` archives already naturally retry because the worker does not advance the watermark past failed sets. The `SkippedPackage` record provides visibility into these failures. The explicit retry/watermark reset is only strictly needed for `SIZE_LIMIT` skips (where the watermark does advance past the skipped message). The UI should present both types but the retry button is most impactful for `SIZE_LIMIT` skips.
|
||||||
|
|
||||||
|
**Performance note:** "Retry All" can cause the worker to re-scan large message ranges. The existing dedup logic (`packageExistsBySourceMessage`) ensures already-ingested packages are skipped quickly, but there is a scanning cost proportional to the number of messages between the reset watermark and the current position.
|
||||||
|
|
||||||
|
### Frontend Changes
|
||||||
|
|
||||||
|
**File:** `src/app/(app)/stls/_components/stl-table.tsx`
|
||||||
|
|
||||||
|
Add a "Skipped / Failed" tab alongside the main packages table.
|
||||||
|
|
||||||
|
**New file:** `src/app/(app)/stls/_components/skipped-packages-tab.tsx`
|
||||||
|
|
||||||
|
Table columns:
|
||||||
|
- **fileName** — archive name
|
||||||
|
- **fileSize** — formatted size
|
||||||
|
- **reason** — color-coded badge: `SIZE_LIMIT` (yellow), `DOWNLOAD_FAILED` (red), `EXTRACT_FAILED` (red), `UPLOAD_FAILED` (red)
|
||||||
|
- **errorMessage** — truncated with expandable tooltip/popover for full text
|
||||||
|
- **channel** — source channel title
|
||||||
|
- **createdAt** — when the skip/failure was recorded
|
||||||
|
|
||||||
|
Actions:
|
||||||
|
- **Retry** button per row — server action that deletes record + resets watermark
|
||||||
|
- **Retry All** button in the header — bulk retry, filterable by reason
|
||||||
|
|
||||||
|
**File:** `src/app/(app)/stls/page.tsx`
|
||||||
|
|
||||||
|
Fetch skipped packages count (for tab badge) alongside existing queries.
|
||||||
|
|
||||||
|
**File:** `src/data/` or `src/lib/telegram/queries.ts`
|
||||||
|
|
||||||
|
Add query functions:
|
||||||
|
- `listSkippedPackages(options)` — paginated list with reason filter
|
||||||
|
- `countSkippedPackages()` — for tab badge
|
||||||
|
- `retrySkippedPackage(id)` — delete record + reset watermark
|
||||||
|
- `retryAllSkippedPackages(reason?)` — bulk retry
|
||||||
|
|
||||||
|
**File:** `src/app/(app)/stls/actions.ts`
|
||||||
|
|
||||||
|
Add server actions:
|
||||||
|
- `retrySkippedPackageAction(id)`
|
||||||
|
- `retryAllSkippedPackagesAction(reason?)`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Files to Create/Modify
|
||||||
|
|
||||||
|
### Create
|
||||||
|
- `src/app/(app)/stls/_components/skipped-packages-tab.tsx` — skipped packages table UI
|
||||||
|
- Prisma migration for `SkippedPackage` model
|
||||||
|
|
||||||
|
### Modify
|
||||||
|
- `worker/src/util/config.ts` — raise default max size
|
||||||
|
- `worker/src/worker.ts` — record skips/failures, clean up on success
|
||||||
|
- `worker/src/db/queries.ts` — add skip record CRUD functions
|
||||||
|
- `prisma/schema.prisma` — add `SkippedPackage` model and `SkipReason` enum
|
||||||
|
- `src/lib/telegram/queries.ts` — modify `searchPackages()` for match counts, add skipped package queries
|
||||||
|
- `src/lib/telegram/types.ts` — add `matchedFileCount`/`matchedByContent` to `PackageListItem`, add skipped package types
|
||||||
|
- `src/app/(app)/stls/page.tsx` — pass search term, fetch skipped count, add tab
|
||||||
|
- `src/app/(app)/stls/_components/stl-table.tsx` — accept search prop, render tabs
|
||||||
|
- `src/app/(app)/stls/_components/package-columns.tsx` — render match badge
|
||||||
|
- `src/app/(app)/stls/_components/package-files-drawer.tsx` — accept highlightTerm, highlight matching files, auto-expand matched folders
|
||||||
|
- `src/app/(app)/stls/actions.ts` — add retry server actions
|
||||||
246
docs/superpowers/specs/2026-03-25-package-grouping-design.md
Normal file
246
docs/superpowers/specs/2026-03-25-package-grouping-design.md
Normal file
@@ -0,0 +1,246 @@
|
|||||||
|
# Package Grouping Design
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Add the ability to group related packages that were posted together in a Telegram channel (e.g., "DUNGEON BLOCKS - Colossal Dungeon" with 6 separate archive files). Groups appear as collapsible rows in the STL files table, with support for both automatic detection via Telegram album IDs and manual grouping through the UI.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- Automatically detect and group files posted together in Telegram (same `media_album_id`)
|
||||||
|
- Display groups as collapsed rows in the STL table with aggregated metadata
|
||||||
|
- Allow manual grouping/ungrouping of packages via the UI
|
||||||
|
- Support editable group names and preview images
|
||||||
|
- Enable "Send All" to deliver every package in a group via the bot
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
- Merging grouped packages into a single Package record (each stays independent)
|
||||||
|
- Time-proximity heuristics for grouping (too error-prone)
|
||||||
|
- Grouping across different source channels
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### New `PackageGroup` Table
|
||||||
|
|
||||||
|
```prisma
|
||||||
|
model PackageGroup {
|
||||||
|
id String @id @default(cuid())
|
||||||
|
name String
|
||||||
|
mediaAlbumId String?
|
||||||
|
sourceChannelId String
|
||||||
|
previewData Bytes?
|
||||||
|
createdAt DateTime @default(now())
|
||||||
|
updatedAt DateTime @updatedAt
|
||||||
|
|
||||||
|
packages Package[]
|
||||||
|
sourceChannel TelegramChannel @relation(fields: [sourceChannelId], references: [id], onDelete: Cascade)
|
||||||
|
|
||||||
|
@@unique([mediaAlbumId, sourceChannelId])
|
||||||
|
@@index([sourceChannelId])
|
||||||
|
@@map("package_groups")
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Package Model Changes
|
||||||
|
|
||||||
|
Add optional group membership:
|
||||||
|
|
||||||
|
```prisma
|
||||||
|
model Package {
|
||||||
|
// ... existing fields ...
|
||||||
|
packageGroupId String?
|
||||||
|
packageGroup PackageGroup? @relation(fields: [packageGroupId], references: [id], onDelete: SetNull)
|
||||||
|
|
||||||
|
@@index([packageGroupId])
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### TelegramChannel Model Changes
|
||||||
|
|
||||||
|
Add back-relation for the new `PackageGroup` model:
|
||||||
|
|
||||||
|
```prisma
|
||||||
|
model TelegramChannel {
|
||||||
|
// ... existing fields and relations ...
|
||||||
|
packageGroups PackageGroup[]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Decisions
|
||||||
|
|
||||||
|
- `mediaAlbumId` is `String?` (TDLib int64 stringified) — only used for dedup lookups, avoids BigInt complexity
|
||||||
|
- `@@unique([mediaAlbumId, sourceChannelId])` prevents duplicate album-derived groups when re-scanning. PostgreSQL treats NULLs as distinct in unique constraints, so manually-created groups (with `mediaAlbumId = null`) are not constrained by this — which is correct behavior
|
||||||
|
- Idempotency for album groups uses `findFirst({ where: { mediaAlbumId, sourceChannelId } })` + conditional `create`, not `upsert`, because Prisma does not support `upsert` on compound unique keys with nullable fields
|
||||||
|
- `onDelete: SetNull` on `Package.packageGroup` means dissolving a group automatically unlinks all members
|
||||||
|
- `onDelete: Cascade` on `PackageGroup.sourceChannel` means deleting a channel cleans up its groups
|
||||||
|
- `sourceTopicId` is omitted from `PackageGroup` — it can be inferred from member packages, and manual groups may span topics
|
||||||
|
- `@@map("package_groups")` follows the project's snake_case table naming convention
|
||||||
|
- `previewData` stores JPEG thumbnail bytes directly on the group (same pattern as Package)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Worker Changes
|
||||||
|
|
||||||
|
### TelegramMessage Interface
|
||||||
|
|
||||||
|
Add optional `mediaAlbumId` field:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
export interface TelegramMessage {
|
||||||
|
id: bigint;
|
||||||
|
fileName: string;
|
||||||
|
fileId: string;
|
||||||
|
fileSize: bigint;
|
||||||
|
date: Date;
|
||||||
|
mediaAlbumId?: string; // Absent or "0" when not part of an album
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The field is optional to minimize call-site changes. The grouping step treats `undefined` and `"0"` equivalently as "not part of an album."
|
||||||
|
|
||||||
|
### TelegramPhoto Interface
|
||||||
|
|
||||||
|
Add optional `mediaAlbumId` field:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
export interface TelegramPhoto {
|
||||||
|
id: bigint;
|
||||||
|
date: Date;
|
||||||
|
caption: string;
|
||||||
|
fileId: string;
|
||||||
|
fileSize: number;
|
||||||
|
mediaAlbumId?: string; // For album-to-preview correlation
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Channel Scanning
|
||||||
|
|
||||||
|
In `getChannelMessages()`, read `media_album_id` from the TDLib message object (already present in TDLib responses, just not captured today). Add `media_album_id?: string` to the `TdMessage` interface and pass through to both `TelegramMessage` and `TelegramPhoto`.
|
||||||
|
|
||||||
|
The document pass and photo pass already run as separate loops over `searchChatMessages`. Both loops capture `media_album_id` independently. Correlation happens at grouping time: album photos are matched to album documents by comparing their `mediaAlbumId` values, not at scan time.
|
||||||
|
|
||||||
|
### Group Creation (Post-Processing)
|
||||||
|
|
||||||
|
After each scan cycle's packages are individually processed (downloaded, hashed, uploaded, indexed), a post-processing step handles grouping:
|
||||||
|
|
||||||
|
1. Collect all packages from the current scan batch that share the same non-zero `mediaAlbumId`
|
||||||
|
2. For each distinct `mediaAlbumId`, check if a `PackageGroup` already exists via `findFirst({ where: { mediaAlbumId, sourceChannelId } })`
|
||||||
|
3. If no group exists, create one:
|
||||||
|
- **Name:** caption of the first message in the album (falls back to first file's base name)
|
||||||
|
- **Preview:** find a `TelegramPhoto` from the scan's `photos[]` array with the same `mediaAlbumId`. If found, download via `downloadPhotoThumbnail`. If not, the group starts with no preview (can be added in UI later)
|
||||||
|
4. Link all member packages via an idempotent `updateMany` — sets `packageGroupId` on all packages whose `sourceMessageId` is in the album's message set. This handles both newly-indexed packages and previously-indexed ones that were created in an earlier partial scan (e.g., if one package failed and was retried later)
|
||||||
|
|
||||||
|
The per-package pipeline is unchanged — each file is still downloaded, hashed, deduped, split, uploaded, and indexed independently. Grouping is a layer on top.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Query Layer
|
||||||
|
|
||||||
|
### Paginated Listing with Groups
|
||||||
|
|
||||||
|
The STL table shows "display items" — either a group (collapsed) or a standalone package. Pagination operates on display items so that a group occupies exactly one slot regardless of member count.
|
||||||
|
|
||||||
|
**Two-step query approach** (handles filters correctly):
|
||||||
|
|
||||||
|
**Step 1 — Find matching display item IDs:**
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Find all group IDs and standalone package IDs where at least one member matches filters
|
||||||
|
SELECT DISTINCT COALESCE(p."packageGroupId", p.id) AS display_id,
|
||||||
|
CASE WHEN p."packageGroupId" IS NOT NULL THEN 'group' ELSE 'package' END AS display_type,
|
||||||
|
MAX(p."indexedAt") AS sort_date
|
||||||
|
FROM packages p
|
||||||
|
LEFT JOIN package_groups pg ON pg.id = p."packageGroupId"
|
||||||
|
WHERE 1=1
|
||||||
|
-- Optional filters applied here (creator, tags, search text, channelId)
|
||||||
|
GROUP BY COALESCE(p."packageGroupId", p.id),
|
||||||
|
CASE WHEN p."packageGroupId" IS NOT NULL THEN 'group' ELSE 'package' END
|
||||||
|
ORDER BY sort_date DESC
|
||||||
|
LIMIT $1 OFFSET $2
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2 — Fetch full data:**
|
||||||
|
|
||||||
|
For groups on the current page, fetch all member packages (including those that didn't match filters — the group appears because at least one member matched, but the expanded view shows all members). For standalone packages, fetch the full package data.
|
||||||
|
|
||||||
|
**Count query** (for pagination total):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT COUNT(*) FROM (
|
||||||
|
SELECT DISTINCT COALESCE(p."packageGroupId", p.id)
|
||||||
|
FROM packages p
|
||||||
|
WHERE 1=1
|
||||||
|
-- Same filters as step 1
|
||||||
|
) AS display_items
|
||||||
|
```
|
||||||
|
|
||||||
|
### Group Row Aggregates
|
||||||
|
|
||||||
|
Computed in the step 2 fetch: total file size (sum), total file count (sum), combined tags (array union), member package count per group. These populate the collapsed group row.
|
||||||
|
|
||||||
|
### Search
|
||||||
|
|
||||||
|
`searchPackages` adds `PackageGroup.name` to search targets via a `LEFT JOIN` to `package_groups`. If any package in a group matches by name/file content, or the group name matches, the whole group appears.
|
||||||
|
|
||||||
|
### Filtering
|
||||||
|
|
||||||
|
Creator/tag filters apply to member packages. A group appears if any member matches the filter. The group row shows aggregates of all members (not just matching ones).
|
||||||
|
|
||||||
|
### New Query Functions
|
||||||
|
|
||||||
|
| Function | Purpose |
|
||||||
|
|----------|---------|
|
||||||
|
| `listDisplayItems(page, limit, filters)` | Two-step paginated query returning groups + standalone packages |
|
||||||
|
| `getDisplayItemCount(filters)` | Count of display items for pagination total |
|
||||||
|
| `getPackageGroup(groupId)` | Group metadata + all member packages |
|
||||||
|
| `updatePackageGroupName(groupId, name)` | Rename group |
|
||||||
|
| `updatePackageGroupPreview(groupId, previewData)` | Replace group preview |
|
||||||
|
| `addPackagesToGroup(packageIds, groupId)` | Manual grouping — add to existing group |
|
||||||
|
| `removePackageFromGroup(packageId)` | Ungroup single package |
|
||||||
|
| `createManualGroup(name, packageIds)` | Create new group from UI |
|
||||||
|
| `dissolveGroup(groupId)` | Ungroup all members, delete group record |
|
||||||
|
|
||||||
|
For manual grouping of packages that already belong to different groups: the UI first dissolves empty source groups (groups where all members were moved), then links the selected packages to the target group. Non-selected members of source groups remain in their original group.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## UI Changes
|
||||||
|
|
||||||
|
### STL Table — Group Rows
|
||||||
|
|
||||||
|
- **Collapsed (default):** Single row showing preview thumbnail, group name (editable inline), archive type badge ("Mixed" if heterogeneous), combined size, combined file count, combined tags (editable), source channel, latest `indexedAt`, actions
|
||||||
|
- **Expanded:** Chevron toggle reveals member packages as indented sub-rows with their existing columns and per-package actions
|
||||||
|
- Chevron icon on the left of the row toggles expand/collapse
|
||||||
|
|
||||||
|
**Loading strategy:** Member packages for all groups on the current page are prefetched in a single batched query during the step 2 fetch. This means expand/collapse is instant (no on-demand loading) and avoids per-row loading states.
|
||||||
|
|
||||||
|
### Group Row Actions
|
||||||
|
|
||||||
|
- **Send All** — Queues bot send requests for every package in the group. Checks for existing PENDING/SENDING requests per package to avoid duplicates.
|
||||||
|
- **View Files** — Opens file drawer showing all member packages' files, separated by package name headers
|
||||||
|
- **Dissolve Group** — Ungroups all members (confirmation required)
|
||||||
|
|
||||||
|
### Individual Package Actions (Within a Group)
|
||||||
|
|
||||||
|
- Existing: Send, View Files
|
||||||
|
- New: "Remove from group" in dropdown menu
|
||||||
|
|
||||||
|
### Manual Grouping
|
||||||
|
|
||||||
|
- Checkbox selection column on package rows
|
||||||
|
- When 2+ packages selected, a "Group Selected" button appears in the table toolbar
|
||||||
|
- Prompts for a group name, creates the group
|
||||||
|
- If selected packages belong to existing groups, those packages are moved to the new group. Source groups that become empty are automatically dissolved.
|
||||||
|
|
||||||
|
### Preview Editing
|
||||||
|
|
||||||
|
- Click the group's preview thumbnail to upload a replacement image
|
||||||
|
- Same upload flow as individual packages (existing component reuse)
|
||||||
|
|
||||||
|
### No Changes To
|
||||||
|
|
||||||
|
- Skipped/failed packages tab
|
||||||
|
- Package detail drawer internals
|
||||||
|
- Search UI (just broader matching behind the scenes)
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
-- AlterTable
|
||||||
|
ALTER TABLE "packages" ADD COLUMN "packageGroupId" TEXT;
|
||||||
|
|
||||||
|
-- CreateTable
|
||||||
|
CREATE TABLE "package_groups" (
|
||||||
|
"id" TEXT NOT NULL,
|
||||||
|
"name" TEXT NOT NULL,
|
||||||
|
"mediaAlbumId" TEXT,
|
||||||
|
"sourceChannelId" TEXT NOT NULL,
|
||||||
|
"previewData" BYTEA,
|
||||||
|
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
"updatedAt" TIMESTAMP(3) NOT NULL,
|
||||||
|
|
||||||
|
CONSTRAINT "package_groups_pkey" PRIMARY KEY ("id")
|
||||||
|
);
|
||||||
|
|
||||||
|
-- CreateIndex
|
||||||
|
CREATE INDEX "package_groups_sourceChannelId_idx" ON "package_groups"("sourceChannelId");
|
||||||
|
|
||||||
|
-- CreateIndex
|
||||||
|
CREATE UNIQUE INDEX "package_groups_mediaAlbumId_sourceChannelId_key" ON "package_groups"("mediaAlbumId", "sourceChannelId");
|
||||||
|
|
||||||
|
-- CreateIndex
|
||||||
|
CREATE INDEX "packages_packageGroupId_idx" ON "packages"("packageGroupId");
|
||||||
|
|
||||||
|
-- AddForeignKey
|
||||||
|
ALTER TABLE "packages" ADD CONSTRAINT "packages_packageGroupId_fkey" FOREIGN KEY ("packageGroupId") REFERENCES "package_groups"("id") ON DELETE SET NULL ON UPDATE CASCADE;
|
||||||
|
|
||||||
|
-- AddForeignKey
|
||||||
|
ALTER TABLE "package_groups" ADD CONSTRAINT "package_groups_sourceChannelId_fkey" FOREIGN KEY ("sourceChannelId") REFERENCES "telegram_channels"("id") ON DELETE CASCADE ON UPDATE CASCADE;
|
||||||
@@ -432,6 +432,7 @@ model TelegramChannel {
|
|||||||
accountMaps AccountChannelMap[]
|
accountMaps AccountChannelMap[]
|
||||||
packages Package[]
|
packages Package[]
|
||||||
skippedPackages SkippedPackage[]
|
skippedPackages SkippedPackage[]
|
||||||
|
packageGroups PackageGroup[]
|
||||||
|
|
||||||
@@index([type, isActive])
|
@@index([type, isActive])
|
||||||
@@index([category])
|
@@index([category])
|
||||||
@@ -474,10 +475,12 @@ model Package {
|
|||||||
tags String[] @default([])
|
tags String[] @default([])
|
||||||
previewData Bytes? // JPEG thumbnail from nearby Telegram photo (stored as raw bytes)
|
previewData Bytes? // JPEG thumbnail from nearby Telegram photo (stored as raw bytes)
|
||||||
previewMsgId BigInt? // Telegram message ID of the matched photo
|
previewMsgId BigInt? // Telegram message ID of the matched photo
|
||||||
|
packageGroupId String?
|
||||||
indexedAt DateTime @default(now())
|
indexedAt DateTime @default(now())
|
||||||
createdAt DateTime @default(now())
|
createdAt DateTime @default(now())
|
||||||
|
|
||||||
sourceChannel TelegramChannel @relation(fields: [sourceChannelId], references: [id])
|
sourceChannel TelegramChannel @relation(fields: [sourceChannelId], references: [id])
|
||||||
|
packageGroup PackageGroup? @relation(fields: [packageGroupId], references: [id], onDelete: SetNull)
|
||||||
files PackageFile[]
|
files PackageFile[]
|
||||||
ingestionRun IngestionRun? @relation(fields: [ingestionRunId], references: [id])
|
ingestionRun IngestionRun? @relation(fields: [ingestionRunId], references: [id])
|
||||||
ingestionRunId String?
|
ingestionRunId String?
|
||||||
@@ -491,6 +494,7 @@ model Package {
|
|||||||
@@index([indexedAt])
|
@@index([indexedAt])
|
||||||
@@index([archiveType])
|
@@index([archiveType])
|
||||||
@@index([creator])
|
@@index([creator])
|
||||||
|
@@index([packageGroupId])
|
||||||
@@map("packages")
|
@@map("packages")
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -512,6 +516,23 @@ model PackageFile {
|
|||||||
@@map("package_files")
|
@@map("package_files")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
model PackageGroup {
|
||||||
|
id String @id @default(cuid())
|
||||||
|
name String
|
||||||
|
mediaAlbumId String?
|
||||||
|
sourceChannelId String
|
||||||
|
previewData Bytes?
|
||||||
|
createdAt DateTime @default(now())
|
||||||
|
updatedAt DateTime @updatedAt
|
||||||
|
|
||||||
|
packages Package[]
|
||||||
|
sourceChannel TelegramChannel @relation(fields: [sourceChannelId], references: [id], onDelete: Cascade)
|
||||||
|
|
||||||
|
@@unique([mediaAlbumId, sourceChannelId])
|
||||||
|
@@index([sourceChannelId])
|
||||||
|
@@map("package_groups")
|
||||||
|
}
|
||||||
|
|
||||||
model IngestionRun {
|
model IngestionRun {
|
||||||
id String @id @default(cuid())
|
id String @id @default(cuid())
|
||||||
accountId String
|
accountId String
|
||||||
|
|||||||
@@ -1,10 +1,11 @@
|
|||||||
"use client";
|
"use client";
|
||||||
|
|
||||||
import { type ColumnDef } from "@tanstack/react-table";
|
import { type ColumnDef } from "@tanstack/react-table";
|
||||||
import { FileArchive, Eye } from "lucide-react";
|
import { FileArchive, Eye, ChevronRight, Layers, Ungroup, Send, ImagePlus } from "lucide-react";
|
||||||
import { DataTableColumnHeader } from "@/components/shared/data-table-column-header";
|
import { DataTableColumnHeader } from "@/components/shared/data-table-column-header";
|
||||||
import { Badge } from "@/components/ui/badge";
|
import { Badge } from "@/components/ui/badge";
|
||||||
import { Button } from "@/components/ui/button";
|
import { Button } from "@/components/ui/button";
|
||||||
|
import { Checkbox } from "@/components/ui/checkbox";
|
||||||
import { SendToTelegramButton } from "./send-to-telegram-button";
|
import { SendToTelegramButton } from "./send-to-telegram-button";
|
||||||
|
|
||||||
export interface PackageRow {
|
export interface PackageRow {
|
||||||
@@ -25,6 +26,34 @@ export interface PackageRow {
|
|||||||
};
|
};
|
||||||
matchedFileCount: number;
|
matchedFileCount: number;
|
||||||
matchedByContent: boolean;
|
matchedByContent: boolean;
|
||||||
|
packageGroupId?: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface GroupHeaderRow {
|
||||||
|
_rowType: "group";
|
||||||
|
id: string;
|
||||||
|
name: string;
|
||||||
|
hasPreview: boolean;
|
||||||
|
totalFileSize: string;
|
||||||
|
totalFileCount: number;
|
||||||
|
packageCount: number;
|
||||||
|
combinedTags: string[];
|
||||||
|
archiveTypes: ("ZIP" | "RAR" | "SEVEN_Z" | "DOCUMENT")[];
|
||||||
|
latestIndexedAt: string;
|
||||||
|
sourceChannel: { id: string; title: string };
|
||||||
|
_expanded: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PackageTableRow extends PackageRow {
|
||||||
|
_rowType: "package";
|
||||||
|
_groupId: string | null;
|
||||||
|
_isGroupMember: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type StlTableRow = GroupHeaderRow | PackageTableRow;
|
||||||
|
|
||||||
|
function isGroupRow(row: StlTableRow): row is GroupHeaderRow {
|
||||||
|
return row._rowType === "group";
|
||||||
}
|
}
|
||||||
|
|
||||||
interface PackageColumnsProps {
|
interface PackageColumnsProps {
|
||||||
@@ -32,9 +61,17 @@ interface PackageColumnsProps {
|
|||||||
onSetCreator: (pkg: PackageRow) => void;
|
onSetCreator: (pkg: PackageRow) => void;
|
||||||
onSetTags: (pkg: PackageRow) => void;
|
onSetTags: (pkg: PackageRow) => void;
|
||||||
searchTerm: string;
|
searchTerm: string;
|
||||||
|
onToggleGroup: (groupId: string) => void;
|
||||||
|
onRenameGroup: (groupId: string, currentName: string) => void;
|
||||||
|
onDissolveGroup: (groupId: string) => void;
|
||||||
|
onSendAllInGroup: (groupId: string) => void;
|
||||||
|
onRemoveFromGroup: (packageId: string) => void;
|
||||||
|
onGroupPreviewUpload: (groupId: string) => void;
|
||||||
|
selectedPackages: Set<string>;
|
||||||
|
onToggleSelect: (packageId: string) => void;
|
||||||
}
|
}
|
||||||
|
|
||||||
function formatBytes(bytesStr: string): string {
|
export function formatBytes(bytesStr: string): string {
|
||||||
const bytes = Number(bytesStr);
|
const bytes = Number(bytesStr);
|
||||||
if (bytes === 0) return "0 B";
|
if (bytes === 0) return "0 B";
|
||||||
const k = 1024;
|
const k = 1024;
|
||||||
@@ -61,107 +98,254 @@ function PreviewCell({ pkg }: { pkg: PackageRow }) {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function GroupPreviewCell({
|
||||||
|
group,
|
||||||
|
onUpload,
|
||||||
|
}: {
|
||||||
|
group: GroupHeaderRow;
|
||||||
|
onUpload: (groupId: string) => void;
|
||||||
|
}) {
|
||||||
|
if (group.hasPreview) {
|
||||||
|
return (
|
||||||
|
<button
|
||||||
|
className="relative group/preview cursor-pointer"
|
||||||
|
onClick={() => onUpload(group.id)}
|
||||||
|
title="Click to change preview image"
|
||||||
|
>
|
||||||
|
<img
|
||||||
|
src={`/api/groups/${group.id}/preview`}
|
||||||
|
alt=""
|
||||||
|
className="h-9 w-9 rounded-md object-cover bg-muted"
|
||||||
|
loading="lazy"
|
||||||
|
/>
|
||||||
|
<div className="absolute inset-0 flex items-center justify-center rounded-md bg-black/50 opacity-0 group-hover/preview:opacity-100 transition-opacity">
|
||||||
|
<ImagePlus className="h-3.5 w-3.5 text-white" />
|
||||||
|
</div>
|
||||||
|
</button>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
<button
|
||||||
|
className="flex h-9 w-9 items-center justify-center rounded-md bg-muted hover:bg-muted/80 transition-colors cursor-pointer"
|
||||||
|
onClick={() => onUpload(group.id)}
|
||||||
|
title="Click to add preview image"
|
||||||
|
>
|
||||||
|
<Layers className="h-4 w-4 text-muted-foreground" />
|
||||||
|
</button>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
export function getPackageColumns({
|
export function getPackageColumns({
|
||||||
onViewFiles,
|
onViewFiles,
|
||||||
onSetCreator,
|
onSetCreator,
|
||||||
onSetTags,
|
onSetTags,
|
||||||
searchTerm,
|
searchTerm,
|
||||||
}: PackageColumnsProps): ColumnDef<PackageRow, unknown>[] {
|
onToggleGroup,
|
||||||
|
onRenameGroup,
|
||||||
|
onDissolveGroup,
|
||||||
|
onSendAllInGroup,
|
||||||
|
onRemoveFromGroup,
|
||||||
|
onGroupPreviewUpload,
|
||||||
|
selectedPackages,
|
||||||
|
onToggleSelect,
|
||||||
|
}: PackageColumnsProps): ColumnDef<StlTableRow, unknown>[] {
|
||||||
return [
|
return [
|
||||||
|
{
|
||||||
|
id: "select",
|
||||||
|
header: "",
|
||||||
|
cell: ({ row }) => {
|
||||||
|
const data = row.original;
|
||||||
|
if (isGroupRow(data)) return null;
|
||||||
|
return (
|
||||||
|
<Checkbox
|
||||||
|
checked={selectedPackages.has(data.id)}
|
||||||
|
onCheckedChange={() => onToggleSelect(data.id)}
|
||||||
|
aria-label="Select package"
|
||||||
|
className="translate-y-[2px]"
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
},
|
||||||
|
enableHiding: false,
|
||||||
|
enableSorting: false,
|
||||||
|
size: 32,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
id: "preview",
|
id: "preview",
|
||||||
header: "",
|
header: "",
|
||||||
cell: ({ row }) => <PreviewCell pkg={row.original} />,
|
cell: ({ row }) => {
|
||||||
|
const data = row.original;
|
||||||
|
if (isGroupRow(data)) {
|
||||||
|
return (
|
||||||
|
<div className="flex items-center gap-1">
|
||||||
|
<button
|
||||||
|
className="shrink-0 p-0.5 cursor-pointer"
|
||||||
|
onClick={() => onToggleGroup(data.id)}
|
||||||
|
aria-label={data._expanded ? "Collapse group" : "Expand group"}
|
||||||
|
>
|
||||||
|
<ChevronRight
|
||||||
|
className={`h-4 w-4 text-muted-foreground transition-transform ${
|
||||||
|
data._expanded ? "rotate-90" : ""
|
||||||
|
}`}
|
||||||
|
/>
|
||||||
|
</button>
|
||||||
|
<GroupPreviewCell group={data} onUpload={onGroupPreviewUpload} />
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
<div className={data._isGroupMember ? "pl-5" : ""}>
|
||||||
|
<PreviewCell pkg={data} />
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
},
|
||||||
enableHiding: false,
|
enableHiding: false,
|
||||||
enableSorting: false,
|
enableSorting: false,
|
||||||
size: 52,
|
size: 72,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
accessorKey: "fileName",
|
accessorKey: "fileName",
|
||||||
header: ({ column }) => <DataTableColumnHeader column={column} title="File Name" />,
|
header: ({ column }) => <DataTableColumnHeader column={column} title="File Name" />,
|
||||||
cell: ({ row }) => (
|
cell: ({ row }) => {
|
||||||
|
const data = row.original;
|
||||||
|
if (isGroupRow(data)) {
|
||||||
|
return (
|
||||||
<div className="min-w-0">
|
<div className="min-w-0">
|
||||||
<div className="flex items-center gap-2">
|
<div className="flex items-center gap-2">
|
||||||
<span className="font-medium truncate max-w-[300px]">{row.original.fileName}</span>
|
<button
|
||||||
{row.original.isMultipart && (
|
className="font-semibold truncate max-w-[300px] cursor-pointer hover:underline text-left"
|
||||||
|
onClick={() => onRenameGroup(data.id, data.name)}
|
||||||
|
title="Click to rename group"
|
||||||
|
>
|
||||||
|
{data.name}
|
||||||
|
</button>
|
||||||
|
<Badge variant="secondary" className="text-[10px] shrink-0">
|
||||||
|
{data.packageCount} pkg{data.packageCount !== 1 ? "s" : ""}
|
||||||
|
</Badge>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
<div className="min-w-0">
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<span className="font-medium truncate max-w-[300px]">{data.fileName}</span>
|
||||||
|
{data.isMultipart && (
|
||||||
<Badge variant="outline" className="text-[10px] shrink-0">
|
<Badge variant="outline" className="text-[10px] shrink-0">
|
||||||
Multi
|
Multi
|
||||||
</Badge>
|
</Badge>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
{searchTerm && row.original.matchedByContent && (
|
{searchTerm && data.matchedByContent && (
|
||||||
<button
|
<button
|
||||||
className="text-[11px] text-amber-500 hover:text-amber-400 hover:underline cursor-pointer mt-0.5"
|
className="text-[11px] text-amber-500 hover:text-amber-400 hover:underline cursor-pointer mt-0.5"
|
||||||
onClick={() => onViewFiles(row.original)}
|
onClick={() => onViewFiles(data)}
|
||||||
>
|
>
|
||||||
{row.original.matchedFileCount.toLocaleString()} file match{row.original.matchedFileCount !== 1 ? "es" : ""}
|
{data.matchedFileCount.toLocaleString()} file match{data.matchedFileCount !== 1 ? "es" : ""}
|
||||||
</button>
|
</button>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
),
|
);
|
||||||
|
},
|
||||||
enableHiding: false,
|
enableHiding: false,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
accessorKey: "archiveType",
|
accessorKey: "archiveType",
|
||||||
header: ({ column }) => <DataTableColumnHeader column={column} title="Type" />,
|
header: ({ column }) => <DataTableColumnHeader column={column} title="Type" />,
|
||||||
cell: ({ row }) => (
|
cell: ({ row }) => {
|
||||||
|
const data = row.original;
|
||||||
|
if (isGroupRow(data)) {
|
||||||
|
const types = data.archiveTypes;
|
||||||
|
if (types.length === 1) {
|
||||||
|
return (
|
||||||
<Badge variant="secondary" className="text-[10px]">
|
<Badge variant="secondary" className="text-[10px]">
|
||||||
{row.original.archiveType}
|
{types[0]}
|
||||||
</Badge>
|
</Badge>
|
||||||
),
|
);
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
<Badge variant="secondary" className="text-[10px]">
|
||||||
|
Mixed
|
||||||
|
</Badge>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
<Badge variant="secondary" className="text-[10px]">
|
||||||
|
{data.archiveType}
|
||||||
|
</Badge>
|
||||||
|
);
|
||||||
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
accessorKey: "fileSize",
|
accessorKey: "fileSize",
|
||||||
header: ({ column }) => <DataTableColumnHeader column={column} title="Size" />,
|
header: ({ column }) => <DataTableColumnHeader column={column} title="Size" />,
|
||||||
cell: ({ row }) => (
|
cell: ({ row }) => {
|
||||||
|
const data = row.original;
|
||||||
|
const size = isGroupRow(data) ? data.totalFileSize : data.fileSize;
|
||||||
|
return (
|
||||||
<span className="text-sm text-muted-foreground">
|
<span className="text-sm text-muted-foreground">
|
||||||
{formatBytes(row.original.fileSize)}
|
{formatBytes(size)}
|
||||||
</span>
|
</span>
|
||||||
),
|
);
|
||||||
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
accessorKey: "fileCount",
|
accessorKey: "fileCount",
|
||||||
header: ({ column }) => <DataTableColumnHeader column={column} title="Files" />,
|
header: ({ column }) => <DataTableColumnHeader column={column} title="Files" />,
|
||||||
cell: ({ row }) => (
|
cell: ({ row }) => {
|
||||||
|
const data = row.original;
|
||||||
|
const count = isGroupRow(data) ? data.totalFileCount : data.fileCount;
|
||||||
|
return (
|
||||||
<span className="text-sm">
|
<span className="text-sm">
|
||||||
{row.original.fileCount.toLocaleString()}
|
{count.toLocaleString()}
|
||||||
</span>
|
</span>
|
||||||
),
|
);
|
||||||
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
accessorKey: "creator",
|
accessorKey: "creator",
|
||||||
header: ({ column }) => <DataTableColumnHeader column={column} title="Creator" />,
|
header: ({ column }) => <DataTableColumnHeader column={column} title="Creator" />,
|
||||||
cell: ({ row }) => (
|
cell: ({ row }) => {
|
||||||
|
const data = row.original;
|
||||||
|
if (isGroupRow(data)) {
|
||||||
|
return <span className="text-sm text-muted-foreground">{"\u2014"}</span>;
|
||||||
|
}
|
||||||
|
return (
|
||||||
<button
|
<button
|
||||||
className="text-sm text-muted-foreground truncate max-w-[160px] block hover:text-foreground hover:underline cursor-pointer text-left"
|
className="text-sm text-muted-foreground truncate max-w-[160px] block hover:text-foreground hover:underline cursor-pointer text-left"
|
||||||
onClick={() => onSetCreator(row.original)}
|
onClick={() => onSetCreator(data)}
|
||||||
title="Click to edit creator"
|
title="Click to edit creator"
|
||||||
>
|
>
|
||||||
{row.original.creator || "\u2014"}
|
{data.creator || "\u2014"}
|
||||||
</button>
|
</button>
|
||||||
),
|
);
|
||||||
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
id: "tags",
|
id: "tags",
|
||||||
header: ({ column }) => <DataTableColumnHeader column={column} title="Tags" />,
|
header: ({ column }) => <DataTableColumnHeader column={column} title="Tags" />,
|
||||||
cell: ({ row }) => {
|
cell: ({ row }) => {
|
||||||
const tags = row.original.tags;
|
const data = row.original;
|
||||||
|
const tags = isGroupRow(data) ? data.combinedTags : data.tags;
|
||||||
if (tags.length === 0) {
|
if (tags.length === 0) {
|
||||||
|
if (isGroupRow(data)) {
|
||||||
|
return <span className="text-sm text-muted-foreground">{"\u2014"}</span>;
|
||||||
|
}
|
||||||
return (
|
return (
|
||||||
<button
|
<button
|
||||||
className="text-sm text-muted-foreground hover:text-foreground cursor-pointer"
|
className="text-sm text-muted-foreground hover:text-foreground cursor-pointer"
|
||||||
onClick={() => onSetTags(row.original)}
|
onClick={() => onSetTags(data)}
|
||||||
title="Click to add tags"
|
title="Click to add tags"
|
||||||
>
|
>
|
||||||
{"\u2014"}
|
{"\u2014"}
|
||||||
</button>
|
</button>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
const clickHandler = isGroupRow(data) ? undefined : () => onSetTags(data as PackageTableRow);
|
||||||
return (
|
return (
|
||||||
<button
|
<button
|
||||||
className="flex flex-wrap gap-1 cursor-pointer"
|
className={`flex flex-wrap gap-1 ${clickHandler ? "cursor-pointer" : "cursor-default"}`}
|
||||||
onClick={() => onSetTags(row.original)}
|
onClick={clickHandler}
|
||||||
title="Click to edit tags"
|
title={clickHandler ? "Click to edit tags" : undefined}
|
||||||
>
|
>
|
||||||
{tags.map((tag) => (
|
{tags.map((tag) => (
|
||||||
<Badge
|
<Badge
|
||||||
@@ -175,7 +359,10 @@ export function getPackageColumns({
|
|||||||
</button>
|
</button>
|
||||||
);
|
);
|
||||||
},
|
},
|
||||||
accessorFn: (row) => row.tags.join(", "),
|
accessorFn: (row) => {
|
||||||
|
if (isGroupRow(row)) return row.combinedTags.join(", ");
|
||||||
|
return row.tags.join(", ");
|
||||||
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
id: "channel",
|
id: "channel",
|
||||||
@@ -190,31 +377,73 @@ export function getPackageColumns({
|
|||||||
{
|
{
|
||||||
accessorKey: "indexedAt",
|
accessorKey: "indexedAt",
|
||||||
header: ({ column }) => <DataTableColumnHeader column={column} title="Indexed" />,
|
header: ({ column }) => <DataTableColumnHeader column={column} title="Indexed" />,
|
||||||
cell: ({ row }) => (
|
cell: ({ row }) => {
|
||||||
|
const data = row.original;
|
||||||
|
const date = isGroupRow(data) ? data.latestIndexedAt : data.indexedAt;
|
||||||
|
return (
|
||||||
<span className="text-sm text-muted-foreground">
|
<span className="text-sm text-muted-foreground">
|
||||||
{new Date(row.original.indexedAt).toLocaleDateString()}
|
{new Date(date).toLocaleDateString()}
|
||||||
</span>
|
</span>
|
||||||
),
|
);
|
||||||
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
id: "actions",
|
id: "actions",
|
||||||
cell: ({ row }) => (
|
cell: ({ row }) => {
|
||||||
|
const data = row.original;
|
||||||
|
if (isGroupRow(data)) {
|
||||||
|
return (
|
||||||
|
<div className="flex items-center gap-0.5">
|
||||||
|
<Button
|
||||||
|
variant="ghost"
|
||||||
|
size="icon"
|
||||||
|
className="h-8 w-8"
|
||||||
|
onClick={() => onSendAllInGroup(data.id)}
|
||||||
|
title="Send all packages in group"
|
||||||
|
>
|
||||||
|
<Send className="h-4 w-4" />
|
||||||
|
</Button>
|
||||||
|
<Button
|
||||||
|
variant="ghost"
|
||||||
|
size="icon"
|
||||||
|
className="h-8 w-8"
|
||||||
|
onClick={() => onDissolveGroup(data.id)}
|
||||||
|
title="Dissolve group"
|
||||||
|
>
|
||||||
|
<Ungroup className="h-4 w-4" />
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return (
|
||||||
<div className="flex items-center gap-0.5">
|
<div className="flex items-center gap-0.5">
|
||||||
<SendToTelegramButton
|
<SendToTelegramButton
|
||||||
packageId={row.original.id}
|
packageId={data.id}
|
||||||
packageName={row.original.fileName}
|
packageName={data.fileName}
|
||||||
variant="icon"
|
variant="icon"
|
||||||
/>
|
/>
|
||||||
<Button
|
<Button
|
||||||
variant="ghost"
|
variant="ghost"
|
||||||
size="icon"
|
size="icon"
|
||||||
className="h-8 w-8"
|
className="h-8 w-8"
|
||||||
onClick={() => onViewFiles(row.original)}
|
onClick={() => onViewFiles(data)}
|
||||||
>
|
>
|
||||||
<Eye className="h-4 w-4" />
|
<Eye className="h-4 w-4" />
|
||||||
</Button>
|
</Button>
|
||||||
|
{data._isGroupMember && (
|
||||||
|
<Button
|
||||||
|
variant="ghost"
|
||||||
|
size="icon"
|
||||||
|
className="h-8 w-8"
|
||||||
|
onClick={() => onRemoveFromGroup(data.id)}
|
||||||
|
title="Remove from group"
|
||||||
|
>
|
||||||
|
<Ungroup className="h-3.5 w-3.5" />
|
||||||
|
</Button>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
),
|
);
|
||||||
|
},
|
||||||
enableHiding: false,
|
enableHiding: false,
|
||||||
},
|
},
|
||||||
];
|
];
|
||||||
|
|||||||
@@ -1,11 +1,17 @@
|
|||||||
"use client";
|
"use client";
|
||||||
|
|
||||||
import { useState, useCallback, useTransition } from "react";
|
import { useState, useCallback, useTransition, useMemo, useRef } from "react";
|
||||||
import { useRouter, usePathname, useSearchParams } from "next/navigation";
|
import { useRouter, usePathname, useSearchParams } from "next/navigation";
|
||||||
import { toast } from "sonner";
|
import { toast } from "sonner";
|
||||||
import { Search } from "lucide-react";
|
import { Search, Layers } from "lucide-react";
|
||||||
import { useDataTable } from "@/hooks/use-data-table";
|
import { useDataTable } from "@/hooks/use-data-table";
|
||||||
import { getPackageColumns, type PackageRow } from "./package-columns";
|
import {
|
||||||
|
getPackageColumns,
|
||||||
|
type PackageRow,
|
||||||
|
type StlTableRow,
|
||||||
|
type PackageTableRow,
|
||||||
|
type GroupHeaderRow,
|
||||||
|
} from "./package-columns";
|
||||||
import { PackageFilesDrawer } from "./package-files-drawer";
|
import { PackageFilesDrawer } from "./package-files-drawer";
|
||||||
import { IngestionStatus } from "./ingestion-status";
|
import { IngestionStatus } from "./ingestion-status";
|
||||||
import { SkippedPackagesTab } from "./skipped-packages-tab";
|
import { SkippedPackagesTab } from "./skipped-packages-tab";
|
||||||
@@ -14,6 +20,7 @@ import { DataTablePagination } from "@/components/shared/data-table-pagination";
|
|||||||
import { DataTableViewOptions } from "@/components/shared/data-table-view-options";
|
import { DataTableViewOptions } from "@/components/shared/data-table-view-options";
|
||||||
import { PageHeader } from "@/components/shared/page-header";
|
import { PageHeader } from "@/components/shared/page-header";
|
||||||
import { Input } from "@/components/ui/input";
|
import { Input } from "@/components/ui/input";
|
||||||
|
import { Button } from "@/components/ui/button";
|
||||||
import {
|
import {
|
||||||
Select,
|
Select,
|
||||||
SelectContent,
|
SelectContent,
|
||||||
@@ -21,14 +28,31 @@ import {
|
|||||||
SelectTrigger,
|
SelectTrigger,
|
||||||
SelectValue,
|
SelectValue,
|
||||||
} from "@/components/ui/select";
|
} from "@/components/ui/select";
|
||||||
|
import {
|
||||||
|
Dialog,
|
||||||
|
DialogContent,
|
||||||
|
DialogDescription,
|
||||||
|
DialogFooter,
|
||||||
|
DialogHeader,
|
||||||
|
DialogTitle,
|
||||||
|
} from "@/components/ui/dialog";
|
||||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
|
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
|
||||||
import { Badge } from "@/components/ui/badge";
|
import { Badge } from "@/components/ui/badge";
|
||||||
import type { IngestionAccountStatus } from "@/lib/telegram/types";
|
import type { DisplayItem, IngestionAccountStatus } from "@/lib/telegram/types";
|
||||||
import type { SkippedRow } from "./skipped-columns";
|
import type { SkippedRow } from "./skipped-columns";
|
||||||
import { updatePackageCreator, updatePackageTags } from "../actions";
|
import {
|
||||||
|
updatePackageCreator,
|
||||||
|
updatePackageTags,
|
||||||
|
renameGroupAction,
|
||||||
|
dissolveGroupAction,
|
||||||
|
createGroupAction,
|
||||||
|
removeFromGroupAction,
|
||||||
|
sendAllInGroupAction,
|
||||||
|
updateGroupPreviewAction,
|
||||||
|
} from "../actions";
|
||||||
|
|
||||||
interface StlTableProps {
|
interface StlTableProps {
|
||||||
data: PackageRow[];
|
data: DisplayItem[];
|
||||||
pageCount: number;
|
pageCount: number;
|
||||||
totalCount: number;
|
totalCount: number;
|
||||||
ingestionStatus: IngestionAccountStatus[];
|
ingestionStatus: IngestionAccountStatus[];
|
||||||
@@ -58,6 +82,88 @@ export function StlTable({
|
|||||||
const [viewPkg, setViewPkg] = useState<PackageRow | null>(null);
|
const [viewPkg, setViewPkg] = useState<PackageRow | null>(null);
|
||||||
const [, startTransition] = useTransition();
|
const [, startTransition] = useTransition();
|
||||||
|
|
||||||
|
// Group expansion state
|
||||||
|
const [expandedGroups, setExpandedGroups] = useState<Set<string>>(new Set());
|
||||||
|
|
||||||
|
// Package selection state (for manual grouping)
|
||||||
|
const [selectedPackages, setSelectedPackages] = useState<Set<string>>(new Set());
|
||||||
|
|
||||||
|
// Create group dialog state
|
||||||
|
const [createGroupOpen, setCreateGroupOpen] = useState(false);
|
||||||
|
const [groupName, setGroupName] = useState("");
|
||||||
|
|
||||||
|
// Group preview upload ref
|
||||||
|
const previewInputRef = useRef<HTMLInputElement>(null);
|
||||||
|
const [uploadGroupId, setUploadGroupId] = useState<string | null>(null);
|
||||||
|
|
||||||
|
const toggleGroup = useCallback((groupId: string) => {
|
||||||
|
setExpandedGroups((prev) => {
|
||||||
|
const next = new Set(prev);
|
||||||
|
if (next.has(groupId)) {
|
||||||
|
next.delete(groupId);
|
||||||
|
} else {
|
||||||
|
next.add(groupId);
|
||||||
|
}
|
||||||
|
return next;
|
||||||
|
});
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const toggleSelect = useCallback((packageId: string) => {
|
||||||
|
setSelectedPackages((prev) => {
|
||||||
|
const next = new Set(prev);
|
||||||
|
if (next.has(packageId)) {
|
||||||
|
next.delete(packageId);
|
||||||
|
} else {
|
||||||
|
next.add(packageId);
|
||||||
|
}
|
||||||
|
return next;
|
||||||
|
});
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
// Flatten DisplayItem[] into StlTableRow[] based on expansion state
|
||||||
|
const tableRows: StlTableRow[] = useMemo(() => {
|
||||||
|
const rows: StlTableRow[] = [];
|
||||||
|
for (const item of data) {
|
||||||
|
if (item.type === "package") {
|
||||||
|
rows.push({
|
||||||
|
...item.data,
|
||||||
|
_rowType: "package" as const,
|
||||||
|
_groupId: null,
|
||||||
|
_isGroupMember: false,
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
const group = item.data;
|
||||||
|
const isExpanded = expandedGroups.has(group.id);
|
||||||
|
rows.push({
|
||||||
|
_rowType: "group" as const,
|
||||||
|
id: group.id,
|
||||||
|
name: group.name,
|
||||||
|
hasPreview: group.hasPreview,
|
||||||
|
totalFileSize: group.totalFileSize,
|
||||||
|
totalFileCount: group.totalFileCount,
|
||||||
|
packageCount: group.packageCount,
|
||||||
|
combinedTags: group.combinedTags,
|
||||||
|
archiveTypes: group.archiveTypes,
|
||||||
|
latestIndexedAt: group.latestIndexedAt,
|
||||||
|
sourceChannel: group.sourceChannel,
|
||||||
|
_expanded: isExpanded,
|
||||||
|
});
|
||||||
|
if (isExpanded) {
|
||||||
|
for (const pkg of group.packages) {
|
||||||
|
rows.push({
|
||||||
|
...pkg,
|
||||||
|
_rowType: "package" as const,
|
||||||
|
_groupId: group.id,
|
||||||
|
_isGroupMember: true,
|
||||||
|
packageGroupId: group.id,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return rows;
|
||||||
|
}, [data, expandedGroups]);
|
||||||
|
|
||||||
const updateSearch = useCallback(
|
const updateSearch = useCallback(
|
||||||
(value: string) => {
|
(value: string) => {
|
||||||
setSearchValue(value);
|
setSearchValue(value);
|
||||||
@@ -103,6 +209,131 @@ export function StlTable({
|
|||||||
[router, pathname, searchParams]
|
[router, pathname, searchParams]
|
||||||
);
|
);
|
||||||
|
|
||||||
|
const handleRenameGroup = useCallback(
|
||||||
|
(groupId: string, currentName: string) => {
|
||||||
|
const value = prompt("Enter group name:", currentName);
|
||||||
|
if (value === null || value.trim() === currentName) return;
|
||||||
|
startTransition(async () => {
|
||||||
|
const result = await renameGroupAction(groupId, value);
|
||||||
|
if (result.success) {
|
||||||
|
toast.success(`Group renamed to "${value.trim()}"`);
|
||||||
|
router.refresh();
|
||||||
|
} else {
|
||||||
|
toast.error(result.error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
},
|
||||||
|
[router]
|
||||||
|
);
|
||||||
|
|
||||||
|
const handleDissolveGroup = useCallback(
|
||||||
|
(groupId: string) => {
|
||||||
|
if (!confirm("Dissolve this group? Packages will become standalone items.")) return;
|
||||||
|
startTransition(async () => {
|
||||||
|
const result = await dissolveGroupAction(groupId);
|
||||||
|
if (result.success) {
|
||||||
|
toast.success("Group dissolved");
|
||||||
|
setExpandedGroups((prev) => {
|
||||||
|
const next = new Set(prev);
|
||||||
|
next.delete(groupId);
|
||||||
|
return next;
|
||||||
|
});
|
||||||
|
router.refresh();
|
||||||
|
} else {
|
||||||
|
toast.error(result.error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
},
|
||||||
|
[router]
|
||||||
|
);
|
||||||
|
|
||||||
|
const handleSendAllInGroup = useCallback(
|
||||||
|
(groupId: string) => {
|
||||||
|
if (!confirm("Send all packages in this group to your Telegram?")) return;
|
||||||
|
startTransition(async () => {
|
||||||
|
const result = await sendAllInGroupAction(groupId);
|
||||||
|
if (result.success) {
|
||||||
|
toast.success("Group packages queued for sending");
|
||||||
|
router.refresh();
|
||||||
|
} else {
|
||||||
|
toast.error(result.error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
},
|
||||||
|
[router]
|
||||||
|
);
|
||||||
|
|
||||||
|
const handleRemoveFromGroup = useCallback(
|
||||||
|
(packageId: string) => {
|
||||||
|
startTransition(async () => {
|
||||||
|
const result = await removeFromGroupAction(packageId);
|
||||||
|
if (result.success) {
|
||||||
|
toast.success("Package removed from group");
|
||||||
|
router.refresh();
|
||||||
|
} else {
|
||||||
|
toast.error(result.error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
},
|
||||||
|
[router]
|
||||||
|
);
|
||||||
|
|
||||||
|
const handleCreateGroup = useCallback(() => {
|
||||||
|
if (selectedPackages.size < 2) return;
|
||||||
|
setGroupName("");
|
||||||
|
setCreateGroupOpen(true);
|
||||||
|
}, [selectedPackages.size]);
|
||||||
|
|
||||||
|
const submitCreateGroup = useCallback(() => {
|
||||||
|
if (!groupName.trim() || selectedPackages.size < 2) return;
|
||||||
|
const ids = Array.from(selectedPackages);
|
||||||
|
startTransition(async () => {
|
||||||
|
const result = await createGroupAction(groupName, ids);
|
||||||
|
if (result.success) {
|
||||||
|
toast.success(`Group "${groupName.trim()}" created`);
|
||||||
|
setSelectedPackages(new Set());
|
||||||
|
setCreateGroupOpen(false);
|
||||||
|
router.refresh();
|
||||||
|
} else {
|
||||||
|
toast.error(result.error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}, [groupName, selectedPackages, router]);
|
||||||
|
|
||||||
|
// Group preview upload handler (Task 12)
|
||||||
|
const handleGroupPreviewUpload = useCallback((groupId: string) => {
|
||||||
|
setUploadGroupId(groupId);
|
||||||
|
// Trigger file input after state update
|
||||||
|
setTimeout(() => {
|
||||||
|
previewInputRef.current?.click();
|
||||||
|
}, 0);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const handlePreviewFileChange = useCallback(
|
||||||
|
(e: React.ChangeEvent<HTMLInputElement>) => {
|
||||||
|
const file = e.target.files?.[0];
|
||||||
|
if (!file || !uploadGroupId) return;
|
||||||
|
|
||||||
|
const formData = new FormData();
|
||||||
|
formData.append("file", file);
|
||||||
|
|
||||||
|
startTransition(async () => {
|
||||||
|
const result = await updateGroupPreviewAction(uploadGroupId, formData);
|
||||||
|
if (result.success) {
|
||||||
|
toast.success("Group preview updated");
|
||||||
|
router.refresh();
|
||||||
|
} else {
|
||||||
|
toast.error(result.error);
|
||||||
|
}
|
||||||
|
setUploadGroupId(null);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Reset input so the same file can be selected again
|
||||||
|
e.target.value = "";
|
||||||
|
},
|
||||||
|
[uploadGroupId, router]
|
||||||
|
);
|
||||||
|
|
||||||
const columns = getPackageColumns({
|
const columns = getPackageColumns({
|
||||||
onViewFiles: (pkg) => setViewPkg(pkg),
|
onViewFiles: (pkg) => setViewPkg(pkg),
|
||||||
searchTerm,
|
searchTerm,
|
||||||
@@ -136,9 +367,17 @@ export function StlTable({
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
},
|
},
|
||||||
|
onToggleGroup: toggleGroup,
|
||||||
|
onRenameGroup: handleRenameGroup,
|
||||||
|
onDissolveGroup: handleDissolveGroup,
|
||||||
|
onSendAllInGroup: handleSendAllInGroup,
|
||||||
|
onRemoveFromGroup: handleRemoveFromGroup,
|
||||||
|
onGroupPreviewUpload: handleGroupPreviewUpload,
|
||||||
|
selectedPackages,
|
||||||
|
onToggleSelect: toggleSelect,
|
||||||
});
|
});
|
||||||
|
|
||||||
const { table } = useDataTable({ data, columns, pageCount });
|
const { table } = useDataTable({ data: tableRows, columns, pageCount });
|
||||||
|
|
||||||
const activeTag = searchParams.get("tag") ?? "";
|
const activeTag = searchParams.get("tag") ?? "";
|
||||||
|
|
||||||
@@ -191,11 +430,37 @@ export function StlTable({
|
|||||||
</Select>
|
</Select>
|
||||||
)}
|
)}
|
||||||
<DataTableViewOptions table={table} />
|
<DataTableViewOptions table={table} />
|
||||||
|
{selectedPackages.size >= 2 && (
|
||||||
|
<Button
|
||||||
|
variant="outline"
|
||||||
|
size="sm"
|
||||||
|
className="h-9 gap-1.5"
|
||||||
|
onClick={handleCreateGroup}
|
||||||
|
>
|
||||||
|
<Layers className="h-3.5 w-3.5" />
|
||||||
|
Group {selectedPackages.size} Selected
|
||||||
|
</Button>
|
||||||
|
)}
|
||||||
|
{selectedPackages.size > 0 && selectedPackages.size < 2 && (
|
||||||
|
<span className="text-xs text-muted-foreground">
|
||||||
|
Select at least 2 packages to group
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<DataTable
|
<DataTable
|
||||||
table={table}
|
table={table}
|
||||||
emptyMessage="No packages found. Archives will appear here after ingestion."
|
emptyMessage="No packages found. Archives will appear here after ingestion."
|
||||||
|
rowClassName={(row) => {
|
||||||
|
const data = row.original as StlTableRow;
|
||||||
|
if (data._rowType === "group") {
|
||||||
|
return "bg-muted/30 border-border";
|
||||||
|
}
|
||||||
|
if (data._rowType === "package" && (data as PackageTableRow)._isGroupMember) {
|
||||||
|
return "bg-muted/10";
|
||||||
|
}
|
||||||
|
return "";
|
||||||
|
}}
|
||||||
/>
|
/>
|
||||||
<DataTablePagination table={table} totalCount={totalCount} />
|
<DataTablePagination table={table} totalCount={totalCount} />
|
||||||
</TabsContent>
|
</TabsContent>
|
||||||
@@ -217,6 +482,47 @@ export function StlTable({
|
|||||||
}}
|
}}
|
||||||
highlightTerm={searchTerm}
|
highlightTerm={searchTerm}
|
||||||
/>
|
/>
|
||||||
|
|
||||||
|
{/* Create Group Dialog */}
|
||||||
|
<Dialog open={createGroupOpen} onOpenChange={setCreateGroupOpen}>
|
||||||
|
<DialogContent className="sm:max-w-md">
|
||||||
|
<DialogHeader>
|
||||||
|
<DialogTitle>Create Package Group</DialogTitle>
|
||||||
|
<DialogDescription>
|
||||||
|
Group {selectedPackages.size} selected packages together. Enter a name for the group.
|
||||||
|
</DialogDescription>
|
||||||
|
</DialogHeader>
|
||||||
|
<div className="py-4">
|
||||||
|
<Input
|
||||||
|
placeholder="Group name..."
|
||||||
|
value={groupName}
|
||||||
|
onChange={(e) => setGroupName(e.target.value)}
|
||||||
|
onKeyDown={(e) => {
|
||||||
|
if (e.key === "Enter") submitCreateGroup();
|
||||||
|
}}
|
||||||
|
autoFocus
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<DialogFooter>
|
||||||
|
<Button variant="outline" onClick={() => setCreateGroupOpen(false)}>
|
||||||
|
Cancel
|
||||||
|
</Button>
|
||||||
|
<Button onClick={submitCreateGroup} disabled={!groupName.trim()}>
|
||||||
|
<Layers className="h-4 w-4 mr-1" />
|
||||||
|
Create Group
|
||||||
|
</Button>
|
||||||
|
</DialogFooter>
|
||||||
|
</DialogContent>
|
||||||
|
</Dialog>
|
||||||
|
|
||||||
|
{/* Hidden file input for group preview upload (Task 12) */}
|
||||||
|
<input
|
||||||
|
ref={previewInputRef}
|
||||||
|
type="file"
|
||||||
|
accept="image/jpeg,image/png,image/webp"
|
||||||
|
className="hidden"
|
||||||
|
onChange={handlePreviewFileChange}
|
||||||
|
/>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,6 +4,13 @@ import { auth } from "@/lib/auth";
|
|||||||
import { prisma } from "@/lib/prisma";
|
import { prisma } from "@/lib/prisma";
|
||||||
import type { ActionResult } from "@/types/api.types";
|
import type { ActionResult } from "@/types/api.types";
|
||||||
import { revalidatePath } from "next/cache";
|
import { revalidatePath } from "next/cache";
|
||||||
|
import {
|
||||||
|
updatePackageGroupName,
|
||||||
|
updatePackageGroupPreview,
|
||||||
|
createManualGroup,
|
||||||
|
removePackageFromGroup,
|
||||||
|
dissolveGroup,
|
||||||
|
} from "@/lib/telegram/queries";
|
||||||
|
|
||||||
const ALLOWED_IMAGE_TYPES = [
|
const ALLOWED_IMAGE_TYPES = [
|
||||||
"image/jpeg",
|
"image/jpeg",
|
||||||
@@ -322,3 +329,186 @@ export async function retryAllSkippedPackagesAction(
|
|||||||
return { success: false, error: "Failed to retry skipped packages" };
|
return { success: false, error: "Failed to retry skipped packages" };
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export async function renameGroupAction(
|
||||||
|
groupId: string,
|
||||||
|
name: string
|
||||||
|
): Promise<ActionResult> {
|
||||||
|
const session = await auth();
|
||||||
|
if (!session?.user?.id) return { success: false, error: "Unauthorized" };
|
||||||
|
|
||||||
|
if (!name.trim()) {
|
||||||
|
return { success: false, error: "Group name cannot be empty" };
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
await updatePackageGroupName(groupId, name);
|
||||||
|
revalidatePath("/stls");
|
||||||
|
return { success: true, data: undefined };
|
||||||
|
} catch {
|
||||||
|
return { success: false, error: "Failed to rename group" };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function dissolveGroupAction(
|
||||||
|
groupId: string
|
||||||
|
): Promise<ActionResult> {
|
||||||
|
const session = await auth();
|
||||||
|
if (!session?.user?.id) return { success: false, error: "Unauthorized" };
|
||||||
|
|
||||||
|
try {
|
||||||
|
await dissolveGroup(groupId);
|
||||||
|
revalidatePath("/stls");
|
||||||
|
return { success: true, data: undefined };
|
||||||
|
} catch {
|
||||||
|
return { success: false, error: "Failed to dissolve group" };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function createGroupAction(
|
||||||
|
name: string,
|
||||||
|
packageIds: string[]
|
||||||
|
): Promise<ActionResult> {
|
||||||
|
const session = await auth();
|
||||||
|
if (!session?.user?.id) return { success: false, error: "Unauthorized" };
|
||||||
|
|
||||||
|
if (!name.trim()) {
|
||||||
|
return { success: false, error: "Group name cannot be empty" };
|
||||||
|
}
|
||||||
|
if (packageIds.length < 2) {
|
||||||
|
return { success: false, error: "At least 2 packages are required to create a group" };
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
await createManualGroup(name, packageIds);
|
||||||
|
revalidatePath("/stls");
|
||||||
|
return { success: true, data: undefined };
|
||||||
|
} catch (err) {
|
||||||
|
const message = err instanceof Error ? err.message : "Failed to create group";
|
||||||
|
return { success: false, error: message };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function removeFromGroupAction(
|
||||||
|
packageId: string
|
||||||
|
): Promise<ActionResult> {
|
||||||
|
const session = await auth();
|
||||||
|
if (!session?.user?.id) return { success: false, error: "Unauthorized" };
|
||||||
|
|
||||||
|
try {
|
||||||
|
await removePackageFromGroup(packageId);
|
||||||
|
revalidatePath("/stls");
|
||||||
|
return { success: true, data: undefined };
|
||||||
|
} catch {
|
||||||
|
return { success: false, error: "Failed to remove package from group" };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function updateGroupPreviewAction(
|
||||||
|
groupId: string,
|
||||||
|
formData: FormData
|
||||||
|
): Promise<ActionResult> {
|
||||||
|
const session = await auth();
|
||||||
|
if (!session?.user?.id) return { success: false, error: "Unauthorized" };
|
||||||
|
|
||||||
|
const file = formData.get("file");
|
||||||
|
if (!(file instanceof File)) {
|
||||||
|
return { success: false, error: "No file provided" };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!ALLOWED_IMAGE_TYPES.includes(file.type as (typeof ALLOWED_IMAGE_TYPES)[number])) {
|
||||||
|
return { success: false, error: "Only JPG, PNG, and WebP images are accepted" };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (file.size > MAX_IMAGE_SIZE) {
|
||||||
|
return { success: false, error: "Image must be smaller than 2 MB" };
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const arrayBuffer = await file.arrayBuffer();
|
||||||
|
const buffer = Buffer.from(arrayBuffer);
|
||||||
|
await updatePackageGroupPreview(groupId, buffer);
|
||||||
|
revalidatePath("/stls");
|
||||||
|
return { success: true, data: undefined };
|
||||||
|
} catch {
|
||||||
|
return { success: false, error: "Failed to upload group preview image" };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function sendAllInGroupAction(
|
||||||
|
groupId: string
|
||||||
|
): Promise<ActionResult> {
|
||||||
|
const session = await auth();
|
||||||
|
if (!session?.user?.id) return { success: false, error: "Unauthorized" };
|
||||||
|
|
||||||
|
try {
|
||||||
|
const telegramLink = await prisma.telegramLink.findUnique({
|
||||||
|
where: { userId: session.user.id },
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!telegramLink) {
|
||||||
|
return { success: false, error: "No linked Telegram account. Link one in Settings." };
|
||||||
|
}
|
||||||
|
|
||||||
|
const group = await prisma.packageGroup.findUnique({
|
||||||
|
where: { id: groupId },
|
||||||
|
select: {
|
||||||
|
packages: {
|
||||||
|
select: { id: true, destChannelId: true, destMessageId: true, fileName: true },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!group) {
|
||||||
|
return { success: false, error: "Group not found" };
|
||||||
|
}
|
||||||
|
|
||||||
|
const sendablePackages = group.packages.filter(
|
||||||
|
(p) => p.destChannelId && p.destMessageId
|
||||||
|
);
|
||||||
|
|
||||||
|
if (sendablePackages.length === 0) {
|
||||||
|
return { success: false, error: "No packages in this group have been uploaded to a destination channel" };
|
||||||
|
}
|
||||||
|
|
||||||
|
let queued = 0;
|
||||||
|
for (const pkg of sendablePackages) {
|
||||||
|
// Only create if no existing PENDING/SENDING request for this package+link combo
|
||||||
|
const existing = await prisma.botSendRequest.findFirst({
|
||||||
|
where: {
|
||||||
|
packageId: pkg.id,
|
||||||
|
telegramLinkId: telegramLink.id,
|
||||||
|
status: { in: ["PENDING", "SENDING"] },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!existing) {
|
||||||
|
const sendRequest = await prisma.botSendRequest.create({
|
||||||
|
data: {
|
||||||
|
packageId: pkg.id,
|
||||||
|
telegramLinkId: telegramLink.id,
|
||||||
|
requestedByUserId: session.user.id,
|
||||||
|
status: "PENDING",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
// Notify the bot via pg_notify
|
||||||
|
try {
|
||||||
|
await prisma.$queryRawUnsafe(
|
||||||
|
`SELECT pg_notify('bot_send', $1)`,
|
||||||
|
sendRequest.id
|
||||||
|
);
|
||||||
|
} catch {
|
||||||
|
// Best-effort — the bot also polls periodically
|
||||||
|
}
|
||||||
|
|
||||||
|
queued++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
revalidatePath("/stls");
|
||||||
|
return { success: true, data: undefined };
|
||||||
|
} catch {
|
||||||
|
return { success: false, error: "Failed to send group packages" };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,7 +1,8 @@
|
|||||||
import { auth } from "@/lib/auth";
|
import { auth } from "@/lib/auth";
|
||||||
import { redirect } from "next/navigation";
|
import { redirect } from "next/navigation";
|
||||||
import { listPackages, searchPackages, getIngestionStatus, getAllPackageTags, listSkippedPackages, countSkippedPackages } from "@/lib/telegram/queries";
|
import { listDisplayItems, searchPackages, getIngestionStatus, getAllPackageTags, listSkippedPackages, countSkippedPackages } from "@/lib/telegram/queries";
|
||||||
import { StlTable } from "./_components/stl-table";
|
import { StlTable } from "./_components/stl-table";
|
||||||
|
import type { DisplayItem, PackageListItem } from "@/lib/telegram/types";
|
||||||
|
|
||||||
interface Props {
|
interface Props {
|
||||||
searchParams: Promise<Record<string, string | string[] | undefined>>;
|
searchParams: Promise<Record<string, string | string[] | undefined>>;
|
||||||
@@ -31,7 +32,7 @@ export default async function StlFilesPage({ searchParams }: Props) {
|
|||||||
limit: perPage,
|
limit: perPage,
|
||||||
searchIn: "both",
|
searchIn: "both",
|
||||||
})
|
})
|
||||||
: listPackages({
|
: listDisplayItems({
|
||||||
page,
|
page,
|
||||||
limit: perPage,
|
limit: perPage,
|
||||||
creator,
|
creator,
|
||||||
@@ -44,6 +45,11 @@ export default async function StlFilesPage({ searchParams }: Props) {
|
|||||||
countSkippedPackages(),
|
countSkippedPackages(),
|
||||||
]);
|
]);
|
||||||
|
|
||||||
|
// For search results, wrap as DisplayItem[]; for non-search, already DisplayItem[]
|
||||||
|
const displayItems: DisplayItem[] = search
|
||||||
|
? (result as { items: PackageListItem[] }).items.map((item) => ({ type: "package" as const, data: item }))
|
||||||
|
: (result as { items: DisplayItem[] }).items;
|
||||||
|
|
||||||
// Fetch skipped packages only if on that tab
|
// Fetch skipped packages only if on that tab
|
||||||
const skippedResult = tab === "skipped"
|
const skippedResult = tab === "skipped"
|
||||||
? await listSkippedPackages({ page, limit: perPage })
|
? await listSkippedPackages({ page, limit: perPage })
|
||||||
@@ -51,7 +57,7 @@ export default async function StlFilesPage({ searchParams }: Props) {
|
|||||||
|
|
||||||
return (
|
return (
|
||||||
<StlTable
|
<StlTable
|
||||||
data={result.items}
|
data={displayItems}
|
||||||
pageCount={result.pagination.totalPages}
|
pageCount={result.pagination.totalPages}
|
||||||
totalCount={result.pagination.total}
|
totalCount={result.pagination.total}
|
||||||
ingestionStatus={ingestionStatus}
|
ingestionStatus={ingestionStatus}
|
||||||
|
|||||||
36
src/app/api/groups/[id]/preview/route.ts
Normal file
36
src/app/api/groups/[id]/preview/route.ts
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
import { NextResponse } from "next/server";
|
||||||
|
import { prisma } from "@/lib/prisma";
|
||||||
|
import { authenticateApiRequest } from "@/lib/telegram/api-auth";
|
||||||
|
|
||||||
|
export async function GET(
|
||||||
|
request: Request,
|
||||||
|
{ params }: { params: Promise<{ id: string }> }
|
||||||
|
) {
|
||||||
|
const authResult = await authenticateApiRequest(request);
|
||||||
|
if ("error" in authResult) return authResult.error;
|
||||||
|
|
||||||
|
const { id } = await params;
|
||||||
|
|
||||||
|
const group = await prisma.packageGroup.findUnique({
|
||||||
|
where: { id },
|
||||||
|
select: { previewData: true },
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!group || !group.previewData) {
|
||||||
|
return new NextResponse(null, { status: 404 });
|
||||||
|
}
|
||||||
|
|
||||||
|
const buffer =
|
||||||
|
group.previewData instanceof Buffer
|
||||||
|
? group.previewData
|
||||||
|
: Buffer.from(group.previewData);
|
||||||
|
|
||||||
|
return new NextResponse(buffer, {
|
||||||
|
status: 200,
|
||||||
|
headers: {
|
||||||
|
"Content-Type": "image/jpeg",
|
||||||
|
"Content-Length": String(buffer.length),
|
||||||
|
"Cache-Control": "public, max-age=3600, immutable",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
"use client";
|
"use client";
|
||||||
|
|
||||||
import { type Table as TanStackTable, flexRender } from "@tanstack/react-table";
|
import { type Table as TanStackTable, type Row, flexRender } from "@tanstack/react-table";
|
||||||
import {
|
import {
|
||||||
Table,
|
Table,
|
||||||
TableBody,
|
TableBody,
|
||||||
@@ -10,13 +10,15 @@ import {
|
|||||||
TableRow,
|
TableRow,
|
||||||
} from "@/components/ui/table";
|
} from "@/components/ui/table";
|
||||||
import { EmptyState } from "./empty-state";
|
import { EmptyState } from "./empty-state";
|
||||||
|
import { cn } from "@/lib/utils";
|
||||||
|
|
||||||
interface DataTableProps<TData> {
|
interface DataTableProps<TData> {
|
||||||
table: TanStackTable<TData>;
|
table: TanStackTable<TData>;
|
||||||
emptyMessage?: string;
|
emptyMessage?: string;
|
||||||
|
rowClassName?: (row: Row<TData>) => string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function DataTable<TData>({ table, emptyMessage }: DataTableProps<TData>) {
|
export function DataTable<TData>({ table, emptyMessage, rowClassName }: DataTableProps<TData>) {
|
||||||
return (
|
return (
|
||||||
<div className="rounded-md border border-border">
|
<div className="rounded-md border border-border">
|
||||||
<Table>
|
<Table>
|
||||||
@@ -36,7 +38,10 @@ export function DataTable<TData>({ table, emptyMessage }: DataTableProps<TData>)
|
|||||||
<TableBody>
|
<TableBody>
|
||||||
{table.getRowModel().rows?.length ? (
|
{table.getRowModel().rows?.length ? (
|
||||||
table.getRowModel().rows.map((row) => (
|
table.getRowModel().rows.map((row) => (
|
||||||
<TableRow key={row.id} className="h-10 border-border hover:bg-muted/50">
|
<TableRow
|
||||||
|
key={row.id}
|
||||||
|
className={cn("h-10 border-border hover:bg-muted/50", rowClassName?.(row))}
|
||||||
|
>
|
||||||
{row.getVisibleCells().map((cell) => (
|
{row.getVisibleCells().map((cell) => (
|
||||||
<TableCell key={cell.id} className="py-1.5 text-sm">
|
<TableCell key={cell.id} className="py-1.5 text-sm">
|
||||||
{flexRender(cell.column.columnDef.cell, cell.getContext())}
|
{flexRender(cell.column.columnDef.cell, cell.getContext())}
|
||||||
|
|||||||
@@ -5,6 +5,8 @@ import type {
|
|||||||
PackageFileItem,
|
PackageFileItem,
|
||||||
IngestionAccountStatus,
|
IngestionAccountStatus,
|
||||||
SkippedPackageItem,
|
SkippedPackageItem,
|
||||||
|
DisplayItem,
|
||||||
|
PackageGroupRow,
|
||||||
} from "./types";
|
} from "./types";
|
||||||
|
|
||||||
export async function listPackages(options: {
|
export async function listPackages(options: {
|
||||||
@@ -73,6 +75,177 @@ export async function listPackages(options: {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export async function listDisplayItems(options: {
|
||||||
|
page: number;
|
||||||
|
limit: number;
|
||||||
|
channelId?: string;
|
||||||
|
creator?: string;
|
||||||
|
tag?: string;
|
||||||
|
sortBy: "indexedAt" | "fileName" | "fileSize";
|
||||||
|
order: "asc" | "desc";
|
||||||
|
}): Promise<{ items: DisplayItem[]; pagination: { page: number; limit: number; total: number; totalPages: number } }> {
|
||||||
|
const { page, limit, channelId, creator, tag, sortBy, order } = options;
|
||||||
|
|
||||||
|
// Build WHERE clause fragments for raw SQL
|
||||||
|
const conditions: string[] = [];
|
||||||
|
const params: unknown[] = [];
|
||||||
|
let paramIdx = 1;
|
||||||
|
|
||||||
|
if (channelId) {
|
||||||
|
conditions.push(`p."sourceChannelId" = $${paramIdx++}`);
|
||||||
|
params.push(channelId);
|
||||||
|
}
|
||||||
|
if (creator) {
|
||||||
|
conditions.push(`p."creator" = $${paramIdx++}`);
|
||||||
|
params.push(creator);
|
||||||
|
}
|
||||||
|
if (tag) {
|
||||||
|
conditions.push(`$${paramIdx++} = ANY(p."tags")`);
|
||||||
|
params.push(tag);
|
||||||
|
}
|
||||||
|
|
||||||
|
const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(" AND ")}` : "";
|
||||||
|
|
||||||
|
const sortCol = sortBy === "fileName" ? `"fileName"` : sortBy === "fileSize" ? `"fileSize"` : `"indexedAt"`;
|
||||||
|
const sortDir = order === "asc" ? "ASC" : "DESC";
|
||||||
|
|
||||||
|
// Step 1: Count display items
|
||||||
|
const countResult = await prisma.$queryRawUnsafe<[{ count: bigint }]>(
|
||||||
|
`SELECT COUNT(*) AS count FROM (
|
||||||
|
SELECT DISTINCT COALESCE(p."packageGroupId", p."id") AS display_id
|
||||||
|
FROM packages p
|
||||||
|
${whereClause}
|
||||||
|
) AS display_items`,
|
||||||
|
...params
|
||||||
|
);
|
||||||
|
const total = Number(countResult[0].count);
|
||||||
|
|
||||||
|
// Step 2: Get display item IDs for this page
|
||||||
|
const limitParam = paramIdx++;
|
||||||
|
const offsetParam = paramIdx++;
|
||||||
|
const displayRows = await prisma.$queryRawUnsafe<
|
||||||
|
{ display_id: string; display_type: string }[]
|
||||||
|
>(
|
||||||
|
`SELECT
|
||||||
|
COALESCE(p."packageGroupId", p."id") AS display_id,
|
||||||
|
CASE WHEN p."packageGroupId" IS NOT NULL THEN 'group' ELSE 'package' END AS display_type,
|
||||||
|
MAX(p.${sortCol}) AS sort_value
|
||||||
|
FROM packages p
|
||||||
|
${whereClause}
|
||||||
|
GROUP BY COALESCE(p."packageGroupId", p."id"),
|
||||||
|
CASE WHEN p."packageGroupId" IS NOT NULL THEN 'group' ELSE 'package' END
|
||||||
|
ORDER BY sort_value ${sortDir}
|
||||||
|
LIMIT $${limitParam} OFFSET $${offsetParam}`,
|
||||||
|
...params, limit, (page - 1) * limit
|
||||||
|
);
|
||||||
|
|
||||||
|
// Step 3: Fetch full data
|
||||||
|
const groupIds = displayRows.filter((r) => r.display_type === "group").map((r) => r.display_id);
|
||||||
|
const packageIds = displayRows.filter((r) => r.display_type === "package").map((r) => r.display_id);
|
||||||
|
|
||||||
|
const standalonePackages = packageIds.length > 0
|
||||||
|
? await prisma.package.findMany({
|
||||||
|
where: { id: { in: packageIds } },
|
||||||
|
select: {
|
||||||
|
id: true, fileName: true, fileSize: true, contentHash: true,
|
||||||
|
archiveType: true, fileCount: true, isMultipart: true,
|
||||||
|
indexedAt: true, creator: true, tags: true, previewData: true,
|
||||||
|
sourceChannel: { select: { id: true, title: true } },
|
||||||
|
},
|
||||||
|
})
|
||||||
|
: [];
|
||||||
|
|
||||||
|
const groups = groupIds.length > 0
|
||||||
|
? await prisma.packageGroup.findMany({
|
||||||
|
where: { id: { in: groupIds } },
|
||||||
|
select: {
|
||||||
|
id: true, name: true, previewData: true,
|
||||||
|
sourceChannel: { select: { id: true, title: true } },
|
||||||
|
packages: {
|
||||||
|
select: {
|
||||||
|
id: true, fileName: true, fileSize: true, contentHash: true,
|
||||||
|
archiveType: true, fileCount: true, isMultipart: true,
|
||||||
|
indexedAt: true, creator: true, tags: true, previewData: true,
|
||||||
|
sourceChannel: { select: { id: true, title: true } },
|
||||||
|
},
|
||||||
|
orderBy: { indexedAt: "desc" },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
: [];
|
||||||
|
|
||||||
|
// Build DisplayItem array in the original sort order
|
||||||
|
const packageMap = new Map(standalonePackages.map((p) => [p.id, p]));
|
||||||
|
const groupMap = new Map(groups.map((g) => [g.id, g]));
|
||||||
|
|
||||||
|
const items: DisplayItem[] = displayRows.map((row) => {
|
||||||
|
if (row.display_type === "package") {
|
||||||
|
const pkg = packageMap.get(row.display_id)!;
|
||||||
|
return {
|
||||||
|
type: "package" as const,
|
||||||
|
data: {
|
||||||
|
id: pkg.id,
|
||||||
|
fileName: pkg.fileName,
|
||||||
|
fileSize: pkg.fileSize.toString(),
|
||||||
|
contentHash: pkg.contentHash,
|
||||||
|
archiveType: pkg.archiveType,
|
||||||
|
fileCount: pkg.fileCount,
|
||||||
|
isMultipart: pkg.isMultipart,
|
||||||
|
hasPreview: pkg.previewData !== null,
|
||||||
|
creator: pkg.creator,
|
||||||
|
tags: pkg.tags,
|
||||||
|
indexedAt: pkg.indexedAt.toISOString(),
|
||||||
|
sourceChannel: pkg.sourceChannel,
|
||||||
|
matchedFileCount: 0,
|
||||||
|
matchedByContent: false,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
} else {
|
||||||
|
const grp = groupMap.get(row.display_id)!;
|
||||||
|
const allTags = [...new Set(grp.packages.flatMap((p) => p.tags))];
|
||||||
|
const archiveTypes = [...new Set(grp.packages.map((p) => p.archiveType))] as PackageGroupRow["archiveTypes"];
|
||||||
|
return {
|
||||||
|
type: "group" as const,
|
||||||
|
data: {
|
||||||
|
id: grp.id,
|
||||||
|
name: grp.name,
|
||||||
|
hasPreview: grp.previewData !== null,
|
||||||
|
totalFileSize: grp.packages.reduce((sum, p) => sum + p.fileSize, BigInt(0)).toString(),
|
||||||
|
totalFileCount: grp.packages.reduce((sum, p) => sum + p.fileCount, 0),
|
||||||
|
packageCount: grp.packages.length,
|
||||||
|
combinedTags: allTags,
|
||||||
|
archiveTypes,
|
||||||
|
latestIndexedAt: grp.packages.length > 0
|
||||||
|
? grp.packages[0].indexedAt.toISOString()
|
||||||
|
: new Date().toISOString(),
|
||||||
|
sourceChannel: grp.sourceChannel,
|
||||||
|
packages: grp.packages.map((pkg) => ({
|
||||||
|
id: pkg.id,
|
||||||
|
fileName: pkg.fileName,
|
||||||
|
fileSize: pkg.fileSize.toString(),
|
||||||
|
contentHash: pkg.contentHash,
|
||||||
|
archiveType: pkg.archiveType,
|
||||||
|
fileCount: pkg.fileCount,
|
||||||
|
isMultipart: pkg.isMultipart,
|
||||||
|
hasPreview: pkg.previewData !== null,
|
||||||
|
creator: pkg.creator,
|
||||||
|
tags: pkg.tags,
|
||||||
|
indexedAt: pkg.indexedAt.toISOString(),
|
||||||
|
sourceChannel: pkg.sourceChannel,
|
||||||
|
matchedFileCount: 0,
|
||||||
|
matchedByContent: false,
|
||||||
|
})),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
items,
|
||||||
|
pagination: { page, limit, total, totalPages: Math.ceil(total / limit) },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
export async function getPackageById(
|
export async function getPackageById(
|
||||||
id: string
|
id: string
|
||||||
): Promise<PackageDetail | null> {
|
): Promise<PackageDetail | null> {
|
||||||
@@ -203,7 +376,16 @@ export async function searchPackages(options: {
|
|||||||
).map((p) => p.id)
|
).map((p) => p.id)
|
||||||
: [];
|
: [];
|
||||||
|
|
||||||
const allIds = [...new Set([...fileMatchedIds, ...packageNameIds])];
|
// Also match by group name
|
||||||
|
const groupNameMatches = await prisma.package.findMany({
|
||||||
|
where: {
|
||||||
|
packageGroup: { name: { contains: q, mode: "insensitive" } },
|
||||||
|
},
|
||||||
|
select: { id: true },
|
||||||
|
});
|
||||||
|
const groupMatchedIds = groupNameMatches.map((p) => p.id);
|
||||||
|
|
||||||
|
const allIds = [...new Set([...fileMatchedIds, ...packageNameIds, ...groupMatchedIds])];
|
||||||
|
|
||||||
const [items, total] = await Promise.all([
|
const [items, total] = await Promise.all([
|
||||||
prisma.package.findMany({
|
prisma.package.findMany({
|
||||||
@@ -388,3 +570,103 @@ export async function listSkippedPackages(options: {
|
|||||||
export async function countSkippedPackages(): Promise<number> {
|
export async function countSkippedPackages(): Promise<number> {
|
||||||
return prisma.skippedPackage.count();
|
return prisma.skippedPackage.count();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export async function getPackageGroup(groupId: string) {
|
||||||
|
return prisma.packageGroup.findUnique({
|
||||||
|
where: { id: groupId },
|
||||||
|
select: {
|
||||||
|
id: true, name: true, previewData: true, mediaAlbumId: true,
|
||||||
|
sourceChannelId: true, createdAt: true,
|
||||||
|
sourceChannel: { select: { id: true, title: true } },
|
||||||
|
packages: {
|
||||||
|
select: {
|
||||||
|
id: true, fileName: true, fileSize: true, archiveType: true,
|
||||||
|
fileCount: true, creator: true, tags: true,
|
||||||
|
},
|
||||||
|
orderBy: { indexedAt: "desc" },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function updatePackageGroupName(groupId: string, name: string) {
|
||||||
|
return prisma.packageGroup.update({
|
||||||
|
where: { id: groupId },
|
||||||
|
data: { name: name.trim() },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function updatePackageGroupPreview(groupId: string, previewData: Buffer) {
|
||||||
|
return prisma.packageGroup.update({
|
||||||
|
where: { id: groupId },
|
||||||
|
data: { previewData: new Uint8Array(previewData) },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function createManualGroup(name: string, packageIds: string[]) {
|
||||||
|
// Verify all packages belong to the same channel
|
||||||
|
const pkgs = await prisma.package.findMany({
|
||||||
|
where: { id: { in: packageIds } },
|
||||||
|
select: { sourceChannelId: true },
|
||||||
|
});
|
||||||
|
if (pkgs.length === 0) {
|
||||||
|
throw new Error("No matching packages found");
|
||||||
|
}
|
||||||
|
const channelIds = new Set(pkgs.map((p) => p.sourceChannelId));
|
||||||
|
if (channelIds.size > 1) {
|
||||||
|
throw new Error("Cannot group packages from different channels");
|
||||||
|
}
|
||||||
|
|
||||||
|
const firstPkg = pkgs[0];
|
||||||
|
const group = await prisma.packageGroup.create({
|
||||||
|
data: {
|
||||||
|
name: name.trim(),
|
||||||
|
sourceChannelId: firstPkg.sourceChannelId,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
await prisma.package.updateMany({
|
||||||
|
where: { id: { in: packageIds } },
|
||||||
|
data: { packageGroupId: group.id },
|
||||||
|
});
|
||||||
|
|
||||||
|
// Clean up empty groups left behind
|
||||||
|
await prisma.packageGroup.deleteMany({
|
||||||
|
where: { packages: { none: {} }, id: { not: group.id } },
|
||||||
|
});
|
||||||
|
|
||||||
|
return group;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function addPackagesToGroup(packageIds: string[], groupId: string) {
|
||||||
|
await prisma.package.updateMany({
|
||||||
|
where: { id: { in: packageIds } },
|
||||||
|
data: { packageGroupId: groupId },
|
||||||
|
});
|
||||||
|
await prisma.packageGroup.deleteMany({
|
||||||
|
where: { packages: { none: {} } },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function removePackageFromGroup(packageId: string) {
|
||||||
|
const pkg = await prisma.package.findUniqueOrThrow({
|
||||||
|
where: { id: packageId },
|
||||||
|
select: { packageGroupId: true },
|
||||||
|
});
|
||||||
|
if (!pkg.packageGroupId) return;
|
||||||
|
await prisma.package.update({
|
||||||
|
where: { id: packageId },
|
||||||
|
data: { packageGroupId: null },
|
||||||
|
});
|
||||||
|
await prisma.packageGroup.deleteMany({
|
||||||
|
where: { id: pkg.packageGroupId, packages: { none: {} } },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function dissolveGroup(groupId: string) {
|
||||||
|
await prisma.package.updateMany({
|
||||||
|
where: { packageGroupId: groupId },
|
||||||
|
data: { packageGroupId: null },
|
||||||
|
});
|
||||||
|
await prisma.packageGroup.delete({ where: { id: groupId } });
|
||||||
|
}
|
||||||
|
|||||||
@@ -68,6 +68,24 @@ export interface PaginatedResponse<T> {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface PackageGroupRow {
|
||||||
|
id: string;
|
||||||
|
name: string;
|
||||||
|
hasPreview: boolean;
|
||||||
|
totalFileSize: string;
|
||||||
|
totalFileCount: number;
|
||||||
|
packageCount: number;
|
||||||
|
combinedTags: string[];
|
||||||
|
archiveTypes: ("ZIP" | "RAR" | "SEVEN_Z" | "DOCUMENT")[];
|
||||||
|
latestIndexedAt: string;
|
||||||
|
sourceChannel: { id: string; title: string };
|
||||||
|
packages: PackageListItem[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export type DisplayItem =
|
||||||
|
| { type: "package"; data: PackageListItem }
|
||||||
|
| { type: "group"; data: PackageGroupRow };
|
||||||
|
|
||||||
export interface IngestionAccountStatus {
|
export interface IngestionAccountStatus {
|
||||||
id: string;
|
id: string;
|
||||||
displayName: string | null;
|
displayName: string | null;
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ export interface TelegramMessage {
|
|||||||
fileId: string;
|
fileId: string;
|
||||||
fileSize: bigint;
|
fileSize: bigint;
|
||||||
date: Date;
|
date: Date;
|
||||||
|
mediaAlbumId?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ArchiveSet {
|
export interface ArchiveSet {
|
||||||
|
|||||||
@@ -6,8 +6,12 @@ import { childLogger } from "../util/logger.js";
|
|||||||
|
|
||||||
const log = childLogger("split");
|
const log = childLogger("split");
|
||||||
|
|
||||||
/** 2GB in bytes — Telegram's file size limit */
|
/**
|
||||||
const MAX_PART_SIZE = 2n * 1024n * 1024n * 1024n;
|
* 1950 MiB — safely under Telegram's 2GB upload limit.
|
||||||
|
* At exactly 2GiB, TDLib's internal 512KB chunking can exceed Telegram's
|
||||||
|
* 4000-part threshold, causing FILE_PARTS_INVALID errors.
|
||||||
|
*/
|
||||||
|
const MAX_PART_SIZE = 1950n * 1024n * 1024n;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Split a file into ≤2GB parts using byte-level splitting.
|
* Split a file into ≤2GB parts using byte-level splitting.
|
||||||
|
|||||||
@@ -535,3 +535,53 @@ export async function deleteSkippedPackage(
|
|||||||
where: { sourceChannelId, sourceMessageId },
|
where: { sourceChannelId, sourceMessageId },
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export async function createOrFindPackageGroup(input: {
|
||||||
|
mediaAlbumId: string;
|
||||||
|
sourceChannelId: string;
|
||||||
|
name: string;
|
||||||
|
previewData?: Buffer | null;
|
||||||
|
}): Promise<string> {
|
||||||
|
// findFirst + conditional create (Prisma doesn't support upsert on nullable compound unique)
|
||||||
|
const existing = await db.packageGroup.findFirst({
|
||||||
|
where: {
|
||||||
|
mediaAlbumId: input.mediaAlbumId,
|
||||||
|
sourceChannelId: input.sourceChannelId,
|
||||||
|
},
|
||||||
|
select: { id: true },
|
||||||
|
});
|
||||||
|
|
||||||
|
if (existing) return existing.id;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const group = await db.packageGroup.create({
|
||||||
|
data: {
|
||||||
|
mediaAlbumId: input.mediaAlbumId,
|
||||||
|
sourceChannelId: input.sourceChannelId,
|
||||||
|
name: input.name,
|
||||||
|
previewData: input.previewData ? new Uint8Array(input.previewData) : undefined,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return group.id;
|
||||||
|
} catch (err) {
|
||||||
|
// Handle race condition: another process created the group between our findFirst and create
|
||||||
|
if (err instanceof Error && err.message.includes("Unique constraint")) {
|
||||||
|
const raced = await db.packageGroup.findFirst({
|
||||||
|
where: { mediaAlbumId: input.mediaAlbumId, sourceChannelId: input.sourceChannelId },
|
||||||
|
select: { id: true },
|
||||||
|
});
|
||||||
|
if (raced) return raced.id;
|
||||||
|
}
|
||||||
|
throw err;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function linkPackagesToGroup(
|
||||||
|
packageIds: string[],
|
||||||
|
groupId: string
|
||||||
|
): Promise<void> {
|
||||||
|
await db.package.updateMany({
|
||||||
|
where: { id: { in: packageIds } },
|
||||||
|
data: { packageGroupId: groupId },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|||||||
79
worker/src/grouping.ts
Normal file
79
worker/src/grouping.ts
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
import type { Client } from "tdl";
|
||||||
|
import type { TelegramPhoto } from "./preview/match.js";
|
||||||
|
import { downloadPhotoThumbnail } from "./tdlib/download.js";
|
||||||
|
import { createOrFindPackageGroup, linkPackagesToGroup } from "./db/queries.js";
|
||||||
|
import { childLogger } from "./util/logger.js";
|
||||||
|
import { db } from "./db/client.js";
|
||||||
|
|
||||||
|
const log = childLogger("grouping");
|
||||||
|
|
||||||
|
export interface IndexedPackageRef {
|
||||||
|
packageId: string;
|
||||||
|
sourceMessageId: bigint;
|
||||||
|
mediaAlbumId?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* After a scan cycle's packages are individually indexed, detect album groups
|
||||||
|
* and create PackageGroup records linking the members.
|
||||||
|
*/
|
||||||
|
export async function processAlbumGroups(
|
||||||
|
client: Client,
|
||||||
|
sourceChannelId: string,
|
||||||
|
indexedPackages: IndexedPackageRef[],
|
||||||
|
photos: TelegramPhoto[]
|
||||||
|
): Promise<void> {
|
||||||
|
// Group indexed packages by mediaAlbumId
|
||||||
|
const albumMap = new Map<string, IndexedPackageRef[]>();
|
||||||
|
for (const pkg of indexedPackages) {
|
||||||
|
if (!pkg.mediaAlbumId || pkg.mediaAlbumId === "0") continue;
|
||||||
|
const group = albumMap.get(pkg.mediaAlbumId) ?? [];
|
||||||
|
group.push(pkg);
|
||||||
|
albumMap.set(pkg.mediaAlbumId, group);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (albumMap.size === 0) return;
|
||||||
|
|
||||||
|
log.info({ albumCount: albumMap.size }, "Detected album groups to process");
|
||||||
|
|
||||||
|
for (const [albumId, members] of albumMap) {
|
||||||
|
if (members.length < 2) continue;
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Find the first package's fileName for the group name fallback
|
||||||
|
const firstPkg = await db.package.findFirst({
|
||||||
|
where: { id: { in: members.map((m) => m.packageId) } },
|
||||||
|
orderBy: { sourceMessageId: "asc" },
|
||||||
|
select: { id: true, fileName: true },
|
||||||
|
});
|
||||||
|
|
||||||
|
// Try to find a caption from the album's photo message
|
||||||
|
const albumPhoto = photos.find((p) => p.mediaAlbumId === albumId);
|
||||||
|
const groupName = albumPhoto?.caption || firstPkg?.fileName || "Unnamed Group";
|
||||||
|
|
||||||
|
// Download preview from album photo if available
|
||||||
|
let previewData: Buffer | null = null;
|
||||||
|
if (albumPhoto) {
|
||||||
|
previewData = await downloadPhotoThumbnail(client, albumPhoto.fileId);
|
||||||
|
}
|
||||||
|
|
||||||
|
const groupId = await createOrFindPackageGroup({
|
||||||
|
mediaAlbumId: albumId,
|
||||||
|
sourceChannelId,
|
||||||
|
name: groupName,
|
||||||
|
previewData,
|
||||||
|
});
|
||||||
|
|
||||||
|
// Idempotent link — safe to re-run if some packages were indexed in prior scans
|
||||||
|
const packageIds = members.map((m) => m.packageId);
|
||||||
|
await linkPackagesToGroup(packageIds, groupId);
|
||||||
|
|
||||||
|
log.info(
|
||||||
|
{ albumId, groupId, groupName, memberCount: packageIds.length },
|
||||||
|
"Linked packages to album group"
|
||||||
|
);
|
||||||
|
} catch (err) {
|
||||||
|
log.warn({ albumId, err }, "Failed to create album group — packages still indexed individually");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -10,6 +10,7 @@ export interface TelegramPhoto {
|
|||||||
/** The smallest photo size available — used as thumbnail. */
|
/** The smallest photo size available — used as thumbnail. */
|
||||||
fileId: string;
|
fileId: string;
|
||||||
fileSize: number;
|
fileSize: number;
|
||||||
|
mediaAlbumId?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ArchiveRef {
|
export interface ArchiveRef {
|
||||||
|
|||||||
@@ -35,6 +35,7 @@ interface TdPhotoSize {
|
|||||||
interface TdMessage {
|
interface TdMessage {
|
||||||
id: number;
|
id: number;
|
||||||
date: number;
|
date: number;
|
||||||
|
media_album_id?: string;
|
||||||
content: {
|
content: {
|
||||||
_: string;
|
_: string;
|
||||||
document?: {
|
document?: {
|
||||||
@@ -211,6 +212,7 @@ export async function getChannelMessages(
|
|||||||
fileId: String(doc.document.id),
|
fileId: String(doc.document.id),
|
||||||
fileSize: BigInt(doc.document.size),
|
fileSize: BigInt(doc.document.size),
|
||||||
date: new Date(msg.date * 1000),
|
date: new Date(msg.date * 1000),
|
||||||
|
mediaAlbumId: msg.media_album_id && msg.media_album_id !== "0" ? msg.media_album_id : undefined,
|
||||||
});
|
});
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -227,6 +229,7 @@ export async function getChannelMessages(
|
|||||||
caption,
|
caption,
|
||||||
fileId: String(smallest.photo.id),
|
fileId: String(smallest.photo.id),
|
||||||
fileSize: smallest.photo.size || smallest.photo.expected_size,
|
fileSize: smallest.photo.size || smallest.photo.expected_size,
|
||||||
|
mediaAlbumId: msg.media_album_id && msg.media_album_id !== "0" ? msg.media_album_id : undefined,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -353,11 +356,14 @@ export async function downloadFile(
|
|||||||
return new Promise<void>((resolve, reject) => {
|
return new Promise<void>((resolve, reject) => {
|
||||||
let lastLoggedPercent = 0;
|
let lastLoggedPercent = 0;
|
||||||
let settled = false;
|
let settled = false;
|
||||||
|
let downloadStarted = false; // True once TDLib reports is_downloading_active
|
||||||
|
let lastProgressBytes = 0;
|
||||||
|
let lastProgressTime = Date.now();
|
||||||
|
|
||||||
// Timeout: 15 minutes per GB, minimum 10 minutes
|
// Timeout: 20 minutes per GB, minimum 15 minutes
|
||||||
const timeoutMs = Math.max(
|
const timeoutMs = Math.max(
|
||||||
10 * 60_000,
|
15 * 60_000,
|
||||||
(totalBytes / (1024 * 1024 * 1024)) * 15 * 60_000
|
(totalBytes / (1024 * 1024 * 1024)) * 20 * 60_000
|
||||||
);
|
);
|
||||||
const timer = setTimeout(() => {
|
const timer = setTimeout(() => {
|
||||||
if (!settled) {
|
if (!settled) {
|
||||||
@@ -371,6 +377,23 @@ export async function downloadFile(
|
|||||||
}
|
}
|
||||||
}, timeoutMs);
|
}, timeoutMs);
|
||||||
|
|
||||||
|
// Stall detection: no progress for 5 minutes after download started → reject
|
||||||
|
const STALL_TIMEOUT_MS = 5 * 60_000;
|
||||||
|
const stallChecker = setInterval(() => {
|
||||||
|
if (settled || !downloadStarted) return;
|
||||||
|
const stallMs = Date.now() - lastProgressTime;
|
||||||
|
if (stallMs >= STALL_TIMEOUT_MS) {
|
||||||
|
settled = true;
|
||||||
|
cleanup();
|
||||||
|
reject(
|
||||||
|
new Error(
|
||||||
|
`Download stalled for ${fileName} — no progress for ${Math.round(stallMs / 60_000)}min ` +
|
||||||
|
`(${lastProgressBytes}/${totalBytes} bytes)`
|
||||||
|
)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}, 30_000);
|
||||||
|
|
||||||
// Listen for file update events to track progress
|
// Listen for file update events to track progress
|
||||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
const handleUpdate = (update: any) => {
|
const handleUpdate = (update: any) => {
|
||||||
@@ -382,6 +405,17 @@ export async function downloadFile(
|
|||||||
const percent =
|
const percent =
|
||||||
totalBytes > 0 ? Math.round((downloaded / totalBytes) * 100) : 0;
|
totalBytes > 0 ? Math.round((downloaded / totalBytes) * 100) : 0;
|
||||||
|
|
||||||
|
// Track whether the download has actually started
|
||||||
|
if (file.local.is_downloading_active) {
|
||||||
|
downloadStarted = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reset stall timer when bytes advance
|
||||||
|
if (downloaded > lastProgressBytes) {
|
||||||
|
lastProgressBytes = downloaded;
|
||||||
|
lastProgressTime = Date.now();
|
||||||
|
}
|
||||||
|
|
||||||
// Log at every 10% increment
|
// Log at every 10% increment
|
||||||
if (percent >= lastLoggedPercent + 10) {
|
if (percent >= lastLoggedPercent + 10) {
|
||||||
lastLoggedPercent = percent - (percent % 10);
|
lastLoggedPercent = percent - (percent % 10);
|
||||||
@@ -412,8 +446,11 @@ export async function downloadFile(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Download stopped without completing (network error, cancelled, etc.)
|
// Download stopped without completing — only if it had actually started.
|
||||||
|
// TDLib may emit an initial updateFile with is_downloading_active=false
|
||||||
|
// before the download begins; ignoring that prevents false positives.
|
||||||
if (
|
if (
|
||||||
|
downloadStarted &&
|
||||||
!file.local.is_downloading_active &&
|
!file.local.is_downloading_active &&
|
||||||
!file.local.is_downloading_completed
|
!file.local.is_downloading_completed
|
||||||
) {
|
) {
|
||||||
@@ -432,6 +469,7 @@ export async function downloadFile(
|
|||||||
|
|
||||||
const cleanup = () => {
|
const cleanup = () => {
|
||||||
clearTimeout(timer);
|
clearTimeout(timer);
|
||||||
|
clearInterval(stallChecker);
|
||||||
client.off("update", handleUpdate);
|
client.off("update", handleUpdate);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@@ -201,6 +201,7 @@ export async function getTopicMessages(
|
|||||||
messages?: {
|
messages?: {
|
||||||
id: number;
|
id: number;
|
||||||
date: number;
|
date: number;
|
||||||
|
media_album_id?: string;
|
||||||
content: {
|
content: {
|
||||||
_: string;
|
_: string;
|
||||||
document?: {
|
document?: {
|
||||||
@@ -248,6 +249,7 @@ export async function getTopicMessages(
|
|||||||
fileId: String(doc.document.id),
|
fileId: String(doc.document.id),
|
||||||
fileSize: BigInt(doc.document.size),
|
fileSize: BigInt(doc.document.size),
|
||||||
date: new Date(msg.date * 1000),
|
date: new Date(msg.date * 1000),
|
||||||
|
mediaAlbumId: msg.media_album_id && msg.media_album_id !== "0" ? msg.media_album_id : undefined,
|
||||||
});
|
});
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -263,6 +265,7 @@ export async function getTopicMessages(
|
|||||||
caption,
|
caption,
|
||||||
fileId: String(smallest.photo.id),
|
fileId: String(smallest.photo.id),
|
||||||
fileSize: smallest.photo.size || smallest.photo.expected_size,
|
fileSize: smallest.photo.size || smallest.photo.expected_size,
|
||||||
|
mediaAlbumId: msg.media_album_id && msg.media_album_id !== "0" ? msg.media_album_id : undefined,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -93,11 +93,13 @@ async function sendAndWaitForUpload(
|
|||||||
let settled = false;
|
let settled = false;
|
||||||
let lastLoggedPercent = 0;
|
let lastLoggedPercent = 0;
|
||||||
let tempMsgId: number | null = null;
|
let tempMsgId: number | null = null;
|
||||||
|
let uploadStarted = false;
|
||||||
|
let lastProgressTime = Date.now();
|
||||||
|
|
||||||
// Timeout: 15 minutes per GB, minimum 10 minutes
|
// Timeout: 20 minutes per GB, minimum 15 minutes
|
||||||
const timeoutMs = Math.max(
|
const timeoutMs = Math.max(
|
||||||
10 * 60_000,
|
15 * 60_000,
|
||||||
(fileSizeMB / 1024) * 15 * 60_000
|
(fileSizeMB / 1024) * 20 * 60_000
|
||||||
);
|
);
|
||||||
|
|
||||||
const timer = setTimeout(() => {
|
const timer = setTimeout(() => {
|
||||||
@@ -112,12 +114,31 @@ async function sendAndWaitForUpload(
|
|||||||
}
|
}
|
||||||
}, timeoutMs);
|
}, timeoutMs);
|
||||||
|
|
||||||
|
// Stall detection: no progress for 5 minutes after upload started → reject
|
||||||
|
const STALL_TIMEOUT_MS = 5 * 60_000;
|
||||||
|
const stallChecker = setInterval(() => {
|
||||||
|
if (settled || !uploadStarted) return;
|
||||||
|
const stallMs = Date.now() - lastProgressTime;
|
||||||
|
if (stallMs >= STALL_TIMEOUT_MS) {
|
||||||
|
settled = true;
|
||||||
|
cleanup();
|
||||||
|
reject(
|
||||||
|
new Error(
|
||||||
|
`Upload stalled for ${fileName} — no progress for ${Math.round(stallMs / 60_000)}min`
|
||||||
|
)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}, 30_000);
|
||||||
|
|
||||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
const handleUpdate = (update: any) => {
|
const handleUpdate = (update: any) => {
|
||||||
// Track upload progress via updateFile events
|
// Track upload progress via updateFile events
|
||||||
if (update?._ === "updateFile") {
|
if (update?._ === "updateFile") {
|
||||||
const file = update.file;
|
const file = update.file;
|
||||||
if (file?.remote?.is_uploading_active && file.expected_size > 0) {
|
if (file?.remote?.is_uploading_active && file.expected_size > 0) {
|
||||||
|
uploadStarted = true;
|
||||||
|
lastProgressTime = Date.now();
|
||||||
|
|
||||||
const uploaded = file.remote.uploaded_size ?? 0;
|
const uploaded = file.remote.uploaded_size ?? 0;
|
||||||
const total = file.expected_size;
|
const total = file.expected_size;
|
||||||
const percent = Math.round((uploaded / total) * 100);
|
const percent = Math.round((uploaded / total) * 100);
|
||||||
@@ -165,6 +186,7 @@ async function sendAndWaitForUpload(
|
|||||||
|
|
||||||
const cleanup = () => {
|
const cleanup = () => {
|
||||||
clearTimeout(timer);
|
clearTimeout(timer);
|
||||||
|
clearInterval(stallChecker);
|
||||||
client.off("update", handleUpdate);
|
client.off("update", handleUpdate);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@@ -47,6 +47,7 @@ import { readRarContents } from "./archive/rar-reader.js";
|
|||||||
import { read7zContents } from "./archive/sevenz-reader.js";
|
import { read7zContents } from "./archive/sevenz-reader.js";
|
||||||
import { byteLevelSplit, concatenateFiles } from "./archive/split.js";
|
import { byteLevelSplit, concatenateFiles } from "./archive/split.js";
|
||||||
import { uploadToChannel } from "./upload/channel.js";
|
import { uploadToChannel } from "./upload/channel.js";
|
||||||
|
import { processAlbumGroups, type IndexedPackageRef } from "./grouping.js";
|
||||||
import type { TelegramAccount, TelegramChannel } from "@prisma/client";
|
import type { TelegramAccount, TelegramChannel } from "@prisma/client";
|
||||||
import type { Client } from "tdl";
|
import type { Client } from "tdl";
|
||||||
|
|
||||||
@@ -722,10 +723,11 @@ async function processArchiveSets(
|
|||||||
|
|
||||||
// Track the highest message ID that was successfully processed
|
// Track the highest message ID that was successfully processed
|
||||||
let maxProcessedId: bigint | null = null;
|
let maxProcessedId: bigint | null = null;
|
||||||
|
const indexedPackageRefs: IndexedPackageRef[] = [];
|
||||||
|
|
||||||
for (let setIdx = 0; setIdx < archiveSets.length; setIdx++) {
|
for (let setIdx = 0; setIdx < archiveSets.length; setIdx++) {
|
||||||
try {
|
try {
|
||||||
await processOneArchiveSet(
|
const packageId = await processOneArchiveSet(
|
||||||
ctx,
|
ctx,
|
||||||
archiveSets[setIdx],
|
archiveSets[setIdx],
|
||||||
setIdx,
|
setIdx,
|
||||||
@@ -734,6 +736,15 @@ async function processArchiveSets(
|
|||||||
ingestionRunId
|
ingestionRunId
|
||||||
);
|
);
|
||||||
|
|
||||||
|
if (packageId) {
|
||||||
|
const firstPart = archiveSets[setIdx].parts[0];
|
||||||
|
indexedPackageRefs.push({
|
||||||
|
packageId,
|
||||||
|
sourceMessageId: firstPart.id,
|
||||||
|
mediaAlbumId: firstPart.mediaAlbumId,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
// Set completed (ingested or confirmed duplicate) — advance watermark
|
// Set completed (ingested or confirmed duplicate) — advance watermark
|
||||||
const setMaxId = archiveSets[setIdx].parts.reduce(
|
const setMaxId = archiveSets[setIdx].parts.reduce(
|
||||||
(max, p) => (p.id > max ? p.id : max),
|
(max, p) => (p.id > max ? p.id : max),
|
||||||
@@ -771,6 +782,16 @@ async function processArchiveSets(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Post-processing: group packages by Telegram album ID
|
||||||
|
if (indexedPackageRefs.length > 0) {
|
||||||
|
await processAlbumGroups(
|
||||||
|
ctx.client,
|
||||||
|
channel.id,
|
||||||
|
indexedPackageRefs,
|
||||||
|
scanResult.photos
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
return maxProcessedId;
|
return maxProcessedId;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -784,7 +805,7 @@ async function processOneArchiveSet(
|
|||||||
totalSets: number,
|
totalSets: number,
|
||||||
previewMatches: Map<string, { id: bigint; fileId: string }>,
|
previewMatches: Map<string, { id: bigint; fileId: string }>,
|
||||||
ingestionRunId: string
|
ingestionRunId: string
|
||||||
): Promise<void> {
|
): Promise<string | null> {
|
||||||
const {
|
const {
|
||||||
client, runId, channelTitle, channel,
|
client, runId, channelTitle, channel,
|
||||||
destChannelTelegramId, destChannelId,
|
destChannelTelegramId, destChannelId,
|
||||||
@@ -814,7 +835,7 @@ async function processOneArchiveSet(
|
|||||||
totalFiles: totalSets,
|
totalFiles: totalSets,
|
||||||
zipsDuplicate: counters.zipsDuplicate,
|
zipsDuplicate: counters.zipsDuplicate,
|
||||||
});
|
});
|
||||||
return;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Size guard: skip archives that exceed WORKER_MAX_ZIP_SIZE_MB ──
|
// ── Size guard: skip archives that exceed WORKER_MAX_ZIP_SIZE_MB ──
|
||||||
@@ -848,7 +869,7 @@ async function processOneArchiveSet(
|
|||||||
partCount: archiveSet.parts.length,
|
partCount: archiveSet.parts.length,
|
||||||
accountId: ctx.accountId,
|
accountId: ctx.accountId,
|
||||||
});
|
});
|
||||||
return;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
const tempPaths: string[] = [];
|
const tempPaths: string[] = [];
|
||||||
@@ -954,7 +975,7 @@ async function processOneArchiveSet(
|
|||||||
totalFiles: totalSets,
|
totalFiles: totalSets,
|
||||||
zipsDuplicate: counters.zipsDuplicate,
|
zipsDuplicate: counters.zipsDuplicate,
|
||||||
});
|
});
|
||||||
return;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Reading metadata ──
|
// ── Reading metadata ──
|
||||||
@@ -999,7 +1020,7 @@ async function processOneArchiveSet(
|
|||||||
(sum, p) => sum + p.fileSize,
|
(sum, p) => sum + p.fileSize,
|
||||||
0n
|
0n
|
||||||
);
|
);
|
||||||
const MAX_UPLOAD_SIZE = 2n * 1024n * 1024n * 1024n;
|
const MAX_UPLOAD_SIZE = 1950n * 1024n * 1024n; // Match split.ts MAX_PART_SIZE
|
||||||
const hasOversizedPart = archiveSet.parts.some((p) => p.fileSize > MAX_UPLOAD_SIZE);
|
const hasOversizedPart = archiveSet.parts.some((p) => p.fileSize > MAX_UPLOAD_SIZE);
|
||||||
|
|
||||||
if (hasOversizedPart) {
|
if (hasOversizedPart) {
|
||||||
@@ -1127,7 +1148,7 @@ async function processOneArchiveSet(
|
|||||||
tags.push(channel.category);
|
tags.push(channel.category);
|
||||||
}
|
}
|
||||||
|
|
||||||
await createPackageWithFiles({
|
const pkg = await createPackageWithFiles({
|
||||||
contentHash,
|
contentHash,
|
||||||
fileName: archiveName,
|
fileName: archiveName,
|
||||||
fileSize: totalSize,
|
fileSize: totalSize,
|
||||||
@@ -1166,6 +1187,8 @@ async function processOneArchiveSet(
|
|||||||
{ fileName: archiveName, contentHash, fileCount: entries.length, creator },
|
{ fileName: archiveName, contentHash, fileCount: entries.length, creator },
|
||||||
"Archive ingested"
|
"Archive ingested"
|
||||||
);
|
);
|
||||||
|
|
||||||
|
return pkg.id;
|
||||||
} finally {
|
} finally {
|
||||||
// ALWAYS delete temp files and the set directory
|
// ALWAYS delete temp files and the set directory
|
||||||
await deleteFiles([...tempPaths, ...splitPaths]);
|
await deleteFiles([...tempPaths, ...splitPaths]);
|
||||||
|
|||||||
Reference in New Issue
Block a user