mirror of
https://github.com/xCyanGrizzly/DragonsStash.git
synced 2026-05-10 22:01:16 +00:00
Merge branch 'main' of https://github.com/xCyanGrizzly/DragonsStash
This commit is contained in:
@@ -0,0 +1,964 @@
|
||||
# Multi-Part Send Fix & Kickstarter Package Linking
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Fix multi-part package forwarding so all archive parts reach the user, and add UI to link STL packages to kickstarters with "send all" capability.
|
||||
|
||||
**Architecture:** Two independent subsystems. (A) Store all destination message IDs when the worker uploads multi-part archives, then have the bot forward every part. (B) Add a package-linker dialog in the kickstarter UI using the existing `linkPackages` action, plus a "send all" action that queues every linked package.
|
||||
|
||||
**Tech Stack:** Prisma (schema + migration), TypeScript worker/bot services, Next.js App Router (server actions + React client components), shadcn/ui, TanStack Table.
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
### Subsystem A — Multi-Part Send Fix
|
||||
|
||||
| Action | File | Responsibility |
|
||||
|--------|------|----------------|
|
||||
| Modify | `prisma/schema.prisma` | Add `destMessageIds BigInt[]` to Package |
|
||||
| Create | `prisma/migrations/<ts>_add_dest_message_ids/migration.sql` | Migration SQL |
|
||||
| Modify | `worker/src/upload/channel.ts` | Return all message IDs from `uploadToChannel` |
|
||||
| Modify | `worker/src/db/queries.ts` | Add `destMessageIds` to `CreatePackageInput` and `createPackageWithFiles` |
|
||||
| Modify | `worker/src/worker.ts` | Pass all message IDs when creating package |
|
||||
| Modify | `bot/src/db/queries.ts` | Include `destMessageIds` in `getPendingSendRequest` |
|
||||
| Modify | `bot/src/send-listener.ts` | Forward all parts, not just the first |
|
||||
|
||||
### Subsystem B — Kickstarter Package Linking UI
|
||||
|
||||
| Action | File | Responsibility |
|
||||
|--------|------|----------------|
|
||||
| Create | `src/app/(app)/kickstarters/_components/package-linker-dialog.tsx` | Dialog with package search + selection for linking |
|
||||
| Modify | `src/app/(app)/kickstarters/_components/kickstarter-columns.tsx` | Add "Link Packages" and "Send All" actions to row menu |
|
||||
| Modify | `src/app/(app)/kickstarters/_components/kickstarter-table.tsx` | Wire up new dialogs + state |
|
||||
| Modify | `src/app/(app)/kickstarters/actions.ts` | Add `sendAllKickstarterPackages` action |
|
||||
| Modify | `src/data/kickstarter.queries.ts` | Add query to search packages for linking |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Add `destMessageIds` to Prisma Schema + Migration
|
||||
|
||||
**Files:**
|
||||
- Modify: `prisma/schema.prisma:470-471`
|
||||
- Create: migration SQL
|
||||
|
||||
- [ ] **Step 1: Add field to schema**
|
||||
|
||||
In `prisma/schema.prisma`, add `destMessageIds` after `destMessageId`:
|
||||
|
||||
```prisma
|
||||
destMessageId BigInt?
|
||||
destMessageIds BigInt[] @default([])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Create migration SQL manually**
|
||||
|
||||
Create the migration directory and SQL file. The migration adds the column with a default and backfills existing rows by copying `destMessageId` into the array where it's non-null:
|
||||
|
||||
```sql
|
||||
-- AlterTable
|
||||
ALTER TABLE "packages" ADD COLUMN "destMessageIds" BIGINT[] DEFAULT ARRAY[]::BIGINT[];
|
||||
|
||||
-- Backfill: copy existing destMessageId into the array
|
||||
UPDATE "packages"
|
||||
SET "destMessageIds" = ARRAY["destMessageId"]
|
||||
WHERE "destMessageId" IS NOT NULL;
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Apply migration to database**
|
||||
|
||||
```bash
|
||||
docker exec dragonsstash-db psql -U dragons -d dragonsstash -f - < migration.sql
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Regenerate Prisma client**
|
||||
|
||||
Use the app container (which has node/prisma) to regenerate:
|
||||
|
||||
```bash
|
||||
docker exec dragonsstash npx prisma generate
|
||||
```
|
||||
|
||||
Or, if running locally with node: `npx prisma generate`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add prisma/schema.prisma prisma/migrations/
|
||||
git commit -m "feat: add destMessageIds field to Package for multi-part forwarding"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Worker — Return All Message IDs from Upload
|
||||
|
||||
**Files:**
|
||||
- Modify: `worker/src/upload/channel.ts:10-12,25-74`
|
||||
|
||||
- [ ] **Step 1: Update UploadResult interface**
|
||||
|
||||
In `worker/src/upload/channel.ts`, change the interface to include all IDs:
|
||||
|
||||
```typescript
|
||||
export interface UploadResult {
|
||||
messageId: bigint;
|
||||
messageIds: bigint[];
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Collect all message IDs in uploadToChannel**
|
||||
|
||||
Replace the upload loop to track all message IDs:
|
||||
|
||||
```typescript
|
||||
export async function uploadToChannel(
|
||||
client: Client,
|
||||
chatId: bigint,
|
||||
filePaths: string[],
|
||||
caption?: string
|
||||
): Promise<UploadResult> {
|
||||
const allMessageIds: bigint[] = [];
|
||||
|
||||
for (let i = 0; i < filePaths.length; i++) {
|
||||
const filePath = filePaths[i];
|
||||
const fileCaption = i === 0 && caption ? caption : undefined;
|
||||
|
||||
const fileName = path.basename(filePath);
|
||||
let fileSizeMB = 0;
|
||||
try {
|
||||
const s = await stat(filePath);
|
||||
fileSizeMB = Math.round(s.size / (1024 * 1024));
|
||||
} catch {
|
||||
// Non-critical
|
||||
}
|
||||
|
||||
log.info(
|
||||
{ chatId: Number(chatId), fileName, sizeMB: fileSizeMB, part: i + 1, total: filePaths.length },
|
||||
"Uploading file to channel"
|
||||
);
|
||||
|
||||
const serverMsgId = await sendWithRetry(client, chatId, filePath, fileCaption, fileName, fileSizeMB);
|
||||
allMessageIds.push(serverMsgId);
|
||||
|
||||
// Rate limit delay between uploads
|
||||
if (i < filePaths.length - 1) {
|
||||
await sleep(config.apiDelayMs);
|
||||
}
|
||||
}
|
||||
|
||||
if (allMessageIds.length === 0) {
|
||||
throw new Error("Upload failed: no messages sent");
|
||||
}
|
||||
|
||||
log.info(
|
||||
{ chatId: Number(chatId), messageId: Number(allMessageIds[0]), files: filePaths.length },
|
||||
"All uploads confirmed by Telegram"
|
||||
);
|
||||
|
||||
return { messageId: allMessageIds[0], messageIds: allMessageIds };
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add worker/src/upload/channel.ts
|
||||
git commit -m "feat: return all message IDs from uploadToChannel for multi-part"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Worker — Store All Message IDs in Database
|
||||
|
||||
**Files:**
|
||||
- Modify: `worker/src/db/queries.ts:104-155`
|
||||
- Modify: `worker/src/worker.ts:1056-1086`
|
||||
|
||||
- [ ] **Step 1: Add destMessageIds to CreatePackageInput**
|
||||
|
||||
In `worker/src/db/queries.ts`, add the field to the interface:
|
||||
|
||||
```typescript
|
||||
export interface CreatePackageInput {
|
||||
// ... existing fields ...
|
||||
destMessageId?: bigint;
|
||||
destMessageIds?: bigint[];
|
||||
// ... rest ...
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Store destMessageIds in createPackageWithFiles**
|
||||
|
||||
In the `db.package.create` call inside `createPackageWithFiles`, add:
|
||||
|
||||
```typescript
|
||||
destMessageIds: input.destMessageIds ?? (input.destMessageId ? [input.destMessageId] : []),
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Pass messageIds from worker pipeline**
|
||||
|
||||
In `worker/src/worker.ts`, the upload section (around line 1068-1085) currently does:
|
||||
|
||||
```typescript
|
||||
destResult = await uploadToChannel(client, destChannelTelegramId, uploadPaths);
|
||||
```
|
||||
|
||||
After this, when calling `createPackageWithFiles`, add `destMessageIds`:
|
||||
|
||||
```typescript
|
||||
const pkg = await createPackageWithFiles({
|
||||
// ... existing fields ...
|
||||
destMessageId: destResult.messageId,
|
||||
destMessageIds: destResult.messageIds,
|
||||
// ... rest ...
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add worker/src/db/queries.ts worker/src/worker.ts
|
||||
git commit -m "feat: store all multi-part message IDs in package record"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Bot — Forward All Parts
|
||||
|
||||
**Files:**
|
||||
- Modify: `bot/src/db/queries.ts:110-132`
|
||||
- Modify: `bot/src/send-listener.ts:105-169`
|
||||
- Modify: `bot/src/tdlib/client.ts:66-122`
|
||||
|
||||
- [ ] **Step 1: Include destMessageIds in bot query**
|
||||
|
||||
In `bot/src/db/queries.ts`, add `destMessageIds` to the `getPendingSendRequest` select:
|
||||
|
||||
```typescript
|
||||
package: {
|
||||
select: {
|
||||
id: true,
|
||||
fileName: true,
|
||||
fileSize: true,
|
||||
fileCount: true,
|
||||
creator: true,
|
||||
tags: true,
|
||||
archiveType: true,
|
||||
destChannelId: true,
|
||||
destMessageId: true,
|
||||
destMessageIds: true, // <-- ADD THIS
|
||||
isMultipart: true, // <-- ADD THIS (for logging)
|
||||
partCount: true, // <-- ADD THIS (for logging)
|
||||
previewData: true,
|
||||
sourceChannel: { select: { title: true, telegramId: true } },
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add copyMultipleMessagesToUser helper**
|
||||
|
||||
In `bot/src/tdlib/client.ts`, add a new export after `copyMessageToUser`:
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Send multiple document messages from a channel to a user's DM.
|
||||
* Used for multi-part archives where each part is a separate Telegram message.
|
||||
* Sends parts sequentially with a small delay to avoid rate limits.
|
||||
*/
|
||||
export async function copyMultipleMessagesToUser(
|
||||
fromChatId: bigint,
|
||||
messageIds: bigint[],
|
||||
toUserId: bigint
|
||||
): Promise<void> {
|
||||
for (let i = 0; i < messageIds.length; i++) {
|
||||
await copyMessageToUser(fromChatId, messageIds[i], toUserId);
|
||||
// Small delay between parts to avoid rate limits
|
||||
if (i < messageIds.length - 1) {
|
||||
await new Promise((resolve) => setTimeout(resolve, 1000));
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Update processSendRequest to forward all parts**
|
||||
|
||||
In `bot/src/send-listener.ts`, update the import to include the new function:
|
||||
|
||||
```typescript
|
||||
import { copyMessageToUser, copyMultipleMessagesToUser, sendTextMessage, sendPhotoMessage } from "./tdlib/client.js";
|
||||
```
|
||||
|
||||
Then replace the single `copyMessageToUser` call (around line 157) with logic that forwards all parts:
|
||||
|
||||
```typescript
|
||||
// Forward the actual archive file(s) from destination channel
|
||||
const messageIds = pkg.destMessageIds as bigint[] | undefined;
|
||||
if (messageIds && messageIds.length > 1) {
|
||||
log.info(
|
||||
{ requestId, parts: messageIds.length },
|
||||
"Sending multi-part archive"
|
||||
);
|
||||
await copyMultipleMessagesToUser(
|
||||
destChannel.telegramId,
|
||||
messageIds,
|
||||
targetUserId
|
||||
);
|
||||
} else {
|
||||
// Single part or legacy (no destMessageIds populated)
|
||||
await copyMessageToUser(
|
||||
destChannel.telegramId,
|
||||
pkg.destMessageId,
|
||||
targetUserId
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add bot/src/db/queries.ts bot/src/send-listener.ts bot/src/tdlib/client.ts
|
||||
git commit -m "feat: forward all parts of multi-part archives via bot"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Rebuild & Deploy Worker + Bot
|
||||
|
||||
- [ ] **Step 1: Rebuild worker image**
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.dev.yml build worker
|
||||
docker tag dragonsstash-worker:latest git.samagsteribbe.nl/admin/dragonsstash-worker:latest
|
||||
docker compose -p dragonsstash -f /opt/stacks/DragonsStash/docker-compose.yml up -d worker
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Rebuild bot image**
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.dev.yml build bot
|
||||
docker tag dragonsstash-bot:latest git.samagsteribbe.nl/admin/dragonsstash-bot:latest
|
||||
docker compose -p dragonsstash -f /opt/stacks/DragonsStash/docker-compose.yml up -d bot
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Verify bot startup**
|
||||
|
||||
```bash
|
||||
docker logs dragonsstash-bot --tail=20
|
||||
```
|
||||
|
||||
Expected: Bot starts cleanly, "Send listener started" message.
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Kickstarter — Package Search Query
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/data/kickstarter.queries.ts`
|
||||
|
||||
- [ ] **Step 1: Add searchPackagesForLinking query**
|
||||
|
||||
Append to `src/data/kickstarter.queries.ts`:
|
||||
|
||||
```typescript
|
||||
export async function searchPackagesForLinking(query: string, limit = 20) {
|
||||
if (!query || query.length < 2) return [];
|
||||
|
||||
return prisma.package.findMany({
|
||||
where: {
|
||||
OR: [
|
||||
{ fileName: { contains: query, mode: "insensitive" } },
|
||||
{ creator: { contains: query, mode: "insensitive" } },
|
||||
],
|
||||
},
|
||||
orderBy: { indexedAt: "desc" },
|
||||
take: limit,
|
||||
select: {
|
||||
id: true,
|
||||
fileName: true,
|
||||
fileSize: true,
|
||||
archiveType: true,
|
||||
creator: true,
|
||||
fileCount: true,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export async function getLinkedPackageIds(kickstarterId: string): Promise<string[]> {
|
||||
const links = await prisma.kickstarterPackage.findMany({
|
||||
where: { kickstarterId },
|
||||
select: { packageId: true },
|
||||
});
|
||||
return links.map((l) => l.packageId);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add src/data/kickstarter.queries.ts
|
||||
git commit -m "feat: add package search query for kickstarter linking"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Kickstarter — Package Linker Dialog Component
|
||||
|
||||
**Files:**
|
||||
- Create: `src/app/(app)/kickstarters/_components/package-linker-dialog.tsx`
|
||||
|
||||
- [ ] **Step 1: Create the package linker dialog**
|
||||
|
||||
This component provides a search input to find packages and checkboxes to select/deselect them. It calls the existing `linkPackages` action on save.
|
||||
|
||||
```tsx
|
||||
"use client";
|
||||
|
||||
import { useState, useTransition, useCallback, useEffect } from "react";
|
||||
import { Search, Package, X, Loader2 } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { linkPackages } from "../actions";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import { Checkbox } from "@/components/ui/checkbox";
|
||||
import {
|
||||
Dialog,
|
||||
DialogContent,
|
||||
DialogDescription,
|
||||
DialogFooter,
|
||||
DialogHeader,
|
||||
DialogTitle,
|
||||
} from "@/components/ui/dialog";
|
||||
import { ScrollArea } from "@/components/ui/scroll-area";
|
||||
|
||||
interface PackageResult {
|
||||
id: string;
|
||||
fileName: string;
|
||||
fileSize: bigint;
|
||||
archiveType: string;
|
||||
creator: string | null;
|
||||
fileCount: number;
|
||||
}
|
||||
|
||||
interface PackageLinkerDialogProps {
|
||||
open: boolean;
|
||||
onOpenChange: (open: boolean) => void;
|
||||
kickstarterId: string;
|
||||
kickstarterName: string;
|
||||
initialPackageIds: string[];
|
||||
}
|
||||
|
||||
function formatSize(bytes: bigint | number): string {
|
||||
const b = Number(bytes);
|
||||
if (b >= 1024 * 1024 * 1024) return `${(b / (1024 * 1024 * 1024)).toFixed(1)} GB`;
|
||||
if (b >= 1024 * 1024) return `${(b / (1024 * 1024)).toFixed(0)} MB`;
|
||||
return `${(b / 1024).toFixed(0)} KB`;
|
||||
}
|
||||
|
||||
export function PackageLinkerDialog({
|
||||
open,
|
||||
onOpenChange,
|
||||
kickstarterId,
|
||||
kickstarterName,
|
||||
initialPackageIds,
|
||||
}: PackageLinkerDialogProps) {
|
||||
const [isPending, startTransition] = useTransition();
|
||||
const [searchQuery, setSearchQuery] = useState("");
|
||||
const [searchResults, setSearchResults] = useState<PackageResult[]>([]);
|
||||
const [isSearching, setIsSearching] = useState(false);
|
||||
const [selectedIds, setSelectedIds] = useState<Set<string>>(new Set(initialPackageIds));
|
||||
|
||||
// Reset state when dialog opens
|
||||
useEffect(() => {
|
||||
if (open) {
|
||||
setSelectedIds(new Set(initialPackageIds));
|
||||
setSearchQuery("");
|
||||
setSearchResults([]);
|
||||
}
|
||||
}, [open, initialPackageIds]);
|
||||
|
||||
const doSearch = useCallback(async (query: string) => {
|
||||
if (query.length < 2) {
|
||||
setSearchResults([]);
|
||||
return;
|
||||
}
|
||||
setIsSearching(true);
|
||||
try {
|
||||
const res = await fetch(`/api/packages/search?q=${encodeURIComponent(query)}&limit=20`);
|
||||
if (res.ok) {
|
||||
const data = await res.json();
|
||||
setSearchResults(data.packages ?? []);
|
||||
}
|
||||
} catch {
|
||||
// Ignore search errors
|
||||
} finally {
|
||||
setIsSearching(false);
|
||||
}
|
||||
}, []);
|
||||
|
||||
// Debounced search
|
||||
useEffect(() => {
|
||||
const timer = setTimeout(() => doSearch(searchQuery), 300);
|
||||
return () => clearTimeout(timer);
|
||||
}, [searchQuery, doSearch]);
|
||||
|
||||
function togglePackage(id: string) {
|
||||
setSelectedIds((prev) => {
|
||||
const next = new Set(prev);
|
||||
if (next.has(id)) next.delete(id);
|
||||
else next.add(id);
|
||||
return next;
|
||||
});
|
||||
}
|
||||
|
||||
function handleSave() {
|
||||
startTransition(async () => {
|
||||
const result = await linkPackages(kickstarterId, Array.from(selectedIds));
|
||||
if (result.success) {
|
||||
toast.success(`Linked ${selectedIds.size} package(s) to "${kickstarterName}"`);
|
||||
onOpenChange(false);
|
||||
} else {
|
||||
toast.error(result.error);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
return (
|
||||
<Dialog open={open} onOpenChange={onOpenChange}>
|
||||
<DialogContent className="sm:max-w-lg">
|
||||
<DialogHeader>
|
||||
<DialogTitle>Link Packages</DialogTitle>
|
||||
<DialogDescription>
|
||||
Search and select STL packages to link to “{kickstarterName}”.
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
|
||||
<div className="space-y-3">
|
||||
{/* Selected count */}
|
||||
{selectedIds.size > 0 && (
|
||||
<div className="flex items-center gap-2 text-sm text-muted-foreground">
|
||||
<Package className="h-4 w-4" />
|
||||
{selectedIds.size} package(s) selected
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
className="h-6 px-2 text-xs"
|
||||
onClick={() => setSelectedIds(new Set())}
|
||||
>
|
||||
Clear all
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Search input */}
|
||||
<div className="relative">
|
||||
<Search className="absolute left-2.5 top-2.5 h-4 w-4 text-muted-foreground" />
|
||||
<Input
|
||||
placeholder="Search packages by name or creator..."
|
||||
value={searchQuery}
|
||||
onChange={(e) => setSearchQuery(e.target.value)}
|
||||
className="pl-9"
|
||||
autoFocus
|
||||
/>
|
||||
{isSearching && (
|
||||
<Loader2 className="absolute right-2.5 top-2.5 h-4 w-4 animate-spin text-muted-foreground" />
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Results */}
|
||||
<ScrollArea className="h-[300px] rounded-md border">
|
||||
<div className="p-2 space-y-1">
|
||||
{searchResults.length === 0 && searchQuery.length >= 2 && !isSearching && (
|
||||
<p className="text-sm text-muted-foreground text-center py-8">
|
||||
No packages found
|
||||
</p>
|
||||
)}
|
||||
{searchQuery.length < 2 && (
|
||||
<p className="text-sm text-muted-foreground text-center py-8">
|
||||
Type at least 2 characters to search
|
||||
</p>
|
||||
)}
|
||||
{searchResults.map((pkg) => (
|
||||
<label
|
||||
key={pkg.id}
|
||||
className="flex items-center gap-3 p-2 rounded-md hover:bg-muted/50 cursor-pointer"
|
||||
>
|
||||
<Checkbox
|
||||
checked={selectedIds.has(pkg.id)}
|
||||
onCheckedChange={() => togglePackage(pkg.id)}
|
||||
/>
|
||||
<div className="flex-1 min-w-0">
|
||||
<p className="text-sm font-medium truncate">{pkg.fileName}</p>
|
||||
<div className="flex items-center gap-2 text-xs text-muted-foreground">
|
||||
{pkg.creator && <span>{pkg.creator}</span>}
|
||||
<span>{formatSize(pkg.fileSize)}</span>
|
||||
<Badge variant="outline" className="text-[10px] h-4 px-1">
|
||||
{pkg.archiveType}
|
||||
</Badge>
|
||||
{pkg.fileCount > 0 && <span>{pkg.fileCount} files</span>}
|
||||
</div>
|
||||
</div>
|
||||
{selectedIds.has(pkg.id) && (
|
||||
<X className="h-3.5 w-3.5 text-muted-foreground shrink-0" />
|
||||
)}
|
||||
</label>
|
||||
))}
|
||||
</div>
|
||||
</ScrollArea>
|
||||
</div>
|
||||
|
||||
<DialogFooter>
|
||||
<Button variant="outline" onClick={() => onOpenChange(false)}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button onClick={handleSave} disabled={isPending}>
|
||||
{isPending ? <Loader2 className="h-4 w-4 animate-spin mr-1" /> : null}
|
||||
Save ({selectedIds.size})
|
||||
</Button>
|
||||
</DialogFooter>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add src/app/(app)/kickstarters/_components/package-linker-dialog.tsx
|
||||
git commit -m "feat: add package linker dialog for kickstarters"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Package Search API Route
|
||||
|
||||
**Files:**
|
||||
- Create: `src/app/api/packages/search/route.ts`
|
||||
|
||||
- [ ] **Step 1: Create the API route**
|
||||
|
||||
The package linker dialog needs a client-side fetch for debounced search. Create a lightweight API route:
|
||||
|
||||
```typescript
|
||||
import { NextResponse } from "next/server";
|
||||
import { auth } from "@/lib/auth";
|
||||
import { searchPackagesForLinking } from "@/data/kickstarter.queries";
|
||||
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
export async function GET(request: Request) {
|
||||
const session = await auth();
|
||||
if (!session?.user?.id) {
|
||||
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
|
||||
}
|
||||
|
||||
const { searchParams } = new URL(request.url);
|
||||
const query = searchParams.get("q") ?? "";
|
||||
const limit = Math.min(Number(searchParams.get("limit") ?? "20"), 50);
|
||||
|
||||
const packages = await searchPackagesForLinking(query, limit);
|
||||
|
||||
// Serialize BigInt for JSON
|
||||
const serialized = packages.map((p) => ({
|
||||
...p,
|
||||
fileSize: p.fileSize.toString(),
|
||||
}));
|
||||
|
||||
return NextResponse.json({ packages: serialized });
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add src/app/api/packages/search/route.ts
|
||||
git commit -m "feat: add package search API route for kickstarter linking"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 9: Kickstarter — Send All Packages Action
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/app/(app)/kickstarters/actions.ts`
|
||||
|
||||
- [ ] **Step 1: Add sendAllKickstarterPackages action**
|
||||
|
||||
Append to `src/app/(app)/kickstarters/actions.ts`:
|
||||
|
||||
```typescript
|
||||
export async function sendAllKickstarterPackages(
|
||||
kickstarterId: string
|
||||
): Promise<ActionResult<{ queued: number }>> {
|
||||
const session = await auth();
|
||||
if (!session?.user?.id) return { success: false, error: "Unauthorized" };
|
||||
|
||||
try {
|
||||
const telegramLink = await prisma.telegramLink.findUnique({
|
||||
where: { userId: session.user.id },
|
||||
});
|
||||
|
||||
if (!telegramLink) {
|
||||
return { success: false, error: "No linked Telegram account. Link one in Settings." };
|
||||
}
|
||||
|
||||
const kickstarter = await prisma.kickstarter.findFirst({
|
||||
where: { id: kickstarterId, userId: session.user.id },
|
||||
select: {
|
||||
packages: {
|
||||
select: {
|
||||
package: {
|
||||
select: { id: true, destChannelId: true, destMessageId: true, fileName: true },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
if (!kickstarter) {
|
||||
return { success: false, error: "Kickstarter not found" };
|
||||
}
|
||||
|
||||
const sendablePackages = kickstarter.packages
|
||||
.map((lnk) => lnk.package)
|
||||
.filter((p) => p.destChannelId && p.destMessageId);
|
||||
|
||||
if (sendablePackages.length === 0) {
|
||||
return { success: false, error: "No linked packages are available for sending" };
|
||||
}
|
||||
|
||||
let queued = 0;
|
||||
for (const pkg of sendablePackages) {
|
||||
const existing = await prisma.botSendRequest.findFirst({
|
||||
where: {
|
||||
packageId: pkg.id,
|
||||
telegramLinkId: telegramLink.id,
|
||||
status: { in: ["PENDING", "SENDING"] },
|
||||
},
|
||||
});
|
||||
|
||||
if (!existing) {
|
||||
const sendRequest = await prisma.botSendRequest.create({
|
||||
data: {
|
||||
packageId: pkg.id,
|
||||
telegramLinkId: telegramLink.id,
|
||||
requestedByUserId: session.user.id,
|
||||
status: "PENDING",
|
||||
},
|
||||
});
|
||||
|
||||
try {
|
||||
await prisma.$queryRawUnsafe(
|
||||
`SELECT pg_notify('bot_send', $1)`,
|
||||
sendRequest.id
|
||||
);
|
||||
} catch {
|
||||
// Best-effort
|
||||
}
|
||||
|
||||
queued++;
|
||||
}
|
||||
}
|
||||
|
||||
revalidatePath(REVALIDATE_PATH);
|
||||
return { success: true, data: { queued } };
|
||||
} catch {
|
||||
return { success: false, error: "Failed to send packages" };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add src/app/(app)/kickstarters/actions.ts
|
||||
git commit -m "feat: add sendAllKickstarterPackages action"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 10: Kickstarter Table — Wire Up Link & Send Actions
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/app/(app)/kickstarters/_components/kickstarter-columns.tsx`
|
||||
- Modify: `src/app/(app)/kickstarters/_components/kickstarter-table.tsx`
|
||||
|
||||
- [ ] **Step 1: Add actions to column menu**
|
||||
|
||||
In `kickstarter-columns.tsx`, add `Link2` and `Send` imports from lucide-react, add `onLinkPackages` and `onSendAll` to props, and add menu items:
|
||||
|
||||
```typescript
|
||||
import { MoreHorizontal, Pencil, Trash2, ExternalLink, Link2, Send } from "lucide-react";
|
||||
|
||||
// Update interface:
|
||||
interface KickstarterColumnsProps {
|
||||
onEdit: (kickstarter: KickstarterRow) => void;
|
||||
onDelete: (id: string) => void;
|
||||
onLinkPackages: (kickstarter: KickstarterRow) => void;
|
||||
onSendAll: (kickstarter: KickstarterRow) => void;
|
||||
}
|
||||
```
|
||||
|
||||
In the actions column dropdown, add between Edit and the separator:
|
||||
|
||||
```tsx
|
||||
<DropdownMenuItem onClick={() => onLinkPackages(row.original)}>
|
||||
<Link2 className="mr-2 h-3.5 w-3.5" />
|
||||
Link Packages
|
||||
</DropdownMenuItem>
|
||||
{row.original._count.packages > 0 && (
|
||||
<DropdownMenuItem onClick={() => onSendAll(row.original)}>
|
||||
<Send className="mr-2 h-3.5 w-3.5" />
|
||||
Send All ({row.original._count.packages})
|
||||
</DropdownMenuItem>
|
||||
)}
|
||||
```
|
||||
|
||||
Update the function signature to destructure the new props:
|
||||
|
||||
```typescript
|
||||
export function getKickstarterColumns({
|
||||
onEdit,
|
||||
onDelete,
|
||||
onLinkPackages,
|
||||
onSendAll,
|
||||
}: KickstarterColumnsProps): ColumnDef<KickstarterRow, unknown>[] {
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Wire up state in kickstarter-table.tsx**
|
||||
|
||||
Add imports and state for the new dialogs:
|
||||
|
||||
```typescript
|
||||
import { PackageLinkerDialog } from "./package-linker-dialog";
|
||||
import { sendAllKickstarterPackages } from "../actions";
|
||||
|
||||
// Inside KickstarterTable:
|
||||
const [linkTarget, setLinkTarget] = useState<KickstarterRow | null>(null);
|
||||
const [sendAllTarget, setSendAllTarget] = useState<KickstarterRow | null>(null);
|
||||
```
|
||||
|
||||
Update the columns call:
|
||||
|
||||
```typescript
|
||||
const columns = getKickstarterColumns({
|
||||
onEdit: (kickstarter) => {
|
||||
setEditKickstarter(kickstarter);
|
||||
setModalOpen(true);
|
||||
},
|
||||
onDelete: (id) => setDeleteId(id),
|
||||
onLinkPackages: (kickstarter) => setLinkTarget(kickstarter),
|
||||
onSendAll: (kickstarter) => {
|
||||
startTransition(async () => {
|
||||
const result = await sendAllKickstarterPackages(kickstarter.id);
|
||||
if (result.success) {
|
||||
toast.success(`Queued ${result.data!.queued} package(s) for delivery`);
|
||||
} else {
|
||||
toast.error(result.error);
|
||||
}
|
||||
});
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
Add the `PackageLinkerDialog` before the closing `</div>` of the component's return:
|
||||
|
||||
```tsx
|
||||
{linkTarget && (
|
||||
<PackageLinkerDialog
|
||||
open={!!linkTarget}
|
||||
onOpenChange={(open) => !open && setLinkTarget(null)}
|
||||
kickstarterId={linkTarget.id}
|
||||
kickstarterName={linkTarget.name}
|
||||
initialPackageIds={[]}
|
||||
/>
|
||||
)}
|
||||
```
|
||||
|
||||
Note: `initialPackageIds` is `[]` because the table doesn't fetch linked packages. The dialog will start empty but preserve selections during the session. For a better UX, we fetch the linked IDs when the dialog opens — see step 3.
|
||||
|
||||
- [ ] **Step 3: Fetch initial linked packages when dialog opens**
|
||||
|
||||
To populate the dialog with already-linked packages, add an API route or use a server action. The simplest approach: modify the `PackageLinkerDialog` to fetch linked IDs on mount.
|
||||
|
||||
In `package-linker-dialog.tsx`, add to the `useEffect` that runs when `open` changes:
|
||||
|
||||
```typescript
|
||||
useEffect(() => {
|
||||
if (open) {
|
||||
setSearchQuery("");
|
||||
setSearchResults([]);
|
||||
// Fetch currently linked packages
|
||||
fetch(`/api/packages/linked?kickstarterId=${kickstarterId}`)
|
||||
.then((res) => res.json())
|
||||
.then((data) => {
|
||||
if (data.packageIds) {
|
||||
setSelectedIds(new Set(data.packageIds));
|
||||
}
|
||||
})
|
||||
.catch(() => {});
|
||||
}
|
||||
}, [open, kickstarterId]);
|
||||
```
|
||||
|
||||
Create the API route at `src/app/api/packages/linked/route.ts`:
|
||||
|
||||
```typescript
|
||||
import { NextResponse } from "next/server";
|
||||
import { auth } from "@/lib/auth";
|
||||
import { getLinkedPackageIds } from "@/data/kickstarter.queries";
|
||||
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
export async function GET(request: Request) {
|
||||
const session = await auth();
|
||||
if (!session?.user?.id) {
|
||||
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
|
||||
}
|
||||
|
||||
const { searchParams } = new URL(request.url);
|
||||
const kickstarterId = searchParams.get("kickstarterId");
|
||||
if (!kickstarterId) {
|
||||
return NextResponse.json({ error: "kickstarterId required" }, { status: 400 });
|
||||
}
|
||||
|
||||
const packageIds = await getLinkedPackageIds(kickstarterId);
|
||||
return NextResponse.json({ packageIds });
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add src/app/(app)/kickstarters/_components/ src/app/api/packages/
|
||||
git commit -m "feat: wire up package linking and send-all in kickstarter table"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 11: Rebuild & Deploy App
|
||||
|
||||
- [ ] **Step 1: Rebuild app image**
|
||||
|
||||
```bash
|
||||
docker compose build app # or equivalent for the production compose
|
||||
docker tag dragonsstash:latest git.samagsteribbe.nl/admin/dragonsstash:latest
|
||||
docker compose -p dragonsstash -f /opt/stacks/DragonsStash/docker-compose.yml up -d app
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify app startup**
|
||||
|
||||
```bash
|
||||
docker logs dragonsstash --tail=20
|
||||
```
|
||||
|
||||
Expected: App starts cleanly, health check passes.
|
||||
|
||||
- [ ] **Step 3: Manual test**
|
||||
|
||||
1. Go to Kickstarters tab
|
||||
2. Open a kickstarter's row menu → "Link Packages"
|
||||
3. Search for a package, select it, save
|
||||
4. Verify the package count column updates
|
||||
5. Use "Send All" to queue all linked packages for Telegram delivery
|
||||
472
docs/superpowers/plans/2026-03-30-grouping-audit-report.md
Normal file
472
docs/superpowers/plans/2026-03-30-grouping-audit-report.md
Normal file
@@ -0,0 +1,472 @@
|
||||
# Dragonstash Grouping System Audit & Enhancement Report
|
||||
|
||||
## Appendix: Real-World Failure Cases (2026-03-29/30)
|
||||
|
||||
These skipped packages reveal two concrete issues:
|
||||
|
||||
### Issue A: `WORKER_MAX_ZIP_SIZE_MB` was 4 GB — blocking all large multipart archives
|
||||
|
||||
| File | Parts | Total Size | Status |
|
||||
|------|-------|-----------|--------|
|
||||
| DM-Stash - Guide to Tharador - Complete STL | 19 | 70.5 GB | SIZE_LIMIT |
|
||||
| DM-Stash - 2023-05 - Greywinds All-in | 16 | 58.9 GB | SIZE_LIMIT |
|
||||
| Axolote Gaming - Castle of the Vampire Lord | 10 | 18 GB | SIZE_LIMIT |
|
||||
| Dungeon Blocks - THE ULTIMATE DUNGEON | 5 | 7.6 GB | SIZE_LIMIT |
|
||||
| Dungeon Blocks - The Toxic sewer | 4 | 6.2 GB | SIZE_LIMIT |
|
||||
| Soulmist | 4 | 6.3 GB | SIZE_LIMIT |
|
||||
| Medieval Town PT1 | 3 | 5.7 GB | SIZE_LIMIT |
|
||||
| Knight Models - Game Of Thrones | 3 | 5.5 GB | SIZE_LIMIT |
|
||||
| Dungeon Blocks - The Lost Cave | 3 | 4.9 GB | SIZE_LIMIT |
|
||||
| El Miniaturista 2025-05 Fulgrim Part II and III | 5 | 4.7 GB | SIZE_LIMIT |
|
||||
|
||||
**Root cause:** Production env had `WORKER_MAX_ZIP_SIZE_MB=4096`. The default in code is 204800 (200 GB), but docker-compose.yml defaulted to 4096.
|
||||
|
||||
**Fix applied:** Raised to 204800 in `/opt/stacks/DragonsStash/.env`. Worker restarted. These archives will be retried on the next ingestion cycle. The worker downloads parts individually (each under 2-4 GB), concatenates, re-splits at 1950 MiB for upload. Peak temp disk usage for the 70.5 GB archive: ~211 GB (353 GB available).
|
||||
|
||||
**Code fix:** `MAX_PART_SIZE` is now configurable via `MAX_PART_SIZE_MB` env var (was hardcoded at 1950). Set to 3900 for Telegram Premium accounts to avoid unnecessary splitting.
|
||||
|
||||
### Issue B: Download failure at 98% (DE1-Supported.7z)
|
||||
|
||||
| File | Size | Error |
|
||||
|------|------|-------|
|
||||
| DE1-Supported.7z | 1.9 GB | Download stopped unexpectedly at 2043674624/2078338541 bytes (98%) |
|
||||
|
||||
**Root cause:** Download stalled near completion with no retry mechanism.
|
||||
|
||||
**Fix applied:** Earlier in this session, download retry logic was added (max 3 retries with `cancelDownloadFile` before each retry). This file will be retried automatically on next ingestion cycle.
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 1: Audit Report — Current State
|
||||
|
||||
### 1.1 Grouping Signal Stack (Current)
|
||||
|
||||
The system currently uses exactly **one automatic grouping signal**:
|
||||
|
||||
| Priority | Signal | Status | Location |
|
||||
|----------|--------|--------|----------|
|
||||
| 1 | `mediaAlbumId` | Implemented | `worker/src/grouping.ts:26-33` |
|
||||
| 2 | Manual override | Implemented | `src/lib/telegram/queries.ts:606-639` |
|
||||
|
||||
**How it works:**
|
||||
- `processAlbumGroups()` in `worker/src/grouping.ts` groups indexed packages by `mediaAlbumId` (filtering out "0" and null)
|
||||
- For albums with 2+ members: creates `PackageGroup`, links packages, assigns name from album photo caption or first filename
|
||||
- Manual grouping via UI: select 2+ packages, enter name, creates group in `createManualGroup()`
|
||||
|
||||
**What does NOT exist:**
|
||||
- No `message_thread_id` (forum topic) scoping
|
||||
- No project/month pattern extraction from filenames
|
||||
- No creator/sender grouping
|
||||
- No time-window + sender clustering
|
||||
- No reply chain analysis
|
||||
- No ZIP internal path prefix matching
|
||||
- No caption fuzzy matching
|
||||
- No staging queue for ungrouped files
|
||||
|
||||
### 1.2 Multipart Archive Detection (`worker/src/archive/multipart.ts`)
|
||||
|
||||
This is a **separate system** from display grouping. `groupArchiveSets()` groups Telegram messages into `ArchiveSet[]` based on filename patterns:
|
||||
|
||||
- `.zip.001`, `.zip.002` → ZIP_NUMBERED
|
||||
- `.z01`, `.z02`, `.zip` → ZIP_LEGACY
|
||||
- `.part1.rar`, `.part2.rar` → RAR_PART
|
||||
- `.r00`, `.r01`, `.rar` → RAR_LEGACY
|
||||
|
||||
These are grouped by `format:baseName.toLowerCase()` key. This is about **reassembling split archives**, not UI grouping. An `ArchiveSet` becomes a single `Package` in the database.
|
||||
|
||||
### 1.3 TDLib Ingestion Handler
|
||||
|
||||
**Pipeline in `worker/src/worker.ts:801-1197`:**
|
||||
```
|
||||
processOneArchiveSet():
|
||||
1. Early skip check (source message ID)
|
||||
2. Size guard (maxZipSizeMB)
|
||||
3. Download all parts
|
||||
4. Compute SHA-256 hash
|
||||
5. Check hash dedup
|
||||
6. Read archive metadata
|
||||
7. Split/repack if needed
|
||||
8. Upload to destination
|
||||
9. Download preview
|
||||
10. Extract fallback preview
|
||||
11. Resolve creator
|
||||
12. Index in database
|
||||
13. Cleanup temp files
|
||||
```
|
||||
|
||||
**Post-indexing:** `processAlbumGroups()` is called once per channel/topic scan to create album-based groups.
|
||||
|
||||
**Gaps:**
|
||||
- Messages are never "dropped" silently — failures go to `SkippedPackage` table with reason
|
||||
- Watermark only advances past successfully processed sets (failed sets block advancement)
|
||||
- No messages are missed within a channel, but there's no audit to verify completeness after the fact
|
||||
|
||||
### 1.4 Hash Verification
|
||||
|
||||
**What IS verified:**
|
||||
| Check | Where | When |
|
||||
|-------|-------|------|
|
||||
| Download file size | `download.ts:verifyAndMove()` | After each file download |
|
||||
| SHA-256 content hash | `worker.ts:952` | After download, used for dedup |
|
||||
| Telegram upload confirmation | `channel.ts:updateMessageSendSucceeded` | Waits for server ACK |
|
||||
|
||||
**What is NOT verified:**
|
||||
| Gap | Impact |
|
||||
|-----|--------|
|
||||
| No hash after upload | Can't detect Telegram-side corruption |
|
||||
| No hash after split | Split files could be silently corrupted |
|
||||
| CRC-32 extracted but never checked | ZIP/RAR per-file integrity not validated |
|
||||
| No end-to-end hash | Split files have different hash than original |
|
||||
| No periodic audit job | Stale/missing data never detected |
|
||||
|
||||
### 1.5 File Size Limit
|
||||
|
||||
| Setting | Value | Configurable? | Location |
|
||||
|---------|-------|---------------|----------|
|
||||
| `MAX_PART_SIZE` | 1950 MiB | **Hardcoded** | `worker/src/archive/split.ts:14` |
|
||||
| `MAX_UPLOAD_SIZE` | 1950 MiB | **Hardcoded** | `worker/src/worker.ts:1023` |
|
||||
| `maxZipSizeMB` | 200 GB | `WORKER_MAX_ZIP_SIZE_MB` env var | `worker/src/util/config.ts:6` |
|
||||
|
||||
The 1950 MiB limit is deliberately below 2 GiB to avoid TDLib's `FILE_PARTS_INVALID` error. There is **no Premium awareness** — all accounts are treated as non-Premium.
|
||||
|
||||
### 1.6 Search Implementation
|
||||
|
||||
- **No fuzzy search** — uses Prisma's `contains` with `mode: "insensitive"` (translates to PostgreSQL `ILIKE`)
|
||||
- **No full-text search infrastructure** — no `tsvector`, no GiST/GIN indexes
|
||||
- **Indexes:** B-tree on `fileName`, `creator`, `archiveType`, `indexedAt`, plus `PackageFile.fileName` and `extension`
|
||||
- Search works for substring matching but won't match typos or similar names
|
||||
|
||||
### 1.7 Notification Infrastructure
|
||||
|
||||
- **pg_notify channels:** `bot_send`, `new_package` (bot), plus 7 worker channels
|
||||
- **Bot subscriptions:** pattern-match (case-insensitive substring) on `fileName` and `creator`
|
||||
- **UI notifications:** Sonner toast (ephemeral only)
|
||||
- **No persistent notification store** — no database model for notifications
|
||||
- **No notification UI panel** in the web app
|
||||
- **No alerts for:** grouping conflicts, hash mismatches, missing parts, upload failures (beyond SkippedPackage table)
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 2: Revised Grouping Signal Stack
|
||||
|
||||
### Recommended Implementation Plan
|
||||
|
||||
I recommend an **incremental approach** — implement signals in phases, starting with highest-value/lowest-risk.
|
||||
|
||||
### Phase 1: Foundation (Required Before Other Signals)
|
||||
|
||||
#### Signal 9: Manual Override Persistence
|
||||
**Status:** Partially implemented. Manual groups exist but don't influence future auto-grouping.
|
||||
|
||||
**Implementation:**
|
||||
- Add `groupingSource` field to `PackageGroup`: `"ALBUM" | "MANUAL" | "AUTO_PATTERN" | "AUTO_TIME" | "AUTO_REPLY" | "AUTO_ZIP" | "AUTO_CAPTION"`
|
||||
- Manual groups already persist. What's missing is the **training feedback** where a manual grouping teaches the system to auto-group similar future files.
|
||||
- This requires a `GroupingRule` model (see schema diff below) that stores learned patterns from manual overrides.
|
||||
|
||||
#### Ungrouped Staging Queue
|
||||
**Implementation:**
|
||||
- After ingestion, packages without a `packageGroupId` are naturally "ungrouped"
|
||||
- Add a filter/tab to the STL page: "Ungrouped" showing packages where `packageGroupId IS NULL`
|
||||
- No schema change needed — just a query filter
|
||||
|
||||
### Phase 2: High-Value Automatic Signals
|
||||
|
||||
#### Signal 1: `mediaAlbumId` (Already Implemented)
|
||||
No changes needed. This is working correctly.
|
||||
|
||||
#### Signal 2: `message_thread_id` Forum Topic Scoping
|
||||
**Status:** Already used for scan scoping (worker scans by topic), but not used as a grouping signal.
|
||||
|
||||
**Implementation:**
|
||||
- `sourceTopicId` is already stored on `Package` (schema line 469)
|
||||
- Use it as a **scoping constraint** for all other signals: time-window, caption matching, etc. only apply within the same topic
|
||||
- No additional schema changes needed
|
||||
|
||||
#### Signal 5: Time Window + Sender Grouping
|
||||
**Implementation:**
|
||||
- After album grouping, find ungrouped packages from the same source channel + topic
|
||||
- Within a configurable window (default 5 min), cluster by proximity
|
||||
- Since we don't have `sender_id` from the source channel (TDLib `searchChatMessages` doesn't return it for channels), this becomes **time-window within topic/channel**
|
||||
- New config: `AUTO_GROUP_TIME_WINDOW_MINUTES` (default: 5)
|
||||
|
||||
#### Signal 3: Project/Month Pattern Extraction
|
||||
**Implementation:**
|
||||
- Extract date patterns from filenames/captions: `YYYY-MM`, `YYYY_MM`, `MonthName Year`
|
||||
- Extract project slugs: common prefix before separator (e.g., "ProjectName - File1.zip" and "ProjectName - File2.zip")
|
||||
- Group packages with matching patterns from the same channel
|
||||
- This should run as a **post-processing pass** after time-window grouping, merging small time-window groups that share a pattern
|
||||
|
||||
#### Signal 4: Creator Grouping
|
||||
**Implementation:**
|
||||
- The `creator` field is already extracted from filenames and stored per-package
|
||||
- Within a channel, if multiple ungrouped packages have the same `creator` and were indexed within the same ingestion run, auto-group them
|
||||
- Lower priority than time-window (might create overly broad groups)
|
||||
|
||||
### Phase 3: Advanced Signals
|
||||
|
||||
#### Signal 6: Reply Chain
|
||||
**Implementation:**
|
||||
- TDLib messages have `reply_to_message_id` but this isn't currently captured during scanning
|
||||
- Would need to modify `getChannelMessages()` in `download.ts` to extract `reply_to_message_id`
|
||||
- Then: if message B replies to message A, and both are archives, group them
|
||||
- **Moderate complexity**, deferred to Phase 3
|
||||
|
||||
#### Signal 7: ZIP Internal Path Prefix
|
||||
**Implementation:**
|
||||
- Already have `PackageFile.path` stored for each file inside an archive
|
||||
- After indexing, find the common root folder across all files
|
||||
- If two packages share the same root prefix and same channel, suggest grouping
|
||||
- This is a **post-hoc analysis** that could run as a background job
|
||||
|
||||
#### Signal 8: Caption Fuzzy Match
|
||||
**Implementation:**
|
||||
- Currently captions from source messages are NOT stored (only photo captions for preview matching)
|
||||
- Would need to capture `msg.content?.caption?.text` during scanning and store on Package
|
||||
- Then: fuzzy-match captions from nearby messages in same channel
|
||||
- **Requires schema change + scan modification**, deferred to Phase 3
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 3: Schema Diff
|
||||
|
||||
All changes are **additive** — no columns dropped, no types changed.
|
||||
|
||||
```prisma
|
||||
// ── PackageGroup additions ──
|
||||
model PackageGroup {
|
||||
// ... existing fields ...
|
||||
groupingSource GroupingSource @default(MANUAL) // NEW: how this group was created
|
||||
}
|
||||
|
||||
// NEW enum
|
||||
enum GroupingSource {
|
||||
ALBUM // From Telegram mediaAlbumId
|
||||
MANUAL // User-created via UI
|
||||
AUTO_PATTERN // Filename/date pattern matching
|
||||
AUTO_TIME // Time-window clustering
|
||||
AUTO_REPLY // Reply chain
|
||||
AUTO_ZIP // ZIP path prefix
|
||||
AUTO_CAPTION // Caption fuzzy match
|
||||
}
|
||||
|
||||
// ── Package additions ──
|
||||
model Package {
|
||||
// ... existing fields ...
|
||||
sourceCaption String? // NEW: caption text from source Telegram message
|
||||
}
|
||||
|
||||
// ── New model: GroupingRule (training from manual overrides) ──
|
||||
model GroupingRule {
|
||||
id String @id @default(cuid())
|
||||
sourceChannelId String
|
||||
pattern String // Regex or glob pattern learned from manual grouping
|
||||
signalType GroupingSource // Which signal this rule applies to
|
||||
confidence Float @default(1.0)
|
||||
createdAt DateTime @default(now())
|
||||
createdByGroupId String? // The manual group that spawned this rule
|
||||
|
||||
sourceChannel TelegramChannel @relation(fields: [sourceChannelId], references: [id], onDelete: Cascade)
|
||||
|
||||
@@index([sourceChannelId])
|
||||
@@map("grouping_rules")
|
||||
}
|
||||
|
||||
// ── New model: SystemNotification ──
|
||||
model SystemNotification {
|
||||
id String @id @default(cuid())
|
||||
type NotificationType
|
||||
severity NotificationSeverity @default(INFO)
|
||||
title String
|
||||
message String
|
||||
context Json? // Structured data: packageId, groupId, sourceMessageId, etc.
|
||||
isRead Boolean @default(false)
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
@@index([isRead, createdAt])
|
||||
@@index([type])
|
||||
@@map("system_notifications")
|
||||
}
|
||||
|
||||
enum NotificationType {
|
||||
HASH_MISMATCH
|
||||
MISSING_PART
|
||||
UPLOAD_FAILED
|
||||
DOWNLOAD_FAILED
|
||||
GROUPING_CONFLICT
|
||||
INTEGRITY_AUDIT
|
||||
}
|
||||
|
||||
enum NotificationSeverity {
|
||||
INFO
|
||||
WARNING
|
||||
ERROR
|
||||
}
|
||||
|
||||
// ── Config additions (worker/src/util/config.ts) ──
|
||||
// maxPartSizeMB: parseInt(process.env.MAX_PART_SIZE_MB ?? "1950", 10)
|
||||
// autoGroupTimeWindowMinutes: parseInt(process.env.AUTO_GROUP_TIME_WINDOW_MINUTES ?? "5", 10)
|
||||
// telegramPremium: process.env.TELEGRAM_PREMIUM === "true"
|
||||
```
|
||||
|
||||
**Migration notes:**
|
||||
- All new fields are optional/have defaults — zero-risk to existing data
|
||||
- `GroupingSource` enum added with `@default(MANUAL)` — existing groups unaffected
|
||||
- `GroupingRule` and `SystemNotification` are new tables — no impact on existing
|
||||
- Backfill: set `groupingSource = ALBUM` for groups where `mediaAlbumId IS NOT NULL`
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 4: Notification Contract
|
||||
|
||||
### Event Shape
|
||||
|
||||
```typescript
|
||||
interface SystemNotificationEvent {
|
||||
type: NotificationType;
|
||||
severity: "INFO" | "WARNING" | "ERROR";
|
||||
title: string;
|
||||
message: string;
|
||||
context: {
|
||||
packageId?: string;
|
||||
groupId?: string;
|
||||
sourceChannelId?: string;
|
||||
sourceMessageId?: bigint;
|
||||
fileName?: string;
|
||||
partNumber?: number;
|
||||
totalParts?: number;
|
||||
expectedHash?: string;
|
||||
actualHash?: string;
|
||||
reason?: string;
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### Where Notifications Fire
|
||||
|
||||
| Event | Where | Trigger |
|
||||
|-------|-------|---------|
|
||||
| `HASH_MISMATCH` | `worker/src/worker.ts` after split | SHA-256 of concatenated split parts != original hash |
|
||||
| `MISSING_PART` | Periodic audit job (new) | Group has `partCount > 1` but fewer than `partCount` dest messages exist |
|
||||
| `UPLOAD_FAILED` | `worker/src/worker.ts` catch block | Upload fails after all retries exhausted |
|
||||
| `DOWNLOAD_FAILED` | `worker/src/worker.ts` catch block | Download fails after all retries |
|
||||
| `GROUPING_CONFLICT` | Auto-grouping pass (new) | Two signals suggest different groups for the same package |
|
||||
| `INTEGRITY_AUDIT` | Periodic job (new) | Scheduled check finds inconsistencies |
|
||||
|
||||
### Delivery
|
||||
|
||||
1. **Database:** Always persisted to `SystemNotification` table
|
||||
2. **pg_notify:** `SELECT pg_notify('system_notification', jsonPayload)` for real-time
|
||||
3. **Web UI:** Notification bell/panel that polls or listens for new notifications
|
||||
4. **Telegram (optional):** Forward critical notifications to admin via bot
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 5: Feature Flag Plan
|
||||
|
||||
### Runtime Configuration (Environment Variables)
|
||||
|
||||
| Flag | Type | Default | Purpose |
|
||||
|------|------|---------|---------|
|
||||
| `TELEGRAM_PREMIUM` | boolean | `false` | Enable 4GB upload limit |
|
||||
| `MAX_PART_SIZE_MB` | number | `1950` | Split threshold in MiB (overrides hardcoded value) |
|
||||
| `AUTO_GROUP_ENABLED` | boolean | `false` | Enable automatic grouping beyond album |
|
||||
| `AUTO_GROUP_TIME_WINDOW_MINUTES` | number | `5` | Time-window clustering threshold |
|
||||
| `AUTO_GROUP_PATTERN_ENABLED` | boolean | `false` | Enable filename/date pattern grouping |
|
||||
| `INTEGRITY_AUDIT_ENABLED` | boolean | `false` | Enable periodic integrity audit |
|
||||
| `INTEGRITY_AUDIT_INTERVAL_HOURS` | number | `24` | How often to run the audit |
|
||||
|
||||
### Premium Mode Behavior
|
||||
|
||||
When `TELEGRAM_PREMIUM=true`:
|
||||
1. `MAX_PART_SIZE_MB` defaults to `3900` (safely under 4 GiB) instead of `1950`
|
||||
2. Files under 4 GB: uploaded as-is (no splitting)
|
||||
3. Files over 4 GB: split using existing `byteLevelSplit()` at the new threshold
|
||||
4. Existing split/rejoin logic is **kept as fallback** — never removed
|
||||
5. `isMultipart` and `partCount` continue to track actual upload state
|
||||
|
||||
### Implementation in `split.ts`:
|
||||
|
||||
```typescript
|
||||
// Replace hardcoded constant with config-driven:
|
||||
const MAX_PART_SIZE = BigInt(config.maxPartSizeMB) * 1024n * 1024n;
|
||||
```
|
||||
|
||||
And in `config.ts`:
|
||||
```typescript
|
||||
maxPartSizeMB: parseInt(
|
||||
process.env.MAX_PART_SIZE_MB ??
|
||||
(process.env.TELEGRAM_PREMIUM === "true" ? "3900" : "1950"),
|
||||
10
|
||||
),
|
||||
```
|
||||
|
||||
### Rollout Strategy
|
||||
|
||||
1. **All flags default to off** — zero behavior change on deploy
|
||||
2. Enable `TELEGRAM_PREMIUM` first (simple, well-understood)
|
||||
3. Enable `AUTO_GROUP_ENABLED` on a **per-channel basis** (see test plan) before globally
|
||||
4. Enable `INTEGRITY_AUDIT_ENABLED` after manual validation
|
||||
5. Pattern-based grouping enabled last (highest complexity)
|
||||
|
||||
---
|
||||
|
||||
## Deliverable 6: Test Plan
|
||||
|
||||
### Phase 0: Pre-Implementation Validation
|
||||
|
||||
Before touching any code, verify the current system baseline:
|
||||
|
||||
1. **Pick one test channel** with known content (a mix of albums, single files, and multipart archives)
|
||||
2. Run an ingestion cycle and record: number of packages, groups, skipped
|
||||
3. Verify all album-based groups are correct
|
||||
4. Note any ungrouped files that "should" be grouped
|
||||
5. This becomes the **regression baseline**
|
||||
|
||||
### Phase 1: Premium Mode Testing
|
||||
|
||||
1. Set `TELEGRAM_PREMIUM=true` and `MAX_PART_SIZE_MB=3900`
|
||||
2. Manually upload a 3 GB test file to a source channel
|
||||
3. Trigger ingestion — verify it uploads as a single message (not split)
|
||||
4. Manually upload a 5 GB test file
|
||||
5. Trigger ingestion — verify it splits at ~3.9 GB threshold
|
||||
6. Verify `isMultipart`, `partCount`, `destMessageIds` are correct
|
||||
7. Send the package via bot — verify all parts arrive
|
||||
|
||||
### Phase 2: Time-Window Grouping Testing
|
||||
|
||||
1. Enable `AUTO_GROUP_ENABLED=true` on the test channel only
|
||||
2. Post 3 files to the channel within 2 minutes (no album)
|
||||
3. Trigger ingestion — verify they auto-group
|
||||
4. Post 2 files 10 minutes apart
|
||||
5. Trigger ingestion — verify they stay ungrouped
|
||||
6. Manually group them — verify `GroupingRule` is created
|
||||
7. Post similar files — verify auto-grouping kicks in
|
||||
|
||||
### Phase 3: Manual QA via API
|
||||
|
||||
Add a **test endpoint** (dev-only) that accepts a fake message payload and runs it through the grouping pipeline without hitting Telegram:
|
||||
|
||||
```
|
||||
POST /api/dev/test-grouping
|
||||
Body: { messages: [...], channelId: "..." }
|
||||
Response: { suggestedGroups: [...] }
|
||||
```
|
||||
|
||||
This allows testing grouping logic against crafted scenarios without waiting for real Telegram messages.
|
||||
|
||||
### Phase 4: Integrity Audit Testing
|
||||
|
||||
1. Enable `INTEGRITY_AUDIT_ENABLED=true`
|
||||
2. Manually corrupt a record (set wrong `contentHash` in DB)
|
||||
3. Run audit — verify `HASH_MISMATCH` notification is created
|
||||
4. Delete one `destMessageId` from a multipart package's `destMessageIds`
|
||||
5. Run audit — verify `MISSING_PART` notification is created
|
||||
6. Check notification UI shows both
|
||||
|
||||
### Regression Checks After Each Phase
|
||||
|
||||
- Re-run ingestion on test channel — same number of packages/groups as baseline
|
||||
- Search for known filenames — still returns correct results
|
||||
- Send a package via bot — still delivers correctly
|
||||
- Album groups unchanged
|
||||
- Manual groups unchanged
|
||||
@@ -0,0 +1,67 @@
|
||||
# Grouping Phase 1: Foundation + Time-Window Grouping
|
||||
|
||||
> **For agentic workers:** Use superpowers:subagent-driven-development to implement this plan.
|
||||
|
||||
**Goal:** Add grouping infrastructure (schema, enums, notifications model), an ungrouped staging queue in the UI, and time-window auto-grouping as the first automatic signal beyond album grouping.
|
||||
|
||||
**Architecture:** Schema changes lay the foundation. Ungrouped tab is a query filter. Time-window grouping runs as a post-processing pass after album grouping in the worker pipeline.
|
||||
|
||||
**Tech Stack:** Prisma schema + migration, worker TypeScript, Next.js App Router.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Schema Migration
|
||||
|
||||
**Files:**
|
||||
- Modify: `prisma/schema.prisma`
|
||||
- Create: migration SQL
|
||||
|
||||
Add:
|
||||
1. `GroupingSource` enum: `ALBUM`, `MANUAL`, `AUTO_TIME`, `AUTO_PATTERN`, `AUTO_REPLY`, `AUTO_ZIP`, `AUTO_CAPTION`
|
||||
2. `groupingSource GroupingSource @default(MANUAL)` on `PackageGroup`
|
||||
3. `SystemNotification` model with `type`, `severity`, `title`, `message`, `context` (Json), `isRead`
|
||||
4. `NotificationType` enum: `HASH_MISMATCH`, `MISSING_PART`, `UPLOAD_FAILED`, `DOWNLOAD_FAILED`, `GROUPING_CONFLICT`, `INTEGRITY_AUDIT`
|
||||
5. `NotificationSeverity` enum: `INFO`, `WARNING`, `ERROR`
|
||||
|
||||
Backfill: `UPDATE package_groups SET "groupingSource" = 'ALBUM' WHERE "mediaAlbumId" IS NOT NULL`
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Ungrouped Staging Tab in STL Page
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/lib/telegram/queries.ts` — add `listUngroupedPackages()` query
|
||||
- Modify: `src/app/(app)/stls/page.tsx` — add tab parameter support
|
||||
- Modify: `src/app/(app)/stls/_components/stl-table.tsx` — add "Ungrouped" tab
|
||||
|
||||
Add a tab next to the existing "Skipped" tab that shows packages where `packageGroupId IS NULL`. Uses the existing `PackageListItem` type and table rendering. This gives users a clear view of files that need manual grouping.
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Time-Window Auto-Grouping in Worker
|
||||
|
||||
**Files:**
|
||||
- Create: `worker/src/grouping.ts` — add `processTimeWindowGroups()` after existing `processAlbumGroups()`
|
||||
- Modify: `worker/src/worker.ts` — call time-window grouping after album grouping
|
||||
- Modify: `worker/src/util/config.ts` — add `autoGroupTimeWindowMinutes` config
|
||||
|
||||
After album grouping completes, find remaining ungrouped packages from the same channel scan. Cluster packages whose `sourceMessageId` timestamps are within the configured window (default 5 minutes). Create groups for clusters of 2+ with `groupingSource = AUTO_TIME` and name derived from the common filename prefix or first file's base name.
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Hash Verification After Split
|
||||
|
||||
**Files:**
|
||||
- Modify: `worker/src/worker.ts` — add hash re-check after concat+split
|
||||
- Modify: `worker/src/archive/hash.ts` — (no changes needed, reuse `hashParts`)
|
||||
|
||||
After `concatenateFiles()` + `byteLevelSplit()`, re-hash the split parts and compare to the original `contentHash`. If mismatch, log error and create a `SystemNotification` (once that table exists). This closes the integrity gap identified in the audit.
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Build & Deploy
|
||||
|
||||
Rebuild worker and app images. Deploy. Verify:
|
||||
- Worker logs show `maxPartSizeMB` and new `autoGroupTimeWindowMinutes` in config
|
||||
- Ungrouped tab visible in STL page
|
||||
- Previously-skipped large archives begin processing
|
||||
Reference in New Issue
Block a user