25 Commits

Author SHA1 Message Date
6178ff3b08 docs: add Task 3 (creator filter combobox) to plan
All checks were successful
continuous-integration/drone/push Build is passing
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 09:42:37 +02:00
57842c6d95 Merge feat/send-all-from-creator: send-all-from-creator button + searchable creator filter 2026-07-04 09:42:08 +02:00
b3c49c3794 feat(stls): add searchable creator filter combobox to toolbar 2026-07-04 09:36:51 +02:00
20e60bc6af feat(stls): add 'send all from creator' toolbar button 2026-07-04 09:26:36 +02:00
50e89719bb feat(stls): add sendAllFromCreatorAction server action 2026-07-04 09:20:36 +02:00
1cae855c26 docs: implementation plan for send-all-from-creator button
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 09:15:58 +02:00
b0baf72f0a docs: spec for send-all-from-creator button on STL page
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 09:14:33 +02:00
0cf5fcd3a7 fix(stls): stop loading preview blobs in list queries (OOM fix)
All checks were successful
continuous-integration/drone/push Build is passing
The STLs list queries selected the full previewData JPEG bytes (avg
~700KB, up to 2MB, and every group member's preview) only to compute a
hasPreview boolean. Under concurrent page loads this piled blobs into
the Node heap, exceeded the 512MB container limit, crashed the process,
and surfaced as repeated connection errors while browsing.

Replace the byte-loading select with a fetchPreviewFlags() helper that
checks `previewData IS NOT NULL` in SQL (IDs only, no bytes). Applied to
listPackages, listDisplayItems, searchPackages, and listUngroupedPackages.

Verified: same concurrent load that drove memory to 508/512MB and forced
a restart now peaks at 156MB (30%) with zero failed requests and no restart.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 19:05:34 +02:00
3b7202a662 harden(telegram): reject non-positive topic id in disableActiveTopic
All checks were successful
continuous-integration/drone/push Build is passing
BigInt("") / BigInt("0") don't throw; guard against them since the
action is exported and admin-callable independent of the UI button.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 12:15:03 +02:00
23f8e91c50 feat(telegram): add "skip & disable topic" on the live worker panel
Track the forum topic currently being processed on IngestionRun
(currentTopicId + currentAccountChannelMapId; additive migration) and
expose it on the live status. processArchiveSets gains an optional
shouldStop callback polled before each archive set; the forum branch
passes a live isTopicFetchEnabled check, so disabling a topic mid-run
lets the in-flight file finish, then skips the rest of that topic.

A new disableActiveTopic server action sets the topic's fetchEnabled
false (upsert), and the worker status panel shows a "Skip & disable
this topic" button while a topic is being processed. Future runs skip
the topic via the existing live per-topic read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 12:10:56 +02:00
c749d03376 Merge fix/topics-drawer-scroll-and-filter: topics drawer scroll + title filter
All checks were successful
continuous-integration/drone/push Build is passing
2026-06-22 09:05:46 +02:00
d7771887a1 fix(worker): silence false-positive integrity-test notifications
The pre-upload integrity test is advisory (never blocks upload), but it
raised a WARNING SystemNotification whenever `7z t` failed — most often
because large 7z archives OOM-kill the test process (SIGKILL / exit 137)
in the memory-limited worker container, which is a tool limitation, not
corruption. Classify failures as encrypted | corrupt | inconclusive;
suppress notifications for inconclusive (debug log only) while still
proceeding with the upload as before. Genuine corruption now uses the
INTEGRITY_AUDIT notification type instead of the misleading HASH_MISMATCH;
encrypted archives still notify.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 08:57:25 +02:00
26bc43299a fix(telegram): make topics drawer scrollable + add topic title filter
The Radix ScrollArea viewport did not get a bounded height inside the
flex-column, max-h, vertically-centred dialog, so a long topic list
overflowed the dialog instead of scrolling (last topics unreachable).
Switch to a native flex-1 min-h-0 overflow-y-auto container, which is
the canonical touch-friendly flex-scroll pattern.

Also add a "Filter topics..." input (matches by topic title) so any
topic can be reached regardless of list length; reset the filter on
close; and add a toast fallback message on toggle failure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 08:50:10 +02:00
8b500a1610 feat(stls): enlarge candidate images in preview picker
All checks were successful
continuous-integration/drone/push Build is passing
Each loaded candidate gets an enlarge button that opens the ImageLightbox
without changing selection, so the right preview is easier to choose.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 04:57:40 +02:00
f0a9d3b4da feat(stls): click-to-enlarge preview in package files drawer
The drawer preview image opens the ImageLightbox; replacing the image
stays available via the existing Upload/Pick Preview buttons. No-preview
upload affordance unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 04:48:13 +02:00
b90317c007 feat(stls): click-to-enlarge package preview thumbnail in table
Package preview thumbnails open the ImageLightbox on click (hover shows a
maximize affordance). No-preview cells unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 04:44:52 +02:00
6f8ddcca81 feat(stls): add reusable ImageLightbox for full-size preview viewing
Radix Dialog-based lightbox (Esc / overlay / close button to dismiss),
image shown object-contain capped to the viewport. No new dependencies.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 04:17:42 +02:00
25ff067ea0 feat(telegram): add Topics action to forum channels in Channels tab
Forum source channels get a Topics menu item that opens the TopicsDrawer.
Non-forum channels are unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 04:15:46 +02:00
7e58cc29fa feat(telegram): add TopicsDrawer for per-topic fetch toggles
Lists a channel's topics with an enable/disable Switch each (optimistic,
revert on failure). Empty state explains topics appear after the next scan.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 01:24:57 +02:00
5a4e358eee feat(telegram): add setTopicFetchEnabled server action
Admin-guarded toggle of TopicProgress.fetchEnabled by row id. Persists
immediately; the worker honours it on its next live per-topic read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 01:22:45 +02:00
c31afc5b92 feat(telegram): add GET /api/telegram/channels/[channelId]/topics
Returns per-topic fetch state for a channel, mirroring the existing
account-links route auth pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 01:19:46 +02:00
6324d64870 feat(telegram): expose isForum on ChannelRow and add listChannelTopics
ChannelRow now carries isForum so the UI can show the Topics action for
forum channels only. listChannelTopics returns per-topic fetch state for
a channel from TopicProgress.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 01:16:42 +02:00
974769350b feat(worker): honour per-topic fetch toggle live in scan loop
Eager-persist a TopicProgress row for every discovered topic at run
start, and read the live fetchEnabled flag per topic so a mid-run
disable skips topics not yet started. Disabled topics are skipped
before any TDLib scan or fetch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 01:13:10 +02:00
f9b82f1654 feat(worker): add ensureTopicProgressRows + isTopicFetchEnabled helpers
ensureTopicProgressRows inserts missing topic rows only (skipDuplicates),
leaving watermarks/scan-state/fetchEnabled untouched. isTopicFetchEnabled
reads the live per-topic flag for mid-run skip decisions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 00:27:24 +02:00
7146a5cf0d feat(db): add fetchEnabled flag to TopicProgress
Additive Postgres migration (default true) that backs the per-topic
fetch enable/disable toggle. Existing rows backfill to enabled, so
current behaviour is unchanged. Migration applied to the live DB via
CI prisma migrate deploy (deferred, per environment).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 00:25:21 +02:00
25 changed files with 1646 additions and 143 deletions

View File

@@ -0,0 +1,454 @@
# Send All From Creator Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add a toolbar button on the STL packages page that queues every sendable package from the currently-filtered creator for delivery to the user's Telegram.
**Architecture:** A new server action `sendAllFromCreatorAction(creatorName)` mirrors the existing `sendAllInGroupAction`: it fetches sendable packages for the creator, dedups against live `BotSendRequest`s, creates a `BotSendRequest` per package, and fires `pg_notify('bot_send', id)`. The bot's existing `send-listener` consumes these unchanged. The client table renders a "Send all from [Creator]" button in the packages-tab toolbar only when `?creator=<name>` is active, wired to the action with a confirm dialog and a count toast.
**Tech Stack:** Next.js 16 App Router, TypeScript, Prisma v7, server actions, `pg_notify`, sonner toasts, lucide-react icons.
**Testing note:** This repo has no test framework (`CLAUDE.md`: "Testing is manual"). Verification for each task is `npm run lint` + `npm run build`, plus manual UI checks at the end. Do not scaffold a test harness.
---
### Task 1: Add `sendAllFromCreatorAction` server action
**Files:**
- Modify: `src/app/(app)/stls/actions.ts` (append new action after `sendAllInGroupAction`, which ends at line 591)
Reference implementation to mirror: `sendAllInGroupAction` at `src/app/(app)/stls/actions.ts:515-591`.
Existing imports already present in the file (do not re-add): `auth` from `@/lib/auth`, `prisma` from `@/lib/prisma`, `ActionResult` from `@/types/api.types`, `revalidatePath` from `next/cache`.
`ActionResult` is generic: `ActionResult<T = void>` (`src/types/api.types.ts`).
- [ ] **Step 1: Append the new action**
Add this to the end of `src/app/(app)/stls/actions.ts`:
```ts
export async function sendAllFromCreatorAction(
creatorName: string
): Promise<ActionResult<{ queued: number; skipped: number }>> {
const session = await auth();
if (!session?.user?.id) return { success: false, error: "Unauthorized" };
const creator = creatorName.trim();
if (!creator) {
return { success: false, error: "No creator specified" };
}
try {
const telegramLink = await prisma.telegramLink.findUnique({
where: { userId: session.user.id },
});
if (!telegramLink) {
return { success: false, error: "No linked Telegram account. Link one in Settings." };
}
const sendablePackages = await prisma.package.findMany({
where: {
creator,
destChannelId: { not: null },
destMessageId: { not: null },
},
select: { id: true },
});
if (sendablePackages.length === 0) {
return { success: false, error: "No uploaded packages found for this creator" };
}
let queued = 0;
let skipped = 0;
for (const pkg of sendablePackages) {
// Only create if no existing PENDING/SENDING request for this package+link combo
const existing = await prisma.botSendRequest.findFirst({
where: {
packageId: pkg.id,
telegramLinkId: telegramLink.id,
status: { in: ["PENDING", "SENDING"] },
},
});
if (existing) {
skipped++;
continue;
}
const sendRequest = await prisma.botSendRequest.create({
data: {
packageId: pkg.id,
telegramLinkId: telegramLink.id,
requestedByUserId: session.user.id,
status: "PENDING",
},
});
// Notify the bot via pg_notify
try {
await prisma.$queryRawUnsafe(
`SELECT pg_notify('bot_send', $1)`,
sendRequest.id
);
} catch {
// Best-effort — the bot also polls periodically
}
queued++;
}
revalidatePath("/stls");
return { success: true, data: { queued, skipped } };
} catch {
return { success: false, error: "Failed to send creator packages" };
}
}
```
- [ ] **Step 2: Verify lint + typecheck pass**
Run: `npm run lint`
Expected: no new errors in `src/app/(app)/stls/actions.ts`.
- [ ] **Step 3: Commit**
```bash
git add src/app/\(app\)/stls/actions.ts
git commit -m "feat(stls): add sendAllFromCreatorAction server action"
```
---
### Task 2: Wire up the toolbar button in the STL table
**Files:**
- Modify: `src/app/(app)/stls/_components/stl-table.tsx`
- lucide import (line 6)
- actions import block (lines 44-54)
- handler (after `handleSendAllInGroup`, which ends at line 278)
- `activeCreator` derivation (near `activeTag` at line 445)
- toolbar JSX (packages-tab toolbar row starting at line 478)
- [ ] **Step 1: Add the `Send` icon to the lucide import**
Change line 6 from:
```ts
import { Search, Layers, Upload } from "lucide-react";
```
to:
```ts
import { Search, Layers, Upload, Send } from "lucide-react";
```
- [ ] **Step 2: Import the new action**
In the actions import block (lines 44-54), add `sendAllFromCreatorAction` next to `sendAllInGroupAction`:
```ts
import {
updatePackageCreator,
updatePackageTags,
renameGroupAction,
dissolveGroupAction,
createGroupAction,
removeFromGroupAction,
sendAllInGroupAction,
sendAllFromCreatorAction,
updateGroupPreviewAction,
mergeGroupsAction,
} from "../actions";
```
- [ ] **Step 3: Derive the active creator from the URL**
Immediately after the `activeTag` line (`src/app/(app)/stls/_components/stl-table.tsx:445`):
```ts
const activeTag = searchParams.get("tag") ?? "";
```
add:
```ts
const activeCreator = searchParams.get("creator") ?? "";
```
- [ ] **Step 4: Add the click handler**
After `handleSendAllInGroup` (which ends at line 278), add:
```ts
const handleSendAllFromCreator = useCallback(() => {
if (!confirm(`Send all packages from "${activeCreator}" to your Telegram?`)) return;
startTransition(async () => {
const result = await sendAllFromCreatorAction(activeCreator);
if (result.success) {
const { queued, skipped } = result.data;
toast.success(
`Queued ${queued} package${queued === 1 ? "" : "s"} from ${activeCreator}` +
(skipped ? ` (${skipped} already queued)` : "")
);
router.refresh();
} else {
toast.error(result.error);
}
});
}, [activeCreator, router]);
```
- [ ] **Step 5: Render the toolbar button**
In the packages-tab toolbar (`flex flex-wrap items-center gap-2` row at line 478), add the button right after the "Upload Files" button (which closes at line 507, before the `selectedPackages.size >= 2` block at line 508):
```tsx
{activeCreator && (
<Button
variant="outline"
size="sm"
className="h-9 gap-1.5"
onClick={handleSendAllFromCreator}
>
<Send className="h-3.5 w-3.5" />
Send all from {activeCreator}
</Button>
)}
```
- [ ] **Step 6: Verify lint + build pass**
Run: `npm run lint && npm run build`
Expected: no new errors; build completes.
- [ ] **Step 7: Manual verification**
1. `npm run dev`, open `/stls`.
2. Click a creator name in the Creator column to apply `?creator=<name>` (or navigate to `/stls?creator=<known creator>`).
3. Confirm the "Send all from [Creator]" button appears in the toolbar.
4. Remove the creator filter → confirm the button disappears.
5. Click the button → confirm the dialog appears; on confirm, a toast reports the queued count.
6. (If a linked Telegram account + uploaded packages exist) confirm packages arrive via the bot.
- [ ] **Step 8: Commit**
```bash
git add src/app/\(app\)/stls/_components/stl-table.tsx
git commit -m "feat(stls): add 'send all from creator' toolbar button"
```
---
### Task 3: Searchable creator filter combobox (makes the filter reachable)
**Why:** Discovered during execution — the `?creator=` filter that reveals the Task 2
button had no UI trigger. The creator table cell click only opens the edit prompt
(`package-columns.tsx:339``onSetCreator`). This task adds a searchable
"All Creators" combobox to the packages-tab toolbar (like the existing Tags select,
but type-to-filter since there can be many creators). The creator cell stays
edit-only.
**Files:**
- Modify: `src/lib/telegram/queries.ts` — add `getAllPackageCreators()`.
- Modify: `src/app/(app)/stls/page.tsx` — fetch + pass `availableCreators`.
- Create: `src/app/(app)/stls/_components/creator-filter.tsx` — the combobox.
- Modify: `src/app/(app)/stls/_components/stl-table.tsx` — new prop, handler, render.
- [ ] **Step 1: Add the distinct-creators query**
Append to `src/lib/telegram/queries.ts` (mirrors `getAllPackageTags` at line 530):
```ts
export async function getAllPackageCreators(): Promise<string[]> {
const result = await prisma.$queryRaw<{ creator: string }[]>`
SELECT DISTINCT creator FROM packages
WHERE creator IS NOT NULL AND creator <> ''
ORDER BY creator
`;
return result.map((r) => r.creator);
}
```
- [ ] **Step 2: Create the combobox component**
Create `src/app/(app)/stls/_components/creator-filter.tsx` (mirrors the
Popover+Command pattern in `src/components/shared/data-table-faceted-filter.tsx`):
```tsx
"use client";
import { useState } from "react";
import { Check, ChevronsUpDown } from "lucide-react";
import { cn } from "@/lib/utils";
import { Button } from "@/components/ui/button";
import {
Command,
CommandEmpty,
CommandGroup,
CommandInput,
CommandItem,
CommandList,
} from "@/components/ui/command";
import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover";
interface CreatorFilterProps {
creators: string[];
value: string; // active creator, "" when none
onChange: (creator: string) => void; // "" clears the filter
}
export function CreatorFilter({ creators, value, onChange }: CreatorFilterProps) {
const [open, setOpen] = useState(false);
return (
<Popover open={open} onOpenChange={setOpen}>
<PopoverTrigger asChild>
<Button
variant="outline"
size="sm"
role="combobox"
aria-expanded={open}
className="h-9 w-[200px] justify-between"
>
<span className="truncate">{value || "All Creators"}</span>
<ChevronsUpDown className="ml-2 h-4 w-4 shrink-0 opacity-50" />
</Button>
</PopoverTrigger>
<PopoverContent className="w-[240px] p-0" align="start">
<Command>
<CommandInput placeholder="Search creators..." className="h-9" />
<CommandList>
<CommandEmpty>No creators found.</CommandEmpty>
<CommandGroup>
<CommandItem
value="__all__"
onSelect={() => {
onChange("");
setOpen(false);
}}
>
<Check
className={cn("mr-2 h-4 w-4", value === "" ? "opacity-100" : "opacity-0")}
/>
All Creators
</CommandItem>
{creators.map((creator) => (
<CommandItem
key={creator}
value={creator}
onSelect={() => {
onChange(creator);
setOpen(false);
}}
>
<Check
className={cn(
"mr-2 h-4 w-4",
value === creator ? "opacity-100" : "opacity-0"
)}
/>
<span className="truncate">{creator}</span>
</CommandItem>
))}
</CommandGroup>
</CommandList>
</Command>
</PopoverContent>
</Popover>
);
}
```
- [ ] **Step 3: Wire the query through the page**
In `src/app/(app)/stls/page.tsx`:
- Add `getAllPackageCreators` to the import from `@/lib/telegram/queries` (line 3).
- Add `getAllPackageCreators()` to the `Promise.all` (line 27) and destructure
`availableCreators`:
```ts
const [result, ingestionStatus, availableTags, availableCreators, skippedCount, ungroupedCount] =
await Promise.all([
// ...existing entries unchanged...
getIngestionStatus(),
getAllPackageTags(),
getAllPackageCreators(),
countSkippedPackages(),
countUngroupedPackages(),
]);
```
(Insert `getAllPackageCreators()` immediately after `getAllPackageTags()`, and add
`availableCreators` in the matching position of the destructure.)
- Pass the prop to `<StlTable>`: `availableCreators={availableCreators}` (next to
`availableTags={availableTags}`).
- [ ] **Step 4: Add prop + handler + render in the table**
In `src/app/(app)/stls/_components/stl-table.tsx`:
- Import the component: `import { CreatorFilter } from "./creator-filter";`
- Add to `StlTableProps` (next to `availableTags: string[];`): `availableCreators: string[];`
- Add to the destructured params (next to `availableTags,`): `availableCreators,`
- Add the handler after `updateTagFilter` (ends ~line 210):
```ts
const updateCreatorFilter = useCallback(
(value: string) => {
const params = new URLSearchParams(searchParams.toString());
if (value) {
params.set("creator", value);
params.set("page", "1");
} else {
params.delete("creator");
}
router.push(`${pathname}?${params.toString()}`, { scroll: false });
},
[router, pathname, searchParams]
);
```
- Render it in the toolbar right after the Tags `Select` block (the
`{availableTags.length > 0 && (...)}` block, ~lines 507-522):
```tsx
{availableCreators.length > 0 && (
<CreatorFilter
creators={availableCreators}
value={activeCreator}
onChange={updateCreatorFilter}
/>
)}
```
- [ ] **Step 5: Verify + manual test**
Run: `npx tsc --noEmit` (filter for the touched files — expect none new). Note: full
`npm run build` fails on a pre-existing stale Prisma client issue in
`src/lib/telegram/*` unrelated to this change; do not attempt to fix it.
Manual: open `/stls`, use the "All Creators" combobox, type to filter, pick a
creator → list filters and the "Send all from [Creator]" button appears; pick
"All Creators" → filter clears and button disappears.
- [ ] **Step 6: Commit**
```bash
git add "src/lib/telegram/queries.ts" "src/app/(app)/stls/page.tsx" \
"src/app/(app)/stls/_components/creator-filter.tsx" \
"src/app/(app)/stls/_components/stl-table.tsx"
git commit -m "feat(stls): add searchable creator filter combobox to toolbar"
```
---
## Self-Review
**Spec coverage:**
- Req 1 (button when filtered by creator) → Task 2 Step 5 (`{activeCreator && ...}`).
- Req 2 (hidden when no filter) → Task 2 Step 5 conditional.
- Req 3 (confirm dialog) → Task 2 Step 4 `confirm(...)`.
- Req 4 (all pages, not current page) → Task 1 queries `prisma.package.findMany` by `creator`, unpaginated.
- Req 5 (skip not-uploaded + dedup live requests) → Task 1 `where` filter on `destChannelId`/`destMessageId` + `existing` check.
- Req 6 (report counts) → Task 1 returns `{ queued, skipped }`; Task 2 Step 4 toast.
- Req 7 (no polling) → Task 2 handler queues + `router.refresh()`, no poll loop.
- Blank-creator guard → Task 1 `creator.trim()` check.
**Placeholder scan:** No TBD/TODO/placeholder steps; all code shown in full.
**Type consistency:** Action returns `ActionResult<{ queued: number; skipped: number }>`; handler destructures `result.data.{queued,skipped}` inside the `result.success` branch (where `data` is typed). `sendAllFromCreatorAction` name matches between Task 1 definition, Task 2 import, and Task 2 call site. `activeCreator` defined once (Step 3) and used in Steps 4-5.

View File

@@ -0,0 +1,170 @@
# Send All From Creator — Design
**Date:** 2026-07-04
**Status:** Approved for planning
## Summary
Add a "Send all from [Creator]" button to the STL packages page that queues every
sendable package from a given creator for delivery to the user's Telegram. The
button appears in the packages-tab toolbar only when the list is filtered by a
single creator (`?creator=<name>`).
This reuses the existing single-send infrastructure (`BotSendRequest` +
`pg_notify('bot_send', id)` + the bot's `send-listener`) and closely mirrors the
existing `sendAllInGroupAction`. No bot, DB schema, or send-listener changes are
required.
## Context
Relevant existing pieces:
- **Creator field:** `Package.creator` is a nullable, indexed `String` on the
`Package` model (`prisma/schema.prisma:476-524`). Filtering by exact creator is
cheap.
- **Creator filter:** The packages list already supports `?creator=<name>`
(`src/app/(app)/stls/page.tsx:22,38`), rendered by the packages column as a
clickable value (`package-columns.tsx`).
- **Single send:** `send-to-telegram-button.tsx``POST /api/telegram/bot/send`
creates a `BotSendRequest` (status `PENDING`) and fires
`pg_notify('bot_send', requestId)`; the button then polls the request to
completion.
- **Existing bulk send (group):** `sendAllInGroupAction(groupId)` in
`src/app/(app)/stls/actions.ts:515-591` fetches the group's packages, filters to
those with `destChannelId` + `destMessageId`, skips any with an existing
`PENDING`/`SENDING` request for the same package + telegram link, creates a
`BotSendRequest` per remaining package, and fires `pg_notify` per package. It is
wired into the table via `handleSendAllInGroup` (`stl-table.tsx:264-278`) with a
`confirm()` dialog and a success toast — no polling.
- **Bot side:** `bot/src/send-listener.ts` processes every `BotSendRequest`
uniformly; nothing there needs to change.
## Requirements
1. When the packages tab is filtered by a single creator (`?creator=<name>` is
present), show a toolbar button labeled **"Send all from [Creator]"**.
2. When no creator filter is active, the button is not rendered.
3. Clicking the button shows a confirmation dialog before sending (consistent with
the single-send and group-send flows, since this pushes files to Telegram).
4. On confirm, queue **all** sendable packages from that creator across all pages —
not just the current page.
5. Skip packages that are not yet uploaded (missing `destChannelId` /
`destMessageId`) and packages that already have a `PENDING`/`SENDING` request for
the same package + telegram link (dedup).
6. Report real counts back to the user via toast, e.g.
"Queued 12 packages from [Creator]" and, when applicable, a skipped count.
7. No polling — the bulk action queues and returns; sends process in the background
via the bot listener (same behavior as group send).
## Design
### Server action — `sendAllFromCreatorAction(creatorName: string)`
New action in `src/app/(app)/stls/actions.ts`, mirroring `sendAllInGroupAction`:
1. Auth check (`auth()`), return `Unauthorized` if no session.
2. Load the user's `TelegramLink`; if none, return
"No linked Telegram account. Link one in Settings."
3. Reject an empty/blank `creatorName` (guard against sending the entire "no
creator" set — see Edge cases).
4. Fetch sendable packages directly with a `where` filter (no in-memory filtering):
```ts
prisma.package.findMany({
where: {
creator: creatorName,
destChannelId: { not: null },
destMessageId: { not: null },
},
select: { id: true },
})
```
5. If none, return "No uploaded packages found for this creator."
6. For each package, skip if an existing `PENDING`/`SENDING` `BotSendRequest`
exists for that package + telegram link; otherwise create a `BotSendRequest`
(status `PENDING`) and fire `pg_notify('bot_send', id)` (best-effort, wrapped in
try/catch — the bot also polls).
7. `revalidatePath("/stls")`.
8. Return counts so the UI can show them. This is a small improvement over
`sendAllInGroupAction`, which currently returns `{ success: true, data: undefined }`.
Shape:
```ts
{ success: true, data: { queued: number, skipped: number } }
```
where `skipped` counts packages that already had a live request. (Packages not
yet uploaded are excluded by the query and not counted as skipped.)
### UI — toolbar button in `stl-table.tsx`
- Derive the active creator from the URL: `const activeCreator =
searchParams.get("creator") ?? "";` (parallel to the existing
`activeTag` at `stl-table.tsx:445`).
- In the packages-tab toolbar (`stl-table.tsx:478` `flex flex-wrap items-center
gap-2` row), render the button only when `activeCreator` is non-empty:
```tsx
{activeCreator && (
<Button variant="outline" size="sm" className="h-9 gap-1.5"
onClick={handleSendAllFromCreator}>
<Send className="h-3.5 w-3.5" />
Send all from {activeCreator}
</Button>
)}
```
(Reuse the icon already used by the send action; place near the Upload / Group
buttons.)
- Add `handleSendAllFromCreator`, modeled on `handleSendAllInGroup`
(`stl-table.tsx:264-278`):
```ts
const handleSendAllFromCreator = useCallback(() => {
if (!confirm(`Send all packages from "${activeCreator}" to your Telegram?`)) return;
startTransition(async () => {
const result = await sendAllFromCreatorAction(activeCreator);
if (result.success) {
const { queued, skipped } = result.data;
toast.success(
`Queued ${queued} package${queued === 1 ? "" : "s"} from ${activeCreator}` +
(skipped ? ` (${skipped} already queued)` : "")
);
router.refresh();
} else {
toast.error(result.error);
}
});
}, [activeCreator, router]);
```
- Import `sendAllFromCreatorAction` alongside the existing `sendAllInGroupAction`
import (`stl-table.tsx:51`).
### Data flow
```
User (filtered by creator) → clicks button → confirm dialog
→ sendAllFromCreatorAction(creatorName)
→ for each sendable, un-queued package: create BotSendRequest + pg_notify
→ returns { queued, skipped } → toast + router.refresh()
(background) bot send-listener consumes each BotSendRequest → delivers to user
```
## Edge cases
- **Blank creator:** The action rejects an empty/whitespace `creatorName` so a
stray `?creator=` never queues every package. The UI already gates the button on
a non-empty `activeCreator`, so this is defense-in-depth.
- **No linked Telegram account:** Returns an error toast, queues nothing.
- **No uploaded packages for creator:** Returns an error toast, queues nothing.
- **All packages already queued:** `queued: 0`, `skipped: N` → toast reflects it.
- **pg_notify failure:** Swallowed per-package (best-effort); the bot's periodic
poll picks the request up. Matches existing behavior.
## Out of scope / non-goals
- No per-row "send all from this creator" action (toolbar-only, per decision).
- No progress polling / live completion tracking for the bulk send.
- No changes to the bot, `send-listener`, `BotSendRequest` schema, or API routes.
- No batching/throttling beyond what the bot listener already does.
## Files touched
- `src/app/(app)/stls/actions.ts` — add `sendAllFromCreatorAction`.
- `src/app/(app)/stls/_components/stl-table.tsx` — derive `activeCreator`, add
`handleSendAllFromCreator`, render the toolbar button, import the new action.

View File

@@ -0,0 +1,5 @@
-- AlterTable: per-topic user-controlled fetch toggle (forum channels)
-- Additive, safe default (true) so existing rows backfill to "enabled" and
-- current behaviour is unchanged. Disabling is non-destructive.
ALTER TABLE "topic_progress"
ADD COLUMN "fetchEnabled" BOOLEAN NOT NULL DEFAULT true;

View File

@@ -0,0 +1,6 @@
-- AlterTable: track the forum topic currently being processed on the live run
-- so the worker status panel can offer a "skip & disable topic" action.
-- Additive, nullable — no data change for existing rows.
ALTER TABLE "ingestion_runs"
ADD COLUMN "currentTopicId" BIGINT,
ADD COLUMN "currentAccountChannelMapId" TEXT;

View File

@@ -582,6 +582,8 @@ model IngestionRun {
totalBytes BigInt? // Total size of current download
downloadPercent Int? // 0-100
lastActivityAt DateTime? // When activity was last updated
currentTopicId BigInt? // Forum topic currently being processed (live "skip topic")
currentAccountChannelMapId String? // AccountChannelMap owning the topic being processed
account TelegramAccount @relation(fields: [accountId], references: [id])
packages Package[]
@@ -597,6 +599,11 @@ model TopicProgress {
accountChannelMapId String
topicId BigInt
topicName String?
/// User-controlled fetch toggle. When false, the worker skips this topic
/// (no scanning, no fetching/transfer). Defaults true so every existing and
/// newly-discovered topic is fetched unless explicitly disabled. Disabling
/// is non-destructive — already-fetched packages are kept.
fetchEnabled Boolean @default(true)
lastProcessedMessageId BigInt?
/// When this topic was last scanned (any reason). Used by recency-skip.
lastScannedAt DateTime?

View File

@@ -7,6 +7,7 @@ import {
Check,
AlertCircle,
ImageOff,
Maximize2,
} from "lucide-react";
import {
Dialog,
@@ -20,6 +21,7 @@ import { Button } from "@/components/ui/button";
import { cn } from "@/lib/utils";
import { toast } from "sonner";
import { setPreviewFromExtract } from "../actions";
import { ImageLightbox } from "./image-lightbox";
interface ArchiveImage {
id: string;
@@ -65,6 +67,7 @@ export function ArchivePreviewPicker({
const [thumbnails, setThumbnails] = useState<Map<string, ThumbnailState>>(new Map());
const [selectedPath, setSelectedPath] = useState<string | null>(null);
const [isPending, startTransition] = useTransition();
const [lightboxSrc, setLightboxSrc] = useState<string | null>(null);
const pollTimers = useRef<Map<string, ReturnType<typeof setInterval>>>(new Map());
// Track which paths have already been requested to avoid re-requesting
const requestedPaths = useRef<Set<string>>(new Set());
@@ -290,71 +293,85 @@ export function ArchivePreviewPicker({
const isFailed = thumbState?.status === "failed";
return (
<button
key={img.id}
type="button"
className={cn(
"relative aspect-square rounded-lg overflow-hidden border-2 transition-all",
"hover:border-primary/50 cursor-pointer group",
isSelected
? "border-primary ring-2 ring-primary/30"
: "border-border",
isFailed && "opacity-60"
)}
onClick={() => {
if (isLoaded) {
setSelectedPath(img.path);
} else if (isFailed) {
// Allow retry on failed
requestedPaths.current.delete(img.path);
requestThumbnail(img.path);
} else if (!thumbState || thumbState.status === "idle") {
requestThumbnail(img.path);
}
}}
title={img.path}
>
{isLoaded && thumbState.imageUrl ? (
<img
src={thumbState.imageUrl}
alt={img.fileName}
className="h-full w-full object-cover"
loading="lazy"
/>
) : isLoading ? (
<div className="h-full w-full flex items-center justify-center bg-muted">
<Loader2 className="h-5 w-5 animate-spin text-muted-foreground" />
</div>
) : isFailed ? (
<div className="h-full w-full flex flex-col items-center justify-center bg-muted gap-1">
<AlertCircle className="h-4 w-4 text-destructive" />
<span className="text-[10px] text-destructive px-1 text-center">
Click to retry
</span>
</div>
) : (
<div className="h-full w-full flex items-center justify-center bg-muted">
<ImageIcon className="h-5 w-5 text-muted-foreground" />
</div>
<div key={img.id} className="group relative">
{isLoaded && thumbState?.imageUrl && (
<button
type="button"
className="absolute top-1.5 left-1.5 z-10 flex h-6 w-6 items-center justify-center rounded-md bg-black/60 text-white opacity-0 transition-opacity hover:bg-black/80 group-hover:opacity-100"
onClick={(e) => {
e.stopPropagation();
setLightboxSrc(thumbState.imageUrl!);
}}
title="Enlarge"
>
<Maximize2 className="h-3.5 w-3.5" />
</button>
)}
<button
type="button"
className={cn(
"relative aspect-square w-full rounded-lg overflow-hidden border-2 transition-all",
"hover:border-primary/50 cursor-pointer",
isSelected
? "border-primary ring-2 ring-primary/30"
: "border-border",
isFailed && "opacity-60"
)}
onClick={() => {
if (isLoaded) {
setSelectedPath(img.path);
} else if (isFailed) {
// Allow retry on failed
requestedPaths.current.delete(img.path);
requestThumbnail(img.path);
} else if (!thumbState || thumbState.status === "idle") {
requestThumbnail(img.path);
}
}}
title={img.path}
>
{isLoaded && thumbState.imageUrl ? (
<img
src={thumbState.imageUrl}
alt={img.fileName}
className="h-full w-full object-cover"
loading="lazy"
/>
) : isLoading ? (
<div className="h-full w-full flex items-center justify-center bg-muted">
<Loader2 className="h-5 w-5 animate-spin text-muted-foreground" />
</div>
) : isFailed ? (
<div className="h-full w-full flex flex-col items-center justify-center bg-muted gap-1">
<AlertCircle className="h-4 w-4 text-destructive" />
<span className="text-[10px] text-destructive px-1 text-center">
Click to retry
</span>
</div>
) : (
<div className="h-full w-full flex items-center justify-center bg-muted">
<ImageIcon className="h-5 w-5 text-muted-foreground" />
</div>
)}
{/* Selection checkmark */}
{isSelected && (
<div className="absolute top-1.5 right-1.5 h-5 w-5 rounded-full bg-primary flex items-center justify-center">
<Check className="h-3 w-3 text-primary-foreground" />
</div>
)}
{/* Selection checkmark */}
{isSelected && (
<div className="absolute top-1.5 right-1.5 h-5 w-5 rounded-full bg-primary flex items-center justify-center">
<Check className="h-3 w-3 text-primary-foreground" />
</div>
)}
{/* File info overlay */}
<div className="absolute bottom-0 left-0 right-0 bg-black/60 px-1.5 py-1 opacity-0 group-hover:opacity-100 transition-opacity">
<p className="text-[10px] text-white truncate">
{img.fileName}
</p>
<p className="text-[9px] text-white/70">
{formatBytes(img.size)}
</p>
</div>
</button>
{/* File info overlay */}
<div className="absolute bottom-0 left-0 right-0 bg-black/60 px-1.5 py-1 opacity-0 group-hover:opacity-100 transition-opacity">
<p className="text-[10px] text-white truncate">
{img.fileName}
</p>
<p className="text-[9px] text-white/70">
{formatBytes(img.size)}
</p>
</div>
</button>
</div>
);
})}
</div>
@@ -394,6 +411,13 @@ export function ArchivePreviewPicker({
</div>
)}
</DialogContent>
<ImageLightbox
src={lightboxSrc}
open={!!lightboxSrc}
onOpenChange={(open) => {
if (!open) setLightboxSrc(null);
}}
/>
</Dialog>
);
}

View File

@@ -0,0 +1,82 @@
"use client";
import { useState } from "react";
import { Check, ChevronsUpDown } from "lucide-react";
import { cn } from "@/lib/utils";
import { Button } from "@/components/ui/button";
import {
Command,
CommandEmpty,
CommandGroup,
CommandInput,
CommandItem,
CommandList,
} from "@/components/ui/command";
import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover";
interface CreatorFilterProps {
creators: string[];
value: string; // active creator, "" when none
onChange: (creator: string) => void; // "" clears the filter
}
export function CreatorFilter({ creators, value, onChange }: CreatorFilterProps) {
const [open, setOpen] = useState(false);
return (
<Popover open={open} onOpenChange={setOpen}>
<PopoverTrigger asChild>
<Button
variant="outline"
size="sm"
role="combobox"
aria-expanded={open}
className="h-9 w-[200px] justify-between"
>
<span className="truncate">{value || "All Creators"}</span>
<ChevronsUpDown className="ml-2 h-4 w-4 shrink-0 opacity-50" />
</Button>
</PopoverTrigger>
<PopoverContent className="w-[240px] p-0" align="start">
<Command>
<CommandInput placeholder="Search creators..." className="h-9" />
<CommandList>
<CommandEmpty>No creators found.</CommandEmpty>
<CommandGroup>
<CommandItem
value="__all__"
onSelect={() => {
onChange("");
setOpen(false);
}}
>
<Check
className={cn("mr-2 h-4 w-4", value === "" ? "opacity-100" : "opacity-0")}
/>
All Creators
</CommandItem>
{creators.map((creator) => (
<CommandItem
key={creator}
value={creator}
onSelect={() => {
onChange(creator);
setOpen(false);
}}
>
<Check
className={cn(
"mr-2 h-4 w-4",
value === creator ? "opacity-100" : "opacity-0"
)}
/>
<span className="truncate">{creator}</span>
</CommandItem>
))}
</CommandGroup>
</CommandList>
</Command>
</PopoverContent>
</Popover>
);
}

View File

@@ -0,0 +1,41 @@
"use client";
import {
Dialog,
DialogContent,
DialogTitle,
} from "@/components/ui/dialog";
interface ImageLightboxProps {
src: string | null;
alt?: string;
open: boolean;
onOpenChange: (open: boolean) => void;
}
/**
* Full-size, in-page preview viewer. Renders the image at native size capped
* to the viewport (object-contain). Dismiss via the close button, Esc, or by
* clicking the overlay.
*/
export function ImageLightbox({
src,
alt = "",
open,
onOpenChange,
}: ImageLightboxProps) {
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent className="w-auto max-w-[95vw] border-0 bg-transparent p-0 shadow-none sm:max-w-[90vw]">
<DialogTitle className="sr-only">Enlarged preview image</DialogTitle>
{src && (
<img
src={src}
alt={alt}
className="mx-auto max-h-[88vh] w-auto max-w-full rounded-lg object-contain"
/>
)}
</DialogContent>
</Dialog>
);
}

View File

@@ -1,12 +1,14 @@
"use client";
import { useState } from "react";
import { type ColumnDef } from "@tanstack/react-table";
import { FileArchive, Eye, ChevronRight, Layers, Ungroup, Send, ImagePlus, GitMerge } from "lucide-react";
import { FileArchive, Eye, ChevronRight, Layers, Ungroup, Send, ImagePlus, GitMerge, Maximize2 } from "lucide-react";
import { DataTableColumnHeader } from "@/components/shared/data-table-column-header";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import { Checkbox } from "@/components/ui/checkbox";
import { SendToTelegramButton } from "./send-to-telegram-button";
import { ImageLightbox } from "./image-lightbox";
export interface PackageRow {
id: string;
@@ -84,14 +86,30 @@ export function formatBytes(bytesStr: string): string {
}
function PreviewCell({ pkg }: { pkg: PackageRow }) {
const [lightboxOpen, setLightboxOpen] = useState(false);
if (pkg.hasPreview) {
const src = `/api/zips/${pkg.id}/preview`;
return (
<img
src={`/api/zips/${pkg.id}/preview`}
alt=""
className="h-9 w-9 rounded-md object-cover bg-muted"
loading="lazy"
/>
<>
<button
type="button"
className="group/preview relative h-9 w-9 overflow-hidden rounded-md bg-muted"
onClick={() => setLightboxOpen(true)}
title="Click to enlarge"
>
<img
src={src}
alt=""
className="h-9 w-9 object-cover"
loading="lazy"
/>
<div className="absolute inset-0 flex items-center justify-center rounded-md bg-black/50 opacity-0 transition-opacity group-hover/preview:opacity-100">
<Maximize2 className="h-3.5 w-3.5 text-white" />
</div>
</button>
<ImageLightbox src={src} open={lightboxOpen} onOpenChange={setLightboxOpen} />
</>
);
}
return (

View File

@@ -13,6 +13,7 @@ import {
Upload,
ImagePlus,
Images,
Maximize2,
} from "lucide-react";
import {
Dialog,
@@ -30,6 +31,7 @@ import type { PackageRow } from "./package-columns";
import { SendToTelegramButton } from "./send-to-telegram-button";
import { uploadPackagePreview } from "../actions";
import { ArchivePreviewPicker } from "./archive-preview-picker";
import { ImageLightbox } from "./image-lightbox";
interface FileItem {
id: string;
@@ -264,6 +266,7 @@ export function PackageFilesDrawer({ pkg, open, onOpenChange, highlightTerm }: P
const [uploading, setUploading] = useState(false);
const [localPreviewUrl, setLocalPreviewUrl] = useState<string | null>(null);
const [showPreviewPicker, setShowPreviewPicker] = useState(false);
const [previewLightboxOpen, setPreviewLightboxOpen] = useState(false);
const fileInputRef = useRef<HTMLInputElement>(null);
const handlePreviewUpload = useCallback(
@@ -384,9 +387,8 @@ export function PackageFilesDrawer({ pkg, open, onOpenChange, highlightTerm }: P
<button
type="button"
className="relative group h-20 w-20 shrink-0 rounded-lg overflow-hidden bg-muted"
onClick={() => fileInputRef.current?.click()}
disabled={uploading}
title="Click to replace preview image"
onClick={() => setPreviewLightboxOpen(true)}
title="Click to enlarge"
>
<img
src={localPreviewUrl ?? `/api/zips/${pkg!.id}/preview`}
@@ -394,11 +396,7 @@ export function PackageFilesDrawer({ pkg, open, onOpenChange, highlightTerm }: P
className="h-full w-full object-cover"
/>
<div className="absolute inset-0 bg-black/50 opacity-0 group-hover:opacity-100 transition-opacity flex items-center justify-center">
{uploading ? (
<Loader2 className="h-5 w-5 text-white animate-spin" />
) : (
<Upload className="h-5 w-5 text-white" />
)}
<Maximize2 className="h-5 w-5 text-white" />
</div>
</button>
) : (
@@ -582,6 +580,11 @@ export function PackageFilesDrawer({ pkg, open, onOpenChange, highlightTerm }: P
}}
/>
)}
<ImageLightbox
src={pkg ? (localPreviewUrl ?? `/api/zips/${pkg.id}/preview`) : null}
open={previewLightboxOpen}
onOpenChange={setPreviewLightboxOpen}
/>
</>
);
}

View File

@@ -3,7 +3,7 @@
import { useState, useCallback, useTransition, useMemo, useRef } from "react";
import { useRouter, usePathname, useSearchParams } from "next/navigation";
import { toast } from "sonner";
import { Search, Layers, Upload } from "lucide-react";
import { Search, Layers, Upload, Send } from "lucide-react";
import { UploadDialog } from "./upload-dialog";
import { useDataTable } from "@/hooks/use-data-table";
import {
@@ -16,6 +16,7 @@ import {
import { PackageFilesDrawer } from "./package-files-drawer";
import { IngestionStatus } from "./ingestion-status";
import { SkippedPackagesTab } from "./skipped-packages-tab";
import { CreatorFilter } from "./creator-filter";
import { DataTable } from "@/components/shared/data-table";
import { DataTablePagination } from "@/components/shared/data-table-pagination";
import { DataTableViewOptions } from "@/components/shared/data-table-view-options";
@@ -49,6 +50,7 @@ import {
createGroupAction,
removeFromGroupAction,
sendAllInGroupAction,
sendAllFromCreatorAction,
updateGroupPreviewAction,
mergeGroupsAction,
} from "../actions";
@@ -59,6 +61,7 @@ interface StlTableProps {
totalCount: number;
ingestionStatus: IngestionAccountStatus[];
availableTags: string[];
availableCreators: string[];
searchTerm: string;
skippedData: SkippedRow[];
skippedPageCount: number;
@@ -74,6 +77,7 @@ export function StlTable({
totalCount,
ingestionStatus,
availableTags,
availableCreators,
searchTerm,
skippedData,
skippedPageCount,
@@ -85,6 +89,7 @@ export function StlTable({
const router = useRouter();
const pathname = usePathname();
const searchParams = useSearchParams();
const activeCreator = searchParams.get("creator") ?? "";
const [searchValue, setSearchValue] = useState(searchParams.get("search") ?? "");
const [viewPkg, setViewPkg] = useState<PackageRow | null>(null);
@@ -207,6 +212,20 @@ export function StlTable({
[router, pathname, searchParams]
);
const updateCreatorFilter = useCallback(
(value: string) => {
const params = new URLSearchParams(searchParams.toString());
if (value) {
params.set("creator", value);
params.set("page", "1");
} else {
params.delete("creator");
}
router.push(`${pathname}?${params.toString()}`, { scroll: false });
},
[router, pathname, searchParams]
);
const activeTab = searchParams.get("tab") ?? "packages";
const updateTab = useCallback(
@@ -277,6 +296,23 @@ export function StlTable({
[router]
);
const handleSendAllFromCreator = useCallback(() => {
if (!confirm(`Send all packages from "${activeCreator}" to your Telegram?`)) return;
startTransition(async () => {
const result = await sendAllFromCreatorAction(activeCreator);
if (result.success) {
const { queued, skipped } = result.data;
toast.success(
`Queued ${queued} package${queued === 1 ? "" : "s"} from ${activeCreator}` +
(skipped ? ` (${skipped} already queued)` : "")
);
router.refresh();
} else {
toast.error(result.error);
}
});
}, [activeCreator, router]);
const handleRemoveFromGroup = useCallback(
(packageId: string) => {
startTransition(async () => {
@@ -500,11 +536,29 @@ export function StlTable({
</SelectContent>
</Select>
)}
{availableCreators.length > 0 && (
<CreatorFilter
creators={availableCreators}
value={activeCreator}
onChange={updateCreatorFilter}
/>
)}
<DataTableViewOptions table={table} />
<Button variant="outline" size="sm" className="h-9" onClick={() => setUploadOpen(true)}>
<Upload className="mr-2 h-4 w-4" />
Upload Files
</Button>
{activeCreator && (
<Button
variant="outline"
size="sm"
className="h-9 gap-1.5"
onClick={handleSendAllFromCreator}
>
<Send className="h-3.5 w-3.5" />
Send all from {activeCreator}
</Button>
)}
{selectedPackages.size >= 2 && (
<Button
variant="outline"

View File

@@ -589,3 +589,82 @@ export async function sendAllInGroupAction(
return { success: false, error: "Failed to send group packages" };
}
}
export async function sendAllFromCreatorAction(
creatorName: string
): Promise<ActionResult<{ queued: number; skipped: number }>> {
const session = await auth();
if (!session?.user?.id) return { success: false, error: "Unauthorized" };
const creator = creatorName.trim();
if (!creator) {
return { success: false, error: "No creator specified" };
}
try {
const telegramLink = await prisma.telegramLink.findUnique({
where: { userId: session.user.id },
});
if (!telegramLink) {
return { success: false, error: "No linked Telegram account. Link one in Settings." };
}
const sendablePackages = await prisma.package.findMany({
where: {
creator,
destChannelId: { not: null },
destMessageId: { not: null },
},
select: { id: true },
});
if (sendablePackages.length === 0) {
return { success: false, error: "No uploaded packages found for this creator" };
}
let queued = 0;
let skipped = 0;
for (const pkg of sendablePackages) {
// Only create if no existing PENDING/SENDING request for this package+link combo
const existing = await prisma.botSendRequest.findFirst({
where: {
packageId: pkg.id,
telegramLinkId: telegramLink.id,
status: { in: ["PENDING", "SENDING"] },
},
});
if (existing) {
skipped++;
continue;
}
const sendRequest = await prisma.botSendRequest.create({
data: {
packageId: pkg.id,
telegramLinkId: telegramLink.id,
requestedByUserId: session.user.id,
status: "PENDING",
},
});
// Notify the bot via pg_notify
try {
await prisma.$queryRawUnsafe(
`SELECT pg_notify('bot_send', $1)`,
sendRequest.id
);
} catch {
// Best-effort — the bot also polls periodically
}
queued++;
}
revalidatePath("/stls");
return { success: true, data: { queued, skipped } };
} catch {
return { success: false, error: "Failed to send creator packages" };
}
}

View File

@@ -1,6 +1,6 @@
import { auth } from "@/lib/auth";
import { redirect } from "next/navigation";
import { listDisplayItems, searchPackages, getIngestionStatus, getAllPackageTags, listSkippedPackages, countSkippedPackages, listUngroupedPackages, countUngroupedPackages } from "@/lib/telegram/queries";
import { listDisplayItems, searchPackages, getIngestionStatus, getAllPackageTags, getAllPackageCreators, listSkippedPackages, countSkippedPackages, listUngroupedPackages, countUngroupedPackages } from "@/lib/telegram/queries";
import { StlTable } from "./_components/stl-table";
import type { DisplayItem, PackageListItem } from "@/lib/telegram/types";
@@ -24,7 +24,7 @@ export default async function StlFilesPage({ searchParams }: Props) {
const tab = (params.tab as string) ?? "packages";
// Fetch packages, ingestion status, tags, and skipped count in parallel
const [result, ingestionStatus, availableTags, skippedCount, ungroupedCount] = await Promise.all([
const [result, ingestionStatus, availableTags, availableCreators, skippedCount, ungroupedCount] = await Promise.all([
search
? searchPackages({
query: search,
@@ -42,6 +42,7 @@ export default async function StlFilesPage({ searchParams }: Props) {
}),
getIngestionStatus(),
getAllPackageTags(),
getAllPackageCreators(),
countSkippedPackages(),
countUngroupedPackages(),
]);
@@ -68,6 +69,7 @@ export default async function StlFilesPage({ searchParams }: Props) {
totalCount={result.pagination.total}
ingestionStatus={ingestionStatus}
availableTags={availableTags}
availableCreators={availableCreators}
searchTerm={search}
skippedData={skippedResult?.items ?? []}
skippedPageCount={skippedResult?.pagination.totalPages ?? 0}

View File

@@ -9,6 +9,7 @@ import {
ArrowUpFromLine,
RefreshCcw,
Tag,
MessagesSquare,
} from "lucide-react";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
@@ -27,6 +28,7 @@ interface ChannelColumnsProps {
onSetType: (id: string, type: "SOURCE" | "DESTINATION") => void;
onRescan: (id: string) => void;
onSetCategory: (id: string, category: string | null) => void;
onManageTopics: (id: string) => void;
}
export function getChannelColumns({
@@ -35,6 +37,7 @@ export function getChannelColumns({
onSetType,
onRescan,
onSetCategory,
onManageTopics,
}: ChannelColumnsProps): ColumnDef<ChannelRow, unknown>[] {
return [
{
@@ -147,6 +150,14 @@ export function getChannelColumns({
Rescan Channel
</DropdownMenuItem>
)}
{row.original.type === "SOURCE" && row.original.isForum && (
<DropdownMenuItem
onClick={() => onManageTopics(row.original.id)}
>
<MessagesSquare className="mr-2 h-3.5 w-3.5" />
Topics
</DropdownMenuItem>
)}
<DropdownMenuItem
onClick={() => {
const cat = prompt("Enter category (e.g. STL, PDF, D&D, Cosplay):", row.original.category ?? "");

View File

@@ -7,6 +7,7 @@ import { getChannelColumns } from "./channel-columns";
import { DestinationCard } from "./destination-card";
import { ChannelPickerDialog } from "./channel-picker-dialog";
import { JoinChannelDialog } from "./join-channel-dialog";
import { TopicsDrawer } from "./topics-drawer";
import {
deleteChannel,
toggleChannelActive,
@@ -32,6 +33,7 @@ export function ChannelsTab({ channels, globalDestination, accounts }: ChannelsT
const [rescanId, setRescanId] = useState<string | null>(null);
const [fetchChannelsAccountId, setFetchChannelsAccountId] = useState<string | null>(null);
const [joinDialogOpen, setJoinDialogOpen] = useState(false);
const [topicsChannelId, setTopicsChannelId] = useState<string | null>(null);
// Find the first authenticated account for "Fetch Channels"
const authenticatedAccounts = accounts.filter((a) => a.authState === "AUTHENTICATED" && a.isActive);
@@ -60,6 +62,7 @@ export function ChannelsTab({ channels, globalDestination, accounts }: ChannelsT
else toast.error(result.error);
});
},
onManageTopics: (id) => setTopicsChannelId(id),
});
const { table } = useDataTable({
@@ -167,6 +170,17 @@ export function ChannelsTab({ channels, globalDestination, accounts }: ChannelsT
open={joinDialogOpen}
onOpenChange={setJoinDialogOpen}
/>
<TopicsDrawer
channelId={topicsChannelId}
channelTitle={
channels.find((c) => c.id === topicsChannelId)?.title
}
open={!!topicsChannelId}
onOpenChange={(open) => {
if (!open) setTopicsChannelId(null);
}}
/>
</div>
);
}

View File

@@ -0,0 +1,182 @@
"use client";
import { useState, useEffect, useCallback, useMemo, useTransition } from "react";
import { toast } from "sonner";
import { Loader2, Search } from "lucide-react";
import {
Dialog,
DialogContent,
DialogDescription,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
import { Input } from "@/components/ui/input";
import { Switch } from "@/components/ui/switch";
import { setTopicFetchEnabled } from "../actions";
interface TopicRow {
id: string;
topicId: string;
topicName: string | null;
fetchEnabled: boolean;
lastScannedAt: string | null;
}
interface TopicsDrawerProps {
channelId: string | null;
channelTitle?: string;
open: boolean;
onOpenChange: (open: boolean) => void;
}
function topicLabel(t: TopicRow): string {
return t.topicName ?? `Topic ${t.topicId}`;
}
export function TopicsDrawer({
channelId,
channelTitle,
open,
onOpenChange,
}: TopicsDrawerProps) {
const [topics, setTopics] = useState<TopicRow[]>([]);
const [loading, setLoading] = useState(false);
const [filter, setFilter] = useState("");
const [, startTransition] = useTransition();
const [pendingId, setPendingId] = useState<string | null>(null);
const fetchTopics = useCallback(async () => {
if (!channelId) return;
setLoading(true);
try {
const res = await fetch(`/api/telegram/channels/${channelId}/topics`);
if (res.ok) setTopics(await res.json());
else toast.error("Failed to load topics");
} catch {
toast.error("Failed to load topics");
}
setLoading(false);
}, [channelId]);
useEffect(() => {
if (open && channelId) fetchTopics();
}, [open, channelId, fetchTopics]);
const handleOpenChange = (next: boolean) => {
// Reset the filter on close so the next open starts clean (the drawer is
// opened per-channel, so close-then-open is the normal channel switch).
if (!next) setFilter("");
onOpenChange(next);
};
const handleToggle = (topic: TopicRow, enabled: boolean) => {
// Optimistic update
setTopics((prev) =>
prev.map((t) => (t.id === topic.id ? { ...t, fetchEnabled: enabled } : t))
);
setPendingId(topic.id);
startTransition(async () => {
const result = await setTopicFetchEnabled(topic.id, enabled);
if (!result.success) {
toast.error(result.error ?? "Failed to update topic");
// Revert on failure
setTopics((prev) =>
prev.map((t) =>
t.id === topic.id ? { ...t, fetchEnabled: !enabled } : t
)
);
}
setPendingId(null);
});
};
const filtered = useMemo(() => {
const q = filter.trim().toLowerCase();
if (!q) return topics;
return topics.filter((t) => topicLabel(t).toLowerCase().includes(q));
}, [topics, filter]);
return (
<Dialog open={open} onOpenChange={handleOpenChange}>
<DialogContent className="sm:max-w-lg max-h-[80vh] flex flex-col gap-0 p-0">
<DialogHeader className="px-6 pt-6 pb-4 border-b border-border space-y-3">
<div className="space-y-1">
<DialogTitle className="truncate pr-8">
Topics{channelTitle ? `: ${channelTitle}` : ""}
</DialogTitle>
<DialogDescription>
Enabled topics are scanned and their files fetched. Disable a topic
to stop fetching new files from it already-fetched files are kept.
</DialogDescription>
</div>
{topics.length > 0 && (
<div className="relative">
<Search className="absolute left-2.5 top-2.5 h-4 w-4 text-muted-foreground" />
<Input
placeholder="Filter topics..."
value={filter}
onChange={(e) => setFilter(e.target.value)}
className="pl-9 h-9"
/>
</div>
)}
</DialogHeader>
{/* Native overflow scroll: the Radix ScrollArea viewport does not get a
bounded height inside this flex-column, max-h, vertically-centred
dialog, so the list overflowed instead of scrolling. flex-1 +
min-h-0 + overflow-y-auto is the canonical, touch-friendly fix. */}
<div className="flex-1 min-h-0 overflow-y-auto">
<div className="px-6 py-4 space-y-2">
{loading ? (
<div className="flex items-center justify-center gap-2 py-12">
<Loader2 className="h-5 w-5 animate-spin text-muted-foreground" />
<span className="text-sm text-muted-foreground">
Loading topics...
</span>
</div>
) : topics.length === 0 ? (
<p className="py-12 text-center text-sm text-muted-foreground">
No topics discovered yet they&apos;ll appear here after the next
scan.
</p>
) : filtered.length === 0 ? (
<p className="py-12 text-center text-sm text-muted-foreground">
No topics match &quot;{filter}&quot;.
</p>
) : (
filtered.map((topic) => (
<div
key={topic.id}
className="flex items-center justify-between gap-3 rounded-md border p-3"
>
<div className="min-w-0">
<p className="truncate text-sm font-medium">
{topicLabel(topic)}
</p>
<p className="text-xs text-muted-foreground">
{topic.fetchEnabled
? "Fetching enabled"
: "Fetching disabled"}
{topic.lastScannedAt
? ` · last scanned ${new Date(
topic.lastScannedAt
).toLocaleDateString()}`
: " · not scanned yet"}
</p>
</div>
<Switch
checked={topic.fetchEnabled}
disabled={pendingId === topic.id}
onCheckedChange={(checked) => handleToggle(topic, checked)}
aria-label={`Toggle fetching for ${topicLabel(topic)}`}
/>
</div>
))
)}
</div>
</div>
</DialogContent>
</Dialog>
);
}

View File

@@ -9,13 +9,14 @@ import {
Radio,
AlertTriangle,
RefreshCw,
SkipForward,
} from "lucide-react";
import { Card, CardContent } from "@/components/ui/card";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import { cn } from "@/lib/utils";
import { toast } from "sonner";
import { triggerIngestion } from "../actions";
import { triggerIngestion, disableActiveTopic } from "../actions";
import type { IngestionAccountStatus } from "@/lib/telegram/types";
interface WorkerStatusPanelProps {
@@ -218,6 +219,24 @@ function RunningStatus({
}: {
run: NonNullable<IngestionAccountStatus["currentRun"]>;
}) {
const [isDisabling, startDisable] = useTransition();
const handleSkipTopic = () => {
const acmId = run.currentAccountChannelMapId;
const topicId = run.currentTopicId;
if (!acmId || !topicId) return;
startDisable(async () => {
const result = await disableActiveTopic(acmId, topicId);
if (result.success) {
toast.success(
"Topic disabled — the current file finishes, then the rest is skipped"
);
} else {
toast.error(result.error ?? "Failed to disable topic");
}
});
};
return (
<div className="space-y-2">
<div className="flex items-center gap-2">
@@ -273,6 +292,27 @@ function RunningStatus({
</span>
)}
</div>
{/* Skip & disable the topic currently being processed */}
{run.currentTopicId && run.currentAccountChannelMapId && (
<div className="pl-6 pt-1">
<Button
variant="outline"
size="sm"
className="h-7 gap-1.5 text-xs"
onClick={handleSkipTopic}
disabled={isDisabling}
title="Finish the current file, skip the rest of this topic, and don't fetch it in future runs"
>
{isDisabling ? (
<Loader2 className="h-3 w-3 animate-spin" />
) : (
<SkipForward className="h-3 w-3" />
)}
Skip &amp; disable this topic
</Button>
</div>
)}
</div>
);
}

View File

@@ -436,6 +436,80 @@ export async function rescanChannel(channelId: string): Promise<ActionResult> {
}
}
// ── Topic fetch toggle ──
export async function setTopicFetchEnabled(
topicProgressId: string,
enabled: boolean
): Promise<ActionResult> {
const admin = await requireAdmin();
if (!admin.success) return admin;
const existing = await prisma.topicProgress.findUnique({
where: { id: topicProgressId },
});
if (!existing) return { success: false, error: "Topic not found" };
try {
await prisma.topicProgress.update({
where: { id: topicProgressId },
data: { fetchEnabled: enabled },
});
revalidatePath(REVALIDATE_PATH);
return { success: true, data: undefined };
} catch {
return { success: false, error: "Failed to update topic" };
}
}
/**
* Disable the topic currently being processed by the worker, identified by its
* account-channel mapping + Telegram topic id (as exposed on the live run
* status). Upserts so a disabled row exists even if the worker hasn't persisted
* the topic yet. The worker honours this live: it finishes the in-flight file
* then skips the rest of the topic, and future runs skip it entirely.
*/
export async function disableActiveTopic(
accountChannelMapId: string,
topicId: string
): Promise<ActionResult> {
const admin = await requireAdmin();
if (!admin.success) return admin;
let topicIdBig: bigint;
try {
topicIdBig = BigInt(topicId);
} catch {
return { success: false, error: "Invalid topic id" };
}
// BigInt("") / BigInt("0") don't throw — reject non-positive ids explicitly
// since this action is exported and admin-callable outside the UI button.
if (topicIdBig <= BigInt(0)) {
return { success: false, error: "Invalid topic id" };
}
try {
await prisma.topicProgress.upsert({
where: {
accountChannelMapId_topicId: {
accountChannelMapId,
topicId: topicIdBig,
},
},
create: {
accountChannelMapId,
topicId: topicIdBig,
fetchEnabled: false,
},
update: { fetchEnabled: false },
});
revalidatePath(REVALIDATE_PATH);
return { success: true, data: undefined };
} catch {
return { success: false, error: "Failed to disable topic" };
}
}
// ── Account-Channel link actions ──
export async function linkChannel(

View File

@@ -0,0 +1,17 @@
import { NextResponse } from "next/server";
import { authenticateApiRequest } from "@/lib/telegram/api-auth";
import { listChannelTopics } from "@/lib/telegram/admin-queries";
export const dynamic = "force-dynamic";
export async function GET(
request: Request,
{ params }: { params: Promise<{ channelId: string }> }
) {
const authResult = await authenticateApiRequest(request, true);
if ("error" in authResult) return authResult.error;
const { channelId } = await params;
const topics = await listChannelTopics(channelId);
return NextResponse.json(topics);
}

View File

@@ -41,6 +41,7 @@ export async function listChannels() {
telegramId: c.telegramId.toString(),
title: c.title,
type: c.type,
isForum: c.isForum,
isActive: c.isActive,
category: c.category,
createdAt: c.createdAt.toISOString(),
@@ -140,3 +141,24 @@ export async function getUnlinkedChannels(accountId: string) {
telegramId: c.telegramId.toString(),
}));
}
// ── Channel topic queries ──
export async function listChannelTopics(channelId: string) {
const rows = await prisma.topicProgress.findMany({
where: { accountChannelMap: { channelId } },
orderBy: [{ fetchEnabled: "desc" }, { topicName: "asc" }],
});
return rows.map((r) => ({
id: r.id,
topicId: r.topicId.toString(),
topicName: r.topicName,
fetchEnabled: r.fetchEnabled,
lastScannedAt: r.lastScannedAt?.toISOString() ?? null,
}));
}
export type ChannelTopicRow = Awaited<
ReturnType<typeof listChannelTopics>
>[number];

View File

@@ -9,6 +9,28 @@ import type {
PackageGroupRow,
} from "./types";
/**
* Returns the subset of the given IDs whose `previewData` column is non-null,
* WITHOUT transferring the (large) image bytes.
*
* List views only need a `hasPreview` boolean per row. Selecting `previewData`
* directly pulls every JPEG blob (avg ~700 KB, up to 2 MB) into the Node heap
* just to compare it against null — under concurrent requests this exhausts the
* container memory limit and crashes the process. This keeps the check in SQL.
*/
async function fetchPreviewFlags(
table: "packages" | "package_groups",
ids: string[]
): Promise<Set<string>> {
if (ids.length === 0) return new Set();
const placeholders = ids.map((_, i) => `$${i + 1}`).join(", ");
const rows = await prisma.$queryRawUnsafe<{ id: string }[]>(
`SELECT id FROM ${table} WHERE "previewData" IS NOT NULL AND id IN (${placeholders})`,
...ids
);
return new Set(rows.map((r) => r.id));
}
export async function listPackages(options: {
page: number;
limit: number;
@@ -40,13 +62,17 @@ export async function listPackages(options: {
indexedAt: true,
creator: true,
tags: true,
previewData: true, // check actual image data, not previewMsgId proxy
sourceChannel: { select: { id: true, title: true } },
},
}),
prisma.package.count({ where }),
]);
const previewIds = await fetchPreviewFlags(
"packages",
items.map((p) => p.id)
);
const mapped: PackageListItem[] = items.map((pkg) => ({
id: pkg.id,
fileName: pkg.fileName,
@@ -55,7 +81,7 @@ export async function listPackages(options: {
archiveType: pkg.archiveType,
fileCount: pkg.fileCount,
isMultipart: pkg.isMultipart,
hasPreview: pkg.previewData !== null,
hasPreview: previewIds.has(pkg.id),
creator: pkg.creator,
tags: pkg.tags,
indexedAt: pkg.indexedAt.toISOString(),
@@ -149,7 +175,7 @@ export async function listDisplayItems(options: {
select: {
id: true, fileName: true, fileSize: true, contentHash: true,
archiveType: true, fileCount: true, isMultipart: true,
indexedAt: true, creator: true, tags: true, previewData: true,
indexedAt: true, creator: true, tags: true,
sourceChannel: { select: { id: true, title: true } },
},
})
@@ -159,13 +185,13 @@ export async function listDisplayItems(options: {
? await prisma.packageGroup.findMany({
where: { id: { in: groupIds } },
select: {
id: true, name: true, previewData: true,
id: true, name: true,
sourceChannel: { select: { id: true, title: true } },
packages: {
select: {
id: true, fileName: true, fileSize: true, contentHash: true,
archiveType: true, fileCount: true, isMultipart: true,
indexedAt: true, creator: true, tags: true, previewData: true,
indexedAt: true, creator: true, tags: true,
sourceChannel: { select: { id: true, title: true } },
},
orderBy: { indexedAt: "desc" },
@@ -174,6 +200,16 @@ export async function listDisplayItems(options: {
})
: [];
// Compute hasPreview flags without transferring image bytes (see fetchPreviewFlags)
const allPackageIds = [
...standalonePackages.map((p) => p.id),
...groups.flatMap((g) => g.packages.map((p) => p.id)),
];
const [packagePreviewIds, groupPreviewIds] = await Promise.all([
fetchPreviewFlags("packages", allPackageIds),
fetchPreviewFlags("package_groups", groups.map((g) => g.id)),
]);
// Build DisplayItem array in the original sort order
const packageMap = new Map(standalonePackages.map((p) => [p.id, p]));
const groupMap = new Map(groups.map((g) => [g.id, g]));
@@ -191,7 +227,7 @@ export async function listDisplayItems(options: {
archiveType: pkg.archiveType,
fileCount: pkg.fileCount,
isMultipart: pkg.isMultipart,
hasPreview: pkg.previewData !== null,
hasPreview: packagePreviewIds.has(pkg.id),
creator: pkg.creator,
tags: pkg.tags,
indexedAt: pkg.indexedAt.toISOString(),
@@ -209,7 +245,7 @@ export async function listDisplayItems(options: {
data: {
id: grp.id,
name: grp.name,
hasPreview: grp.previewData !== null,
hasPreview: groupPreviewIds.has(grp.id),
totalFileSize: grp.packages.reduce((sum, p) => sum + p.fileSize, BigInt(0)).toString(),
totalFileCount: grp.packages.reduce((sum, p) => sum + p.fileCount, 0),
packageCount: grp.packages.length,
@@ -227,7 +263,7 @@ export async function listDisplayItems(options: {
archiveType: pkg.archiveType,
fileCount: pkg.fileCount,
isMultipart: pkg.isMultipart,
hasPreview: pkg.previewData !== null,
hasPreview: packagePreviewIds.has(pkg.id),
creator: pkg.creator,
tags: pkg.tags,
indexedAt: pkg.indexedAt.toISOString(),
@@ -440,13 +476,17 @@ export async function searchPackages(options: {
indexedAt: true,
creator: true,
tags: true,
previewData: true,
sourceChannel: { select: { id: true, title: true } },
},
}),
Promise.resolve(allIds.length),
]);
const previewIds = await fetchPreviewFlags(
"packages",
items.map((p) => p.id)
);
const mapped: PackageListItem[] = items.map((pkg) => ({
id: pkg.id,
fileName: pkg.fileName,
@@ -455,7 +495,7 @@ export async function searchPackages(options: {
archiveType: pkg.archiveType,
fileCount: pkg.fileCount,
isMultipart: pkg.isMultipart,
hasPreview: pkg.previewData !== null,
hasPreview: previewIds.has(pkg.id),
creator: pkg.creator,
tags: pkg.tags,
indexedAt: pkg.indexedAt.toISOString(),
@@ -494,6 +534,15 @@ export async function getAllPackageTags(): Promise<string[]> {
return result.map((r) => r.tag);
}
export async function getAllPackageCreators(): Promise<string[]> {
const result = await prisma.$queryRaw<{ creator: string }[]>`
SELECT DISTINCT creator FROM packages
WHERE creator IS NOT NULL AND creator <> ''
ORDER BY creator
`;
return result.map((r) => r.creator);
}
export async function getIngestionStatus(): Promise<IngestionAccountStatus[]> {
const accounts = await prisma.telegramAccount.findMany({
orderBy: { createdAt: "asc" },
@@ -550,6 +599,8 @@ export async function getIngestionStatus(): Promise<IngestionAccountStatus[]> {
totalBytes: currentRun.totalBytes?.toString() ?? null,
downloadPercent: currentRun.downloadPercent,
lastActivityAt: currentRun.lastActivityAt?.toISOString() ?? null,
currentTopicId: currentRun.currentTopicId?.toString() ?? null,
currentAccountChannelMapId: currentRun.currentAccountChannelMapId,
}
: null,
});
@@ -634,13 +685,17 @@ export async function listUngroupedPackages(options: {
partCount: true,
tags: true,
indexedAt: true,
previewData: true,
sourceChannel: { select: { id: true, title: true } },
},
}),
prisma.package.count({ where }),
]);
const previewIds = await fetchPreviewFlags(
"packages",
items.map((p) => p.id)
);
return {
items: items.map((p) => ({
id: p.id,
@@ -654,7 +709,7 @@ export async function listUngroupedPackages(options: {
partCount: p.partCount,
tags: p.tags,
indexedAt: p.indexedAt.toISOString(),
hasPreview: !!p.previewData,
hasPreview: previewIds.has(p.id),
sourceChannel: p.sourceChannel,
matchedFileCount: 0,
matchedByContent: false,

View File

@@ -123,5 +123,7 @@ export interface IngestionAccountStatus {
totalBytes: string | null; // BigInt serialized as string
downloadPercent: number | null;
lastActivityAt: string | null;
currentTopicId: string | null; // BigInt serialized as string
currentAccountChannelMapId: string | null;
} | null;
}

View File

@@ -5,21 +5,28 @@ import { childLogger } from "../util/logger.js";
const execFileAsync = promisify(execFile);
const log = childLogger("integrity");
export type IntegrityFailureKind = "encrypted" | "corrupt" | "inconclusive";
export type IntegrityResult =
| { ok: true }
| { ok: false; reason: string };
| { ok: false; reason: string; kind: IntegrityFailureKind };
/**
* Test that the archive can be read end-to-end without errors, BEFORE we
* spend bandwidth uploading it to the destination channel. Catches:
* - Truncated downloads (rare given our size check, but cheap to confirm)
* - CRC errors inside the archive
* - Bad central directories
* - Encrypted archives (we report them as failures rather than upload
* a file users can't extract)
* spend bandwidth uploading it to the destination channel.
*
* Returns { ok: true } if the archive is intact. Returns
* { ok: false, reason } otherwise. Logs at warn level on failure.
* Failures are classified so the caller can react appropriately:
* - "encrypted" — password-protected; users can't extract it. Actionable.
* - "corrupt" — genuine CRC / structural error (truncated download, bad
* central directory, CRC mismatch). Actionable.
* - "inconclusive" — the test tool itself was killed (OOM) or timed out,
* typically on very large 7z archives in a memory-limited
* container (exit 137 / SIGKILL). This is a TOOL
* LIMITATION, not corruption — callers should NOT raise a
* user-facing alarm for it.
*
* Returns { ok: true } if the archive is intact, otherwise
* { ok: false, reason, kind }.
*
* For multipart archives, pass the first part. unzip / unrar / 7z all
* auto-discover sibling parts.
@@ -42,7 +49,7 @@ export async function testArchiveIntegrity(
maxBuffer: 10 * 1024 * 1024,
});
if (stderr && stderr.trim()) {
return { ok: false, reason: `unzip -t reported: ${stderr.slice(0, 500)}` };
return { ok: false, kind: "corrupt", reason: `unzip -t reported: ${stderr.slice(0, 500)}` };
}
return { ok: true };
}
@@ -58,7 +65,7 @@ export async function testArchiveIntegrity(
if (/All OK/i.test(combined)) {
return { ok: true };
}
return { ok: false, reason: `unrar t did not report "All OK": ${combined.slice(-500)}` };
return { ok: false, kind: "corrupt", reason: `unrar t did not report "All OK": ${combined.slice(-500)}` };
}
if (archiveType === "SEVEN_Z") {
@@ -70,24 +77,52 @@ export async function testArchiveIntegrity(
if (/Everything is Ok/i.test(combined)) {
return { ok: true };
}
return { ok: false, reason: `7z t did not report "Everything is Ok": ${combined.slice(-500)}` };
return { ok: false, kind: "corrupt", reason: `7z t did not report "Everything is Ok": ${combined.slice(-500)}` };
}
return { ok: false, reason: `Unknown archive type: ${archiveType}` };
return { ok: false, kind: "corrupt", reason: `Unknown archive type: ${archiveType}` };
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
// execFile throws on non-zero exit. Try to extract the most useful part.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const stderr = (err as any)?.stderr as string | undefined;
// execFile throws on non-zero exit, on timeout, and when killed by a signal.
const e = err as {
stderr?: unknown;
stdout?: unknown;
signal?: string | null;
killed?: boolean;
code?: number | string | null;
};
const stderr = typeof e.stderr === "string" ? e.stderr : "";
const stdout = typeof e.stdout === "string" ? e.stdout : "";
const detail = stderr ? `: ${stderr.slice(0, 500)}` : "";
const haystack = `${msg}\n${stdout}\n${stderr}`;
// Specifically flag encrypted archives so the caller can record a more
// specific SkipReason / notification.
if (/password|encrypted|need.*password/i.test(`${msg}${detail}`)) {
return { ok: false, reason: `Archive is encrypted (password protected): ${msg}${detail}` };
// Encrypted archives — users can't extract them, so flag clearly.
if (/password|encrypted|wrong password|enter password/i.test(haystack)) {
return { ok: false, kind: "encrypted", reason: `Archive is encrypted (password protected): ${msg}${detail}` };
}
// Inconclusive — the test tool was killed or timed out rather than
// reporting corruption. Common on large 7z in memory-limited containers,
// where `7z t` gets OOM-killed (SIGKILL / exit 137) mid-decompression.
// That's a tool limitation, not a corrupt archive.
const killedBySignal = e.signal === "SIGKILL" || e.signal === "SIGTERM";
const killedExitCode = e.code === 137 || e.code === 143; // 128 + SIGKILL/SIGTERM
const timedOut = e.killed === true;
const maxBufferExceeded = e.code === "ERR_CHILD_PROCESS_STDIO_MAXBUFFER";
if (killedBySignal || killedExitCode || timedOut || maxBufferExceeded) {
log.debug(
{ err, archiveType, firstPartPath, signal: e.signal, code: e.code, killed: e.killed },
"Archive integrity test inconclusive (tool killed or timed out)"
);
return {
ok: false,
kind: "inconclusive",
reason: `Integrity test could not complete (tool killed or timed out — likely OOM on a large archive): ${msg}`,
};
}
// Genuine failure: a real non-zero exit with error output.
log.debug({ err, archiveType, firstPartPath }, "Archive integrity test failed");
return { ok: false, reason: `Integrity test failed: ${msg}${detail}` };
return { ok: false, kind: "corrupt", reason: `Integrity test failed: ${msg}${detail}` };
}
}

View File

@@ -369,6 +369,8 @@ export interface ActivityUpdate {
downloadedBytes?: bigint | null;
totalBytes?: bigint | null;
downloadPercent?: number | null;
currentTopicId?: bigint | null;
currentAccountChannelMapId?: string | null;
messagesScanned?: number;
zipsFound?: number;
zipsDuplicate?: number;
@@ -396,6 +398,10 @@ export async function updateRunActivity(
...(activity.zipsFound !== undefined && { zipsFound: activity.zipsFound }),
...(activity.zipsDuplicate !== undefined && { zipsDuplicate: activity.zipsDuplicate }),
...(activity.zipsIngested !== undefined && { zipsIngested: activity.zipsIngested }),
...(activity.currentTopicId !== undefined && { currentTopicId: activity.currentTopicId }),
...(activity.currentAccountChannelMapId !== undefined && {
currentAccountChannelMapId: activity.currentAccountChannelMapId,
}),
},
});
}
@@ -410,6 +416,8 @@ const CLEAR_ACTIVITY = {
downloadedBytes: null,
totalBytes: null,
downloadPercent: null,
currentTopicId: null,
currentAccountChannelMapId: null,
lastActivityAt: new Date(),
};
@@ -659,6 +667,45 @@ export async function upsertTopicProgress(
});
}
/**
* Ensure a TopicProgress row exists for every discovered topic so the admin
* UI can list and toggle them — including brand-new topics. Inserts missing
* rows only (fetchEnabled defaults to true); never overwrites watermarks,
* scan-state, or the user's fetchEnabled choice on rows that already exist.
*/
export async function ensureTopicProgressRows(
mappingId: string,
topics: { topicId: bigint; name: string | null }[]
): Promise<void> {
if (topics.length === 0) return;
await db.topicProgress.createMany({
data: topics.map((t) => ({
accountChannelMapId: mappingId,
topicId: t.topicId,
topicName: t.name,
})),
skipDuplicates: true,
});
}
/**
* Read the CURRENT per-topic fetch flag straight from the DB. Returns true
* when no row exists yet (default = enabled). Called fresh inside the topic
* loop so a mid-run toggle is honoured for topics not yet started.
*/
export async function isTopicFetchEnabled(
mappingId: string,
topicId: bigint
): Promise<boolean> {
const row = await db.topicProgress.findUnique({
where: {
accountChannelMapId_topicId: { accountChannelMapId: mappingId, topicId },
},
select: { fetchEnabled: true },
});
return row?.fetchEnabled ?? true;
}
// ── Channel fetch requests (DB-mediated communication with web app) ──
export async function getChannelFetchRequest(requestId: string) {

View File

@@ -37,6 +37,8 @@ import {
updatePackageTopicContext,
upsertChannelScanState,
upsertTopicScanState,
ensureTopicProgressRows,
isTopicFetchEnabled,
} from "./db/queries.js";
import type { ActivityUpdate } from "./db/queries.js";
import { createTdlibClient, closeTdlibClient } from "./tdlib/client.js";
@@ -536,6 +538,8 @@ export async function runWorkerForAccount(
currentActivity: `Enumerating topics in "${channelLabel}"`,
currentStep: "scanning",
currentChannel: channelLabel,
currentTopicId: null,
currentAccountChannelMapId: null,
currentFile: null,
currentFileNum: null,
totalFiles: null,
@@ -548,6 +552,15 @@ export async function runWorkerForAccount(
const rawTopics = await getForumTopicList(client, channel.telegramId);
const topicProgressList = await getTopicProgress(mapping.id);
// Persist a TopicProgress row for every discovered topic so the
// admin UI can list and toggle them — including brand-new topics.
// Inserts missing rows only; existing watermarks / scan-state /
// fetchEnabled choices are left untouched.
await ensureTopicProgressRows(
mapping.id,
rawTopics.map((t) => ({ topicId: t.topicId, name: t.name }))
);
// Process more-specific topics BEFORE "General" so the first
// encounter of any file is in its most specific context. This makes
// newly-created Packages carry useful topic info (e.g., a campaign
@@ -567,6 +580,18 @@ export async function runWorkerForAccount(
for (let tIdx = 0; tIdx < topics.length; tIdx++) {
const topic = topics[tIdx];
try {
// ── Per-topic fetch toggle (live, mid-run honouring) ──
// Read the CURRENT enabled flag straight from the DB (not the
// run-start `topicProgressList` snapshot) so disabling a topic
// mid-run skips it for the remainder of this run.
if (!(await isTopicFetchEnabled(mapping.id, topic.topicId))) {
accountLog.info(
{ channel: channel.title, topic: topic.name },
"Topic fetch disabled by user — skipping"
);
continue;
}
let progress = topicProgressList.find(
(tp) => tp.topicId === topic.topicId
);
@@ -727,6 +752,8 @@ export async function runWorkerForAccount(
currentActivity: `Scanning "${topicLabel}"${topicProgress}`,
currentStep: "scanning",
currentChannel: channelLabel,
currentTopicId: topic.topicId,
currentAccountChannelMapId: mapping.id,
currentFile: null,
currentFileNum: null,
totalFiles: null,
@@ -807,7 +834,11 @@ export async function runWorkerForAccount(
topic.name,
messageId
);
}
},
// shouldStop: re-read the live fetch flag before each archive
// set. A mid-run "disable topic" lets the current file finish,
// then skips the rest of this topic's archives.
async () => !(await isTopicFetchEnabled(mapping.id, topic.topicId))
);
// Sync client back in case it was recreated during upload stall recovery
client = pipelineCtx.client;
@@ -899,6 +930,8 @@ export async function runWorkerForAccount(
currentActivity: `Scanning "${channelLabel}" for new archives`,
currentStep: "scanning",
currentChannel: channelLabel,
currentTopicId: null,
currentAccountChannelMapId: null,
currentFile: null,
currentFileNum: null,
totalFiles: null,
@@ -1180,7 +1213,12 @@ async function processArchiveSets(
* below any failed message ID in this scan). Used by the caller to
* advance the channel/topic watermark incrementally — otherwise a long
* scan that gets killed by worker restart loses all progress. */
onWatermarkAdvance?: (messageId: bigint) => Promise<void>
onWatermarkAdvance?: (messageId: bigint) => Promise<void>,
/** Optional cancellation check, polled before each archive set. When it
* resolves true, processing stops after the set currently in flight (that
* one completes; remaining sets in this scan are skipped). Used by the
* forum branch to honour a mid-run "disable topic". */
shouldStop?: () => Promise<boolean>
): Promise<{ maxProcessedId: bigint | null; minFailedId: bigint | null }> {
const { client, runId, channelTitle, channel, throttled, counters, accountLog } = ctx;
@@ -1258,6 +1296,16 @@ async function processArchiveSets(
const indexedPackageRefs: IndexedPackageRef[] = [];
for (let setIdx = 0; setIdx < archiveSets.length; setIdx++) {
// Cooperative cancellation: if the caller signals stop (e.g. the topic was
// disabled mid-run), skip the remaining archive sets in this scan. The set
// processed in the previous iteration has already completed.
if (shouldStop && (await shouldStop())) {
accountLog.info(
{ channel: channelTitle, processed: setIdx, total: archiveSets.length },
"Stop signal received (topic disabled) — skipping remaining archive sets in this scan"
);
break;
}
try {
const packageId = await processOneArchiveSet(
ctx,
@@ -1918,34 +1966,45 @@ async function processOneArchiveSet(
if (!isMultipartZip) {
const integrity = await testArchiveIntegrity(archType, tempPaths[0]);
if (!integrity.ok) {
// Detect encryption specifically — those won't extract for users
// even if we upload them. Surface clearly via notification but
// STILL proceed: the user can audit and decide what to do.
const isEncrypted = /encrypted/i.test(integrity.reason);
accountLog.warn(
{ fileName: archiveName, reason: integrity.reason.slice(0, 200), isEncrypted },
"Archive integrity test failed — proceeding with upload anyway (advisory check)"
);
try {
await db.systemNotification.create({
data: {
type: isEncrypted ? "UPLOAD_FAILED" : "HASH_MISMATCH",
severity: "WARNING",
title: isEncrypted
? `Archive may be encrypted: ${archiveName}`
: `Integrity test reported issues: ${archiveName}`,
message: integrity.reason.slice(0, 1000),
context: {
fileName: archiveName,
sourceChannelId: channel.id,
sourceMessageId: Number(archiveSet.parts[0].id),
archiveType: archType,
advisory: true,
if (integrity.kind === "inconclusive") {
// The test tool was killed (OOM) or timed out — typically a large
// 7z in a memory-limited container. This is a tool limitation, NOT
// corruption, so log quietly and DON'T raise a notification. The
// upload proceeds exactly as before.
accountLog.debug(
{ fileName: archiveName, reason: integrity.reason.slice(0, 200) },
"Archive integrity test inconclusive — proceeding with upload (advisory check)"
);
} else {
// Encrypted (won't extract for users) or genuinely corrupt — surface
// clearly via notification but STILL proceed: the user can audit and
// decide what to do.
const isEncrypted = integrity.kind === "encrypted";
accountLog.warn(
{ fileName: archiveName, reason: integrity.reason.slice(0, 200), kind: integrity.kind },
"Archive integrity test failed — proceeding with upload anyway (advisory check)"
);
try {
await db.systemNotification.create({
data: {
type: isEncrypted ? "UPLOAD_FAILED" : "INTEGRITY_AUDIT",
severity: "WARNING",
title: isEncrypted
? `Archive may be encrypted: ${archiveName}`
: `Integrity test reported issues: ${archiveName}`,
message: integrity.reason.slice(0, 1000),
context: {
fileName: archiveName,
sourceChannelId: channel.id,
sourceMessageId: Number(archiveSet.parts[0].id),
archiveType: archType,
advisory: true,
},
},
},
});
} catch {
// Best-effort notification
});
} catch {
// Best-effort notification
}
}
}
}