dragonsstash/docs/superpowers/plans/2026-05-26-channel-scan-skip.md

Channel-Scan Skip Optimization Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add DB-persistent "skip if recently scanned and idle" guards so the worker stops re-scanning unchanged forum topics and channels every cycle, especially on restart. For the Model Printing Emporium channel (1,086 forum topics), drops per-cycle searchChatMessages calls from ~1,086 to ~50.

Architecture: Six new persisted columns (lastScannedAt, lastScanFoundArchives, consecutiveEmptyScans) on both AccountChannelMap (for non-forum channels) and TopicProgress (for forum topics). Three config knobs. A top-of-loop skip decision tree that checks recency, backoff, and pending failures before any TDLib call. After the existing SkippedPackage retry pass runs, a cheap getChat / getForumTopicInfo call short-circuits the paginated searchChatMessages when the channel/topic has nothing new server-side. The existing failure-retry semantics (d99a506 watermark cap + 901f32f retry pass) are preserved untouched — a channel sitting on retryable SkippedPackage rows is never skipped.

Tech Stack: TypeScript 5.9, Prisma 7.4 with PostgreSQL, TDLib via tdl 8.1.0, prebuilt-tdlib 1.8.64. No test framework — verification is manual via Docker logs.

---

Spec reference

This plan implements docs/superpowers/specs/2026-05-26-channel-scan-skip-design.md. Read that for the rationale, edge-case analysis, and risk register.

---

File structure

Files modified — no new source files created:

• prisma/schema.prisma — add 3 columns to each of AccountChannelMap and TopicProgress
• prisma/migrations/20260526000000_channel_scan_state/migration.sql — NEW migration file
• worker/src/util/config.ts — add 3 env vars
• worker/src/db/queries.ts — add 2 helper functions
• worker/src/tdlib/chats.ts — add 2 helper functions for last-message lookup
• worker/src/worker.ts — wire skip checks + getChat short-circuit into both forum and non-forum branches; update end-of-scan bookkeeping

The change is logically one feature but split into 6 tasks so each commit stays small and reviewable.

---

Task 1: Schema and migration

Files:

• Modify: prisma/schema.prisma (the AccountChannelMap and TopicProgress models)

• Create: prisma/migrations/20260526000000_channel_scan_state/migration.sql

• Step 1: Add the three columns to AccountChannelMap

Open prisma/schema.prisma and find the model AccountChannelMap block. Replace the existing model with:

model AccountChannelMap {
  id                     String      @id @default(cuid())
  accountId              String
  channelId              String
  role                   ChannelRole @default(READER)
  lastProcessedMessageId BigInt?
  /// When this channel was last scanned (any reason, including skipped scans
  /// that bumped the timestamp). Used by the recency-skip guard.
  lastScannedAt          DateTime?
  /// True if the last scan found archives OR left retryable SkippedPackages
  /// pending. Tracks "this channel has work I might need to revisit" — not
  /// just "I uploaded something this cycle".
  lastScanFoundArchives  Boolean     @default(false)
  /// Number of consecutive cycles where this channel was trulyIdle (no
  /// archives, no failures, no retryables). Drives the backoff that lets
  /// cold channels skip cycles entirely.
  consecutiveEmptyScans  Int         @default(0)
  createdAt              DateTime    @default(now())

  account        TelegramAccount @relation(fields: [accountId], references: [id], onDelete: Cascade)
  channel        TelegramChannel @relation(fields: [channelId], references: [id], onDelete: Cascade)
  topicProgress  TopicProgress[]

  @@unique([accountId, channelId])
  @@index([accountId])
  @@index([channelId])
  @@map("account_channel_map")
}

• Step 2: Add the same three columns to TopicProgress

Find the model TopicProgress block and replace it with:

model TopicProgress {
  id                     String  @id @default(cuid())
  accountChannelMapId    String
  topicId                BigInt
  topicName              String?
  lastProcessedMessageId BigInt?
  /// When this topic was last scanned (any reason). Used by recency-skip.
  lastScannedAt          DateTime?
  /// True if the last scan found archives OR has retryable SkippedPackages
  /// pending for this topic. See AccountChannelMap doc for details.
  lastScanFoundArchives  Boolean   @default(false)
  /// Number of consecutive cycles where this topic was trulyIdle. Drives
  /// backoff for cold topics.
  consecutiveEmptyScans  Int       @default(0)

  accountChannelMap AccountChannelMap @relation(fields: [accountChannelMapId], references: [id], onDelete: Cascade)

  @@unique([accountChannelMapId, topicId])
  @@index([accountChannelMapId])
  @@map("topic_progress")
}

• Step 3: Create the migration file

Create the directory and SQL file:

mkdir -p prisma/migrations/20260526000000_channel_scan_state

Create prisma/migrations/20260526000000_channel_scan_state/migration.sql with this content:

-- AlterTable: per-channel scan-state columns
ALTER TABLE "account_channel_map"
  ADD COLUMN "lastScannedAt" TIMESTAMP(3),
  ADD COLUMN "lastScanFoundArchives" BOOLEAN NOT NULL DEFAULT false,
  ADD COLUMN "consecutiveEmptyScans" INTEGER NOT NULL DEFAULT 0;

-- AlterTable: per-topic scan-state columns (forum channels)
ALTER TABLE "topic_progress"
  ADD COLUMN "lastScannedAt" TIMESTAMP(3),
  ADD COLUMN "lastScanFoundArchives" BOOLEAN NOT NULL DEFAULT false,
  ADD COLUMN "consecutiveEmptyScans" INTEGER NOT NULL DEFAULT 0;

The defaults make this a metadata-only change in PostgreSQL — no table rewrite, runs in milliseconds even on production-size data.

• Step 4: Regenerate the Prisma client and type-check

Run from the project root (uses Docker because we don't have Node locally):

docker run --rm -v "$PWD:/work" -w /work node:20-bookworm-slim sh -c "npx prisma generate 2>&1 | tail -3 && cd worker && npx tsc --noEmit 2>&1 | grep -v 'npm notice'"

Expected: ✔ Generated Prisma Client (v7.4.0) ... and zero TypeScript errors. If you see "npm install" missing, run npm install in /work and /work/worker first.

• Step 5: Commit

git add prisma/schema.prisma prisma/migrations/20260526000000_channel_scan_state/
git commit -m "$(cat <<'EOF'
feat(db): add scan-state columns to AccountChannelMap + TopicProgress

Three new fields on each table:
  - lastScannedAt          — when the worker last touched this scope
  - lastScanFoundArchives  — true if last scan had archives OR pending
                             retryables; tracks "work might need revisit"
  - consecutiveEmptyScans  — counter for cold-channel backoff

Schema change only. Worker logic in follow-up commits. Migration is a
metadata-only ALTER (NOT NULL with default) so it runs in ms even on
21k+ Package rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 2: Config knobs

Files:

• Modify: worker/src/util/config.ts

• Step 1: Add the three config fields

Open worker/src/util/config.ts. Inside the config object literal, after the existing maxSkipAttempts line, add:

  /** Window in which a recent successful empty scan lets us skip the next
   *  scan entirely. Default 5 minutes. */
  skipRecentScanWindowMs: parseInt(
    process.env.WORKER_SKIP_RECENT_SCAN_WINDOW_MS ?? "300000",
    10
  ),
  /** After this many consecutive empty scans, a channel/topic enters
   *  backoff and is only scanned every Nth cycle. */
  emptyScanBackoffThreshold: parseInt(
    process.env.WORKER_EMPTY_SCAN_BACKOFF_THRESHOLD ?? "5",
    10
  ),
  /** While in backoff, scan only every Nth cycle. Default 5 = scan every
   *  fifth cycle = once every ~5 hours given the 60-min default interval. */
  emptyScanBackoffEveryNth: parseInt(
    process.env.WORKER_EMPTY_SCAN_BACKOFF_EVERY_NTH ?? "5",
    10
  ),

The file's existing as const at the end of the object remains unchanged.

• Step 2: Type-check

docker run --rm -v "$PWD:/work" -w /work/worker node:20-bookworm-slim sh -c "npx tsc --noEmit 2>&1 | grep -v 'npm notice'"

Expected: zero errors.

• Step 3: Commit

git add worker/src/util/config.ts
git commit -m "$(cat <<'EOF'
feat(config): add three scan-skip tuning env vars

  WORKER_SKIP_RECENT_SCAN_WINDOW_MS    (default 300000 = 5 min)
  WORKER_EMPTY_SCAN_BACKOFF_THRESHOLD  (default 5 cycles)
  WORKER_EMPTY_SCAN_BACKOFF_EVERY_NTH  (default 5)

All optional with safe defaults. Not yet read by any code — the worker
integration lands in follow-up commits.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 3: DB helpers for atomic scan-state writes

Files:

• Modify: worker/src/db/queries.ts

These helpers wrap the existing watermark writes so callers update all four fields (watermark + 3 new fields) in one Prisma call. That keeps the data consistent and gives us one place to compute consecutiveEmptyScans.

• Step 1: Add upsertChannelScanState and upsertTopicScanState

Open worker/src/db/queries.ts. Find the existing updateLastProcessedMessage function. Right BELOW it, add:

export interface ScanStateUpdate {
  /** New watermark to persist. Use the same value the caller would have
   *  passed to updateLastProcessedMessage / upsertTopicProgress. */
  lastProcessedMessageId: bigint | null;
  /** True if the scan found archives OR has retryable SkippedPackages
   *  pending. The caller computes this via the trulyIdle formula. */
  lastScanFoundArchives: boolean;
  /** Pre-incremented value of consecutiveEmptyScans. Caller passes:
   *    trulyIdle ? prev + 1 : 0
   *  We do the arithmetic outside the helper so the helper stays a pure
   *  setter — easier to reason about. */
  consecutiveEmptyScans: number;
}

/**
 * Atomically update an AccountChannelMap's watermark and scan-state fields.
 * Replaces the older updateLastProcessedMessage for the post-scan write.
 * Sets lastScannedAt = NOW() server-side.
 */
export async function upsertChannelScanState(
  mappingId: string,
  update: ScanStateUpdate
) {
  return db.accountChannelMap.update({
    where: { id: mappingId },
    data: {
      lastProcessedMessageId: update.lastProcessedMessageId ?? undefined,
      lastScannedAt: new Date(),
      lastScanFoundArchives: update.lastScanFoundArchives,
      consecutiveEmptyScans: update.consecutiveEmptyScans,
    },
  });
}

/**
 * Atomically upsert a TopicProgress row with the new watermark + scan-state
 * fields. Same semantics as upsertChannelScanState but for forum topics.
 */
export async function upsertTopicScanState(
  accountChannelMapId: string,
  topicId: bigint,
  topicName: string | null,
  update: ScanStateUpdate
) {
  return db.topicProgress.upsert({
    where: {
      accountChannelMapId_topicId: { accountChannelMapId, topicId },
    },
    create: {
      accountChannelMapId,
      topicId,
      topicName,
      lastProcessedMessageId: update.lastProcessedMessageId,
      lastScannedAt: new Date(),
      lastScanFoundArchives: update.lastScanFoundArchives,
      consecutiveEmptyScans: update.consecutiveEmptyScans,
    },
    update: {
      topicName,
      lastProcessedMessageId: update.lastProcessedMessageId ?? undefined,
      lastScannedAt: new Date(),
      lastScanFoundArchives: update.lastScanFoundArchives,
      consecutiveEmptyScans: update.consecutiveEmptyScans,
    },
  });
}

• Step 2: Type-check

docker run --rm -v "$PWD:/work" -w /work/worker node:20-bookworm-slim sh -c "npx tsc --noEmit 2>&1 | grep -v 'npm notice'"

Expected: zero errors. The new functions reference db.accountChannelMap and db.topicProgress which Prisma generated from the schema in Task 1, so the regen from Step 1.4 must have run.

• Step 3: Commit

git add worker/src/db/queries.ts
git commit -m "$(cat <<'EOF'
feat(db): add upsertChannelScanState / upsertTopicScanState helpers

Wraps the existing watermark write with the three new scan-state
columns from Task 1. Single transaction, sets lastScannedAt=NOW()
server-side. Caller is responsible for computing the trulyIdle bool
and the new consecutiveEmptyScans value (pre-increment vs reset).

Existing updateLastProcessedMessage / upsertTopicProgress are kept
for callers that don't need the new fields (the SkippedPackage retry
pass, which only adjusts the watermark).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 4: TDLib getChat / getForumTopicInfo helpers

Files:

• Modify: worker/src/tdlib/chats.ts

These helpers read the channel's (or topic's) server-side last-message-id from TDLib's local cache. Cheap when the chat has already been loaded — which we already do via loadChats at the top of runWorkerForAccount.

• Step 1: Add getChannelLastMessageId and getForumTopicLastMessageId

Open worker/src/tdlib/chats.ts. Append at the bottom of the file (after the existing exports):

/**
 * Return the chat's server-side last_message.id from TDLib's local cache.
 * Used by the channel-scan-skip guard to short-circuit a paginated
 * searchChatMessages when nothing has changed since our watermark.
 *
 * Returns null when the chat has no last_message (empty channel) or the
 * call fails — callers must treat null as "unknown" and run the scan.
 */
export async function getChannelLastMessageId(
  client: Client,
  chatId: bigint
): Promise<bigint | null> {
  try {
    // eslint-disable-next-line @typescript-eslint/no-explicit-any
    const chat = (await client.invoke({
      _: "getChat",
      chat_id: Number(chatId),
    })) as { last_message?: { id?: number } };
    const id = chat.last_message?.id;
    return id ? BigInt(id) : null;
  } catch (err) {
    log.debug({ err, chatId: chatId.toString() }, "getChannelLastMessageId failed");
    return null;
  }
}

/**
 * Return the forum topic's last_message_id from TDLib. Same purpose as
 * getChannelLastMessageId but scoped to a single topic in a forum
 * supergroup. TDLib's `getForumTopic` returns a `forumTopic` whose `info`
 * field contains the last_message_id.
 *
 * Returns null on failure or empty topic — caller treats as "unknown".
 */
export async function getForumTopicLastMessageId(
  client: Client,
  chatId: bigint,
  topicId: bigint
): Promise<bigint | null> {
  try {
    // eslint-disable-next-line @typescript-eslint/no-explicit-any
    const topic = (await client.invoke({
      _: "getForumTopic",
      chat_id: Number(chatId),
      message_thread_id: Number(topicId),
    })) as { last_message?: { id?: number }; info?: { last_message_id?: number } };
    const id = topic.last_message?.id ?? topic.info?.last_message_id;
    return id ? BigInt(id) : null;
  } catch (err) {
    log.debug(
      { err, chatId: chatId.toString(), topicId: topicId.toString() },
      "getForumTopicLastMessageId failed"
    );
    return null;
  }
}

Note the file already imports Client from tdl and has a log defined at the top — both will be reused.

• Step 2: Type-check

docker run --rm -v "$PWD:/work" -w /work/worker node:20-bookworm-slim sh -c "npx tsc --noEmit 2>&1 | grep -v 'npm notice'"

Expected: zero errors.

• Step 3: Commit

git add worker/src/tdlib/chats.ts
git commit -m "$(cat <<'EOF'
feat(tdlib): add getChannelLastMessageId / getForumTopicLastMessageId

Both read the server-side last message ID from TDLib's local cache.
Used by the channel-scan-skip guard to short-circuit a paginated
searchChatMessages when last_message.id <= our watermark.

Returns null on any failure so the caller can fall back to scanning —
we'd rather waste a scan than miss new content.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 5: Wire the skip + short-circuit into the non-forum branch

Files:

• Modify: worker/src/worker.ts (the non-forum channel branch of runWorkerForAccount)

This task touches just the non-forum branch. The forum branch is Task 6 — same logic, different scope. Splitting them keeps the diff small.

• Step 1: Update imports

Open worker/src/worker.ts. Find the import block from ./db/queries.js (around line 10) and add upsertChannelScanState to the list. Find the import from ./tdlib/chats.js (around line 36) and add getChannelLastMessageId. The resulting imports look like:

import {
  getSourceChannelMappings,
  getGlobalDestinationChannel,
  packageExistsByHash,
  packageExistsBySourceMessage,
  createPackageStub,
  updatePackageWithMetadata,
  createIngestionRun,
  completeIngestionRun,
  failIngestionRun,
  updateLastProcessedMessage,
  updateRunActivity,
  setChannelForum,
  getTopicProgress,
  upsertTopicProgress,
  upsertChannel,
  ensureAccountChannelLink,
  getGlobalSetting,
  getChannelFetchRequest,
  updateFetchRequestStatus,
  getAccountLinkedChannelIds,
  getExistingChannelsByTelegramId,
  getAccountById,
  deleteOrphanedPackageByHash,
  getUploadedPackageByHash,
  upsertSkippedPackage,
  deleteSkippedPackage,
  getCappedSkippedMessageIds,
  findRepostedPackage,
  findPackageByRemoteUniqueId,
  getRetryableSkippedMessageIds,
  updatePackageTopicContext,
  upsertChannelScanState,
} from "./db/queries.js";

And:

import { getAccountChats, joinChatByInviteLink, getChannelLastMessageId } from "./tdlib/chats.js";

• Step 2: Expose cycleCount from the scheduler

The skip logic needs to know which cycle this is for the backoff "every Nth" check. Open worker/src/scheduler.ts and find the let cycleCount = 0; declaration near the top. Above the async function runCycle() declaration, add an exporter:

/** Read-only access to the current cycle counter for code that needs to
 *  apply per-cycle modulo logic (e.g. the cold-channel backoff). */
export function getCurrentCycle(): number {
  return cycleCount;
}

• Step 3: Import getCurrentCycle in worker.ts

In worker/src/worker.ts, just below the existing import { ... } from "./scheduler.js"; line (or add a new import if none exists), make sure getCurrentCycle is imported. If there's no scheduler import yet, add:

import { getCurrentCycle } from "./scheduler.js";

• Step 4: Add the skip decision at the top of the non-forum branch

Find the non-forum branch inside runWorkerForAccount. It starts with the comment // ── Non-forum channel: flat scan (existing behavior) ── (around line 706 after recent commits). Immediately AFTER the else { line and BEFORE the existing await updateRunActivity(...) "Scanning ..." call, insert the skip guard:

        } else {
          // ── Channel-scan-skip guard ──
          // Before any TDLib call, decide whether this channel can be
          // skipped entirely this cycle. Three signals (in order):
          //   1. retryable SkippedPackages exist → MUST scan
          //   2. lastScannedAt within window AND last scan was idle → skip
          //   3. in backoff AND not the Nth cycle → skip
          // See docs/superpowers/specs/2026-05-26-channel-scan-skip-design.md
          try {
            const retryable = await getRetryableSkippedMessageIds({
              accountId: account.id,
              sourceChannelId: channel.id,
              topicId: null,
              cap: config.maxSkipAttempts,
            });
            if (retryable.length === 0 && mapping.lastScannedAt) {
              const sinceLastScanMs = Date.now() - mapping.lastScannedAt.getTime();
              const withinRecencyWindow = sinceLastScanMs < config.skipRecentScanWindowMs;
              const inBackoff = mapping.consecutiveEmptyScans >= config.emptyScanBackoffThreshold;
              const backoffSkipsThisCycle =
                inBackoff && getCurrentCycle() % config.emptyScanBackoffEveryNth !== 0;

              if (
                (withinRecencyWindow && !mapping.lastScanFoundArchives) ||
                backoffSkipsThisCycle
              ) {
                accountLog.debug(
                  {
                    channel: channel.title,
                    sinceLastScanMs,
                    consecutiveEmptyScans: mapping.consecutiveEmptyScans,
                    reason: withinRecencyWindow ? "recent-idle" : "backoff",
                  },
                  "Skipping channel — recently scanned and idle, or in backoff"
                );
                continue;
              }
            }
          } catch (skipErr) {
            // Skip guard is best-effort. If the retryable query fails,
            // fall through and do the normal scan.
            accountLog.warn(
              { err: skipErr, channel: channel.title },
              "Skip guard failed, proceeding with scan"
            );
          }

          // ── Non-forum channel: flat scan (existing behavior) ──
          await updateRunActivity(activeRunId, {

• Step 5: Add the getChat short-circuit after the retry pass

In the same non-forum branch, find where the existing retry pass writes back the watermark (look for the comment // Pull the watermark back below the lowest still-retryable SkippedPackage). After the retry pass completes — the spot where effectiveChannelWatermark is set — and BEFORE the const scanResult = await getChannelMessages(...) call, insert:

          // ── getChat short-circuit ──
          // After the retry pass has settled the effective watermark, ask
          // TDLib for the channel's last_message.id. If it's <= our watermark,
          // no new content exists since last cycle — skip the paginated
          // searchChatMessages entirely. Still update scan-state so the
          // recent-scan skip can kick in next cycle.
          const channelLastId = await getChannelLastMessageId(client, channel.telegramId);
          if (
            channelLastId !== null
            && effectiveChannelWatermark !== null
            && channelLastId <= effectiveChannelWatermark
          ) {
            accountLog.info(
              {
                channel: channel.title,
                channelLastId: channelLastId.toString(),
                watermark: effectiveChannelWatermark.toString(),
              },
              "Channel caught up via getChat — skipping searchChatMessages"
            );
            await upsertChannelScanState(mapping.id, {
              lastProcessedMessageId: effectiveChannelWatermark,
              lastScanFoundArchives: false,
              consecutiveEmptyScans: (mapping.consecutiveEmptyScans ?? 0) + 1,
            });
            continue;
          }

• Step 6: Replace the end-of-scan watermark write with the new helper

In the same non-forum branch, find the existing end-of-scan watermark write. It looks like:

          if (channelWatermark !== null) {
            await updateLastProcessedMessage(mapping.id, channelWatermark);
          }

Replace it with the new helper, computing the trulyIdle value first:

          // ── Persist scan state ──
          // trulyIdle: nothing new this scan AND nothing failed AND no
          // retryable SkippedPackages pending. The retryable check matters —
          // a chronically-failing archive should NEVER let the channel back
          // off, even though zipsFound stays at 0 for it.
          const retryablePendingNow = await getRetryableSkippedMessageIds({
            accountId: account.id,
            sourceChannelId: channel.id,
            topicId: null,
            cap: config.maxSkipAttempts,
          });
          const trulyIdle =
            scanResult.archives.length === 0
            && minFailedId === null
            && retryablePendingNow.length === 0;
          const newConsecutive = trulyIdle
            ? (mapping.consecutiveEmptyScans ?? 0) + 1
            : 0;
          if (channelWatermark !== null) {
            await upsertChannelScanState(mapping.id, {
              lastProcessedMessageId: channelWatermark,
              lastScanFoundArchives: !trulyIdle,
              consecutiveEmptyScans: newConsecutive,
            });
          }

• Step 7: Type-check

docker run --rm -v "$PWD:/work" -w /work/worker node:20-bookworm-slim sh -c "npx tsc --noEmit 2>&1 | grep -v 'npm notice'"

Expected: zero errors. If TS complains that mapping.lastScannedAt doesn't exist, the Prisma client regen from Task 1 didn't take effect — re-run npx prisma generate.

• Step 8: Commit

git add worker/src/worker.ts worker/src/scheduler.ts
git commit -m "$(cat <<'EOF'
feat(worker): non-forum channel-scan-skip + getChat short-circuit

For non-forum channels in runWorkerForAccount, three guards:

  1. Top-of-loop recency/backoff skip — if recently scanned with no
     pending work, or in backoff and not its turn, skip entirely.
     Bypassed when retryable SkippedPackages exist.

  2. After the SkippedPackage retry pass, a getChat short-circuit —
     if TDLib's local cache says the channel's last_message.id <= our
     effective watermark, skip the paginated searchChatMessages.

  3. End-of-scan persists lastScannedAt + lastScanFoundArchives +
     consecutiveEmptyScans via the new upsertChannelScanState helper.
     trulyIdle requires: no archives, no failures, no retryable pending.

Forum-topic branch lands in the next commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 6: Wire the skip + short-circuit into the forum-topic branch

Files:

• Modify: worker/src/worker.ts (the forum branch of runWorkerForAccount)

Same logic as Task 5, scoped to forum topics. The forum branch loops over topics inside a channel, so the skip/short-circuit go inside the topic loop, not the channel loop.

• Step 1: Update imports

Open worker/src/worker.ts. The db/queries.js import block (modified in Task 5) needs upsertTopicScanState too:

import {
  // ... all existing entries from Task 5 ...
  upsertChannelScanState,
  upsertTopicScanState,
} from "./db/queries.js";

And the ./tdlib/chats.js import needs getForumTopicLastMessageId:

import {
  getAccountChats,
  joinChatByInviteLink,
  getChannelLastMessageId,
  getForumTopicLastMessageId,
} from "./tdlib/chats.js";

• Step 2: Add the top-of-topic-loop skip guard

Find the forum branch in runWorkerForAccount. The topic loop starts with for (let tIdx = 0; tIdx < topics.length; tIdx++) {. Inside that loop, find the try { block and the let progress = topicProgressList.find(...) line. Just AFTER the existing General-topic-ID fallback (if (!progress && topic.name === "General") block), and BEFORE the existing SkippedPackage retry pass, insert:

              // ── Topic-scan-skip guard ──
              // Same three-signal decision as the non-forum branch, but
              // scoped to a single topic. Uses `progress` for the persisted
              // scan-state fields (lastScannedAt etc).
              try {
                const retryableForTopic = await getRetryableSkippedMessageIds({
                  accountId: account.id,
                  sourceChannelId: channel.id,
                  topicId: topic.topicId,
                  cap: config.maxSkipAttempts,
                });
                if (retryableForTopic.length === 0 && progress?.lastScannedAt) {
                  const sinceLastScanMs = Date.now() - progress.lastScannedAt.getTime();
                  const withinRecencyWindow = sinceLastScanMs < config.skipRecentScanWindowMs;
                  const inBackoff =
                    (progress.consecutiveEmptyScans ?? 0) >= config.emptyScanBackoffThreshold;
                  const backoffSkipsThisCycle =
                    inBackoff && getCurrentCycle() % config.emptyScanBackoffEveryNth !== 0;

                  if (
                    (withinRecencyWindow && !progress.lastScanFoundArchives) ||
                    backoffSkipsThisCycle
                  ) {
                    accountLog.debug(
                      {
                        channel: channel.title,
                        topic: topic.name,
                        sinceLastScanMs,
                        consecutiveEmptyScans: progress.consecutiveEmptyScans,
                        reason: withinRecencyWindow ? "recent-idle" : "backoff",
                      },
                      "Skipping topic — recently scanned and idle, or in backoff"
                    );
                    continue;
                  }
                }
              } catch (skipErr) {
                accountLog.warn(
                  { err: skipErr, topic: topic.name },
                  "Topic skip guard failed, proceeding with scan"
                );
              }

• Step 3: Add getForumTopicInfo short-circuit after the retry pass

In the same forum branch, find where the topic-level retry pass updates progress (look for the lines after the if (retryable.length > 0) block where progress = { ...(progress ?? ...), lastProcessedMessageId: resetTo } as typeof progress;). The getTopicMessages call comes shortly after. Just BEFORE the const scanResult = await getTopicMessages(...) line, insert:

              // ── getForumTopic short-circuit ──
              // After the retry pass has settled the effective watermark,
              // ask TDLib for the topic's last_message_id. If it's <= our
              // watermark, no new content — skip the paginated search.
              const topicLastId = await getForumTopicLastMessageId(
                client,
                channel.telegramId,
                topic.topicId
              );
              const effectiveTopicWatermark = progress?.lastProcessedMessageId ?? null;
              if (
                topicLastId !== null
                && effectiveTopicWatermark !== null
                && topicLastId <= effectiveTopicWatermark
              ) {
                accountLog.info(
                  {
                    channel: channel.title,
                    topic: topic.name,
                    topicLastId: topicLastId.toString(),
                    watermark: effectiveTopicWatermark.toString(),
                  },
                  "Topic caught up via getForumTopic — skipping searchChatMessages"
                );
                await upsertTopicScanState(mapping.id, topic.topicId, topic.name, {
                  lastProcessedMessageId: effectiveTopicWatermark,
                  lastScanFoundArchives: false,
                  consecutiveEmptyScans: (progress?.consecutiveEmptyScans ?? 0) + 1,
                });
                continue;
              }

• Step 4: Replace the end-of-topic-scan watermark write with the new helper

Find the existing topic-progress write. It's an upsertTopicProgress call at the end of the topic-loop body (after processArchiveSets). Look for:

              if (topicWatermark !== null) {
                await upsertTopicProgress(
                  mapping.id,
                  topic.topicId,
                  topic.name,
                  topicWatermark
                );
              }

Replace it with:

              // ── Persist topic scan state ──
              const retryableTopicPendingNow = await getRetryableSkippedMessageIds({
                accountId: account.id,
                sourceChannelId: channel.id,
                topicId: topic.topicId,
                cap: config.maxSkipAttempts,
              });
              const topicTrulyIdle =
                scanResult.archives.length === 0
                && minFailedId === null
                && retryableTopicPendingNow.length === 0;
              const newTopicConsecutive = topicTrulyIdle
                ? (progress?.consecutiveEmptyScans ?? 0) + 1
                : 0;
              if (topicWatermark !== null) {
                await upsertTopicScanState(mapping.id, topic.topicId, topic.name, {
                  lastProcessedMessageId: topicWatermark,
                  lastScanFoundArchives: !topicTrulyIdle,
                  consecutiveEmptyScans: newTopicConsecutive,
                });
              }

• Step 5: Same treatment for the "no archives found in topic" branch

Earlier in the forum branch, there's a path that handles "scan returned zero archives" — it writes the watermark without going through processArchiveSets. Look for:

              if (scanResult.archives.length === 0) {
                accountLog.info(
                  { channelId: channel.id, topic: topic.name, totalScanned: scanResult.totalScanned },
                  "No new archives in topic"
                );
                // Still advance topic watermark so we don't re-scan these messages next cycle
                if (scanResult.maxScannedMessageId) {
                  await upsertTopicProgress(
                    mapping.id,
                    topic.topicId,
                    topic.name,
                    scanResult.maxScannedMessageId
                  );
                }
                continue;
              }

Replace it with:

              if (scanResult.archives.length === 0) {
                accountLog.info(
                  { channelId: channel.id, topic: topic.name, totalScanned: scanResult.totalScanned },
                  "No new archives in topic"
                );
                // Still advance topic watermark so we don't re-scan these messages next cycle.
                // Counts as truly idle UNLESS retryable SkippedPackages exist for this topic.
                const retryableTopicNoArchives = await getRetryableSkippedMessageIds({
                  accountId: account.id,
                  sourceChannelId: channel.id,
                  topicId: topic.topicId,
                  cap: config.maxSkipAttempts,
                });
                const topicTrulyIdleNoArchives = retryableTopicNoArchives.length === 0;
                if (scanResult.maxScannedMessageId) {
                  await upsertTopicScanState(mapping.id, topic.topicId, topic.name, {
                    lastProcessedMessageId: scanResult.maxScannedMessageId,
                    lastScanFoundArchives: !topicTrulyIdleNoArchives,
                    consecutiveEmptyScans: topicTrulyIdleNoArchives
                      ? (progress?.consecutiveEmptyScans ?? 0) + 1
                      : 0,
                  });
                }
                continue;
              }

• Step 6: Same treatment for the non-forum "no archives" path

Symmetric to Step 5 but for the non-forum branch. Find:

          if (scanResult.archives.length === 0) {
            accountLog.info({ channelId: channel.id, title: channel.title, totalScanned: scanResult.totalScanned }, "No new archives in channel");
            // Still advance watermark to highest scanned message so we don't
            // re-scan these messages next cycle
            if (scanResult.maxScannedMessageId) {
              await updateLastProcessedMessage(mapping.id, scanResult.maxScannedMessageId);
            }
            continue;
          }

Replace with:

          if (scanResult.archives.length === 0) {
            accountLog.info({ channelId: channel.id, title: channel.title, totalScanned: scanResult.totalScanned }, "No new archives in channel");
            const retryableNoArchives = await getRetryableSkippedMessageIds({
              accountId: account.id,
              sourceChannelId: channel.id,
              topicId: null,
              cap: config.maxSkipAttempts,
            });
            const channelTrulyIdleNoArchives = retryableNoArchives.length === 0;
            if (scanResult.maxScannedMessageId) {
              await upsertChannelScanState(mapping.id, {
                lastProcessedMessageId: scanResult.maxScannedMessageId,
                lastScanFoundArchives: !channelTrulyIdleNoArchives,
                consecutiveEmptyScans: channelTrulyIdleNoArchives
                  ? (mapping.consecutiveEmptyScans ?? 0) + 1
                  : 0,
              });
            }
            continue;
          }

• Step 7: Type-check

docker run --rm -v "$PWD:/work" -w /work/worker node:20-bookworm-slim sh -c "npx tsc --noEmit 2>&1 | grep -v 'npm notice'"

Expected: zero errors.

• Step 8: Full build to be sure runtime is also clean

docker run --rm -v "$PWD:/work" -w /work/worker node:20-bookworm-slim sh -c "npm run build 2>&1 | tail -3"

Expected: silent success (just the > tsc line).

• Step 9: Commit

git add worker/src/worker.ts
git commit -m "$(cat <<'EOF'
feat(worker): forum-topic scan-skip + getForumTopic short-circuit

Mirror of the non-forum guards from the previous commit, scoped to
forum topics inside the topic loop:

  - Top-of-topic-loop recency/backoff skip
  - getForumTopic short-circuit after the SkippedPackage retry pass
  - upsertTopicScanState for the end-of-scan persistence (both the
    archives-found path and the no-archives path)
  - Symmetric no-archives path for the non-forum branch

Same trulyIdle definition throughout: no archives this scan, no
failures this scan, no retryable SkippedPackage rows pending. Channels
with chronic failures stay out of backoff because their counter never
increments.

For MPE specifically (1,086 forum topics), per-cycle searchChatMessages
calls drop from ~1,086 to roughly the count of topics with new
activity in the last 5 minutes — typically <50.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 7: Manual verification against the live worker

This project has no automated tests. Verification is by Docker deploy + log inspection. Each step below corresponds to one risk identified in the design spec.

• Step 1: Rebuild and restart the worker

docker compose up -d --build worker

Wait for the migration to apply (visible in the dragonsstash app's startup logs — the worker container shares the DB). Expected: Migration 20260526000000_channel_scan_state ... applied.

• Step 2: Confirm the first cycle scans everything

docker logs --since 5m dragonsstash-worker 2>&1 | grep -E "(Scanning|caught up|Skipping channel|Skipping topic)" | tail -30

Expected: many "Scanning forum channel by topic" lines for MPE and other forums. No "Skipping" lines yet because all rows have lastScannedAt = NULL.

• Step 3: Trigger an immediate second cycle (from admin UI's Trigger Cycle button, or via psql)

docker exec dragonsstash-db psql -U admin -d dragonsstash -c "SELECT pg_notify('ingestion_trigger', '');"

Wait ~20 seconds.

• Step 4: Confirm the second cycle skips idle channels/topics

docker logs --since 1m dragonsstash-worker 2>&1 | grep -E "(Skipping channel|Skipping topic|caught up via)" | wc -l

Expected: a number >> 100 for an MPE-sized account. If you see 0, the recency window may be set wrong — confirm WORKER_SKIP_RECENT_SCAN_WINDOW_MS is unset or 300000.

• Step 5: Verify failure retry is NOT broken

Pick a SkippedPackage row with attemptCount < 5 (the default cap):

docker exec dragonsstash-db psql -U admin -d dragonsstash -c "
SELECT id, \"fileName\", \"attemptCount\", \"sourceChannelId\", \"sourceMessageId\"
FROM skipped_packages
WHERE \"attemptCount\" < 5
ORDER BY \"createdAt\" DESC
LIMIT 1;
"

Note the sourceChannelId. Trigger a cycle and confirm the channel containing that row is not skipped:

docker logs --since 30s dragonsstash-worker 2>&1 | grep "Skipping channel" | grep "<channel-title>"

Expected: no match (the channel was scanned, not skipped). And:

docker logs --since 30s dragonsstash-worker 2>&1 | grep "Resetting channel watermark to retry skipped messages"

Expected: at least one line indicating the retry pass ran.

• Step 6: Verify backoff kicks in

Pick a topic that's been idle for a while (e.g., MPE → "Erevan's Guide [Completed]"). After 5+ clean cycles, confirm:

docker exec dragonsstash-db psql -U admin -d dragonsstash -c "
SELECT \"topicName\", \"consecutiveEmptyScans\"
FROM topic_progress
WHERE \"consecutiveEmptyScans\" >= 5
ORDER BY \"consecutiveEmptyScans\" DESC
LIMIT 10;
"

Expected: at least a few topics with counter >= 5. Subsequent cycles should log "Skipping topic ... reason: backoff" for them, scanning only every 5th cycle.

• Step 7: Verify restart safety

Restart the worker container:

docker compose restart worker

Tail the logs during the first post-restart cycle:

docker logs --since 30s -f dragonsstash-worker 2>&1 | grep -E "(Skipping|caught up|Scanning .*topic)" | head -30

Expected: many "Skipping" / "caught up" lines well before any heavy scan starts. The first cycle after restart should take a fraction of the time it used to.

• Step 8: If anything is wrong, do not commit further — the implementation commits from Tasks 1-6 produced the code.

If a regression is detected (e.g., failure retries no longer fire), revert the relevant task commit with git revert <hash> and investigate. The changes are layered so reverting later tasks first is safe (config knobs and DB columns are inert without the worker.ts logic).

---

Self-review

Spec coverage

Walking each spec requirement against tasks:

• ✅ Schema columns on AccountChannelMap → Task 1, Step 1
• ✅ Schema columns on TopicProgress → Task 1, Step 2
• ✅ Migration SQL → Task 1, Step 3
• ✅ Three config knobs → Task 2
• ✅ DB helpers (upsertChannelScanState, upsertTopicScanState) → Task 3
• ✅ TDLib helpers (getChannelLastMessageId, getForumTopicLastMessageId) → Task 4
• ✅ Top-of-loop skip guard — non-forum → Task 5, Step 4
• ✅ Top-of-loop skip guard — forum topic → Task 6, Step 2
• ✅ getChat short-circuit — non-forum → Task 5, Step 5
• ✅ getForumTopic short-circuit — forum → Task 6, Step 3
• ✅ End-of-scan bookkeeping — non-forum (both archives-found and no-archives paths) → Task 5 Step 6 + Task 6 Step 6
• ✅ End-of-scan bookkeeping — forum (both paths) → Task 6 Steps 4 + 5
• ✅ consecutiveEmptyScans reset/increment logic → embedded in the bookkeeping steps
• ✅ cycleCount % N backoff modulo → Task 5 Step 4 / Task 6 Step 2 via getCurrentCycle()
• ✅ getCurrentCycle() exporter from scheduler → Task 5, Step 2
• ✅ "trulyIdle" formula consistent across all four bookkeeping sites → uses getRetryableSkippedMessageIds, scanResult.archives.length, minFailedId
• ✅ Manual verification covering each edge case → Task 7

No gaps.

Placeholder scan

Searched for "TBD", "TODO", "implement later", "fill in details", "Add appropriate error handling", "handle edge cases" — none present. All code blocks are complete; every step that says "do X" shows X verbatim.

Type consistency

• ScanStateUpdate interface (Task 3) — lastProcessedMessageId: bigint | null, lastScanFoundArchives: boolean, consecutiveEmptyScans: number. All call sites in Tasks 5 and 6 pass these names.
• upsertChannelScanState(mappingId, update) — Task 5 always passes mapping.id. Consistent.
• upsertTopicScanState(accountChannelMapId, topicId, topicName, update) — Task 6 always passes mapping.id, topic.topicId, topic.name, update. Consistent.
• getChannelLastMessageId(client, chatId) returns bigint | null — Task 5 Step 5 checks if (channelLastId !== null && ... && channelLastId <= ...). Consistent.
• getForumTopicLastMessageId(client, chatId, topicId) — same shape and same Task 6 usage. Consistent.
• getCurrentCycle() returns number — Task 5/6 use % config.emptyScanBackoffEveryNth which is also number. Consistent.
• config.skipRecentScanWindowMs / emptyScanBackoffThreshold / emptyScanBackoffEveryNth — all number, all read with parseInt. Consistent with Date.now() - X.getTime() (also number) comparisons.

Build green at every commit

Each task ends with a type-check step before its commit, so the tree should compile after every commit individually. Task 6 adds a full build for extra safety.

---

Execution Handoff

Plan complete and saved to docs/superpowers/plans/2026-05-26-channel-scan-skip.md. Two execution options:

1. Subagent-Driven (recommended) — I dispatch a fresh subagent per task, review between tasks, fast iteration.

2. Inline Execution — Execute tasks in this session using executing-plans, batch execution with checkpoints.

Which approach?