mirror of
https://github.com/xCyanGrizzly/DragonsStash.git
synced 2026-09-21 05:21:43 +00:00
docs: design for forward-priority ingestion pipeline
Prioritize native Telegram forwarding over download+reupload for source channels that allow it, reusing the ranged archive-listing readers to keep indexing complete without a local download. Falls back to the existing download+reupload pipeline per-channel (when forwarding is blocked) and per-archive (when ranged listing fails).
This commit is contained in:
@@ -89,7 +89,14 @@
|
||||
"Bash(wait:*)",
|
||||
"WebSearch",
|
||||
"Bash(SKILL_CREATOR_PATH=\"C:\\\\Users\\\\A00963355\\\\.claude\\\\plugins\\\\cache\\\\claude-plugins-official\\\\skill-creator\\\\d5c15b861cd2\\\\skills\\\\skill-creator\" && WORKSPACE=\"C:\\\\Users\\\\A00963355\\\\OneDrive - Amaris Zorggroep\\\\Documents\\\\VScodeProjects\\\\DragonsStash\\\\.claude\\\\skills\\\\tdlib-telegram-workspace\\\\iteration-1\" && python \"$SKILL_CREATOR_PATH/eval-viewer/generate_review.py\" \"$WORKSPACE\" --skill-name \"tdlib-telegram\" --benchmark \"$WORKSPACE/benchmark.json\" --static \"$WORKSPACE/review.html\" 2>&1)",
|
||||
"Bash(start:*)"
|
||||
"Bash(start:*)",
|
||||
"Bash(npm run:*)",
|
||||
"Bash(DATABASE_URL=\"postgresql://dragons:stash@localhost:5432/dragonsstash\" npx prisma migrate dev --name add-skipped-packages)",
|
||||
"Bash(git checkout:*)",
|
||||
"Bash(DATABASE_URL=\"postgresql://dragons:stash@localhost:5432/dragonsstash?schema=public\" npx prisma migrate dev --name add_package_groups 2>&1)",
|
||||
"Bash(psql:*)",
|
||||
"Bash(git log:*)",
|
||||
"Bash(git merge:*)"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
# Forward-priority ingestion — design
|
||||
|
||||
**Date:** 2026-07-30
|
||||
**Status:** Approved (design), pending spec review → implementation plan
|
||||
|
||||
## Problem
|
||||
|
||||
The worker ingests every archive the same way regardless of whether it needs to: download the
|
||||
full file from the source channel, then re-upload the full file to the destination (archive)
|
||||
channel. That download+reupload round-trip was originally necessary because some source channels
|
||||
have "restrict saving content" (protected content) enabled, which blocks Telegram-native
|
||||
forwarding — for those channels there is no alternative to moving the bytes through the worker.
|
||||
|
||||
But most source channels do NOT restrict forwarding. For those, the round-trip is pure waste:
|
||||
Telegram can copy the message from source chat to destination chat server-side, with no bytes
|
||||
ever passing through the worker. The worker still needs to end up with the same outcome it has
|
||||
today — a destination-channel copy, a dedup-safe identity, and a full inner-file listing — just
|
||||
without paying for a download and re-upload to get there.
|
||||
|
||||
Separately, `feat/ranged-archive-listing` (merged to master ahead of this feature) already built
|
||||
exactly the missing piece: reading a ZIP/RAR/7z archive's inner-file listing via small ranged
|
||||
reads against the file wherever it currently lives (source channel, destination channel — doesn't
|
||||
matter), with no full download. It was built for backfilling listings onto already-deduped
|
||||
placeholder packages. This feature generalizes that same capability to fresh ingestion, and pairs
|
||||
it with a new native-forward upload path.
|
||||
|
||||
## Goals
|
||||
|
||||
- For channels that allow forwarding: skip download and re-upload entirely for new archives. Use
|
||||
Telegram-native forwarding from source chat to destination chat, and the existing ranged-listing
|
||||
readers to index inner files, with no full download in the common case.
|
||||
- For channels that block forwarding (or when forwarding isn't yet known): keep today's
|
||||
download+reupload pipeline exactly as-is.
|
||||
- Every ingested package — regardless of path — ends up with the same outcome as today: a
|
||||
`Package` row with a valid dedup identity, `destMessageId`/`destMessageIds`, creator, tags, and a
|
||||
full inner-file listing (`PackageFile` rows). Indexing completeness must not regress.
|
||||
- If the cheap ranged listing fails for a specific archive (bad/unsupported header, CLI error,
|
||||
etc.) in an otherwise-forwarding-eligible channel, fall back to today's full download+reupload
|
||||
pipeline for that one archive — never forward with an empty or partial listing.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No ranged single-entry preview extraction. Forward-path packages still get a preview when a
|
||||
channel photo message matches (cheap, unrelated to archive bytes); when there's no matching
|
||||
photo, forward-path packages simply have no preview, same as any package where preview
|
||||
extraction fails today. In-archive preview extraction (unzip/unrar/7z against a local file) stays
|
||||
as a download-path-only feature. May be revisited as a follow-up if it turns out to matter.
|
||||
- No reprocessing of already-ingested packages. This only changes behavior for newly-scanned
|
||||
archives going forward.
|
||||
- No change to the bot's user-delivery leg (`bot/src/tdlib/client.ts` `copyMessageToUser`) — it
|
||||
already sends via `inputFileRemote` with no download, and is unaffected by this feature.
|
||||
- No change to `config.maxZipSizeMB` or the multipart byte-level split/repack logic. The existing
|
||||
size guard runs before either path is chosen, so nothing above the cap reaches the forward path's
|
||||
fallback-to-download step either. Splitting simply never engages on the forward path — a
|
||||
forwarded message is already within whatever size Telegram accepted when it was first uploaded.
|
||||
|
||||
## Approaches considered
|
||||
|
||||
**A — Branch inside the existing pipeline (chosen).** Add one fork point in
|
||||
`processOneArchiveSet`, immediately after the existing pre-download dedup checks: if the channel
|
||||
allows forwarding, attempt the ranged-listing + forward path; on any failure, fall through into
|
||||
today's download-based code for that one archive, unchanged. Smallest diff; reuses the existing
|
||||
dedup/retry/watermark machinery as-is; matches the file's existing forum-vs-non-forum branching
|
||||
style.
|
||||
|
||||
**B — Separate pipeline per channel.** Decide once per channel and route the whole channel through
|
||||
either a "forward module" or the existing "download module." Cleaner separation on paper, but
|
||||
duplicates the SkippedPackage/stall/watermark bookkeeping that currently lives once in
|
||||
`processArchiveSets`/`processOneArchiveSet` — higher regression risk in a large orchestration file
|
||||
with no tests at that level. Rejected.
|
||||
|
||||
**C — Strategy-object refactor.** Extract an `IngestStrategy` interface (`download` / `forward`)
|
||||
and slim `processOneArchiveSet` to delegate to it. The more "proper" abstraction, but it's a
|
||||
structural refactor of already-battle-tested code that doesn't need it for this feature to work.
|
||||
Rejected — can revisit later if a third strategy ever appears.
|
||||
|
||||
## Sequencing
|
||||
|
||||
`feat/ranged-archive-listing` merges to master first, as-is (it's complete and serves a different
|
||||
purpose already). This feature is built on a fresh branch off master afterward.
|
||||
|
||||
## Components
|
||||
|
||||
### 1. `TelegramChannel.allowsForwarding` (new column, new migration)
|
||||
|
||||
`Boolean?` — nullable, `null` means "not yet checked". Refreshed from TDLib's chat
|
||||
protected-content flag (exact field name to be confirmed against the pinned `tdl`/TDLib version
|
||||
via docs lookup during implementation — expected to be `chat.has_protected_content`) at the same
|
||||
point the worker already calls `getChat` per channel per cycle, mirroring the existing
|
||||
`isForum`/`setChannelForum` read-and-persist pattern precisely. `null` or `false` both route to the
|
||||
download path — a channel never uses the forward path on unverified permission.
|
||||
|
||||
### 2. Shared ranged-listing dispatcher
|
||||
|
||||
`readScannedListingRanged` (plus `RangedPart`, `tdlibRangeReader`, and the format-specific
|
||||
ZIP/RAR/7z readers) currently live inside `provenance-backfill.ts`. Promote the dispatcher (and
|
||||
whatever it depends on) into a shared module (e.g. `worker/src/archive/ranged/dispatch.ts`) so
|
||||
`worker.ts` can call the same no-download listing logic for fresh ingestion without a circular
|
||||
import. `provenance-backfill.ts` switches to importing from the new shared location; behavior
|
||||
unchanged for the existing backfill path.
|
||||
|
||||
### 3. `forwardArchiveToChannel` (new, `worker/src/upload/forward.ts`)
|
||||
|
||||
Mirrors `uploadToChannel`'s shape and return type (`{ messageId, messageIds }`). Uses TDLib
|
||||
`forwardMessages` to copy all parts of an archive set from the source chat to the destination chat
|
||||
in one batch call (message IDs in original order), wrapped in the same flood-wait/retry handling
|
||||
style as `uploadToChannel`. Followed by the same destination read-back verification style as
|
||||
today's post-upload check (`getMessage` on each new destination message ID, confirm a document is
|
||||
present).
|
||||
|
||||
### 4. Dedup identity for forward-path packages
|
||||
|
||||
`Package.contentHash` stays a required unique string, but forward-path packages can't hash real
|
||||
bytes. Derivation order:
|
||||
1. If the ranged listing's CRC32s are complete (ZIP/RAR today) — hash the sorted CRC32 list into a
|
||||
synthetic `fingerprint:<hash>` value, reusing `archive/fingerprint.ts`'s existing
|
||||
`crcFingerprint`.
|
||||
2. Otherwise (7z, or any incomplete-CRC case) — synthesize `forward:<remoteUniqueId>`, following
|
||||
the existing `rebuild:`-prefixed placeholder-hash precedent in `rebuild.ts`.
|
||||
|
||||
Additionally, extend repost detection: before committing to the forward path, compare the new
|
||||
listing's CRC fingerprint (via the existing `compareFingerprints`/`fingerprintsMatch` logic already
|
||||
used in `provenance-backfill.ts`'s ambiguous-candidate disambiguation) against recent Packages
|
||||
sharing the same file name + size. A fingerprint match is treated as a duplicate and skipped, same
|
||||
as today's `findRepostedPackage` handling — this is what lets a forwarded copy and a previously
|
||||
fully-downloaded copy of the same archive still dedupe against each other, despite never sharing a
|
||||
byte-hash-derived `contentHash`.
|
||||
|
||||
### 5. Fork point in `processOneArchiveSet`
|
||||
|
||||
All existing pre-download checks run first, completely unchanged, in the same order:
|
||||
`remote.unique_id` match → `packageExistsBySourceMessage` → `findRepostedPackage` (name+size) →
|
||||
cross-channel provenance backfill → size guard (`maxZipSizeMB`).
|
||||
|
||||
Then:
|
||||
|
||||
```
|
||||
if channel.allowsForwarding === true:
|
||||
entries = readScannedListingRanged(archiveType, client, scannedParts)
|
||||
if entries is not null:
|
||||
contentHash = deriveForwardContentHash(entries, remoteUniqueId)
|
||||
if fingerprintRepostCheck(entries, fileName, fileSize) finds a match:
|
||||
→ treat as duplicate, skip (same bookkeeping as today's dup path)
|
||||
destResult = forwardArchiveToChannel(client, sourceChatId, partMessageIds, destChatId)
|
||||
creator, tags ← derived from entries/filename/channel/topic, same as today
|
||||
preview ← channel-photo match only (no in-archive extraction)
|
||||
createPackageStub(...) + updatePackageWithMetadata(...), same as today
|
||||
counters.zipsForwarded++
|
||||
→ done
|
||||
else:
|
||||
→ fall through into the existing download/hash/split/upload flow below, unchanged
|
||||
(log the fallback for observability)
|
||||
else:
|
||||
→ existing download/hash/split/upload flow, completely unchanged
|
||||
```
|
||||
|
||||
### 6. Observability
|
||||
|
||||
New `zipsForwarded` counter alongside the existing `zipsFound`/`zipsDuplicate`/`zipsIngested`/
|
||||
`zipsBackfilled` counters, surfaced the same way (run activity, ingestion run summary). A WARN-level
|
||||
log line when a forwarding-eligible archive falls back to download (mirrors the existing
|
||||
`confidence: "ranged" | "full-download-fallback"` logging convention from the ranged-listing
|
||||
backfill work), so the fallback rate is visible without digging through debug logs.
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
scan → pre-download dedup + size guard (unchanged)
|
||||
→ channel.allowsForwarding?
|
||||
true → ranged listing
|
||||
ok → fingerprint dedup check → forward → stub + entries + tags (no in-archive preview) → done
|
||||
null → [fall through] existing download pipeline
|
||||
false/unknown → existing download pipeline (unchanged)
|
||||
```
|
||||
|
||||
## Error handling
|
||||
|
||||
- `forwardMessages` failure (permission revoked mid-run, rate limit, transient Telegram error) —
|
||||
same `SkippedPackage`/`SystemNotification` bookkeeping as today's upload failures. Extend
|
||||
`inferSkipReason` to recognize forward-specific error text the same way it already recognizes
|
||||
upload errors.
|
||||
- Fingerprint-repost check finds multiple ambiguous same-name/size candidates that can't be
|
||||
uniquely disambiguated — same `INTEGRITY_AUDIT` notification pattern already used in
|
||||
`provenance-backfill.ts`: don't guess, surface for manual triage.
|
||||
- `allowsForwarding` unknown (channel just linked, not yet scanned by the refresh point) — treated
|
||||
as `false`; the download path runs. No channel uses an unverified forwarding permission.
|
||||
- Ranged listing throwing instead of returning `null` — treated identically to returning `null`
|
||||
(fall through to download), consistent with how the existing ranged readers already treat
|
||||
internal errors (they catch and return `null` themselves).
|
||||
|
||||
## Testing
|
||||
|
||||
- Unit tests (vitest, alongside the existing `archive/*.test.ts` and `archive/ranged/*.test.ts`
|
||||
files): the dedup-identity derivation function (fingerprint-hash vs remoteUniqueId-fallback
|
||||
branches), the extended fingerprint-based repost check, and `forwardArchiveToChannel`'s
|
||||
request-building logic against a mocked TDLib client — same style as the existing ranged-reader
|
||||
tests (pure logic, no live TDLib).
|
||||
- Live verification (manual — matches this repo's existing convention that the large
|
||||
`worker.ts`/`worker.py`-equivalent orchestration function has no automated test coverage and is
|
||||
verified live post-deploy): one forwarding-enabled test channel and one protected-content test
|
||||
channel. Confirm forward-path packages land with correct entries/tags/dedup identity and
|
||||
`destMessageIds`; confirm the protected channel still goes through the unchanged full pipeline;
|
||||
confirm a deliberately-unparseable archive in a forwarding-enabled channel correctly falls back
|
||||
to download+reupload and still ends up fully indexed.
|
||||
|
||||
## Rollout
|
||||
|
||||
Local build + deploy, following the same recipe as the ranged-archive-listing work: build
|
||||
`worker/Dockerfile` locally, recreate the `dragonsstash-worker` container from the local image (no
|
||||
`pull`, no GitHub push required). New DB migration for `TelegramChannel.allowsForwarding`. No
|
||||
changes required to the bot or app services.
|
||||
Reference in New Issue
Block a user