dragonsstash/docs/superpowers/specs/2026-07-27-ranged-archive-listing-design.md

# Ranged inner-file listing for RAR & 7z — design

**Date:** 2026-07-27
**Status:** Approved (design), pending spec review → implementation plan

## Problem

The reindex/provenance-backfill path (`worker/src/provenance-backfill.ts`) can index an
archive's inner files *without* re-downloading it, by reading the file listing from a small
ranged read of the copy already in the source/destination channel. This works **only for
ZIP** today (ZIP keeps its central directory in a tail that `parseZipCentralDirectoryFromTail`
reads). For **RAR and 7z**, `tryProvenanceBackfill` backfills provenance (creator, source
channel, `remoteUniqueId`) so the file is skipped on re-scan and never re-downloaded — but it
leaves the inner listing empty (`fileCount = 0`), because `scannedEntries` is only computed for
ZIP.

Scope of the gap (rebuild placeholders with `fileCount = 0`, as of 2026-07-27):

| Type | Count | Total | Avg | Max | Multipart |
|---|---|---|---|---|---|
| 7z | 19,646 | 14 TB | 0.73 GB | 3.9 GB | 0 |
| RAR | 12,780 | 13 TB | 1.04 GB | 116 GB | 1,016 |

A "just download the whole archive" fallback for all of these means ~27 TB of re-downloads —
the exact cost this path exists to avoid.

## Goals

- Index the inner files (names + sizes; CRCs where cheaply available) of RAR and 7z
  placeholders **without** downloading the whole archive in the common case.
- Reuse the existing, battle-tested CLI listing parsers (`parse7zOutput`,
  `parseUnrarTechnical`) rather than reimplementing filename/size/CRC extraction.
- Keep cost proportional to **file count**, not archive size (so even the 116 GB RAR is cheap).
- Guarantee a listing for the rare archives the cheap path can't handle, via a full-download
  fallback that respects the existing max-size guard.

## Non-goals

- No change to ingestion of genuinely-new files (those are downloaded in full to be re-uploaded
  regardless, so a cheap listing does not help there). This feature only affects the
  provenance-backfill / skip path.
- No new ZIP behaviour — the existing ZIP tail reader stays as-is.
- Not attempting to list password-encrypted-header archives from ranged reads (no password);
  those take the fallback and, if oversized, are flagged.

## Approach (chosen: "harvest header regions → sparse file → native CLI")

Do the *minimum* binary parsing needed to locate an archive's header bytes, fetch only those
via ranged reads, write them into a sparse temp file at their true offsets (data regions left
as unwritten zero holes → ~no disk use), then run the real `7z l` / `unrar lt` and reuse the
existing parsers. The native tools do the hard parsing (7z's LZMA-encoded headers, RAR's two
format versions, Unicode names) — we only compute where the headers are.

Rejected alternatives: full native TS parsers (most custom binary code, highest risk);
RAR5-quick-open-only (most community RARs lack it → collapses to ~13 TB of RAR downloads).

## Components

New directory `worker/src/archive/ranged/`, one focused module per concern, all returning the
existing `FileEntry[]` type from `zip-reader.ts`:

- `sparse-list.ts` — `listFromSparse(parts, runner, parse) → FileEntry[] | null`, where each
  `part` is `{ fileName, size, regions: {offset, bytes}[] }`. For each part it writes a sparse
  temp file (`truncate` to `size`, then write only the header `regions` at their offsets),
  co-locates all parts in one temp dir under their real names, invokes the supplied CLI runner
  (`7z l` / `unrar lt`) on the first part, feeds stdout to the supplied `parse` fn
  (`parse7zOutput` / `parseUnrarTechnical`), and cleans up. Single-part archives are just the
  one-element case. Returns `null` on CLI error / empty parse.
- `sevenz-ranged.ts` — `readSevenZListingRanged(client, parts) → FileEntry[] | null`.
- `rar-ranged.ts` — `readRarListingRanged(client, parts) → FileEntry[] | null`.
- Dispatcher in `provenance-backfill.ts`: `readScannedListingRanged(archiveType, client, parts)`
  replacing the current `if (archiveType === "ZIP")` branch; the destination-copy read in
  `resolveCandidateFingerprintEntries` gets the same dispatch.

`FileEntry` shape (unchanged): `{ path, fileName, extension, compressedSize, uncompressedSize, crc32 }`.

### 7z ranged listing

7z layout: 32-byte signature header at offset 0 → packed streams → end header (lists files) at
the end; the signature header stores the end header's location.

1. Ranged-read `[0, 32)`; validate magic `37 7A BC AF 27 1C`. Read LE `uint64`
   `NextHeaderOffset` (byte 12) and `NextHeaderSize` (byte 20). End header is at absolute offset
   `32 + NextHeaderOffset`, length `NextHeaderSize`.
2. Ranged-read `[32 + NextHeaderOffset, NextHeaderSize)` — the "next header".
3. Branch on the next header's first byte (a 7z property id):
   - **`0x01` (kHeader, plain/uncompressed header):** two regions suffice —
     `{0: sigHeader}` and `{32+NextHeaderOffset: endHeader}`.
   - **`0x17` (kEncodedHeader, LZMA-compressed header):** the next header is only a *descriptor*
     whose `PackInfo` points at a packed header stream stored **in the middle** of the file (not
     at EOF). Parse the descriptor's `StreamsInfo → kPackInfo (0x06)` to read `PackPos` and the
     `PackSize`s (7z variable-length "numbers"; sum them). Ranged-read the contiguous packed
     region `[32 + PackPos, Σ PackSize)` and add it as a **third** sparse region. `7z l` then
     decodes the header from that region.
   - **anything else:** return `null` (→ fallback).
4. `listFromSparse` with the 2 or 3 regions, runner = `7z l`. `parse7zOutput` yields
   names+sizes (`crc32: null`, as today). Return `null` on bad magic / read failure / CLI error.

**Why the third region is required (spike finding, 2026-07-27):** the original two-region
(start+end) reconstruction was proven insufficient in a live test — `7z l` rejected it with
"Cannot open the file as [7z] archive" because these archives use an *encoded* header whose
compressed bytes live in a packed stream in the file body (a sparse hole), not at EOF. The
`0x17` branch fetches exactly that packed region. `7z l` still never touches the file-data
packed streams (it only lists), so those gaps stay sparse. The `read7zNumber` reader (7z's
base-128-ish variable-length integer with a first-byte length mask) and the minimal
`kPackInfo` walk are the only new 7z binary parsing; `7z l` still does the actual file listing.
All 7z placeholders are single-part.

### RAR ranged listing

RAR has no index; walk the block chain, parsing only each block's **size fields** to step
forward and harvest header bytes.

1. Read first ~16 bytes; detect **RAR4** (`52 61 72 21 1A 07 00`) vs **RAR5** (`…07 01 00`) and
   the signature length.
2. From just after the signature, loop:
   - Ranged-read a header chunk (start 8 KB; if parsed `HeaderSize` exceeds it — long filenames
     — re-read exactly).
   - Minimal block-extent parse:
     - RAR5: `CRC32(4)` + vint `HeaderSize` + vint `HeaderType` + vint `HeaderFlags`; if the
       "extra area" flag (`0x0001`) → vint `ExtraAreaSize`; if the "data present" flag
       (`0x0002`) → vint `DataSize`. Next block = `pos + 4 + len(HeaderSize vint) + HeaderSize
       + DataSize`.
     - RAR4: `HEAD_CRC(2)` + `HEAD_TYPE(1)` + `HEAD_FLAGS(2)` + `HEAD_SIZE(2)`; if flag `0x8000`
       → `ADD_SIZE(4)`. Next block = `pos + HEAD_SIZE + ADD_SIZE`.
   - Harvest `[blockOffset, blockOffset + HeaderSize)` into the regions list.
   - Stop at the end-of-archive block or EOF.
3. `listFromSparse` (headers present, data sparse) → `unrar lt` → `parseUnrarTechnical`. RAR
   headers carry CRC32, so RAR contributes CRCs (fingerprint disambiguation keeps working).

**Multipart RAR** (1,016): each volume starts with its own signature + headers. Walk **each
part from its own signature**, reconstruct one sparse temp file per part with correct names
(`name.part1.rar`, `.part2.rar`, …) co-located in a temp dir, and run `unrar lt` on part 1 —
`unrar` auto-discovers co-located siblings (per the existing reader's note). The global-offset →
`(part, offsetInPart)` mapping reuses the multipart size math the ZIP path already uses.

## Fallback & integration

- A ranged reader returning `null` = cheap read failed (bad magic, read error, walk gave up, or
  CLI error on the sparse file) → **full-download fallback**: download the whole archive, run the
  existing `readRarContents` / `read7zContents`, backfill.
- The fallback is gated by `config.maxZipSizeMB` (the same guard used at ingest). Over the cap →
  no download; write a `SystemNotification` (`INTEGRITY_AUDIT`, WARNING) and leave the listing
  empty for manual review. This ensures nothing pathological (e.g. the 116 GB RAR) is pulled.
- Downstream is unchanged: `compareFingerprints` already treats null/incomplete CRCs as
  "incomplete" (name-size path), and `backfillProvenance` writes entries when the candidate's
  `fileCount === 0`.
- Observability: reuse the `zipsBackfilled` counter; add structured logs with
  `confidence: "ranged" | "full-download-fallback"` and a WARN on fallback so miss-rate is
  visible.

## Risks & de-risking spike (do before the full build)

On 3–4 real placeholder archives per format:

1. Confirm `downloadFileRange` returns correct bytes at **arbitrary (non-tail) offsets** —
   currently only tail-verified in production. Underpins everything; if it fails, stop and
   rethink. (Note: `range-download.ts` flags absolute-offset behaviour as pending live
   verification; tail reads are proven by the 43 ZIP backfills done 2026-07-26.)
2. Confirm `7z l` and `unrar lt` list correctly from a **sparse reconstructed file** — single
   part first, then multipart RAR (the highest-risk case).

If multipart-RAR sparse reconstruction proves unreliable in the spike, multipart RAR uses the
full-download fallback (respecting the size cap → oversized ones flagged, not downloaded).

## Testing

- **Unit (vitest, alongside `central-directory.test.ts`):** 7z signature-header parse; RAR4 &
  RAR5 block-extent walk against committed small fixtures; `sparse-list` writes the correct
  regions. Pure logic, no TDLib.
- **Live post-deploy:** watch `zipsBackfilled` climb for RAR/7z via the ranged path; spot-check
  a handful of backfilled packages' `package_files` against a real `unrar lt` / `7z l` on a full
  download of the same file; confirm the fallback/flag path fires on a deliberately-broken case.

## Rollout

Local build + deploy (no GitHub push required), per the established recipe: build
`worker/Dockerfile` locally, recreate the `dragonsstash-worker` container from the local image
(no `pull`). No new DB migration. The scheduler re-runs hourly and will backfill RAR/7z
placeholders on subsequent cycles.