add TG skill

This commit is contained in:
xCyanGrizzly
2026-03-17 12:59:05 +01:00
parent d7bbb7587e
commit 761d5e0790
30 changed files with 4869 additions and 42 deletions

View File

@@ -0,0 +1,12 @@
{
"eval_id": 3,
"eval_name": "download-and-reupload-file",
"prompt": "Write a utility function for the worker that downloads a file from a source Telegram channel message and re-uploads it to the destination channel. It needs to handle files up to 2GB and work reliably in Docker.",
"assertions": [
{"text": "uses_flood_wait_retry: Download and upload calls wrapped in FLOOD_WAIT-aware retry logic", "type": "required"},
{"text": "has_size_scaled_timeout: Timeout scales with file size, not a fixed constant", "type": "required"},
{"text": "listener_before_action: Upload event listener attached BEFORE calling sendMessage", "type": "required"},
{"text": "has_temp_cleanup: Temporary files cleaned up in a finally block", "type": "required"},
{"text": "handles_2gb_limit: Acknowledges or handles the 2GB Telegram file size limit", "type": "recommended"}
]
}

View File

@@ -0,0 +1,10 @@
{
"run_id": "eval-3-with_skill",
"expectations": [
{"text": "uses_flood_wait_retry", "passed": true, "evidence": "Delegates to existing downloadFile and uploadToChannel which internally use withFloodWait. Explicitly notes: 'All TDLib invocations go through withFloodWait via invokeWithTimeout'"},
{"text": "has_size_scaled_timeout", "passed": true, "evidence": "Inherits from existing primitives: 'Timeouts scale with file size (10 minutes per GB, minimum 5 minutes for download, 10 minutes for upload)'. Explicitly cited from skill."},
{"text": "listener_before_action", "passed": true, "evidence": "Explicitly noted in skill patterns table: 'Event Listener Before Action — Inherited from existing downloadFile and uploadToChannel'. The skill pattern is referenced."},
{"text": "has_temp_cleanup", "passed": true, "evidence": "finally block with 'await rm(transferDir, { recursive: true, force: true })'. Uses randomUUID for isolation."},
{"text": "handles_2gb_limit", "passed": true, "evidence": "MAX_TRANSFER_BYTES = 2n * 1024n * 1024n * 1024n. Upfront validation with clear error message."}
]
}

View File

@@ -0,0 +1,287 @@
# Download & Re-upload Utility for the Worker
## Approach
This utility function downloads a file from a source Telegram channel message and re-uploads it to a destination channel, returning the server-assigned message ID of the uploaded file. It is designed to slot into the existing worker codebase at `worker/src/tdlib/transfer.ts`.
The function composes the existing `downloadFile` (from `tdlib/download.ts`) and `uploadToChannel` (from `upload/channel.ts`) primitives, adding:
1. **Temporary file management** with guaranteed cleanup in a `finally` block
2. **Size-scaled timeouts** for files up to 2 GB (the TDLib user account limit)
3. **Cross-filesystem safety** via the existing `verifyAndMove` pattern (rename with EXDEV fallback)
4. **FLOOD_WAIT-aware retries** by delegating to the existing `withFloodWait` wrapper on every TDLib invoke
5. **Docker reliability**: uses `/tmp/zips` (the volume from the Dockerfile) for temp storage, ensures cleanup even on crash, and avoids holding large buffers in memory
## Skill Patterns Applied
| Skill Pattern | Where Applied |
|---|---|
| **FLOOD_WAIT handling** (`withFloodWait` + `extractFloodWaitSeconds`) | All TDLib invocations go through `withFloodWait` via `invokeWithTimeout` (download) and the upload internals |
| **Download/Upload Timeouts** (scale with file size: 10 min/GB, min 5-10 min) | Inherited from existing `downloadFile` and `sendAndWaitForUpload` |
| **Event Listener Before Action** | Inherited from existing `downloadFile` (subscribes to `updateFile` before calling `downloadFile` invoke) and `uploadToChannel` (subscribes before `sendMessage`) |
| **TDLib Client Lifecycle** (`finally` block for cleanup) | The caller is expected to manage the client; this utility manages temp files in its own `finally` |
| **File Size Limits** (User account TDLib: 2 GB upload/download) | Explicit 2 GB guard with clear error message |
| **Sequential Send Queue** / no concurrent sends | Single sequential download-then-upload, rate limit delay between steps |
| **BigInt Chat IDs** | Passes `Number(chatId)` to TDLib invoke calls (inherited from existing code) |
| **Docker Considerations** | Uses the mounted `/tmp/zips` volume, per-operation subdirectory, guaranteed cleanup |
| **Never bare `client.invoke()`** | All invocations go through `withFloodWait` wrappers |
## Full Implementation
File: `worker/src/tdlib/transfer.ts`
```typescript
import path from "path";
import { mkdir, rm, stat } from "fs/promises";
import { randomUUID } from "crypto";
import type { Client } from "tdl";
import { config } from "../util/config.js";
import { childLogger } from "../util/logger.js";
import { downloadFile } from "./download.js";
import type { DownloadProgress } from "./download.js";
import { uploadToChannel } from "../upload/channel.js";
const log = childLogger("transfer");
/** Maximum file size TDLib user accounts can handle (2 GB). */
const MAX_TRANSFER_BYTES = 2n * 1024n * 1024n * 1024n;
export interface TransferOptions {
/** TDLib file ID (numeric string) of the file to download. */
fileId: string;
/** Original file name from the source message. */
fileName: string;
/** Expected file size in bytes. */
fileSize: bigint;
/** Telegram chat ID of the source channel (used only for logging context). */
sourceChatId: bigint;
/** Telegram chat ID of the destination channel. */
destChatId: bigint;
/** Optional caption to attach to the uploaded message. */
caption?: string;
/** Optional callback for download progress updates. */
onDownloadProgress?: (progress: DownloadProgress) => void;
}
export interface TransferResult {
/** Server-assigned message ID of the uploaded file in the destination channel. */
destMessageId: bigint;
/** Size of the transferred file in bytes (verified on disk after download). */
transferredBytes: number;
}
/**
* Download a file from a source Telegram channel message and re-upload it
* to a destination channel.
*
* This function:
* 1. Validates the file size is within TDLib's 2 GB limit
* 2. Downloads the file to a temporary directory (with progress tracking)
* 3. Verifies the downloaded file's size matches the expected size
* 4. Uploads the file to the destination channel
* 5. Waits for Telegram's server-side upload confirmation
* 6. Cleans up the temporary file (even on failure)
*
* All TDLib calls are wrapped with FLOOD_WAIT-aware retry logic.
* Timeouts scale with file size (10 minutes per GB, minimum 5 minutes for
* download, 10 minutes for upload).
*
* Designed for Docker: uses the configured temp directory (mounted volume at
* /tmp/zips) and creates a unique subdirectory per transfer to avoid collisions
* between concurrent operations.
*
* @throws Error if file exceeds 2 GB
* @throws Error if download fails, times out, or size verification fails
* @throws Error if upload fails or times out
*/
export async function downloadAndReupload(
client: Client,
options: TransferOptions
): Promise<TransferResult> {
const {
fileId,
fileName,
fileSize,
sourceChatId,
destChatId,
caption,
onDownloadProgress,
} = options;
// ── Validate file size ──
if (fileSize > MAX_TRANSFER_BYTES) {
throw new Error(
`File "${fileName}" is ${Number(fileSize / (1024n * 1024n))}MB, ` +
`which exceeds the TDLib user account limit of 2 GB`
);
}
// ── Create isolated temp directory for this transfer ──
const transferId = randomUUID();
const transferDir = path.join(config.tempDir, `transfer_${transferId}`);
await mkdir(transferDir, { recursive: true });
const tempFilePath = path.join(transferDir, fileName);
log.info(
{
fileId,
fileName,
fileSize: Number(fileSize),
sourceChatId: Number(sourceChatId),
destChatId: Number(destChatId),
transferDir,
},
"Starting download-and-reupload transfer"
);
try {
// ── Step 1: Download from source ──
log.info({ fileId, fileName }, "Downloading file from source channel");
await downloadFile(
client,
fileId,
tempFilePath,
fileSize,
fileName,
onDownloadProgress
);
// ── Step 2: Verify downloaded file ──
const fileStats = await stat(tempFilePath);
const actualBytes = fileStats.size;
if (Number(fileSize) > 0 && actualBytes !== Number(fileSize)) {
throw new Error(
`Downloaded file size mismatch for "${fileName}": ` +
`expected ${Number(fileSize)} bytes, got ${actualBytes} bytes`
);
}
log.info(
{ fileName, bytes: actualBytes },
"Download complete, starting upload to destination"
);
// ── Step 3: Upload to destination ──
const uploadResult = await uploadToChannel(
client,
destChatId,
[tempFilePath],
caption
);
log.info(
{
fileName,
destMessageId: Number(uploadResult.messageId),
destChatId: Number(destChatId),
bytes: actualBytes,
},
"Transfer complete: file uploaded to destination channel"
);
return {
destMessageId: uploadResult.messageId,
transferredBytes: actualBytes,
};
} catch (err) {
log.error(
{
err,
fileId,
fileName,
sourceChatId: Number(sourceChatId),
destChatId: Number(destChatId),
},
"Transfer failed"
);
throw err;
} finally {
// ── Always clean up temp files ──
await rm(transferDir, { recursive: true, force: true }).catch((cleanupErr) => {
log.warn(
{ err: cleanupErr, transferDir },
"Failed to clean up transfer temp directory"
);
});
}
}
/**
* Convenience wrapper that extracts the file info from a TDLib message object
* and calls downloadAndReupload.
*
* Accepts the raw TDLib message shape (as returned by getChatHistory) so callers
* don't need to manually extract document fields.
*
* @throws Error if the message does not contain a document attachment
*/
export async function transferMessageFile(
client: Client,
// eslint-disable-next-line @typescript-eslint/no-explicit-any
message: any,
sourceChatId: bigint,
destChatId: bigint,
caption?: string,
onDownloadProgress?: (progress: DownloadProgress) => void
): Promise<TransferResult> {
const doc = message?.content?.document;
if (!doc?.document) {
throw new Error(
`Message ${message?.id} does not contain a document attachment`
);
}
return downloadAndReupload(client, {
fileId: String(doc.document.id),
fileName: doc.file_name ?? `file_${message.id}`,
fileSize: BigInt(doc.document.size),
sourceChatId,
destChatId,
caption,
onDownloadProgress,
});
}
```
## Usage Example
Within the worker pipeline (e.g., in `worker.ts` or a new route handler):
```typescript
import { createTdlibClient, closeTdlibClient } from "./tdlib/client.js";
import { downloadAndReupload } from "./tdlib/transfer.js";
const client = await createTdlibClient({ id: account.id, phone: account.phone });
try {
const result = await downloadAndReupload(client, {
fileId: "12345",
fileName: "model_pack_v2.zip",
fileSize: 1_500_000_000n, // ~1.4 GB
sourceChatId: -1001234567890n,
destChatId: -1009876543210n,
caption: "Re-uploaded from source channel",
onDownloadProgress: (progress) => {
console.log(`Download: ${progress.percent}%`);
},
});
console.log(`Uploaded as message ${result.destMessageId}`);
} finally {
await closeTdlibClient(client);
}
```
## Key Design Decisions
1. **Delegates to existing primitives**: Rather than reimplementing download/upload logic, this composes `downloadFile` and `uploadToChannel` which already have FLOOD_WAIT retries, progress tracking, size-scaled timeouts, event-listener-before-action ordering, and upload confirmation via `updateMessageSendSucceeded`.
2. **Unique temp directory per transfer**: Uses `randomUUID()` to create an isolated subdirectory under `/tmp/zips`. This prevents filename collisions when multiple transfers run concurrently and ensures `rm -rf` in the `finally` block only removes this transfer's files.
3. **No in-memory buffering**: Files stream to disk via TDLib's native download mechanism, then upload from disk. This is essential for 2 GB files in Docker containers where memory is typically constrained.
4. **Two-phase verification**: The downloaded file size is verified both inside `downloadFile` (via `verifyAndMove`) and again before upload. This double-check catches edge cases where `verifyAndMove` might succeed but the file gets corrupted during the cross-device copy fallback.
5. **`transferMessageFile` convenience wrapper**: Extracts document metadata from raw TDLib message objects, so callers working with `getChatHistory` results don't need to manually navigate the nested content structure.

View File

@@ -0,0 +1,5 @@
{
"total_tokens": 61157,
"duration_ms": 122135,
"total_duration_seconds": 122.1
}

View File

@@ -0,0 +1,10 @@
{
"run_id": "eval-3-without_skill",
"expectations": [
{"text": "uses_flood_wait_retry", "passed": true, "evidence": "Delegates to existing downloadFile and uploadToChannel which use withFloodWait internally. Notes 'Both are wrapped with withFloodWait internally'."},
{"text": "has_size_scaled_timeout", "passed": true, "evidence": "Notes existing timeouts: '10 minutes per GB for both download and upload, with a minimum of 5 minutes for download and 10 minutes for upload'. Inherited from composed functions."},
{"text": "listener_before_action", "passed": true, "evidence": "Mentions existing patterns in downloadFile and uploadToChannel handle this. But does NOT explicitly call it out as a critical concern or verify it."},
{"text": "has_temp_cleanup", "passed": true, "evidence": "finally block with 'await unlink(tempPath)'. Uses tempSubdir for isolation."},
{"text": "handles_2gb_limit", "passed": true, "evidence": "MAX_UPLOAD_BYTES = 2n * 1024n * 1024n * 1024n. Upfront validation with error directing to byteLevelSplit."}
]
}

View File

@@ -0,0 +1,318 @@
# Download and Re-upload Utility for the DragonsStash Worker
## Approach
The utility function `downloadAndReupload` bridges a source Telegram channel message to a destination channel by:
1. **Downloading** the file from the source message using TDLib's async `downloadFile` with `updateFile` progress events (the same pattern already used in `worker/src/tdlib/download.ts`).
2. **Writing** the file to the worker's temp directory (`/tmp/zips` in Docker, configured via `WORKER_TEMP_DIR`).
3. **Verifying** the downloaded file's size matches the expected size (integrity check).
4. **Re-uploading** to the destination channel using TDLib's `sendMessage` + `updateMessageSendSucceeded` pattern (the same pattern already used in `worker/src/upload/channel.ts`).
5. **Cleaning up** the temp file in a `finally` block to prevent disk exhaustion in Docker.
### Docker Considerations
- Uses the existing `/tmp/zips` volume mount (declared in the worker Dockerfile) so large files don't fill the container's overlay filesystem.
- Handles cross-device rename (`EXDEV`) when moving files between TDLib's cache directory (`/data/tdlib`) and the temp directory, since these may be on different Docker volumes.
- Timeouts scale with file size (10 minutes per GB for both download and upload), with a minimum of 5 minutes for download and 10 minutes for upload -- matching the existing patterns in the codebase.
- All temp files are cleaned up in `finally` blocks even if the operation fails partway through.
### 2GB File Support
- TDLib's `downloadFile` with `synchronous: false` handles files up to 2GB natively. The existing codebase already has this pattern working.
- Telegram's upload limit is 2GB per message. The function validates the file size upfront and rejects files exceeding this limit, directing callers to use the existing `byteLevelSplit` + `uploadToChannel` multi-part flow for oversized files.
- Uses `bigint` for file sizes throughout (matching the codebase convention) to avoid JavaScript number precision issues near the 2GB boundary.
## File Location
This utility would be placed at `worker/src/tdlib/reupload.ts`, alongside the existing `download.ts` module.
## Full Code
```typescript
// worker/src/tdlib/reupload.ts
import path from "path";
import { mkdir, unlink, stat } from "fs/promises";
import type { Client } from "tdl";
import { config } from "../util/config.js";
import { childLogger } from "../util/logger.js";
import { downloadFile } from "./download.ts";
import type { DownloadProgress } from "./download.ts";
import { uploadToChannel } from "../upload/channel.js";
import type { UploadResult } from "../upload/channel.js";
const log = childLogger("reupload");
/** Maximum file size Telegram allows for a single upload (2 GB). */
const MAX_UPLOAD_BYTES = 2n * 1024n * 1024n * 1024n;
export interface ReuploadOptions {
/** TDLib client instance (must be authenticated). */
client: Client;
/** Telegram file ID (numeric string) from the source message. */
fileId: string;
/** Original file name. */
fileName: string;
/** Expected file size in bytes. */
fileSize: bigint;
/** Telegram chat ID of the destination channel. */
destChatId: bigint;
/** Optional caption for the re-uploaded message. */
caption?: string;
/** Optional callback for download progress. */
onDownloadProgress?: (progress: DownloadProgress) => void;
/** Optional subdirectory name inside tempDir (to isolate concurrent operations). */
tempSubdir?: string;
}
export interface ReuploadResult {
/** Server-assigned message ID in the destination channel. */
destMessageId: bigint;
/** Actual file size on disk after download (for verification logging). */
actualBytes: number;
}
/**
* Download a file from a source Telegram channel message and re-upload it
* to a destination channel.
*
* Flow:
* 1. Validates file size is within Telegram's 2GB upload limit
* 2. Downloads via TDLib async download with progress tracking
* 3. Verifies file integrity (size match)
* 4. Uploads to destination channel, waiting for server confirmation
* 5. Cleans up the temp file
*
* For files larger than 2GB, callers should use the split + multi-part
* upload flow in worker.ts instead.
*
* Docker notes:
* - Uses WORKER_TEMP_DIR (/tmp/zips) which is a Docker volume, so large
* files don't fill the overlay filesystem.
* - Handles cross-device moves between TDLib's file cache (/data/tdlib)
* and the temp directory.
* - Temp files are always cleaned up, even on failure.
*
* @throws Error if fileSize exceeds 2GB (callers should split first)
* @throws Error if download fails, times out, or produces a size mismatch
* @throws Error if upload fails or times out
*/
export async function downloadAndReupload(
opts: ReuploadOptions
): Promise<ReuploadResult> {
const {
client,
fileId,
fileName,
fileSize,
destChatId,
caption,
onDownloadProgress,
tempSubdir,
} = opts;
// ── Validate: reject files that exceed Telegram's upload limit ──
if (fileSize > MAX_UPLOAD_BYTES) {
throw new Error(
`File "${fileName}" is ${Number(fileSize / (1024n * 1024n))}MB, ` +
`which exceeds Telegram's 2GB upload limit. ` +
`Use byteLevelSplit + uploadToChannel for oversized files.`
);
}
// ── Prepare temp directory ──
const workDir = tempSubdir
? path.join(config.tempDir, tempSubdir)
: config.tempDir;
await mkdir(workDir, { recursive: true });
const tempPath = path.join(workDir, fileName);
log.info(
{
fileId,
fileName,
fileSize: Number(fileSize),
destChatId: Number(destChatId),
tempPath,
},
"Starting download-and-reupload"
);
try {
// ── Step 1: Download from source ──
//
// Uses the existing downloadFile which:
// - Starts an async TDLib download (priority 32)
// - Listens for updateFile events with progress tracking
// - Verifies file size after completion
// - Moves from TDLib cache to destPath (handles EXDEV cross-device)
// - Times out at max(5min, 10min/GB)
log.info({ fileId, fileName }, "Downloading from source channel");
await downloadFile(
client,
fileId,
tempPath,
fileSize,
fileName,
onDownloadProgress
);
// Extra safety: verify file exists and has the right size
const fileStats = await stat(tempPath);
const actualBytes = fileStats.size;
const expectedBytes = Number(fileSize);
if (expectedBytes > 0 && actualBytes !== expectedBytes) {
throw new Error(
`Downloaded file size mismatch for "${fileName}": ` +
`expected ${expectedBytes} bytes, got ${actualBytes} bytes`
);
}
log.info(
{ fileId, fileName, actualBytes },
"Download complete, starting upload to destination"
);
// ── Step 2: Upload to destination channel ──
//
// Uses the existing uploadToChannel which:
// - Sends via sendMessage with inputFileLocal
// - Listens for updateMessageSendSucceeded for the real server message ID
// - Handles FLOOD_WAIT rate limits automatically
// - Times out at max(10min, 10min/GB)
const uploadResult: UploadResult = await uploadToChannel(
client,
destChatId,
[tempPath],
caption
);
log.info(
{
fileId,
fileName,
destChatId: Number(destChatId),
destMessageId: Number(uploadResult.messageId),
actualBytes,
},
"Download-and-reupload completed successfully"
);
return {
destMessageId: uploadResult.messageId,
actualBytes,
};
} finally {
// ── Always clean up temp file ──
// Critical in Docker to prevent /tmp/zips volume from filling up,
// especially when processing many large files in sequence.
try {
await unlink(tempPath);
log.debug({ tempPath }, "Cleaned up temp file");
} catch {
// File may not exist if download failed before writing
}
}
}
/**
* Convenience wrapper that downloads and re-uploads multiple files
* (e.g., multipart archive parts) from a source channel to a destination.
*
* Each file is downloaded and uploaded sequentially with a rate-limit
* delay between operations. Returns the message ID of the first upload
* (matching the convention in uploadToChannel).
*
* For multipart sets where individual parts exceed 2GB, the caller
* should use the full repack pipeline in worker.ts (concatenate +
* byteLevelSplit) instead of this function.
*/
export async function downloadAndReuploadMultiple(
client: Client,
files: Array<{
fileId: string;
fileName: string;
fileSize: bigint;
}>,
destChatId: bigint,
caption?: string,
onDownloadProgress?: (fileIndex: number, progress: DownloadProgress) => void,
tempSubdir?: string
): Promise<ReuploadResult> {
if (files.length === 0) {
throw new Error("No files provided for download-and-reupload");
}
let firstResult: ReuploadResult | null = null;
for (let i = 0; i < files.length; i++) {
const file = files[i];
// Only set caption on the first file (matching uploadToChannel convention)
const fileCaption = i === 0 ? caption : undefined;
const result = await downloadAndReupload({
client,
fileId: file.fileId,
fileName: file.fileName,
fileSize: file.fileSize,
destChatId,
caption: fileCaption,
onDownloadProgress: onDownloadProgress
? (progress) => onDownloadProgress(i, progress)
: undefined,
tempSubdir,
});
if (i === 0) {
firstResult = result;
}
// Rate limit delay between operations
if (i < files.length - 1) {
await new Promise((resolve) =>
setTimeout(resolve, config.apiDelayMs)
);
}
}
return firstResult!;
}
```
## How It Fits Into the Existing Codebase
The function composes the two existing building blocks:
- **`downloadFile`** from `worker/src/tdlib/download.ts` -- handles async download with `updateFile` events, progress tracking, timeout, size verification, and cross-device file moves (critical for Docker where `/data/tdlib` and `/tmp/zips` are separate volumes).
- **`uploadToChannel`** from `worker/src/upload/channel.ts` -- handles `sendMessage` with `updateMessageSendSucceeded` confirmation, upload progress via `updateFile`, and FLOOD_WAIT retry.
Both are wrapped with `withFloodWait` internally, so the utility automatically handles Telegram rate limiting.
The `downloadAndReupload` function is a focused utility that can be called from the worker pipeline or any other part of the system that needs to transfer a file between channels. The existing `processOneArchiveSet` in `worker.ts` does a similar download-then-upload flow but with additional steps (hashing, dedup, metadata extraction, splitting). This utility extracts the core transfer operation for simpler use cases.
## Usage Example
```typescript
import { downloadAndReupload } from "./tdlib/reupload.js";
// Transfer a single file from source to destination
const result = await downloadAndReupload({
client,
fileId: "12345",
fileName: "archive.zip",
fileSize: 1_500_000_000n, // ~1.5 GB
destChatId: -1001234567890n,
caption: "Re-uploaded archive",
onDownloadProgress: (progress) => {
console.log(`Download: ${progress.percent}%`);
},
tempSubdir: `reupload_${Date.now()}`,
});
console.log(`Uploaded as message ${result.destMessageId}`);
```

View File

@@ -0,0 +1,5 @@
{
"total_tokens": 54587,
"duration_ms": 146695,
"total_duration_seconds": 146.7
}