addd TG integration

2026-05-11 06:11:15 +00:00 · 2026-03-02 11:57:17 +01:00
parent b427193d17
commit 4d0df6b1a4
35 changed files with 4436 additions and 242 deletions
--- a/README.md
+++ b/README.md
@@ -1,9 +1,11 @@
 # Dragon's Stash

-A self-hosted inventory management system for 3D printing filament, SLA resin, and miniature paints. Built with a dark, data-dense UI inspired by [Spoolman](https://github.com/Donkie/Spoolman).
+A self-hosted inventory management system for 3D printing filament, SLA resin, and miniature paints — with an integrated Telegram archive worker that ingests, indexes, and redistributes archive files. Built with a dark, data-dense UI inspired by [Spoolman](https://github.com/Donkie/Spoolman).

 ## Features

+### Inventory Management
+
 - **Filament tracking** with spool weight, material type, color swatches, and usage logging
 - **SLA resin management** with bottle sizes, resin types, and remaining volume tracking
 - **Miniature paint inventory** with product lines, finishes, and volume tracking
@@ -13,7 +15,18 @@ A self-hosted inventory management system for 3D printing filament, SLA resin, a
 - **Low-stock alerts** with configurable threshold percentage
 - **Dark theme** optimized for workshop environments
 - **Role-based auth** with admin and user roles
- **Docker-ready** for easy self-hosting
+
+### Telegram Archive Worker
+
+- **Channel scanning** — monitors configured Telegram channels (including forum topics) for archive files (ZIP, RAR, 7z)
+- **Multipart detection** — automatically groups related multipart archives (`.part01.rar`, `.z01`, `.001`, etc.)
+- **Content indexing** — extracts file listings from archives and stores them in the database
+- **Destination upload** — re-uploads processed archives to a configured destination channel
+- **Byte-level splitting** — splits files exceeding Telegram's 2GB limit into uploadable chunks
+- **Full repack** — concatenates and re-splits multipart sets where any single part exceeds 2GB
+- **Progress tracking** — resumes from the last successfully processed message on each run
+- **Upload verification** — confirms files reached the destination before marking them complete
+- **Preview matching** — associates photo messages with their corresponding archive sets

 ## Tech Stack

@@ -24,6 +37,8 @@ A self-hosted inventory management system for 3D printing filament, SLA resin, a
 - **UI**: Tailwind CSS, shadcn/ui, Lucide icons
 - **Tables**: TanStack Table v8 with server-side pagination
 - **Validation**: Zod v4 + React Hook Form
+- **Worker**: Node.js + TDLib (via tdl)
+- **Archive handling**: unrar, zlib

 ## Quick Start

@@ -31,6 +46,7 @@ A self-hosted inventory management system for 3D printing filament, SLA resin, a

 - Node.js 20+
 - PostgreSQL 16+ (or Docker)
+- Telegram API credentials (for the worker — get from [my.telegram.org/apps](https://my.telegram.org/apps))

 ### Development Setup

@@ -50,7 +66,7 @@ npm install
 3. Start a PostgreSQL database (using Docker):

 ```bash
-docker compose -f docker-compose.dev.yml up -d
+docker compose -f docker-compose.dev.yml up -d db
 ```

 4. Copy the environment file and update values:
@@ -62,8 +78,8 @@ cp .env.example .env.local
 5. Run database migrations and seed:

 ```bash
-npx prisma migrate dev
-npx prisma db seed
+npx prisma migrate dev     # Run migrations
+npx prisma db seed         # Seed with sample data (admin/user accounts + inventory)
 ```

 6. Start the development server:
@@ -76,20 +92,75 @@ npm run dev
   - **Admin**: admin@dragonsstash.local / password123
   - **User**: user@dragonsstash.local / password123

-### Docker Deployment
+### Running the Worker in Development
+
+To also run the Telegram worker alongside the dev database:

 ```bash
+docker compose -f docker-compose.dev.yml up -d
+```
+
+This starts both the PostgreSQL database and the worker container. The worker reads `TELEGRAM_API_ID` and `TELEGRAM_API_HASH` from your `.env.local` file.
+
+## Docker Deployment
+
+### Full Stack (App + Worker + Database)
+
+Run the entire application from Docker:
+
+```bash
+cp .env.example .env
+# Edit .env — set TELEGRAM_API_ID, TELEGRAM_API_HASH, and a secure AUTH_SECRET
 docker compose up -d
 ```

-This starts both the application and PostgreSQL database. The app will be available at `http://localhost:3000`.
+The app will be available at [http://localhost:3000](http://localhost:3000).

-To seed the database on first run:
+### Seeding the Database
+
+To seed the database with sample data on first run:

 ```bash
 SEED_DATABASE=true docker compose up -d
 ```

+This creates default admin/user accounts and sample inventory data. The seed runs once during the app container's entrypoint (before the Next.js server starts). On subsequent runs without `SEED_DATABASE=true`, seeding is skipped automatically.
+
+You can also seed manually at any time:
+
+```bash
+npx prisma db seed
+```
+
+### Development Mode (DB + Worker Only)
+
+If you prefer to run the Next.js app locally with hot reload:
+
+```bash
+docker compose -f docker-compose.dev.yml up -d   # Start DB + worker
+npm run dev                                        # Start Next.js locally
+```
+
+### Rebuilding After Code Changes
+
+```bash
+docker compose build && docker compose up -d --force-recreate
+```
+
+To rebuild only the worker:
+
+```bash
+docker compose build worker && docker compose up -d worker --force-recreate
+```
+
+### Viewing Logs
+
+```bash
+docker compose logs -f worker   # Worker logs
+docker compose logs -f app      # App logs
+docker compose logs -f db       # Database logs
+```
+
 ## Project Structure

 ```
@@ -116,6 +187,16 @@ src/
  lib/               # Auth config, Prisma client, constants
  schemas/           # Zod validation schemas
  types/             # TypeScript type definitions
+worker/
+  src/
+    archive/         # Archive detection, multipart grouping, byte-level splitting
+    db/              # Prisma queries for packages, progress tracking
+    preview/         # Preview image matching
+    tdlib/           # TDLib client, channel scanning, topic/forum handling
+    upload/          # Telegram upload logic
+    util/            # Config, logger
+    worker.ts        # Main processing pipeline
+    index.ts         # Entry point + scheduler
 prisma/
  schema.prisma      # Database schema
  seed.ts            # Seed data
@@ -125,6 +206,8 @@ prisma/

 Environment variables (see `.env.example`):

+### Application
+
 | Variable | Description | Default |
 |----------|-------------|---------|
 | `DATABASE_URL` | PostgreSQL connection string | Required |
@@ -133,6 +216,20 @@ Environment variables (see `.env.example`):
 | `AUTH_GITHUB_ID` | GitHub OAuth client ID | Optional |
 | `AUTH_GITHUB_SECRET` | GitHub OAuth client secret | Optional |
 | `NEXT_PUBLIC_APP_URL` | Public application URL | `http://localhost:3000` |
+| `SEED_DATABASE` | Seed the database on app container start | `false` |
+
+### Telegram Worker
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `TELEGRAM_API_ID` | Telegram API ID (from [my.telegram.org](https://my.telegram.org/apps)) | Required |
+| `TELEGRAM_API_HASH` | Telegram API hash | Required |
+| `WORKER_INTERVAL_MINUTES` | Scan interval in minutes | `60` |
+| `WORKER_TEMP_DIR` | Temp directory for downloads | `/tmp/zips` |
+| `TDLIB_STATE_DIR` | TDLib session state persistence directory | `/data/tdlib` |
+| `WORKER_MAX_ZIP_SIZE_MB` | Max archive size to process (MB) | `4096` |
+| `MULTIPART_TIMEOUT_HOURS` | Max time span for multipart set parts (0 = no limit) | `0` |
+| `LOG_LEVEL` | Worker log level (`debug`, `info`, `warn`, `error`) | `info` |

 ## Health Check