mirror of
https://github.com/xCyanGrizzly/DragonsStash.git
synced 2026-07-24 07:42:47 +00:00
fix: prevent pool exhaustion that caused 4-hour duplicate check stall
All checks were successful
continuous-integration/drone/push Build is passing
All checks were successful
continuous-integration/drone/push Build is passing
The pg pool had max=5 connections shared between Prisma operations and advisory locks. With 2 account locks held permanently and hash locks from timed-out (but still running) background work, pool.connect() would block forever — causing the Turnbase.7z stall. - Increase pool max from 5 to 15 for headroom - Add 30s connectionTimeoutMillis so pool.connect() throws instead of hanging forever when the pool is exhausted - On startup, terminate zombie PostgreSQL sessions from previous worker instances that hold stale advisory locks Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -27,6 +27,33 @@ async function main(): Promise<void> {
|
||||
await cleanupTempDir();
|
||||
await markStaleRunsAsFailed();
|
||||
|
||||
// Release any advisory locks orphaned by a previous worker instance.
|
||||
// When Docker kills a container, PostgreSQL may keep the session alive
|
||||
// (zombie connections), holding advisory locks that block the new worker.
|
||||
try {
|
||||
const result = await pool.query(`
|
||||
SELECT pid, state, left(query, 80) as query, age(clock_timestamp(), state_change) as idle_time
|
||||
FROM pg_stat_activity
|
||||
WHERE datname = current_database()
|
||||
AND pid != pg_backend_pid()
|
||||
AND state = 'idle'
|
||||
AND query LIKE '%pg_try_advisory_lock%'
|
||||
AND state_change < clock_timestamp() - interval '5 minutes'
|
||||
`);
|
||||
for (const row of result.rows) {
|
||||
log.warn(
|
||||
{ pid: row.pid, idleTime: row.idle_time, query: row.query },
|
||||
"Terminating stale advisory lock session from previous worker"
|
||||
);
|
||||
await pool.query("SELECT pg_terminate_backend($1)", [row.pid]);
|
||||
}
|
||||
if (result.rows.length > 0) {
|
||||
log.info({ terminated: result.rows.length }, "Cleaned up stale advisory lock sessions");
|
||||
}
|
||||
} catch (err) {
|
||||
log.warn({ err }, "Failed to clean up stale advisory locks (non-fatal)");
|
||||
}
|
||||
|
||||
// Verify destination messages exist for all "uploaded" packages.
|
||||
// Resets any packages whose dest message is missing so they get re-processed.
|
||||
await recoverIncompleteUploads();
|
||||
|
||||
Reference in New Issue
Block a user