Files
QuoteForge/bible-local/decisions/2026-10-26-mariadb-single-pool.md
T
Mikhail ChusavitinandClaude Opus 5.5 44a09d4420 fix(sync): stop MariaDB connection leak and cut per-cycle sync waste
- ConnectionManager reuses a stale-but-healthy pool instead of dialing a new
  one every 30s, and closes every pool it replaces (one client held ~30
  connections). Pool capped at 2 connections, idle released after 1 min.
- Background sync every 1 min with ±30% jitter, scheduled from cycle start;
  users-status online threshold = interval × 1.3. One heartbeat per cycle.
- Partnumber books: read headers only, fetch partnumbers_json per book when
  new/incomplete; empty book counts as complete; an incomplete book is
  re-pulled once per process (server book lists BOT_AGIBOT_G2_PERF missing
  from qt_partnumber_book_items).
- NeedSync in one UNION ALL query; SyncQtSettings skips unchanged rewrite.
- Migration 035: server-maintained server_updated_at on qt_configurations and
  qt_projects. Clients pull only changed rows after one full pull per process;
  servers without 035 and old clients keep full pulls.
- Migrator: fresh GORM statement per verify query and history insert; files
  with two verify checks failed to record.

ADL: bible-local/decisions/2026-10-26-mariadb-single-pool.md,
bible-local/decisions/2026-10-26-incremental-config-project-pull.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 20:46:59 +03:00

3.3 KiB
Raw Blame History

Decision: one MariaDB pool per client — reuse, never orphan

Date: 2026-10-26 Status: active

Context

A single client held ~30 MariaDB connections during sync. ConnectionManager.GetDB treated a pool older than pingInterval (30s) as unusable and dialed a brand-new *gorm.DB without closing the previous one. Sync calls getDB() per step, so every ~30s a new pool appeared while the old one kept its idle connections until ConnMaxLifetime (5 min): ~10 orphaned pools × 2 idle + up to 10 active ≈ 30. Each orphaned *sql.DB also leaked its opener goroutine. TryConnect, the background ping-failure path and a failed ping inside dial dropped pools the same way.

Decision

  • GetDB pings the existing pool when it is merely stale and returns it; it dials only when there is no pool or the ping fails.
  • Whenever the manager replaces or drops a pool (GetDB, TryConnect, checkOnlineNow, MarkOffline, Disconnect, failed ping in dial) the old *sql.DB is closed.
  • Pool limits: MaxOpenConns=2, MaxIdleConns=2, ConnMaxLifetime=5m, ConnMaxIdleTime=1m — one client holds at most 2 connections, idle ones are released within a minute. Sync is sequential (one query at a time); the second connection lets UI server queries (active viewers, users status) run alongside a long sync step.
  • Background sync interval (backgroundSyncInterval in cmd/qfs/main.go) is 1 minute with ±30% random jitter (sync.WorkerJitter), so clients do not hit MariaDB in lockstep. The next cycle is scheduled from the start of the previous one, so heartbeats (qt_client_schema_state.last_checked_at) are at most 1.3 × interval apart while a cycle is shorter than that.
  • /api/sync/users-status online threshold = interval × (1 + WorkerJitter).
  • PullPartnumberBooks reads only book headers (id, version, created_at, is_active) each cycle; partnumbers_json (~5 MB for all books) is fetched per book only when the book is new or its local items are incomplete. An empty book counts as complete. An incomplete book is re-pulled at most once per process per version: a server book can list partnumbers missing from qt_partnumber_book_items (seen: BOT_AGIBOT_G2_PERF in PNBOOK-2026-10-21-001), which a re-pull cannot fix; this is logged as WARN with the missing partnumbers.
  • A worker cycle sends one heartbeat: the worker calls EnsureReadinessForSync once and then the unexported pushPendingChanges (the exported PushPendingChanges still checks readiness itself for the manual-sync handlers).
  • NeedSync asks for the latest active pricelist of all three sources in one UNION ALL query (GetLatestActiveIDsBySources): 0.71s → 0.24s.
  • SyncQtSettings skips the local DELETE + INSERT when the server rows equal the cache.
  • Measured against the production server: a cold sync took 171s (pricelists 105s, partnumber books 38s, configurations 23s) and is bound by server reads, not pool size. A warm cycle takes ~1.4s (was ~2.5s plus a 5 MB download).

Consequences

  • Code that keeps a *gorm.DB across a reconnect gets sql: database is closed; sync steps must call getDB() per operation instead of caching the handle.
  • Tests: internal/db/connection_test.go, internal/services/sync/worker_jitter_test.go.