Files
QuoteForge/bible-local/decisions/2026-10-26-incremental-config-project-pull.md
T
Mikhail ChusavitinandClaude Opus 5.5 44a09d4420 fix(sync): stop MariaDB connection leak and cut per-cycle sync waste
- ConnectionManager reuses a stale-but-healthy pool instead of dialing a new
  one every 30s, and closes every pool it replaces (one client held ~30
  connections). Pool capped at 2 connections, idle released after 1 min.
- Background sync every 1 min with ±30% jitter, scheduled from cycle start;
  users-status online threshold = interval × 1.3. One heartbeat per cycle.
- Partnumber books: read headers only, fetch partnumbers_json per book when
  new/incomplete; empty book counts as complete; an incomplete book is
  re-pulled once per process (server book lists BOT_AGIBOT_G2_PERF missing
  from qt_partnumber_book_items).
- NeedSync in one UNION ALL query; SyncQtSettings skips unchanged rewrite.
- Migration 035: server-maintained server_updated_at on qt_configurations and
  qt_projects. Clients pull only changed rows after one full pull per process;
  servers without 035 and old clients keep full pulls.
- Migrator: fresh GORM statement per verify query and history insert; files
  with two verify checks failed to record.

ADL: bible-local/decisions/2026-10-26-mariadb-single-pool.md,
bible-local/decisions/2026-10-26-incremental-config-project-pull.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 20:46:59 +03:00

3.1 KiB

Decision: incremental pull of configurations and projects via server_updated_at

Date: 2026-10-26 Status: active

Context

Opening the configurations or projects list (GET /api/configs, /api/projects, /api/projects/:uuid/configs, on-demand with a 30s cooldown) and the manual sync ran ImportConfigurationsToLocal / ImportProjectsToLocal, which read every server row and rewrote it locally. Measured on production (2026-10-26): configurations 22.2s (4158 rows, ~3.2 MB, 4078 rewritten for one actual change), projects 4.6s (1692 rows), holding a pool connection the whole time. qt_configurations had no change column; qt_projects.updated_at is set by clients (SaveProjectPreservingUpdatedAt), so a change made offline and pushed later carries an old timestamp and cannot drive a filter.

Decision

  • Server migration 035_add_server_updated_at.sql adds server_updated_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6) plus index (server_updated_at, id) to qt_configurations and qt_projects. MariaDB maintains it; no client writes it and it is not in models.Configuration / models.Project, so writes from any client version (GORM Create/Save, raw UPDATE with column lists) keep working and bump the marker on every real change.
  • Client (internal/services/sync/pull_cursor.go):
    • if the column is missing (server not migrated yet) every pull is full, exactly as before; presence is re-checked until found, then cached for the process;
    • the first pull of each table per process is full (restores rows lost locally once per app start), later pulls read server_updated_at >= cursor - 120s in one query;
    • the cursor is the server's NOW(6) read before the pull, stored in app_settings (server_pull_cursor:<table>) together with host:port/database; a cursor from a different server is ignored;
    • the per-row merge (skip pending, keep local deactivation sticky, keep line) is unchanged; only the set of rows read differs.
  • Old clients keep doing full pulls and are unaffected.

Consequences

  • Idle pulls go from ~27s / 3.2 MB to one indexed range query.
  • Hard deletes on the server are not propagated — same as the full pull before.
  • Changes applied by InnoDB ON UPDATE CASCADE (e.g. a qt_projects.uuid change cascading to qt_configurations.project_uuid) do not bump server_updated_at; they are picked up by the next full pull (app start).
  • Cursor comparison happens in the server session time zone. Production is SYSTEM/MSK (no DST); on a server with DST, rows changed around the autumn transition could be missed until the next app start.
  • Applying 035 adds the column and builds an index on both tables (~4k and ~2k rows on production, seconds); qfs -migrate is run once by the developer as usual.
  • Tests: internal/services/sync/incremental_pull_mariadb_test.go (needs QF_TEST_MARIADB_DSN, a disposable MariaDB with the production table DDL; verified on MariaDB 11.8, same as production), migrator fix in internal/models/sql_migration_verify_test.go (two checks pass).