Files
QuoteForge/bible-local/decisions/2026-10-26-incremental-config-project-pull.md
T
Mikhail ChusavitinandClaude Opus 5.5 44a09d4420 fix(sync): stop MariaDB connection leak and cut per-cycle sync waste
- ConnectionManager reuses a stale-but-healthy pool instead of dialing a new
  one every 30s, and closes every pool it replaces (one client held ~30
  connections). Pool capped at 2 connections, idle released after 1 min.
- Background sync every 1 min with ±30% jitter, scheduled from cycle start;
  users-status online threshold = interval × 1.3. One heartbeat per cycle.
- Partnumber books: read headers only, fetch partnumbers_json per book when
  new/incomplete; empty book counts as complete; an incomplete book is
  re-pulled once per process (server book lists BOT_AGIBOT_G2_PERF missing
  from qt_partnumber_book_items).
- NeedSync in one UNION ALL query; SyncQtSettings skips unchanged rewrite.
- Migration 035: server-maintained server_updated_at on qt_configurations and
  qt_projects. Clients pull only changed rows after one full pull per process;
  servers without 035 and old clients keep full pulls.
- Migrator: fresh GORM statement per verify query and history insert; files
  with two verify checks failed to record.

ADL: bible-local/decisions/2026-10-26-mariadb-single-pool.md,
bible-local/decisions/2026-10-26-incremental-config-project-pull.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 20:46:59 +03:00

53 lines
3.1 KiB
Markdown

# Decision: incremental pull of configurations and projects via server_updated_at
**Date:** 2026-10-26
**Status:** active
## Context
Opening the configurations or projects list (`GET /api/configs`, `/api/projects`,
`/api/projects/:uuid/configs`, on-demand with a 30s cooldown) and the manual sync ran
`ImportConfigurationsToLocal` / `ImportProjectsToLocal`, which read **every** server row
and rewrote it locally. Measured on production (2026-10-26): configurations 22.2s
(4158 rows, ~3.2 MB, 4078 rewritten for one actual change), projects 4.6s (1692 rows),
holding a pool connection the whole time. `qt_configurations` had no change column;
`qt_projects.updated_at` is set by clients (`SaveProjectPreservingUpdatedAt`), so a change
made offline and pushed later carries an old timestamp and cannot drive a filter.
## Decision
- Server migration `035_add_server_updated_at.sql` adds
`server_updated_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6)`
plus index `(server_updated_at, id)` to `qt_configurations` and `qt_projects`. MariaDB
maintains it; no client writes it and it is **not** in `models.Configuration` /
`models.Project`, so writes from any client version (GORM `Create`/`Save`, raw
`UPDATE` with column lists) keep working and bump the marker on every real change.
- Client (`internal/services/sync/pull_cursor.go`):
- if the column is missing (server not migrated yet) every pull is full, exactly as
before; presence is re-checked until found, then cached for the process;
- the first pull of each table per process is full (restores rows lost locally once per
app start), later pulls read `server_updated_at >= cursor - 120s` in one query;
- the cursor is the server's `NOW(6)` read *before* the pull, stored in `app_settings`
(`server_pull_cursor:<table>`) together with `host:port/database`; a cursor from a
different server is ignored;
- the per-row merge (skip `pending`, keep local deactivation sticky, keep `line`) is
unchanged; only the set of rows read differs.
- Old clients keep doing full pulls and are unaffected.
## Consequences
- Idle pulls go from ~27s / 3.2 MB to one indexed range query.
- Hard deletes on the server are not propagated — same as the full pull before.
- Changes applied by InnoDB `ON UPDATE CASCADE` (e.g. a `qt_projects.uuid` change
cascading to `qt_configurations.project_uuid`) do not bump `server_updated_at`; they are
picked up by the next full pull (app start).
- Cursor comparison happens in the server session time zone. Production is `SYSTEM`/MSK
(no DST); on a server with DST, rows changed around the autumn transition could be
missed until the next app start.
- Applying 035 adds the column and builds an index on both tables (~4k and ~2k rows on
production, seconds); `qfs -migrate` is run once by the developer as usual.
- Tests: `internal/services/sync/incremental_pull_mariadb_test.go` (needs
`QF_TEST_MARIADB_DSN`, a disposable MariaDB with the production table DDL; verified on
MariaDB 11.8, same as production), migrator fix in
`internal/models/sql_migration_verify_test.go` (`two checks pass`).