- ConnectionManager reuses a stale-but-healthy pool instead of dialing a new one every 30s, and closes every pool it replaces (one client held ~30 connections). Pool capped at 2 connections, idle released after 1 min. - Background sync every 1 min with ±30% jitter, scheduled from cycle start; users-status online threshold = interval × 1.3. One heartbeat per cycle. - Partnumber books: read headers only, fetch partnumbers_json per book when new/incomplete; empty book counts as complete; an incomplete book is re-pulled once per process (server book lists BOT_AGIBOT_G2_PERF missing from qt_partnumber_book_items). - NeedSync in one UNION ALL query; SyncQtSettings skips unchanged rewrite. - Migration 035: server-maintained server_updated_at on qt_configurations and qt_projects. Clients pull only changed rows after one full pull per process; servers without 035 and old clients keep full pulls. - Migrator: fresh GORM statement per verify query and history insert; files with two verify checks failed to record. ADL: bible-local/decisions/2026-10-26-mariadb-single-pool.md, bible-local/decisions/2026-10-26-incremental-config-project-pull.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
53 lines
3.1 KiB
Markdown
53 lines
3.1 KiB
Markdown
# Decision: incremental pull of configurations and projects via server_updated_at
|
|
|
|
**Date:** 2026-10-26
|
|
**Status:** active
|
|
|
|
## Context
|
|
|
|
Opening the configurations or projects list (`GET /api/configs`, `/api/projects`,
|
|
`/api/projects/:uuid/configs`, on-demand with a 30s cooldown) and the manual sync ran
|
|
`ImportConfigurationsToLocal` / `ImportProjectsToLocal`, which read **every** server row
|
|
and rewrote it locally. Measured on production (2026-10-26): configurations 22.2s
|
|
(4158 rows, ~3.2 MB, 4078 rewritten for one actual change), projects 4.6s (1692 rows),
|
|
holding a pool connection the whole time. `qt_configurations` had no change column;
|
|
`qt_projects.updated_at` is set by clients (`SaveProjectPreservingUpdatedAt`), so a change
|
|
made offline and pushed later carries an old timestamp and cannot drive a filter.
|
|
|
|
## Decision
|
|
|
|
- Server migration `035_add_server_updated_at.sql` adds
|
|
`server_updated_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6)`
|
|
plus index `(server_updated_at, id)` to `qt_configurations` and `qt_projects`. MariaDB
|
|
maintains it; no client writes it and it is **not** in `models.Configuration` /
|
|
`models.Project`, so writes from any client version (GORM `Create`/`Save`, raw
|
|
`UPDATE` with column lists) keep working and bump the marker on every real change.
|
|
- Client (`internal/services/sync/pull_cursor.go`):
|
|
- if the column is missing (server not migrated yet) every pull is full, exactly as
|
|
before; presence is re-checked until found, then cached for the process;
|
|
- the first pull of each table per process is full (restores rows lost locally once per
|
|
app start), later pulls read `server_updated_at >= cursor - 120s` in one query;
|
|
- the cursor is the server's `NOW(6)` read *before* the pull, stored in `app_settings`
|
|
(`server_pull_cursor:<table>`) together with `host:port/database`; a cursor from a
|
|
different server is ignored;
|
|
- the per-row merge (skip `pending`, keep local deactivation sticky, keep `line`) is
|
|
unchanged; only the set of rows read differs.
|
|
- Old clients keep doing full pulls and are unaffected.
|
|
|
|
## Consequences
|
|
|
|
- Idle pulls go from ~27s / 3.2 MB to one indexed range query.
|
|
- Hard deletes on the server are not propagated — same as the full pull before.
|
|
- Changes applied by InnoDB `ON UPDATE CASCADE` (e.g. a `qt_projects.uuid` change
|
|
cascading to `qt_configurations.project_uuid`) do not bump `server_updated_at`; they are
|
|
picked up by the next full pull (app start).
|
|
- Cursor comparison happens in the server session time zone. Production is `SYSTEM`/MSK
|
|
(no DST); on a server with DST, rows changed around the autumn transition could be
|
|
missed until the next app start.
|
|
- Applying 035 adds the column and builds an index on both tables (~4k and ~2k rows on
|
|
production, seconds); `qfs -migrate` is run once by the developer as usual.
|
|
- Tests: `internal/services/sync/incremental_pull_mariadb_test.go` (needs
|
|
`QF_TEST_MARIADB_DSN`, a disposable MariaDB with the production table DDL; verified on
|
|
MariaDB 11.8, same as production), migrator fix in
|
|
`internal/models/sql_migration_verify_test.go` (`two checks pass`).
|