Compare commits
3
Commits
d6c498f0cf
...
v3.51
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
44a09d4420 | ||
|
|
8a08270b79 | ||
|
|
cab5d13f16 |
No files matched your search
@@ -196,6 +196,7 @@ PK: lot_name
|
||||
| rental_items | JSON | per-lot New/БУ condition for the Аренда tab; see [02-architecture.md](02-architecture.md#rental--paid-testing-pricing-contract) |
|
||||
| rental_uplift_percent | decimal(8,2) DEFAULT 0 | uplift applied to Estimate buy price for rental "Цена" |
|
||||
| feature_flags | JSON NULL | added by migration 034. Data features the configuration uses (`["sale_row_markups"]`); derived on save, a client refuses to write a config with a flag it doesn't know. See [decisions/2026-09-24-configuration-feature-flags.md](decisions/2026-09-24-configuration-feature-flags.md) |
|
||||
| server_updated_at | timestamp(6) NOT NULL | added by migration 035. Server-maintained change marker (`DEFAULT`/`ON UPDATE CURRENT_TIMESTAMP(6)`, index `(server_updated_at, id)`); never written by clients, used for incremental pulls. See [decisions/2026-10-26-incremental-config-project-pull.md](decisions/2026-10-26-incremental-config-project-pull.md) |
|
||||
|
||||
### qt_lot_metadata
|
||||
PK: lot_name
|
||||
@@ -306,6 +307,7 @@ PK: username
|
||||
| rental_enabled | tinyint(1) DEFAULT 0 | shows the Аренда tab on this project's configurations |
|
||||
| created_at | timestamp | |
|
||||
| updated_at | timestamp ON UPDATE | |
|
||||
| server_updated_at | timestamp(6) NOT NULL | added by migration 035. Server-maintained change marker (`DEFAULT`/`ON UPDATE CURRENT_TIMESTAMP(6)`, index `(server_updated_at, id)`); never written by clients, used for incremental pulls. See [decisions/2026-10-26-incremental-config-project-pull.md](decisions/2026-10-26-incremental-config-project-pull.md) |
|
||||
|
||||
### qt_schema_migrations
|
||||
| Column | Type | Notes |
|
||||
@@ -441,6 +443,9 @@ MariaDB:
|
||||
not the app's 3s-timeout sync connection — a long backfill must not be cut mid-file;
|
||||
- each file, its `-- verify: <description> | <SQL>` queries and history insertion run
|
||||
on one database session, preserving session variables;
|
||||
- every verify query and the history insert use a fresh GORM statement
|
||||
(`Session{NewDB: true}`) on that session; chaining them on one statement leaked state
|
||||
and broke files with two or more verify checks (fixed 2026-10-26 with migration 035);
|
||||
- a file is recorded only after statements and all verify checks succeed. Verify
|
||||
queries return rows only for broken invariants; a row or SQL error stops the run
|
||||
without marking that file applied. Malformed verify headers fail before execution;
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# Decision: incremental pull of configurations and projects via server_updated_at
|
||||
|
||||
**Date:** 2026-10-26
|
||||
**Status:** active
|
||||
|
||||
## Context
|
||||
|
||||
Opening the configurations or projects list (`GET /api/configs`, `/api/projects`,
|
||||
`/api/projects/:uuid/configs`, on-demand with a 30s cooldown) and the manual sync ran
|
||||
`ImportConfigurationsToLocal` / `ImportProjectsToLocal`, which read **every** server row
|
||||
and rewrote it locally. Measured on production (2026-10-26): configurations 22.2s
|
||||
(4158 rows, ~3.2 MB, 4078 rewritten for one actual change), projects 4.6s (1692 rows),
|
||||
holding a pool connection the whole time. `qt_configurations` had no change column;
|
||||
`qt_projects.updated_at` is set by clients (`SaveProjectPreservingUpdatedAt`), so a change
|
||||
made offline and pushed later carries an old timestamp and cannot drive a filter.
|
||||
|
||||
## Decision
|
||||
|
||||
- Server migration `035_add_server_updated_at.sql` adds
|
||||
`server_updated_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6)`
|
||||
plus index `(server_updated_at, id)` to `qt_configurations` and `qt_projects`. MariaDB
|
||||
maintains it; no client writes it and it is **not** in `models.Configuration` /
|
||||
`models.Project`, so writes from any client version (GORM `Create`/`Save`, raw
|
||||
`UPDATE` with column lists) keep working and bump the marker on every real change.
|
||||
- Client (`internal/services/sync/pull_cursor.go`):
|
||||
- if the column is missing (server not migrated yet) every pull is full, exactly as
|
||||
before; presence is re-checked until found, then cached for the process;
|
||||
- the first pull of each table per process is full (restores rows lost locally once per
|
||||
app start), later pulls read `server_updated_at >= cursor - 120s` in one query;
|
||||
- the cursor is the server's `NOW(6)` read *before* the pull, stored in `app_settings`
|
||||
(`server_pull_cursor:<table>`) together with `host:port/database`; a cursor from a
|
||||
different server is ignored;
|
||||
- the per-row merge (skip `pending`, keep local deactivation sticky, keep `line`) is
|
||||
unchanged; only the set of rows read differs.
|
||||
- Old clients keep doing full pulls and are unaffected.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Idle pulls go from ~27s / 3.2 MB to one indexed range query.
|
||||
- Hard deletes on the server are not propagated — same as the full pull before.
|
||||
- Changes applied by InnoDB `ON UPDATE CASCADE` (e.g. a `qt_projects.uuid` change
|
||||
cascading to `qt_configurations.project_uuid`) do not bump `server_updated_at`; they are
|
||||
picked up by the next full pull (app start).
|
||||
- Cursor comparison happens in the server session time zone. Production is `SYSTEM`/MSK
|
||||
(no DST); on a server with DST, rows changed around the autumn transition could be
|
||||
missed until the next app start.
|
||||
- Applying 035 adds the column and builds an index on both tables (~4k and ~2k rows on
|
||||
production, seconds); `qfs -migrate` is run once by the developer as usual.
|
||||
- Tests: `internal/services/sync/incremental_pull_mariadb_test.go` (needs
|
||||
`QF_TEST_MARIADB_DSN`, a disposable MariaDB with the production table DDL; verified on
|
||||
MariaDB 11.8, same as production), migrator fix in
|
||||
`internal/models/sql_migration_verify_test.go` (`two checks pass`).
|
||||
@@ -0,0 +1,54 @@
|
||||
# Decision: one MariaDB pool per client — reuse, never orphan
|
||||
|
||||
**Date:** 2026-10-26
|
||||
**Status:** active
|
||||
|
||||
## Context
|
||||
|
||||
A single client held ~30 MariaDB connections during sync. `ConnectionManager.GetDB`
|
||||
treated a pool older than `pingInterval` (30s) as unusable and dialed a brand-new
|
||||
`*gorm.DB` without closing the previous one. Sync calls `getDB()` per step, so every
|
||||
~30s a new pool appeared while the old one kept its idle connections until
|
||||
`ConnMaxLifetime` (5 min): ~10 orphaned pools × 2 idle + up to 10 active ≈ 30. Each
|
||||
orphaned `*sql.DB` also leaked its opener goroutine. `TryConnect`, the background
|
||||
ping-failure path and a failed ping inside `dial` dropped pools the same way.
|
||||
|
||||
## Decision
|
||||
|
||||
- `GetDB` pings the existing pool when it is merely stale and returns it; it dials only
|
||||
when there is no pool or the ping fails.
|
||||
- Whenever the manager replaces or drops a pool (`GetDB`, `TryConnect`,
|
||||
`checkOnlineNow`, `MarkOffline`, `Disconnect`, failed ping in `dial`) the old
|
||||
`*sql.DB` is closed.
|
||||
- Pool limits: `MaxOpenConns=2`, `MaxIdleConns=2`, `ConnMaxLifetime=5m`,
|
||||
`ConnMaxIdleTime=1m` — one client holds at most 2 connections, idle ones are released
|
||||
within a minute. Sync is sequential (one query at a time); the second connection lets
|
||||
UI server queries (active viewers, users status) run alongside a long sync step.
|
||||
- Background sync interval (`backgroundSyncInterval` in `cmd/qfs/main.go`) is 1 minute
|
||||
with ±30% random jitter (`sync.WorkerJitter`), so clients do not hit MariaDB in
|
||||
lockstep. The next cycle is scheduled from the *start* of the previous one, so
|
||||
heartbeats (`qt_client_schema_state.last_checked_at`) are at most 1.3 × interval apart
|
||||
while a cycle is shorter than that.
|
||||
- `/api/sync/users-status` online threshold = interval × (1 + `WorkerJitter`).
|
||||
- `PullPartnumberBooks` reads only book headers (`id, version, created_at, is_active`)
|
||||
each cycle; `partnumbers_json` (~5 MB for all books) is fetched per book only when the
|
||||
book is new or its local items are incomplete. An empty book counts as complete. An
|
||||
incomplete book is re-pulled at most once per process per version: a server book can
|
||||
list partnumbers missing from `qt_partnumber_book_items` (seen: `BOT_AGIBOT_G2_PERF`
|
||||
in `PNBOOK-2026-10-21-001`), which a re-pull cannot fix; this is logged as `WARN` with
|
||||
the missing partnumbers.
|
||||
- A worker cycle sends one heartbeat: the worker calls `EnsureReadinessForSync` once and
|
||||
then the unexported `pushPendingChanges` (the exported `PushPendingChanges` still checks
|
||||
readiness itself for the manual-sync handlers).
|
||||
- `NeedSync` asks for the latest active pricelist of all three sources in one
|
||||
`UNION ALL` query (`GetLatestActiveIDsBySources`): 0.71s → 0.24s.
|
||||
- `SyncQtSettings` skips the local `DELETE` + `INSERT` when the server rows equal the cache.
|
||||
- Measured against the production server: a cold sync took 171s (pricelists 105s,
|
||||
partnumber books 38s, configurations 23s) and is bound by server reads, not pool
|
||||
size. A warm cycle takes ~1.4s (was ~2.5s plus a 5 MB download).
|
||||
|
||||
## Consequences
|
||||
|
||||
- Code that keeps a `*gorm.DB` across a reconnect gets `sql: database is closed`; sync
|
||||
steps must call `getDB()` per operation instead of caching the handle.
|
||||
- Tests: `internal/db/connection_test.go`, `internal/services/sync/worker_jitter_test.go`.
|
||||
+1
-1
@@ -46,7 +46,7 @@ import (
|
||||
var Version = "dev"
|
||||
var errVendorImportTooLarge = errors.New("vendor workspace file exceeds 1 GiB limit")
|
||||
|
||||
const backgroundSyncInterval = 5 * time.Minute
|
||||
const backgroundSyncInterval = time.Minute
|
||||
const onDemandPullCooldown = 30 * time.Second
|
||||
const startupConsoleWarning = "Не закрывайте консоль иначе приложение не будет работать"
|
||||
|
||||
|
||||
+67
-18
@@ -22,9 +22,10 @@ const (
|
||||
// goroutines must never pay the MySQL dial/read timeout themselves.
|
||||
defaultStatusCheckInterval = 15 * time.Second
|
||||
|
||||
maxOpenConns = 10
|
||||
maxOpenConns = 2
|
||||
maxIdleConns = 2
|
||||
connMaxLifetime = 5 * time.Minute
|
||||
connMaxIdleTime = time.Minute
|
||||
)
|
||||
|
||||
// ConnectionStatus represents the current status of the database connection
|
||||
@@ -49,11 +50,13 @@ type ConnectionManager struct {
|
||||
|
||||
statusCheckInterval time.Duration // background prober cadence (default: 15s)
|
||||
stopStatusLoop chan struct{} // closed by Stop() to end the background loop
|
||||
|
||||
dialFn func() (*gorm.DB, error) // opens a new pool; cm.dial unless overridden in tests
|
||||
}
|
||||
|
||||
// NewConnectionManager creates a new ConnectionManager instance
|
||||
func NewConnectionManager(localDB *localdb.LocalDB) *ConnectionManager {
|
||||
return &ConnectionManager{
|
||||
cm := &ConnectionManager{
|
||||
localDB: localDB,
|
||||
connectTimeout: defaultConnectTimeout,
|
||||
pingInterval: defaultPingInterval,
|
||||
@@ -63,6 +66,8 @@ func NewConnectionManager(localDB *localdb.LocalDB) *ConnectionManager {
|
||||
lastError: nil,
|
||||
lastCheck: time.Time{},
|
||||
}
|
||||
cm.dialFn = cm.dial
|
||||
return cm
|
||||
}
|
||||
|
||||
// Start launches a background goroutine that keeps the online-status cache
|
||||
@@ -137,25 +142,66 @@ func (cm *ConnectionManager) GetDB() (*gorm.DB, error) {
|
||||
return db, err
|
||||
}
|
||||
|
||||
newDB, err := cm.dial()
|
||||
// The cached pool is only stale, not broken: ping it instead of dialing a
|
||||
// new one. Re-dialing here used to orphan the old *sql.DB together with its
|
||||
// idle connections, leaking a pool every pingInterval during sync.
|
||||
cm.mu.RLock()
|
||||
existing := cm.db
|
||||
cm.mu.RUnlock()
|
||||
if existing != nil {
|
||||
if err := cm.ping(existing); err == nil {
|
||||
cm.mu.Lock()
|
||||
cm.lastError = nil
|
||||
cm.lastCheck = time.Now()
|
||||
cm.mu.Unlock()
|
||||
return existing, nil
|
||||
}
|
||||
}
|
||||
|
||||
newDB, err := cm.dialFn()
|
||||
|
||||
cm.mu.Lock()
|
||||
old := cm.db
|
||||
if err != nil {
|
||||
// Drop stale handle so callers don't treat it as an active connection.
|
||||
cm.db = nil
|
||||
cm.lastError = err
|
||||
cm.lastCheck = time.Now()
|
||||
cm.mu.Unlock()
|
||||
closeDB(old)
|
||||
return nil, err
|
||||
}
|
||||
cm.db = newDB
|
||||
cm.lastError = nil
|
||||
cm.lastCheck = time.Now()
|
||||
cm.mu.Unlock()
|
||||
closeDB(old)
|
||||
|
||||
return newDB, nil
|
||||
}
|
||||
|
||||
// ping checks an existing pool with the connect timeout. No locks held.
|
||||
func (cm *ConnectionManager) ping(db *gorm.DB) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), cm.connectTimeout)
|
||||
defer cancel()
|
||||
|
||||
sqlDB, err := db.DB()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return sqlDB.PingContext(ctx)
|
||||
}
|
||||
|
||||
// closeDB releases a pool that is no longer referenced by the manager.
|
||||
func closeDB(db *gorm.DB) {
|
||||
if db == nil {
|
||||
return
|
||||
}
|
||||
if sqlDB, err := db.DB(); err == nil {
|
||||
sqlDB.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// cachedResult returns (db, err, true) if the cached state is still fresh
|
||||
// enough to answer without a new network round-trip: either a live
|
||||
// connection within pingInterval, or a recent failure still within
|
||||
@@ -200,15 +246,17 @@ func (cm *ConnectionManager) dial() (*gorm.DB, error) {
|
||||
return nil, fmt.Errorf("getting sql.DB: %w", err)
|
||||
}
|
||||
|
||||
// Ping with timeout
|
||||
if err = sqlDB.PingContext(ctx); err != nil {
|
||||
return nil, fmt.Errorf("pinging database: %w", err)
|
||||
}
|
||||
|
||||
// Set connection pool settings
|
||||
sqlDB.SetMaxOpenConns(maxOpenConns)
|
||||
sqlDB.SetMaxIdleConns(maxIdleConns)
|
||||
sqlDB.SetConnMaxLifetime(connMaxLifetime)
|
||||
sqlDB.SetConnMaxIdleTime(connMaxIdleTime)
|
||||
|
||||
// Ping with timeout
|
||||
if err = sqlDB.PingContext(ctx); err != nil {
|
||||
sqlDB.Close()
|
||||
return nil, fmt.Errorf("pinging database: %w", err)
|
||||
}
|
||||
|
||||
return db, nil
|
||||
}
|
||||
@@ -264,20 +312,17 @@ func (cm *ConnectionManager) checkOnlineNow() bool {
|
||||
}
|
||||
|
||||
// Perform ping with timeout — no locks held here.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), cm.connectTimeout)
|
||||
defer cancel()
|
||||
|
||||
sqlDB, err := db.DB()
|
||||
if err == nil {
|
||||
err = sqlDB.PingContext(ctx)
|
||||
}
|
||||
err := cm.ping(db)
|
||||
|
||||
cm.mu.Lock()
|
||||
defer cm.mu.Unlock()
|
||||
if err != nil {
|
||||
cm.lastError = err
|
||||
cm.lastCheck = time.Now()
|
||||
cm.db = nil
|
||||
if cm.db == db {
|
||||
cm.db = nil
|
||||
}
|
||||
closeDB(db)
|
||||
return false
|
||||
}
|
||||
cm.lastCheck = time.Now()
|
||||
@@ -292,19 +337,23 @@ func (cm *ConnectionManager) TryConnect() error {
|
||||
cm.connMu.Lock()
|
||||
defer cm.connMu.Unlock()
|
||||
|
||||
newDB, err := cm.dial()
|
||||
newDB, err := cm.dialFn()
|
||||
|
||||
cm.mu.Lock()
|
||||
defer cm.mu.Unlock()
|
||||
old := cm.db
|
||||
if err != nil {
|
||||
cm.db = nil
|
||||
cm.lastError = err
|
||||
cm.lastCheck = time.Now()
|
||||
cm.mu.Unlock()
|
||||
closeDB(old)
|
||||
return err
|
||||
}
|
||||
cm.db = newDB
|
||||
cm.lastError = nil
|
||||
cm.lastCheck = time.Now()
|
||||
cm.mu.Unlock()
|
||||
closeDB(old)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/localdb"
|
||||
"github.com/glebarez/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
func newTestConnectionManager(t *testing.T) (*ConnectionManager, *int) {
|
||||
t.Helper()
|
||||
local, err := localdb.New(filepath.Join(t.TempDir(), "qfs.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("open local db: %v", err)
|
||||
}
|
||||
cm := NewConnectionManager(local)
|
||||
dials := 0
|
||||
cm.dialFn = func() (*gorm.DB, error) {
|
||||
dials++
|
||||
return gorm.Open(sqlite.Open(filepath.Join(t.TempDir(), "remote.db")), &gorm.Config{
|
||||
Logger: logger.Default.LogMode(logger.Silent),
|
||||
})
|
||||
}
|
||||
return cm, &dials
|
||||
}
|
||||
|
||||
func TestGetDBReusesStalePoolInsteadOfRedialing(t *testing.T) {
|
||||
cm, dials := newTestConnectionManager(t)
|
||||
|
||||
first, err := cm.GetDB()
|
||||
if err != nil {
|
||||
t.Fatalf("first GetDB: %v", err)
|
||||
}
|
||||
|
||||
// Simulate pingInterval elapsing between sync steps.
|
||||
cm.mu.Lock()
|
||||
cm.lastCheck = time.Now().Add(-2 * cm.pingInterval)
|
||||
cm.mu.Unlock()
|
||||
|
||||
second, err := cm.GetDB()
|
||||
if err != nil {
|
||||
t.Fatalf("second GetDB: %v", err)
|
||||
}
|
||||
if second != first {
|
||||
t.Fatalf("expected stale but healthy pool to be reused")
|
||||
}
|
||||
if *dials != 1 {
|
||||
t.Fatalf("expected 1 dial, got %d", *dials)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTryConnectClosesReplacedPool(t *testing.T) {
|
||||
cm, _ := newTestConnectionManager(t)
|
||||
|
||||
first, err := cm.GetDB()
|
||||
if err != nil {
|
||||
t.Fatalf("GetDB: %v", err)
|
||||
}
|
||||
if err := cm.TryConnect(); err != nil {
|
||||
t.Fatalf("TryConnect: %v", err)
|
||||
}
|
||||
|
||||
sqlDB, err := first.DB()
|
||||
if err != nil {
|
||||
t.Fatalf("sql.DB: %v", err)
|
||||
}
|
||||
if err := sqlDB.Ping(); err == nil {
|
||||
t.Fatalf("expected replaced pool to be closed")
|
||||
}
|
||||
}
|
||||
@@ -43,7 +43,7 @@ func NewSyncHandler(localDB *localdb.LocalDB, syncService *sync.Service, connMgr
|
||||
syncService: syncService,
|
||||
connMgr: connMgr,
|
||||
autoSyncInterval: autoSyncInterval,
|
||||
onlineGraceFactor: 1.10,
|
||||
onlineGraceFactor: 1 + sync.WorkerJitter, // longest gap between jittered sync cycles
|
||||
tmpl: tmpl,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
package localdb
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// serverPullCursor is the server-side NOW(6) taken before the last successful
|
||||
// incremental pull of a table, bound to the server it was taken from.
|
||||
type serverPullCursor struct {
|
||||
Scope string `json:"scope"`
|
||||
ServerTime string `json:"server_time"`
|
||||
}
|
||||
|
||||
func serverPullCursorKey(table string) string {
|
||||
return "server_pull_cursor:" + table
|
||||
}
|
||||
|
||||
// ServerPullScope identifies the MariaDB server+database a pull cursor belongs to,
|
||||
// so a cursor never survives a switch to another server.
|
||||
func (l *LocalDB) ServerPullScope() (string, error) {
|
||||
settings, err := l.GetSettings()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return fmt.Sprintf("%s:%d/%s", settings.Host, settings.Port, settings.Database), nil
|
||||
}
|
||||
|
||||
// GetServerPullCursor returns the stored server time for table if it was taken
|
||||
// on the server identified by scope.
|
||||
func (l *LocalDB) GetServerPullCursor(table, scope string) (string, bool) {
|
||||
raw, ok := l.getAppSettingValue(serverPullCursorKey(table))
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
var cursor serverPullCursor
|
||||
if err := json.Unmarshal([]byte(raw), &cursor); err != nil || cursor.Scope != scope || cursor.ServerTime == "" {
|
||||
return "", false
|
||||
}
|
||||
return cursor.ServerTime, true
|
||||
}
|
||||
|
||||
// SetServerPullCursor stores the server time a pull of table started at.
|
||||
func (l *LocalDB) SetServerPullCursor(table, scope, serverTime string) error {
|
||||
raw, err := json.Marshal(serverPullCursor{Scope: scope, ServerTime: serverTime})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return l.upsertAppSetting(l.db, serverPullCursorKey(table), string(raw), time.Now())
|
||||
}
|
||||
@@ -72,6 +72,11 @@ func (l *LocalDB) SyncQtSettings(mariaDB *gorm.DB) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
var current []LocalQtSetting
|
||||
if err := l.db.Select("name, value").Find(¤t).Error; err == nil && sameQtSettings(current, rows) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return l.db.Transaction(func(tx *gorm.DB) error {
|
||||
if err := tx.Exec("DELETE FROM local_qt_settings").Error; err != nil {
|
||||
return fmt.Errorf("clearing local_qt_settings: %w", err)
|
||||
@@ -142,3 +147,19 @@ func (l *LocalDB) GetConfiguratorSettings() (*ConfiguratorSettings, error) {
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func sameQtSettings(a, b []LocalQtSetting) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
values := make(map[string]string, len(a))
|
||||
for _, row := range a {
|
||||
values[row.Name] = row.Value
|
||||
}
|
||||
for _, row := range b {
|
||||
if v, ok := values[row.Name]; !ok || v != row.Value {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -38,8 +38,12 @@ func applySQLMigrationFile(db *gorm.DB, filename, content string) error {
|
||||
if err := executeMigrationStatements(conn, filename, splitSQLStatements(content)); err != nil {
|
||||
return err
|
||||
}
|
||||
// Each query gets a fresh statement on the pinned connection: chaining
|
||||
// Raw/Create on conn itself leaks the previous query's state (with two
|
||||
// verify checks the history insert targeted a table named "rows").
|
||||
fresh := func() *gorm.DB { return conn.Session(&gorm.Session{NewDB: true}) }
|
||||
for _, check := range checks {
|
||||
rows, err := conn.Raw(check.query).Rows()
|
||||
rows, err := fresh().Raw(check.query).Rows()
|
||||
if err != nil {
|
||||
return fmt.Errorf("verify migration %s (%s): %w", filename, check.description, err)
|
||||
}
|
||||
@@ -56,7 +60,7 @@ func applySQLMigrationFile(db *gorm.DB, filename, content string) error {
|
||||
return fmt.Errorf("verify migration %s failed: %s", filename, check.description)
|
||||
}
|
||||
}
|
||||
if err := conn.Create(&SQLSchemaMigration{Filename: filename}).Error; err != nil {
|
||||
if err := fresh().Create(&SQLSchemaMigration{Filename: filename}).Error; err != nil {
|
||||
return fmt.Errorf("record migration %s: %w", filename, err)
|
||||
}
|
||||
return nil
|
||||
|
||||
@@ -15,6 +15,7 @@ func TestMigrationVerification(t *testing.T) {
|
||||
{"failed", "-- verify: bad value | SELECT 1 FROM example WHERE value <> 1\nCREATE TEMP TABLE example (value INT); INSERT INTO example VALUES (2);", "bad value"},
|
||||
{"query error", "-- verify: missing table | SELECT 1 FROM missing\nCREATE TEMP TABLE example (value INT);", "missing table"},
|
||||
{"malformed", "-- verify: no query\nCREATE TABLE example (value INT);", "invalid migration verify header"},
|
||||
{"two checks pass", "-- verify: first | SELECT 1 WHERE 0\n-- verify: second | SELECT 1 WHERE 0\nCREATE TEMP TABLE example (value INT);", ""},
|
||||
{"second check fails", "-- verify: first | SELECT 1 WHERE 0\n-- verify: second | SELECT 1\nCREATE TEMP TABLE example (value INT);", "second"},
|
||||
{"execution failure", "-- verify: none | SELECT 1 WHERE 0\nINVALID SQL;", "exec migration"},
|
||||
} {
|
||||
|
||||
@@ -141,6 +141,31 @@ func (r *PartnumberBookRepository) HasAllBookItems(bookID uint) bool {
|
||||
return count == int64(len(book.PartnumbersJSON))
|
||||
}
|
||||
|
||||
// MissingBookItems returns the book's partnumbers that have no local catalog row.
|
||||
func (r *PartnumberBookRepository) MissingBookItems(bookID uint) []string {
|
||||
book, err := r.getBook(bookID)
|
||||
if err != nil || len(book.PartnumbersJSON) == 0 {
|
||||
return nil
|
||||
}
|
||||
var present []string
|
||||
if err := r.db.Model(&localdb.LocalPartnumberBookItem{}).
|
||||
Where("partnumber IN ?", []string(book.PartnumbersJSON)).
|
||||
Pluck("partnumber", &present).Error; err != nil {
|
||||
return nil
|
||||
}
|
||||
have := make(map[string]struct{}, len(present))
|
||||
for _, pn := range present {
|
||||
have[pn] = struct{}{}
|
||||
}
|
||||
var missing []string
|
||||
for _, pn := range book.PartnumbersJSON {
|
||||
if _, ok := have[pn]; !ok {
|
||||
missing = append(missing, pn)
|
||||
}
|
||||
}
|
||||
return missing
|
||||
}
|
||||
|
||||
func (r *PartnumberBookRepository) getBook(bookID uint) (*localdb.LocalPartnumberBook, error) {
|
||||
var book localdb.LocalPartnumberBook
|
||||
if err := r.db.First(&book, bookID).Error; err != nil {
|
||||
|
||||
@@ -156,6 +156,35 @@ func (r *PricelistRepository) GetLatestActiveBySource(source string) (*models.Pr
|
||||
return &pricelist, nil
|
||||
}
|
||||
|
||||
// GetLatestActiveIDsBySources returns, per source, the ID GetLatestActiveBySource
|
||||
// would return, in one round trip. Sources with no such pricelist are absent.
|
||||
func (r *PricelistRepository) GetLatestActiveIDsBySources(sources []string) (map[string]uint, error) {
|
||||
result := make(map[string]uint, len(sources))
|
||||
if len(sources) == 0 {
|
||||
return result, nil
|
||||
}
|
||||
parts := make([]string, 0, len(sources))
|
||||
args := make([]any, 0, len(sources))
|
||||
for _, source := range sources {
|
||||
parts = append(parts, `(SELECT id, source FROM qt_pricelists
|
||||
WHERE is_active = 1 AND source = ?
|
||||
AND EXISTS (SELECT 1 FROM qt_pricelist_items WHERE qt_pricelist_items.pricelist_id = qt_pricelists.id)
|
||||
ORDER BY created_at DESC, id DESC LIMIT 1)`)
|
||||
args = append(args, source)
|
||||
}
|
||||
var rows []struct {
|
||||
ID uint
|
||||
Source string
|
||||
}
|
||||
if err := r.db.Raw(strings.Join(parts, " UNION ALL "), args...).Scan(&rows).Error; err != nil {
|
||||
return nil, fmt.Errorf("getting latest pricelists: %w", err)
|
||||
}
|
||||
for _, row := range rows {
|
||||
result[row.Source] = row.ID
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// Create creates a new pricelist
|
||||
func (r *PricelistRepository) Create(pricelist *models.Pricelist) error {
|
||||
if err := r.db.Create(pricelist).Error; err != nil {
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
package sync_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/models"
|
||||
syncsvc "git.mchus.pro/mchus/quoteforge/internal/services/sync"
|
||||
"gorm.io/driver/mysql"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
// Runs only against a disposable MariaDB that already has the qt_projects and
|
||||
// qt_configurations tables (e.g. a container loaded with SHOW CREATE TABLE from
|
||||
// production). The test truncates them. Example:
|
||||
//
|
||||
// QF_TEST_MARIADB_DSN='root:test@tcp(127.0.0.1:33306)/qf?parseTime=true' go test ./internal/services/sync -run MariaDB
|
||||
func newMariaDBForPullTest(t *testing.T) *gorm.DB {
|
||||
t.Helper()
|
||||
dsn := os.Getenv("QF_TEST_MARIADB_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("QF_TEST_MARIADB_DSN not set")
|
||||
}
|
||||
db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
|
||||
if err != nil {
|
||||
t.Fatalf("open mariadb: %v", err)
|
||||
}
|
||||
mustExec(t, db, "SET FOREIGN_KEY_CHECKS=0")
|
||||
mustExec(t, db, "TRUNCATE TABLE qt_configurations")
|
||||
mustExec(t, db, "TRUNCATE TABLE qt_projects")
|
||||
mustExec(t, db, "SET FOREIGN_KEY_CHECKS=1")
|
||||
for _, table := range []string{"qt_configurations", "qt_projects"} {
|
||||
mustExec(t, db, "ALTER TABLE "+table+" DROP INDEX IF EXISTS idx_"+table+"_server_updated_at")
|
||||
mustExec(t, db, "ALTER TABLE "+table+" DROP COLUMN IF EXISTS server_updated_at")
|
||||
}
|
||||
return db
|
||||
}
|
||||
|
||||
func mustExec(t *testing.T, db *gorm.DB, sql string, args ...any) {
|
||||
t.Helper()
|
||||
if err := db.Exec(sql, args...).Error; err != nil {
|
||||
t.Fatalf("%s: %v", sql, err)
|
||||
}
|
||||
}
|
||||
|
||||
func applyServerUpdatedAtMigration(t *testing.T, db *gorm.DB) {
|
||||
t.Helper()
|
||||
raw, err := os.ReadFile("../../../migrations/035_add_server_updated_at.sql")
|
||||
if err != nil {
|
||||
t.Fatalf("read migration: %v", err)
|
||||
}
|
||||
var body []string
|
||||
for _, line := range strings.Split(string(raw), "\n") {
|
||||
if !strings.HasPrefix(strings.TrimSpace(line), "--") {
|
||||
body = append(body, line)
|
||||
}
|
||||
}
|
||||
for _, stmt := range strings.Split(strings.Join(body, "\n"), ";") {
|
||||
if strings.TrimSpace(stmt) != "" {
|
||||
mustExec(t, db, stmt)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// seedServerConfig writes the way any client version does: through the
|
||||
// models.Configuration struct, which knows nothing about server_updated_at.
|
||||
func seedServerConfig(t *testing.T, db *gorm.DB, uuid, name string) *models.Configuration {
|
||||
t.Helper()
|
||||
cfg := &models.Configuration{
|
||||
UUID: uuid,
|
||||
OwnerUsername: "tester",
|
||||
Name: name,
|
||||
Items: models.ConfigItems{{LotName: "CPU_PULL", Quantity: 1, UnitPrice: 100}},
|
||||
ServerCount: 1,
|
||||
Line: 10,
|
||||
}
|
||||
if err := db.Create(cfg).Error; err != nil {
|
||||
t.Fatalf("seed config %s: %v", uuid, err)
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
func TestIncrementalPullMariaDB(t *testing.T) {
|
||||
serverDB := newMariaDBForPullTest(t)
|
||||
local := newLocalDBForSyncTest(t)
|
||||
if err := local.SaveSettings("127.0.0.1", 33306, "qf", "root", "test"); err != nil {
|
||||
t.Fatalf("save settings: %v", err)
|
||||
}
|
||||
|
||||
// Before migration 035: every pull is full, exactly as before.
|
||||
seedServerConfig(t, serverDB, "cfg-a", "A")
|
||||
seedServerConfig(t, serverDB, "cfg-b", "B")
|
||||
svc := syncsvc.NewServiceWithDB(serverDB, local)
|
||||
for i := 0; i < 2; i++ {
|
||||
res, err := svc.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("pre-migration import: %v", err)
|
||||
}
|
||||
if res.Imported+res.Updated != 2 {
|
||||
t.Fatalf("pre-migration pull %d: expected full pull of 2, got %+v", i, res)
|
||||
}
|
||||
}
|
||||
|
||||
applyServerUpdatedAtMigration(t, serverDB)
|
||||
applyServerUpdatedAtMigration(t, serverDB) // re-runnable
|
||||
// A client without the column in its model still inserts and updates fine.
|
||||
seedServerConfig(t, serverDB, "cfg-c", "C")
|
||||
// Age all rows past the overlap window.
|
||||
mustExec(t, serverDB, "UPDATE qt_configurations SET server_updated_at = NOW(6) - INTERVAL 1 HOUR")
|
||||
|
||||
// First pull after the migration in this process: full, sets the cursor.
|
||||
res, err := svc.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("first post-migration import: %v", err)
|
||||
}
|
||||
if res.Imported+res.Updated != 3 {
|
||||
t.Fatalf("expected full pull of 3, got %+v", res)
|
||||
}
|
||||
|
||||
// Nothing changed on the server: nothing is pulled.
|
||||
res, err = svc.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("idle incremental import: %v", err)
|
||||
}
|
||||
if res.Imported+res.Updated != 0 {
|
||||
t.Fatalf("expected empty incremental pull, got %+v", res)
|
||||
}
|
||||
|
||||
// An old-style client update (gorm Save of the whole struct) bumps the marker.
|
||||
var b models.Configuration
|
||||
if err := serverDB.Where("uuid = ?", "cfg-b").First(&b).Error; err != nil {
|
||||
t.Fatalf("load cfg-b: %v", err)
|
||||
}
|
||||
b.Name = "B renamed"
|
||||
if err := serverDB.Save(&b).Error; err != nil {
|
||||
t.Fatalf("save cfg-b: %v", err)
|
||||
}
|
||||
seedServerConfig(t, serverDB, "cfg-d", "D")
|
||||
|
||||
res, err = svc.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("incremental import: %v", err)
|
||||
}
|
||||
if res.Updated != 1 || res.Imported != 1 {
|
||||
t.Fatalf("expected 1 updated + 1 imported, got %+v", res)
|
||||
}
|
||||
localB, err := local.GetConfigurationByUUID("cfg-b")
|
||||
if err != nil || localB.Name != "B renamed" {
|
||||
t.Fatalf("expected cfg-b renamed locally, got %+v err=%v", localB, err)
|
||||
}
|
||||
|
||||
// A new process does one full pull again.
|
||||
restarted := syncsvc.NewServiceWithDB(serverDB, local)
|
||||
res, err = restarted.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("restart import: %v", err)
|
||||
}
|
||||
if res.Imported+res.Updated != 4 {
|
||||
t.Fatalf("expected full pull of 4 after restart, got %+v", res)
|
||||
}
|
||||
|
||||
// Switching to another server invalidates the cursor.
|
||||
if err := local.SaveSettings("other-host", 3306, "qf", "root", "test"); err != nil {
|
||||
t.Fatalf("save settings: %v", err)
|
||||
}
|
||||
res, err = restarted.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("import after server switch: %v", err)
|
||||
}
|
||||
if res.Imported+res.Updated != 4 {
|
||||
t.Fatalf("expected full pull of 4 after server switch, got %+v", res)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIncrementalProjectPullMariaDB(t *testing.T) {
|
||||
serverDB := newMariaDBForPullTest(t)
|
||||
local := newLocalDBForSyncTest(t)
|
||||
if err := local.SaveSettings("127.0.0.1", 33306, "qf", "root", "test"); err != nil {
|
||||
t.Fatalf("save settings: %v", err)
|
||||
}
|
||||
applyServerUpdatedAtMigration(t, serverDB)
|
||||
|
||||
code := "P-1"
|
||||
project := &models.Project{UUID: "prj-1", OwnerUsername: "tester", Code: code, IsActive: true}
|
||||
if err := serverDB.Create(project).Error; err != nil {
|
||||
t.Fatalf("seed project: %v", err)
|
||||
}
|
||||
mustExec(t, serverDB, "UPDATE qt_projects SET server_updated_at = NOW(6) - INTERVAL 1 HOUR")
|
||||
|
||||
svc := syncsvc.NewServiceWithDB(serverDB, local)
|
||||
if res, err := svc.ImportProjectsToLocal(); err != nil || res.Imported != 1 {
|
||||
t.Fatalf("first project pull: %+v err=%v", res, err)
|
||||
}
|
||||
if res, err := svc.ImportProjectsToLocal(); err != nil || res.Imported+res.Updated != 0 {
|
||||
t.Fatalf("idle project pull: %+v err=%v", res, err)
|
||||
}
|
||||
mustExec(t, serverDB, "UPDATE qt_projects SET name = 'renamed' WHERE uuid = 'prj-1'")
|
||||
if res, err := svc.ImportProjectsToLocal(); err != nil || res.Updated != 1 {
|
||||
t.Fatalf("incremental project pull: %+v err=%v", res, err)
|
||||
}
|
||||
}
|
||||
@@ -23,15 +23,16 @@ func (s *Service) PullPartnumberBooks() (int, error) {
|
||||
|
||||
localBookRepo := repository.NewPartnumberBookRepository(s.localDB.DB())
|
||||
|
||||
// Headers only: partnumbers_json is several MB in total and the worker runs
|
||||
// every minute, so it is fetched per book only when that book needs items.
|
||||
type serverBook struct {
|
||||
ID int `gorm:"column:id"`
|
||||
Version string `gorm:"column:version"`
|
||||
CreatedAt time.Time `gorm:"column:created_at"`
|
||||
IsActive bool `gorm:"column:is_active"`
|
||||
PartnumbersJSON string `gorm:"column:partnumbers_json"`
|
||||
ID int `gorm:"column:id"`
|
||||
Version string `gorm:"column:version"`
|
||||
CreatedAt time.Time `gorm:"column:created_at"`
|
||||
IsActive bool `gorm:"column:is_active"`
|
||||
}
|
||||
var serverBooks []serverBook
|
||||
if err := mariaDB.Raw("SELECT id, version, created_at, is_active, partnumbers_json FROM qt_partnumber_books ORDER BY created_at DESC, id DESC").Scan(&serverBooks).Error; err != nil {
|
||||
if err := mariaDB.Raw("SELECT id, version, created_at, is_active FROM qt_partnumber_books ORDER BY created_at DESC, id DESC").Scan(&serverBooks).Error; err != nil {
|
||||
return 0, fmt.Errorf("querying server partnumber books: %w", err)
|
||||
}
|
||||
slog.Info("partnumber books found on server", "count", len(serverBooks))
|
||||
@@ -40,39 +41,58 @@ func (s *Service) PullPartnumberBooks() (int, error) {
|
||||
for _, sb := range serverBooks {
|
||||
var existing localdb.LocalPartnumberBook
|
||||
err := s.localDB.DB().Where("server_id = ?", sb.ID).First(&existing).Error
|
||||
partnumbers, errPartnumbers := decodeServerPartnumbers(sb.PartnumbersJSON)
|
||||
if errPartnumbers != nil {
|
||||
slog.Error("failed to decode server partnumbers_json", "server_id", sb.ID, "error", errPartnumbers)
|
||||
continue
|
||||
}
|
||||
if err == nil {
|
||||
existing.Version = sb.Version
|
||||
existing.CreatedAt = sb.CreatedAt
|
||||
existing.IsActive = sb.IsActive
|
||||
if existing.Version != sb.Version || existing.IsActive != sb.IsActive {
|
||||
existing.Version = sb.Version
|
||||
existing.CreatedAt = sb.CreatedAt
|
||||
existing.IsActive = sb.IsActive
|
||||
if err := localBookRepo.SaveBook(&existing); err != nil {
|
||||
slog.Error("failed to update local partnumber book header", "server_id", sb.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
if localBookRepo.HasAllBookItems(existing.ID) {
|
||||
slog.Debug("partnumber book already synced, skipping", "server_id", sb.ID, "version", sb.Version)
|
||||
continue
|
||||
}
|
||||
// A server book may list partnumbers that qt_partnumber_book_items
|
||||
// lacks; re-pulling cannot fix that, so try once per process.
|
||||
if !s.markPartnumberBookRepair(sb.ID, sb.Version) {
|
||||
slog.Debug("partnumber book still incomplete after re-pull this session, skipping", "server_id", sb.ID, "version", sb.Version)
|
||||
continue
|
||||
}
|
||||
slog.Info("partnumber book header exists but catalog items are missing, re-pulling items", "server_id", sb.ID, "version", sb.Version)
|
||||
partnumbers, err := fetchServerBookPartnumbers(mariaDB, sb.ID)
|
||||
if err != nil {
|
||||
slog.Error("failed to fetch server partnumbers_json", "server_id", sb.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
existing.PartnumbersJSON = partnumbers
|
||||
if err := localBookRepo.SaveBook(&existing); err != nil {
|
||||
slog.Error("failed to update local partnumber book header", "server_id", sb.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
|
||||
localItemCount := localBookRepo.CountBookItems(existing.ID)
|
||||
if localItemCount > 0 && localBookRepo.HasAllBookItems(existing.ID) {
|
||||
slog.Debug("partnumber book already synced, skipping", "server_id", sb.ID, "version", sb.Version, "items", localItemCount)
|
||||
continue
|
||||
}
|
||||
slog.Info("partnumber book header exists but catalog items are missing, re-pulling items", "server_id", sb.ID, "version", sb.Version)
|
||||
n, err := pullBookItems(mariaDB, localBookRepo, existing.PartnumbersJSON)
|
||||
if err != nil {
|
||||
slog.Error("failed to re-pull items for existing book", "server_id", sb.ID, "error", err)
|
||||
} else {
|
||||
slog.Info("re-pulled items for existing book", "server_id", sb.ID, "version", sb.Version, "items_saved", n)
|
||||
pulled++
|
||||
if missing := localBookRepo.MissingBookItems(existing.ID); len(missing) > 0 {
|
||||
slog.Warn("partnumber book lists partnumbers absent from server qt_partnumber_book_items", "server_id", sb.ID, "version", sb.Version, "missing", missing)
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
slog.Info("pulling new partnumber book", "server_id", sb.ID, "version", sb.Version, "is_active", sb.IsActive)
|
||||
|
||||
partnumbers, err := fetchServerBookPartnumbers(mariaDB, sb.ID)
|
||||
if err != nil {
|
||||
slog.Error("failed to fetch server partnumbers_json", "server_id", sb.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
localBook := &localdb.LocalPartnumberBook{
|
||||
ServerID: sb.ID,
|
||||
Version: sb.Version,
|
||||
@@ -99,6 +119,30 @@ func (s *Service) PullPartnumberBooks() (int, error) {
|
||||
return pulled, nil
|
||||
}
|
||||
|
||||
// markPartnumberBookRepair records a re-pull attempt for a book version and
|
||||
// reports whether this is the first attempt in the current process.
|
||||
func (s *Service) markPartnumberBookRepair(serverID int, version string) bool {
|
||||
s.pnRepairMu.Lock()
|
||||
defer s.pnRepairMu.Unlock()
|
||||
if s.pnRepairAttempts == nil {
|
||||
s.pnRepairAttempts = map[int]string{}
|
||||
}
|
||||
if v, ok := s.pnRepairAttempts[serverID]; ok && v == version {
|
||||
return false
|
||||
}
|
||||
s.pnRepairAttempts[serverID] = version
|
||||
return true
|
||||
}
|
||||
|
||||
// fetchServerBookPartnumbers loads and decodes partnumbers_json of one server book.
|
||||
func fetchServerBookPartnumbers(mariaDB *gorm.DB, serverID int) (localdb.LocalStringList, error) {
|
||||
var raw string
|
||||
if err := mariaDB.Raw("SELECT partnumbers_json FROM qt_partnumber_books WHERE id = ?", serverID).Scan(&raw).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return decodeServerPartnumbers(raw)
|
||||
}
|
||||
|
||||
// pullBookItems fetches catalog items for a partnumber list from MariaDB and saves them to SQLite.
|
||||
// Returns the number of items saved.
|
||||
func pullBookItems(mariaDB *gorm.DB, repo *repository.PartnumberBookRepository, partnumbers localdb.LocalStringList) (int, error) {
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
package sync
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// serverPullOverlap re-reads rows changed shortly before the cursor, covering
|
||||
// transactions that stamped server_updated_at before the cursor but committed
|
||||
// after the previous pull read it.
|
||||
const serverPullOverlap = "INTERVAL 120 SECOND"
|
||||
|
||||
// serverPull describes how one pull of a server table should run. See
|
||||
// bible-local/decisions/2026-10-26-incremental-config-project-pull.md.
|
||||
type serverPull struct {
|
||||
table string
|
||||
scope string
|
||||
since string // cursor; empty = full pull
|
||||
startedAt string // server NOW(6) before reading; becomes the next cursor
|
||||
trackable bool // server has server_updated_at (migration 035)
|
||||
}
|
||||
|
||||
// beginServerPull decides between a full and an incremental pull of table.
|
||||
// The first pull of each table per process is full, so rows lost locally are
|
||||
// restored at least once per app start; later pulls read only rows whose
|
||||
// server_updated_at moved since the stored cursor.
|
||||
func (s *Service) beginServerPull(mariaDB *gorm.DB, table string) serverPull {
|
||||
pull := serverPull{table: table}
|
||||
if !s.serverHasChangeMarker(mariaDB, table) {
|
||||
return pull
|
||||
}
|
||||
if err := mariaDB.Raw("SELECT DATE_FORMAT(NOW(6), '%Y-%m-%d %H:%i:%s.%f')").Scan(&pull.startedAt).Error; err != nil || pull.startedAt == "" {
|
||||
slog.Warn("server pull: failed to read server time, falling back to full pull", "table", table, "error", err)
|
||||
return serverPull{table: table}
|
||||
}
|
||||
scope, err := s.localDB.ServerPullScope()
|
||||
if err != nil {
|
||||
return serverPull{table: table}
|
||||
}
|
||||
pull.scope = scope
|
||||
pull.trackable = true
|
||||
|
||||
s.pullMu.Lock()
|
||||
fullDone := s.fullPullDone[table]
|
||||
s.pullMu.Unlock()
|
||||
if fullDone {
|
||||
if since, ok := s.localDB.GetServerPullCursor(table, scope); ok {
|
||||
pull.since = since
|
||||
}
|
||||
}
|
||||
return pull
|
||||
}
|
||||
|
||||
// incrementalScope narrows query to rows changed since the cursor.
|
||||
func (p serverPull) incrementalScope(query *gorm.DB) *gorm.DB {
|
||||
return query.Where("server_updated_at >= CAST(? AS DATETIME(6)) - "+serverPullOverlap, p.since)
|
||||
}
|
||||
|
||||
// finishServerPull records a successful pull: the cursor moves to the server
|
||||
// time taken before reading, and a full pull marks the table as verified for
|
||||
// this process.
|
||||
func (s *Service) finishServerPull(pull serverPull) {
|
||||
if !pull.trackable {
|
||||
return
|
||||
}
|
||||
if err := s.localDB.SetServerPullCursor(pull.table, pull.scope, pull.startedAt); err != nil {
|
||||
slog.Warn("server pull: failed to store cursor", "table", pull.table, "error", err)
|
||||
return
|
||||
}
|
||||
if pull.since == "" {
|
||||
s.pullMu.Lock()
|
||||
if s.fullPullDone == nil {
|
||||
s.fullPullDone = map[string]bool{}
|
||||
}
|
||||
s.fullPullDone[pull.table] = true
|
||||
s.pullMu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// serverHasChangeMarker reports whether table has server_updated_at. A positive
|
||||
// answer is cached for the process; a negative one is re-checked next time so a
|
||||
// migration applied while the client runs is picked up.
|
||||
func (s *Service) serverHasChangeMarker(mariaDB *gorm.DB, table string) bool {
|
||||
s.pullMu.Lock()
|
||||
known := s.changeMarker[table]
|
||||
s.pullMu.Unlock()
|
||||
if known {
|
||||
return true
|
||||
}
|
||||
var n int64
|
||||
if err := mariaDB.Raw(`
|
||||
SELECT COUNT(*) FROM information_schema.COLUMNS
|
||||
WHERE table_schema = DATABASE() AND table_name = ? AND column_name = 'server_updated_at'
|
||||
`, table).Scan(&n).Error; err != nil || n == 0 {
|
||||
return false
|
||||
}
|
||||
s.pullMu.Lock()
|
||||
if s.changeMarker == nil {
|
||||
s.changeMarker = map[string]bool{}
|
||||
}
|
||||
s.changeMarker[table] = true
|
||||
s.pullMu.Unlock()
|
||||
return true
|
||||
}
|
||||
@@ -30,6 +30,13 @@ type Service struct {
|
||||
directDB *gorm.DB
|
||||
pricelistMu sync.Mutex // prevents concurrent pricelist syncs
|
||||
schemaOnce sync.Once // ensures ensureClientSchemaStateTable runs at most once per process
|
||||
|
||||
pnRepairMu sync.Mutex
|
||||
pnRepairAttempts map[int]string // server book id -> version already re-pulled this process
|
||||
|
||||
pullMu sync.Mutex
|
||||
changeMarker map[string]bool // server table -> has server_updated_at
|
||||
fullPullDone map[string]bool // server table -> full pull succeeded this process
|
||||
}
|
||||
|
||||
// NewService creates a new sync service
|
||||
@@ -118,13 +125,27 @@ func (s *Service) ImportConfigurationsToLocal() (*ConfigImportResult, error) {
|
||||
|
||||
configRepo := repository.NewConfigurationRepository(mariaDB)
|
||||
result := &ConfigImportResult{}
|
||||
pull := s.beginServerPull(mariaDB, "qt_configurations")
|
||||
|
||||
offset := 0
|
||||
const limit = 200
|
||||
for {
|
||||
serverConfigs, _, err := configRepo.ListAll(offset, limit)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing server configurations: %w", err)
|
||||
var serverConfigs []models.Configuration
|
||||
if pull.since != "" {
|
||||
// Incremental: the changed set is small, read it in one go.
|
||||
if offset > 0 {
|
||||
break
|
||||
}
|
||||
if err := pull.incrementalScope(mariaDB.Model(&models.Configuration{})).
|
||||
Order("created_at DESC, id DESC").
|
||||
Find(&serverConfigs).Error; err != nil {
|
||||
return nil, fmt.Errorf("listing changed server configurations: %w", err)
|
||||
}
|
||||
} else {
|
||||
serverConfigs, _, err = configRepo.ListAll(offset, limit)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing server configurations: %w", err)
|
||||
}
|
||||
}
|
||||
if len(serverConfigs) == 0 {
|
||||
break
|
||||
@@ -171,6 +192,7 @@ func (s *Service) ImportConfigurationsToLocal() (*ConfigImportResult, error) {
|
||||
offset += len(serverConfigs)
|
||||
}
|
||||
|
||||
s.finishServerPull(pull)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
@@ -184,13 +206,27 @@ func (s *Service) ImportProjectsToLocal() (*ProjectImportResult, error) {
|
||||
|
||||
projectRepo := repository.NewProjectRepository(mariaDB)
|
||||
result := &ProjectImportResult{}
|
||||
pull := s.beginServerPull(mariaDB, "qt_projects")
|
||||
|
||||
offset := 0
|
||||
const limit = 200
|
||||
for {
|
||||
serverProjects, _, err := projectRepo.List(offset, limit, true)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing server projects: %w", err)
|
||||
var serverProjects []models.Project
|
||||
if pull.since != "" {
|
||||
// Incremental: the changed set is small, read it in one go.
|
||||
if offset > 0 {
|
||||
break
|
||||
}
|
||||
if err := pull.incrementalScope(mariaDB.Model(&models.Project{})).
|
||||
Order("created_at DESC").
|
||||
Find(&serverProjects).Error; err != nil {
|
||||
return nil, fmt.Errorf("listing changed server projects: %w", err)
|
||||
}
|
||||
} else {
|
||||
serverProjects, _, err = projectRepo.List(offset, limit, true)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing server projects: %w", err)
|
||||
}
|
||||
}
|
||||
if len(serverProjects) == 0 {
|
||||
break
|
||||
@@ -247,6 +283,7 @@ func (s *Service) ImportProjectsToLocal() (*ProjectImportResult, error) {
|
||||
offset += len(serverProjects)
|
||||
}
|
||||
|
||||
s.finishServerPull(pull)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
@@ -291,28 +328,30 @@ func (s *Service) NeedSync() (bool, error) {
|
||||
mariaDB, err := s.getDB()
|
||||
if err == nil {
|
||||
pricelistRepo := repository.NewPricelistRepository(mariaDB)
|
||||
sources := []models.PricelistSource{
|
||||
models.PricelistSourceEstimate,
|
||||
models.PricelistSourceWarehouse,
|
||||
models.PricelistSourceCompetitor,
|
||||
sources := []string{
|
||||
string(models.PricelistSourceEstimate),
|
||||
string(models.PricelistSourceWarehouse),
|
||||
string(models.PricelistSourceCompetitor),
|
||||
}
|
||||
allMatch := true
|
||||
for _, source := range sources {
|
||||
latestServer, err := pricelistRepo.GetLatestActiveBySource(string(source))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
latestLocal, err := s.localDB.GetLatestLocalPricelistBySource(string(source))
|
||||
if err != nil {
|
||||
return true, nil
|
||||
}
|
||||
if latestServer.ID != latestLocal.ServerID {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
if allMatch {
|
||||
latestServer, err := pricelistRepo.GetLatestActiveIDsBySources(sources)
|
||||
if err != nil {
|
||||
slog.Warn("failed to check latest server pricelists", "error", err)
|
||||
return false, nil
|
||||
}
|
||||
for _, source := range sources {
|
||||
serverID, ok := latestServer[source]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
latestLocal, err := s.localDB.GetLatestLocalPricelistBySource(source)
|
||||
if err != nil {
|
||||
return true, nil
|
||||
}
|
||||
if serverID != latestLocal.ServerID {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
@@ -941,6 +980,12 @@ func (s *Service) PushPendingChanges() (int, error) {
|
||||
if _, err := s.EnsureReadinessForSync(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return s.pushPendingChanges()
|
||||
}
|
||||
|
||||
// pushPendingChanges pushes without its own readiness check; the caller has
|
||||
// just run EnsureReadinessForSync (and with it the client heartbeat).
|
||||
func (s *Service) pushPendingChanges() (int, error) {
|
||||
|
||||
removed, err := s.localDB.PurgeOrphanConfigurationPendingChanges()
|
||||
if err != nil {
|
||||
|
||||
@@ -3,12 +3,29 @@ package sync
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"math/rand/v2"
|
||||
"time"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/db"
|
||||
)
|
||||
|
||||
// Worker performs background synchronization at regular intervals
|
||||
// WorkerJitter spreads background sync cycles by ±30% of the interval so that
|
||||
// clients started at the same time do not hit MariaDB in lockstep.
|
||||
const WorkerJitter = 0.3
|
||||
|
||||
// jitteredInterval returns base scaled by a random factor in [1-WorkerJitter, 1+WorkerJitter].
|
||||
func jitteredInterval(base time.Duration) time.Duration {
|
||||
factor := 1 - WorkerJitter + rand.Float64()*2*WorkerJitter
|
||||
return time.Duration(float64(base) * factor)
|
||||
}
|
||||
|
||||
// nextCycleDelay returns how long to wait so the next cycle starts a jittered
|
||||
// interval after started; zero if the cycle itself took longer than that.
|
||||
func nextCycleDelay(started time.Time, base time.Duration) time.Duration {
|
||||
return max(jitteredInterval(base)-time.Since(started), 0)
|
||||
}
|
||||
|
||||
// Worker performs background synchronization at jittered intervals
|
||||
type Worker struct {
|
||||
service *Service
|
||||
connMgr *db.ConnectionManager
|
||||
@@ -37,12 +54,14 @@ func (w *Worker) isOnline() bool {
|
||||
func (w *Worker) Start(ctx context.Context) {
|
||||
w.logger.Info("starting background sync worker", "interval", w.interval)
|
||||
|
||||
ticker := time.NewTicker(w.interval)
|
||||
defer ticker.Stop()
|
||||
|
||||
// Run once immediately
|
||||
// Run once immediately. The next cycle is scheduled from the start of the
|
||||
// previous one, so heartbeats are at most (1+WorkerJitter)*interval apart.
|
||||
started := time.Now()
|
||||
w.runSync()
|
||||
|
||||
timer := time.NewTimer(nextCycleDelay(started, w.interval))
|
||||
defer timer.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
@@ -51,8 +70,10 @@ func (w *Worker) Start(ctx context.Context) {
|
||||
case <-w.stopCh:
|
||||
w.logger.Info("background sync worker stopped")
|
||||
return
|
||||
case <-ticker.C:
|
||||
case <-timer.C:
|
||||
started = time.Now()
|
||||
w.runSync()
|
||||
timer.Reset(nextCycleDelay(started, w.interval))
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -79,7 +100,7 @@ func (w *Worker) runSync() {
|
||||
"reason_code", readiness.ReasonCode,
|
||||
"reason_text", readiness.ReasonText,
|
||||
)
|
||||
} else if pushed, err := w.service.PushPendingChanges(); err != nil {
|
||||
} else if pushed, err := w.service.pushPendingChanges(); err != nil {
|
||||
w.logger.Warn("background sync: failed to push pending changes", "error", err)
|
||||
} else if pushed > 0 {
|
||||
w.logger.Info("background sync: pushed pending changes", "count", pushed)
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
package sync
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestJitteredIntervalStaysWithinBounds(t *testing.T) {
|
||||
base := time.Minute
|
||||
lo := time.Duration(float64(base) * (1 - WorkerJitter))
|
||||
hi := time.Duration(float64(base) * (1 + WorkerJitter))
|
||||
seen := map[time.Duration]struct{}{}
|
||||
for i := 0; i < 1000; i++ {
|
||||
d := jitteredInterval(base)
|
||||
if d < lo || d > hi {
|
||||
t.Fatalf("interval %v outside [%v, %v]", d, lo, hi)
|
||||
}
|
||||
seen[d] = struct{}{}
|
||||
}
|
||||
if len(seen) < 100 {
|
||||
t.Fatalf("expected spread of intervals, got %d distinct values", len(seen))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
-- Tables affected: qt_configurations, qt_projects
|
||||
-- recovery.not-started: safe to re-run; ADD COLUMN IF NOT EXISTS / ADD INDEX IF NOT EXISTS
|
||||
-- recovery.partial: ALTER TABLE qt_configurations DROP COLUMN server_updated_at; ALTER TABLE qt_projects DROP COLUMN server_updated_at;
|
||||
-- recovery.completed: no action needed
|
||||
-- verify: qt_configurations.server_updated_at column missing | SELECT 1 FROM information_schema.COLUMNS WHERE table_schema=DATABASE() AND table_name='qt_configurations' AND column_name='server_updated_at' HAVING COUNT(*)=0
|
||||
-- verify: qt_projects.server_updated_at column missing | SELECT 1 FROM information_schema.COLUMNS WHERE table_schema=DATABASE() AND table_name='qt_projects' AND column_name='server_updated_at' HAVING COUNT(*)=0
|
||||
|
||||
-- Server-maintained change marker for incremental client pulls. Never written by
|
||||
-- clients: DEFAULT on INSERT and ON UPDATE on any row change, so clients that do not
|
||||
-- know the column keep working unchanged. See
|
||||
-- bible-local/decisions/2026-10-26-incremental-config-project-pull.md.
|
||||
ALTER TABLE qt_configurations
|
||||
ADD COLUMN IF NOT EXISTS server_updated_at TIMESTAMP(6) NOT NULL
|
||||
DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
|
||||
ADD INDEX IF NOT EXISTS idx_qt_configurations_server_updated_at (server_updated_at, id);
|
||||
|
||||
ALTER TABLE qt_projects
|
||||
ADD COLUMN IF NOT EXISTS server_updated_at TIMESTAMP(6) NOT NULL
|
||||
DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
|
||||
ADD INDEX IF NOT EXISTS idx_qt_projects_server_updated_at (server_updated_at, id);
|
||||
@@ -0,0 +1,26 @@
|
||||
# QuoteForge v3.50
|
||||
|
||||
Дата релиза: 2026-10-26
|
||||
Тег: `v3.50`
|
||||
|
||||
Предыдущий релиз: `v3.49`
|
||||
|
||||
## Ключевые изменения
|
||||
|
||||
- локальный API и первоначальная настройка защищены от CSRF и DNS rebinding; сохранённый пароль не отправляется при изменении хоста, порта или пользователя без явного ввода пароля (#2);
|
||||
- устранены XSS через LOT, BOM и сообщения синхронизации; обработчики корзины и аренды получают LOT через data-атрибуты (#4);
|
||||
- перед сбросом локальной базы создаётся отдельный проверенный архив; несинхронизированные данные защищены отказом без явного принудительного сброса, переменная QFS_RESET_LOCAL_DB больше не запускает сброс (#7);
|
||||
- в настройках подключения MariaDB доступны TLS preferred, required, verify-ca и verify-full, а также CA-сертификат; preferred допускает соединение без TLS, verify-full проверяет сертификат и имя сервера (#5);
|
||||
- SQL-миграции выполняются по одной сессии на файл, запускают verify-проверки до записи в историю и не скрывают ошибки внешних ключей (#3);
|
||||
- минимальная версия Go повышена до 1.27.3, обновлены зависимости; релизная сборка блокируется при достижимых уязвимостях по govulncheck (#6).
|
||||
|
||||
## Проверки
|
||||
|
||||
Пройдены Go-тесты, go vet, проверка XSS и сборки macOS arm64 / Windows amd64.
|
||||
Govulncheck: 0 достижимых уязвимостей; остаётся одна запись на уровне модуля,
|
||||
уязвимый код которого приложение не вызывает.
|
||||
|
||||
## Запуск на macOS
|
||||
|
||||
Снимите карантинный атрибут через терминал: `xattr -d com.apple.quarantine /path/to/qfs-darwin-arm64`
|
||||
После этого бинарник запустится без предупреждения Gatekeeper.
|
||||
Reference in new issue
Block a user