Compare commits

..
13 Commits
Author SHA1 Message Date
mchus 52720d8e38 Merge pull request 'Release notes v3.52' (#10) from docs/release-v3.52 into main 2026-10-26 22:58:47 +03:00
Mikhail Chusavitin 0ca30afba3 docs: release notes v3.52 2026-10-26 22:58:17 +03:00
mchus 60b6047279 Merge pull request 'Приёмка #2–#7 и восстановление серверных миграций по bible' (#9) from fix/issue-acceptance-and-migration-safety into main 2026-10-26 22:49:53 +03:00
Mikhail Chusavitin 8569871a17 fix: verify server snapshots and recover whole migration sessions (#3) 2026-10-26 22:48:07 +03:00
Mikhail Chusavitin 3d4101ecc0 test: enforce release failure gates and record platform builds (#6) 2026-10-26 21:22:34 +03:00
Mikhail Chusavitin 46d7ccda4d test: verify TLS policies against disposable MariaDB servers (#5) 2026-10-26 21:10:24 +03:00
Mikhail Chusavitin 2d60fcfb39 test: verify reset CLI and restored startup preserve local work (#7) 2026-10-26 21:07:41 +03:00
Mikhail Chusavitin ffdbe4fbd0 test: exercise XSS regressions in headless Chrome (#4) 2026-10-26 21:03:36 +03:00
Mikhail Chusavitin e8dd03c4f5 test: verify browser trust boundary on both application routers (#2) 2026-10-26 20:59:35 +03:00
Mikhail ChusavitinandClaude Opus 5.5 fa09513e26 docs: release notes v3.51
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 20:52:03 +03:00
Mikhail ChusavitinandClaude Opus 5.5 44a09d4420 fix(sync): stop MariaDB connection leak and cut per-cycle sync waste
- ConnectionManager reuses a stale-but-healthy pool instead of dialing a new
  one every 30s, and closes every pool it replaces (one client held ~30
  connections). Pool capped at 2 connections, idle released after 1 min.
- Background sync every 1 min with ±30% jitter, scheduled from cycle start;
  users-status online threshold = interval × 1.3. One heartbeat per cycle.
- Partnumber books: read headers only, fetch partnumbers_json per book when
  new/incomplete; empty book counts as complete; an incomplete book is
  re-pulled once per process (server book lists BOT_AGIBOT_G2_PERF missing
  from qt_partnumber_book_items).
- NeedSync in one UNION ALL query; SyncQtSettings skips unchanged rewrite.
- Migration 035: server-maintained server_updated_at on qt_configurations and
  qt_projects. Clients pull only changed rows after one full pull per process;
  servers without 035 and old clients keep full pulls.
- Migrator: fresh GORM statement per verify query and history insert; files
  with two verify checks failed to record.

ADL: bible-local/decisions/2026-10-26-mariadb-single-pool.md,
bible-local/decisions/2026-10-26-incremental-config-project-pull.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 20:46:59 +03:00
Mikhail ChusavitinandCodex 8a08270b79 docs: release notes v3.50
Co-Authored-By: Codex <noreply@openai.com>
2026-10-26 10:58:39 +03:00
mchus cab5d13f16 Merge pull request 'Исправления шести issue: безопасность API, защита данных, TLS и миграции' (#8) from fix/open-issues-security-recovery into main 2026-10-26 10:56:07 +03:00
56 changed files with 3493 additions and 243 deletions

No files matched your search

+4 -1
View File
@@ -349,4 +349,7 @@ Rules:
quoted attributes. Never interpolate data into JavaScript event attributes.
Cart and rental controls carry escaped LOT values in `data-lot` and dispatch
through delegated listeners. Sync errors are escaped as text; toast messages use
`textContent`. Run `node scripts/tests/html-security.cjs` for the regression check.
`textContent`. Run `node scripts/tests/html-security.cjs` for the fast regression check and
`node scripts/tests/html-security-browser.cjs` for real DOM parsing and delegated
events in headless Chrome (`CHROME_BIN` overrides the executable). The browser
check uses synthetic data, controlled API responses and a temporary profile.
+27 -2
View File
@@ -196,6 +196,7 @@ PK: lot_name
| rental_items | JSON | per-lot New/БУ condition for the Аренда tab; see [02-architecture.md](02-architecture.md#rental--paid-testing-pricing-contract) |
| rental_uplift_percent | decimal(8,2) DEFAULT 0 | uplift applied to Estimate buy price for rental "Цена" |
| feature_flags | JSON NULL | added by migration 034. Data features the configuration uses (`["sale_row_markups"]`); derived on save, a client refuses to write a config with a flag it doesn't know. See [decisions/2026-09-24-configuration-feature-flags.md](decisions/2026-09-24-configuration-feature-flags.md) |
| server_updated_at | timestamp(6) NOT NULL | added by migration 035. Server-maintained change marker (`DEFAULT`/`ON UPDATE CURRENT_TIMESTAMP(6)`, index `(server_updated_at, id)`); never written by clients, used for incremental pulls. See [decisions/2026-10-26-incremental-config-project-pull.md](decisions/2026-10-26-incremental-config-project-pull.md) |
### qt_lot_metadata
PK: lot_name
@@ -306,6 +307,7 @@ PK: username
| rental_enabled | tinyint(1) DEFAULT 0 | shows the Аренда tab on this project's configurations |
| created_at | timestamp | |
| updated_at | timestamp ON UPDATE | |
| server_updated_at | timestamp(6) NOT NULL | added by migration 035. Server-maintained change marker (`DEFAULT`/`ON UPDATE CURRENT_TIMESTAMP(6)`, index `(server_updated_at, id)`); never written by clients, used for incremental pulls. See [decisions/2026-10-26-incremental-config-project-pull.md](decisions/2026-10-26-incremental-config-project-pull.md) |
### qt_schema_migrations
| Column | Type | Notes |
@@ -439,16 +441,39 @@ MariaDB:
`UPTO` as applied **without running them**, for a server whose schema is already current;
- migrations use their own connection without read/write timeouts (`openMigrationDB`),
not the app's 3s-timeout sync connection — a long backfill must not be cut mid-file;
- server migration commands finish before runtime connection workers start;
- `-migrate` requires a DBA maintenance window with every other writer stopped,
directly granted global ALL PRIVILEGES, and no other connection using the schema.
Do not expand runtime-user privileges; use the existing developer DBA workflow;
- the runner holds a schema advisory lock and one connection for the entire session.
It automatically writes a full consistent snapshot before changes and per-table
snapshots before every file; each archive is restored into a temporary schema
and compared before any corresponding original-schema change. Snapshot or
verification failure stops the run. See [06-backup.md](06-backup.md);
- on a statement/verify failure, per-step snapshots are restored in reverse order,
including migration history; a full-session snapshot is the fallback. A persistent
marker refuses retry after interruption until `qfs -migrate-restore <archive.zip>`
restores and verifies the original state. Connection/recovery failure retains the
marker and archives and emits an explicit restore diagnostic;
- recovery supports InnoDB base tables and their triggers. Views, routines,
events and cross-schema FKs block migrations before changes; new migrations must
stay within supported table SQL. Existing triggers are captured with their
definer, body, order and SQL/charset context, then restored after all data; Verification requires CREATE/DROP DATABASE.
The runner refuses NO_BACKSLASH_ESCAPES sessions;
- each file, its `-- verify: <description> | <SQL>` queries and history insertion run
on one database session, preserving session variables;
- every verify query and the history insert use a fresh GORM statement
(`Session{NewDB: true}`) on that session; chaining them on one statement leaked state
and broke files with two or more verify checks (fixed 2026-10-26 with migration 035);
- a file is recorded only after statements and all verify checks succeed. Verify
queries return rows only for broken invariants; a row or SQL error stops the run
without marking that file applied. Malformed verify headers fail before execution;
- only duplicate column/key names and already-existing objects are tolerated.
FK creation errors (including errno 150/121) remain fatal;
- each file must be safe to re-run after a partial failure (`ADD COLUMN IF NOT EXISTS`,
idempotent `UPDATE ... WHERE <not yet done>`). The current statement splitter assumes
semicolons occur only between statements; do not add semicolons inside literals;
idempotent `UPDATE ... WHERE <not yet done>`). The statement parser preserves
semicolons in literals/quoted identifiers and strips ordinary comments. Malformed
quotes, executable comments and DELIMITER directives are rejected before execution;
- there is no GORM `AutoMigrate` against MariaDB: it retypes production columns to match the
running binary's models. On 2026-09-24 a stale v1.18 binary run with the old AutoMigrate
`-migrate` did exactly that, which is why it was removed.
+1
View File
@@ -65,6 +65,7 @@ They are stored in SQLite and encrypted with `local_encryption.key` unless `QUOT
| `-reset-localdb` | reset after mandatory verified backup; refuses unsynced work |
| `-force-reset-localdb` | with `-reset-localdb`, allow clearing unsynced work; backup remains mandatory |
| `-migrate` | apply pending server SQL migrations (`migrations/*.sql`) and exit |
| `-migrate-restore <archive.zip>` | restore and verify a server migration snapshot during DBA maintenance |
| `-migrate-status` | list server SQL migrations with applied/pending state and exit |
| `-migrate-baseline <file>` | record server migrations up to `<file>` as applied without running them, and exit |
| `-version` | print app version and exit |
+64
View File
@@ -71,3 +71,67 @@ in one transaction and preserve connection settings. Missing legacy tables are
skipped. Use the restore procedure above for the pre-reset archive; retain it
until the reset has been verified. The environment reset switch was removed to
prevent repeated destructive startup.
## Server migration snapshots
`qfs -migrate` writes mandatory archives under
`<backup root>/server-migrations/qfs-mariadb-v1-<label>-<UTC timestamp>-<unique>.zip`.
`QFS_BACKUP_DIR` overrides the root; `QFS_BACKUP_DISABLE` never disables safety
snapshots. Paths are checked outside git, including symlink targets, before
creation and restore. Archives have private filesystem permissions and contain
server table schemas and data, which may be sensitive; never put them in git.
Connection passwords and the local encryption key are not copied into metadata.
One full snapshot precedes the session. Before each file, a separate snapshot
contains a schema definition and row stream for **every** table: this includes
all affected tables even when a legacy Tables-affected header is incomplete.
This intentionally favors recovery over storage/time efficiency for the manual
DBA tool. Retain safety snapshots outside normal rotation until the migration
has been verified; remove verified-session artifacts as operational cleanup.
The versioned ZIP manifest records table schemas, writable column order, row
counts and order-independent row digests. Row streams preserve NULL, empty
strings, binary values and fractional timestamps. Restore temporarily uses
NO_AUTO_VALUE_ON_ZERO to preserve explicit zero IDs and legacy zero dates;
row digests detect coercion and the original session mode is restored. Generated columns are rebuilt
from schema, not inserted. Trigger bodies, definers, firing order and creation
SQL/charset context are preserved. Triggers are recreated after row loading so
restore does not duplicate audit data or alter restored values. CRC/format/digests are checked, then each archive is
restored into a unique temporary schema on the same server and compared for
schema and data equality. The temporary schema is dropped afterward. A failed
or interrupted check must be investigated before retry; abandoned `qfs_verify_`
schemas are DBA cleanup artifacts, never application data.
Stop every other writer for the entire migration/recovery window, including
clients that could reconnect. The runner detects other current schema connections
and uses an advisory lock to exclude a second runner; the lock does not stop
unrelated SQL clients. An active client appearing during failure recovery causes
recovery refusal rather than overwriting its work. InnoDB base tables and triggers are supported; views, routines, events and
cross-schema FKs require explicit recovery support before migration. Triggers
whose recorded database collation differs from the current database collation
also cause refusal until that legacy state has supported recovery. Trigger
routine calls and qualified references to other schemas are refused; source
SQL must have no effects outside the captured schema.
Unsupported schemas stop before changes. Runtime database-user grants stay as
listed in [03-database.md](03-database.md); the tool requires an existing DBA account.
Failure restores step snapshots in reverse order; the full-session archive is
used if step recovery is insufficient. Schema, writable data and migration
history must match the session start before recovery is reported successful.
The persistent `.pending` marker is keyed by configured endpoint and schema, so
server restarts do not hide an interrupted session. Backups survive connection
loss; a failed recovery retains its marker and reports the full snapshot path.
After an interruption, keep clients stopped and use the snapshot path reported
by the runner:
```bash
go run ./cmd/qfs -migrate-restore /outside/repository/server-migrations/qfs-mariadb-v1-session-<timestamp>-<unique>.zip
make migrate-status
make migrate
```
Restore uses the currently saved DBA connection, requires the same database name,
checks the archive by restoring it into a temporary schema first, restores the
original schema, verifies equality and clears the matching pending marker. It
does not require mysql/mysqldump or any operator-installed database utility.
+8
View File
@@ -10,6 +10,12 @@ go run ./cmd/migrate_project_updated_at
go test ./...
go vet ./...
make security-check # pinned govulncheck, blocks reachable vulnerabilities
# Optional live transport checks; owns disposable Docker containers:
go test -tags integration ./internal/mariadb -run TestMariaDBTLSIntegration -count=1
# Snapshot/DDL recovery checks; owns a disposable MariaDB 11.8 container:
go test -tags integration ./internal/models -run TestSQLMigrationSafetyMariaDB -count=1
# Production-dump rehearsal; source path via QFS_TEST_PRODUCTION_DUMP:
go test -tags integration ./internal/models -run TestProductionDumpMigrationMariaDB -count=1 -v
make build-release
make install-hooks
```
@@ -45,6 +51,8 @@ normal Go toolchain selection downloads a suitable compiler when necessary.
The scanner version is pinned in the Makefile. All release build targets and
`scripts/release.sh` require this check to succeed before producing artifacts.
A scanner/database/download failure also stops release.
`node scripts/tests/release-security.cjs` verifies failure stops all release
entry points before artifact creation using an isolated failing scanner.
When updating dependencies: run `go mod tidy`, `go test ./...`, `go vet ./...`,
`make security-check` and the JavaScript regression check documented in
@@ -0,0 +1,52 @@
# Decision: incremental pull of configurations and projects via server_updated_at
**Date:** 2026-10-26
**Status:** active
## Context
Opening the configurations or projects list (`GET /api/configs`, `/api/projects`,
`/api/projects/:uuid/configs`, on-demand with a 30s cooldown) and the manual sync ran
`ImportConfigurationsToLocal` / `ImportProjectsToLocal`, which read **every** server row
and rewrote it locally. Measured on production (2026-10-26): configurations 22.2s
(4158 rows, ~3.2 MB, 4078 rewritten for one actual change), projects 4.6s (1692 rows),
holding a pool connection the whole time. `qt_configurations` had no change column;
`qt_projects.updated_at` is set by clients (`SaveProjectPreservingUpdatedAt`), so a change
made offline and pushed later carries an old timestamp and cannot drive a filter.
## Decision
- Server migration `035_add_server_updated_at.sql` adds
`server_updated_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6)`
plus index `(server_updated_at, id)` to `qt_configurations` and `qt_projects`. MariaDB
maintains it; no client writes it and it is **not** in `models.Configuration` /
`models.Project`, so writes from any client version (GORM `Create`/`Save`, raw
`UPDATE` with column lists) keep working and bump the marker on every real change.
- Client (`internal/services/sync/pull_cursor.go`):
- if the column is missing (server not migrated yet) every pull is full, exactly as
before; presence is re-checked until found, then cached for the process;
- the first pull of each table per process is full (restores rows lost locally once per
app start), later pulls read `server_updated_at >= cursor - 120s` in one query;
- the cursor is the server's `NOW(6)` read *before* the pull, stored in `app_settings`
(`server_pull_cursor:<table>`) together with `host:port/database`; a cursor from a
different server is ignored;
- the per-row merge (skip `pending`, keep local deactivation sticky, keep `line`) is
unchanged; only the set of rows read differs.
- Old clients keep doing full pulls and are unaffected.
## Consequences
- Idle pulls go from ~27s / 3.2 MB to one indexed range query.
- Hard deletes on the server are not propagated — same as the full pull before.
- Changes applied by InnoDB `ON UPDATE CASCADE` (e.g. a `qt_projects.uuid` change
cascading to `qt_configurations.project_uuid`) do not bump `server_updated_at`; they are
picked up by the next full pull (app start).
- Cursor comparison happens in the server session time zone. Production is `SYSTEM`/MSK
(no DST); on a server with DST, rows changed around the autumn transition could be
missed until the next app start.
- Applying 035 adds the column and builds an index on both tables (~4k and ~2k rows on
production, seconds); `qfs -migrate` is run once by the developer as usual.
- Tests: `internal/services/sync/incremental_pull_mariadb_test.go` (needs
`QF_TEST_MARIADB_DSN`, a disposable MariaDB with the production table DDL; verified on
MariaDB 11.8, same as production), migrator fix in
`internal/models/sql_migration_verify_test.go` (`two checks pass`).
@@ -0,0 +1,54 @@
# Decision: one MariaDB pool per client — reuse, never orphan
**Date:** 2026-10-26
**Status:** active
## Context
A single client held ~30 MariaDB connections during sync. `ConnectionManager.GetDB`
treated a pool older than `pingInterval` (30s) as unusable and dialed a brand-new
`*gorm.DB` without closing the previous one. Sync calls `getDB()` per step, so every
~30s a new pool appeared while the old one kept its idle connections until
`ConnMaxLifetime` (5 min): ~10 orphaned pools × 2 idle + up to 10 active ≈ 30. Each
orphaned `*sql.DB` also leaked its opener goroutine. `TryConnect`, the background
ping-failure path and a failed ping inside `dial` dropped pools the same way.
## Decision
- `GetDB` pings the existing pool when it is merely stale and returns it; it dials only
when there is no pool or the ping fails.
- Whenever the manager replaces or drops a pool (`GetDB`, `TryConnect`,
`checkOnlineNow`, `MarkOffline`, `Disconnect`, failed ping in `dial`) the old
`*sql.DB` is closed.
- Pool limits: `MaxOpenConns=2`, `MaxIdleConns=2`, `ConnMaxLifetime=5m`,
`ConnMaxIdleTime=1m` — one client holds at most 2 connections, idle ones are released
within a minute. Sync is sequential (one query at a time); the second connection lets
UI server queries (active viewers, users status) run alongside a long sync step.
- Background sync interval (`backgroundSyncInterval` in `cmd/qfs/main.go`) is 1 minute
with ±30% random jitter (`sync.WorkerJitter`), so clients do not hit MariaDB in
lockstep. The next cycle is scheduled from the *start* of the previous one, so
heartbeats (`qt_client_schema_state.last_checked_at`) are at most 1.3 × interval apart
while a cycle is shorter than that.
- `/api/sync/users-status` online threshold = interval × (1 + `WorkerJitter`).
- `PullPartnumberBooks` reads only book headers (`id, version, created_at, is_active`)
each cycle; `partnumbers_json` (~5 MB for all books) is fetched per book only when the
book is new or its local items are incomplete. An empty book counts as complete. An
incomplete book is re-pulled at most once per process per version: a server book can
list partnumbers missing from `qt_partnumber_book_items` (seen: `BOT_AGIBOT_G2_PERF`
in `PNBOOK-2026-10-21-001`), which a re-pull cannot fix; this is logged as `WARN` with
the missing partnumbers.
- A worker cycle sends one heartbeat: the worker calls `EnsureReadinessForSync` once and
then the unexported `pushPendingChanges` (the exported `PushPendingChanges` still checks
readiness itself for the manual-sync handlers).
- `NeedSync` asks for the latest active pricelist of all three sources in one
`UNION ALL` query (`GetLatestActiveIDsBySources`): 0.71s → 0.24s.
- `SyncQtSettings` skips the local `DELETE` + `INSERT` when the server rows equal the cache.
- Measured against the production server: a cold sync took 171s (pricelists 105s,
partnumber books 38s, configurations 23s) and is bound by server reads, not pool
size. A warm cycle takes ~1.4s (was ~2.5s plus a 5 MB download).
## Consequences
- Code that keeps a `*gorm.DB` across a reconnect gets `sql: database is closed`; sync
steps must call `getDB()` per operation instead of caching the handle.
- Tests: `internal/db/connection_test.go`, `internal/services/sync/worker_jitter_test.go`.
@@ -0,0 +1,62 @@
# Decision: verified snapshots and whole-session recovery for server migrations
**Date:** 2026-10-26
**Status:** active
## Context
Issue #3 fixes FK error propagation, session variables and verify execution.
Its proposed manual-backup exception conflicts with the shared go-database
contract. MariaDB DDL cannot be made rollback-safe by wrapping the migration
files in an ordinary transaction: see [MariaDB implicit commits](https://mariadb.com/docs/server/reference/sql-statements/transactions/sql-statements-that-cause-an-implicit-commit).
## Decision
The migration engine, rather than the operator, owns full-session and pre-file
per-table backups. A Go ZIP/JSON snapshot format avoids external database-client
dependencies. Every archive is restored into a temporary schema and compared
before the corresponding original-schema changes. Per-file archives include all
tables to cover incomplete legacy affected-table headers and dynamic SQL.
On failure, restore each step snapshot in reverse order; use the full-session
snapshot as fallback. Include migration history in recovery. Persist a recovery
marker before applying any changes, refuse an interrupted session on retry and
ship an explicit `-migrate-restore` path. Restore failures retain archives and
produce diagnostics instead of claiming success.
The production dump contains 27 triggers, so the implementation preserves
InnoDB tables and trigger bodies, definers, order and SQL/charset context.
Triggers are recreated after row loading to avoid firing them during restore;
verification binds them to the temporary schema and performs no later DML.
Views, routines, events, cross-schema FKs and legacy trigger/database collation
mismatches are refused before changes. It requires an existing
DBA account with directly granted global ALL PRIVILEGES so object/client
visibility cannot silently hide unsupported state, plus a maintenance window
without other writers. Do not grant these privileges to the runtime user. The
advisory migration lock excludes other runners, not arbitrary SQL clients.
The parser supports SQL literals and quoted identifiers containing semicolons,
escaped/doubled quotes and ordinary comments. It rejects malformed input,
executable comments and DELIMITER rather than silently misinterpreting them.
Existing merged SQL files are unchanged.
## Consequences
Safety snapshots and restore verification add disk usage, temporary server schema
creation and runtime to the developer-only migration command. Normal local-first
CRUD and sync gain no MariaDB dependency. Migration commands start no background
connection worker. Before production use, rehearse on a current production dump
in MariaDB matching production version and collation; synthetic tests alone do
not satisfy that gate. Unsupported schema features need explicit recovery support
before such schemas may be migrated by this tool.
## Acceptance
Rehearsed on the supplied 2026-10-26 production dump (45 tables, 27 triggers),
using an isolated MariaDB 11.8.3 container and utf8mb4_uca1400_ai_ci. The test
verifies full restore, actual migration 035 on the reconstructed pre-035 clone,
reverse recovery after a later DDL/verify failure, successful retry and exact
restoration of the original dump state. Synthetic tests additionally exercise
FK errno 150, interrupted-session recovery, NULL/empty/binary/decimal/timestamp
values, zero auto-increment IDs and legacy zero dates. No production changes
are performed by these checks.
+105 -21
View File
@@ -2,19 +2,19 @@
Repository: https://git.mchus.pro/mchus/QuoteForge
All issue pages were retrieved, including closed issues: seven total, six open.
Issue [#1](https://git.mchus.pro/mchus/QuoteForge/issues/1) was already closed and
is excluded from implementation. Work is sequential on
`fix/open-issues-security-recovery`, with one implementation commit per open issue.
Scope: issues #2-#7. Initial fixes were merged into `main` in
[PR #8](https://git.mchus.pro/mchus/QuoteForge/pulls/8), with release notes in v3.50.
The follow-up adds completed acceptance checks and server migration recovery
required by the shared rules. Each issue has its own implementation/acceptance commit.
| Order | Issue | Implementation and acceptance | Status |
| --- | --- | --- | --- |
| 1 | [#2](https://git.mchus.pro/mchus/QuoteForge/issues/2) | Block foreign Host/Origin/Referer and cross-site browser requests in both routers; validate API body types; refuse saved-password reuse for a changed host/port/user. Middleware attack cases and setup credential tests must pass. | Implemented |
| 2 | [#4](https://git.mchus.pro/mchus/QuoteForge/issues/4) | Move LOT-bearing cart/rental events to delegated listeners and escaped data attributes; escape BOM and sync output; share HTML escaping; render toast as text. Malicious LOT regression checks and template tests must pass. | Implemented |
| 3 | [#7](https://git.mchus.pro/mchus/QuoteForge/issues/7) | Require a fresh verified archive outside rotation before reset, refuse unsynced/legacy states without explicit force, use transactional deletion, remove environment-triggered reset. Verify refusal, ZIP restore, backup failure and rollback. | Implemented |
| 4 | [#5](https://git.mchus.pro/mchus/QuoteForge/issues/5) | Persist TLS mode/CA in SQLite, expose setup controls, share policy across setup/sync/migration DSNs. Verify certificate trust, hostname checks, plaintext fallback policy and settings round-trip. | Implemented |
| 5 | [#3](https://git.mchus.pro/mchus/QuoteForge/issues/3) | Keep each migration file in one session, execute verify headers before history insertion, propagate FK failures. Verify failed queries/checks never enter history. | Implemented |
| 6 | [#6](https://git.mchus.pro/mchus/QuoteForge/issues/6) | Require Go 1.27.3, update vulnerable dependencies, block release on pinned govulncheck. Full tests, vet, security scan and macOS/Windows builds must pass. | Implemented |
| 1 | [#2](https://git.mchus.pro/mchus/QuoteForge/issues/2) | Block foreign Host/Origin/Referer and cross-site browser requests in both routers; validate API body types; refuse saved-password reuse for a changed host/port/user. Middleware and real setup/runtime router tests reject attacks without changing saved credentials or user data. | Acceptance passed |
| 2 | [#4](https://git.mchus.pro/mchus/QuoteForge/issues/4) | Move LOT-bearing cart/rental events to delegated listeners and escaped data attributes; escape BOM and sync output; share HTML escaping; render toast as text. Malicious LOT regression checks and headless Chrome DOM checks pass. | Acceptance passed |
| 3 | [#7](https://git.mchus.pro/mchus/QuoteForge/issues/7) | Require a fresh verified archive outside rotation before reset, refuse unsynced/legacy states without explicit force, use transactional deletion, remove environment-triggered reset. Verified refusal, ZIP restore, backup failure, rollback and CLI startup after restore. | Acceptance passed |
| 4 | [#5](https://git.mchus.pro/mchus/QuoteForge/issues/5) | Persist TLS mode/CA in SQLite, expose setup controls, share policy across setup/sync/migration DSNs. Verified certificate trust, hostname checks, fallback policy, settings round-trip and live MariaDB 11.8 transport. | Acceptance passed |
| 5 | [#3](https://git.mchus.pro/mchus/QuoteForge/issues/3) | Keep each migration file in one session, execute verify headers before history insertion, propagate FK failures. Failed queries/checks restore the entire session. Verified snapshots, literal-aware SQL parsing, trigger preservation and crash recovery pass synthetic tests and production-dump rehearsal on MariaDB 11.8.3. | Acceptance passed |
| 6 | [#6](https://git.mchus.pro/mchus/QuoteForge/issues/6) | Require Go 1.27.3, update vulnerable dependencies, block release on pinned govulncheck. Tests, vet, security scan, cross-platform builds and release failure checks pass. | Acceptance passed |
## Decisions
@@ -28,19 +28,103 @@ is excluded from implementation. Work is sequential on
encryption. The setup form explains this distinction.
- `go 1.27.3` enforces the minimum compiler, including with `GOTOOLCHAIN=local`.
A redundant `toolchain go1.27.3` is removed by Go tooling and is unnecessary.
- The local ignored `.claude/skills/release/SKILL.md` was updated with the security
prerequisite; the shared Makefile and release script enforce it in git.
## Validation and handoff
Passed: `go test ./...`, `go vet ./...`, `node scripts/tests/html-security.cjs`,
`make security-check`, shell syntax and diff checks, and CGO-free builds for
macOS arm64 and Windows amd64.
The original implementation PR is merged. Current acceptance results are listed
below. Govulncheck reports zero reachable vulnerabilities and one module-level
advisory whose affected code is not called. Initial cross-platform builds and
all Go/JavaScript checks pass; final verification also covers the recovery change.
Production-dump rehearsal also passes on MariaDB 11.8.3 with the production
collation. No production migration or new release is performed by this work.
Govulncheck reports zero reachable vulnerabilities and one module-level advisory
whose affected code is not called. Gin's transitive quic-go was explicitly raised
to v0.59.1 after the first scan identified a reachable advisory in v0.59.0.
## Acceptance follow-up
The checks use temporary SQLite databases and synthetic certificates. They do
not exercise a live MariaDB server or browser automation. Review the draft PR
before merge; issue state changes and a production release remain separate steps.
### Issue #2
`go test ./cmd/qfs -run TestRequestBoundaryOnApplicationRouters -count=1`
exercises the production setup and runtime route constructors with temporary
SQLite state. Foreign setup forms, opaque origins and rebinding hosts are
rejected, text/plain JSON receives 415, and both setup handlers reject saved
password reuse for a changed destination. Local reads still succeed. Saved
credentials, project/configuration counts and the pending queue remain unchanged.
The setup router construction was extracted without changing its routes so
this regression exercises the same middleware registration as startup.
### Issue #4
`node scripts/tests/html-security-browser.cjs` passes in headless Chrome with
an isolated temporary profile. It executes the actual template rendering
functions in a browser DOM with controlled API responses. Both cart layouts,
rental, parsed BOM, toast, pending sync errors, error history and repair errors
preserve malicious strings as text. Real click/change/focus events preserve
LOT values through delegated handlers; no injected elements or payload execution
are observed. This is a rendering integration check, not a full application
navigation test. Set `CHROME_BIN` outside macOS to select a Chrome executable.
### Issue #7
`go test ./cmd/qfs ./internal/localdb -run TestReset -count=1` covers reset
safety and a subprocess executing the real `main` flag/startup path. With
rotating backups disabled, the environment switch does not reset pending work,
normal reset refuses it, and explicit forced reset creates a verified archive.
The test restores ZIP contents and reopens the database through normal startup:
connection settings and the pending queue survive. It deliberately selects
`-migrate-status` with an unavailable loopback database to stop startup before
HTTP/browser launch; no production database or user state is touched.
### Issue #5
`go test -tags integration ./internal/mariadb -run TestMariaDBTLSIntegration -count=1`
starts its own disposable MariaDB 11.8 containers on random loopback ports, with
TLS enabled and disabled, and removes both afterward. The test verifies the
actual session cipher: preferred encrypts with a TLS server and falls back only
with a plaintext server; required and verification modes reject plaintext.
Correct trust succeeds, unrelated roots and wrong hostname fail. Certificates,
private keys and database storage are temporary; no saved user DSN is used.
The matching image must already exist locally (`docker pull mariadb:11.8`).
Setup policy, local settings and synthetic TLS tests also pass.
### Issue #6
Go 1.27.3: `go test ./...`, `go vet ./...` and `make security-check` pass.
CGO-free builds produce macOS arm64 Mach-O and Windows amd64 PE executables
outside the repository. `node scripts/tests/release-security.cjs` substitutes a
failing Go scanner in an isolated directory and confirms all five release build
targets and `scripts/release.sh` stop before compilation or artifact creation.
The scanner reports zero reachable vulnerabilities and one uncalled module-level
advisory. No release is published by these acceptance checks.
### Issue #3
The parser, mandatory server snapshots, restore-readiness checks, reverse step
recovery and explicit crash restore are implemented; see
[server migration recovery decision](decisions/2026-10-26-server-migration-recovery.md).
`go test -tags integration ./internal/models -run TestSQLMigrationSafetyMariaDB -count=1`
passes on a disposable MariaDB 11.8: session variables and two verify checks,
semicolon literals, a later failed verify, a real FK errno 150 after successful
DDL, reverse recovery of prior files, interrupted-session refusal/restore/retry,
unsupported-schema refusal, active-client refusal and backup failure.
Restored schema, binary/NULL/empty/decimal/timestamp data and history are compared
to the original snapshot. Unit tests cover snapshot format/digests, symlink path
refusal and all embedded migration SQL syntax.
The supplied `2026-10-26_RFQ_LOG.sql` dump (45 tables, 27 triggers) is loaded
read-only into a new disposable MariaDB **11.8.3** container with
`utf8mb4_uca1400_ai_ci`, matching the production version and collation.
`TestProductionDumpMigrationMariaDB` verifies a full snapshot restore, rebuilds
the pre-035 schema only in that clone, applies the real embedded migration 035,
forces a subsequent DDL/verify failure, verifies reverse recovery against the
complete pre-session snapshot, then applies 035 successfully and restores the
exact original dump state. Trigger bodies, definers and order are preserved;
triggers never fire during row loading. No production schema or dump file is changed.
Run this optional, destructive-to-its-own-container rehearsal with an existing
`mariadb:11.8.3` image and `QFS_TEST_PRODUCTION_DUMP=/outside/repository/dump.sql`:
```bash
go test -tags integration ./internal/models -run TestProductionDumpMigrationMariaDB -count=1 -v
```
The environment variable supplies the source file only; no externally supplied
DSN is accepted. The test owns and removes its container and temporary archives.
+19 -47
View File
@@ -46,7 +46,7 @@ import (
var Version = "dev"
var errVendorImportTooLarge = errors.New("vendor workspace file exceeds 1 GiB limit")
const backgroundSyncInterval = 5 * time.Minute
const backgroundSyncInterval = time.Minute
const onDemandPullCooldown = 30 * time.Second
const startupConsoleWarning = "Не закрывайте консоль иначе приложение не будет работать"
@@ -62,6 +62,7 @@ func main() {
resetLocalDB := flag.Bool("reset-localdb", false, "reset local SQLite data after a verified safety backup (keeps connection settings)")
forceResetLocalDB := flag.Bool("force-reset-localdb", false, "allow -reset-localdb to clear unsynced work after a verified backup")
migrate := flag.Bool("migrate", false, "apply pending server (MariaDB) SQL migrations and exit")
migrateRestore := flag.String("migrate-restore", "", "restore a verified server migration snapshot during maintenance and exit")
migrateStatus := flag.Bool("migrate-status", false, "list server SQL migrations with applied/pending state and exit")
migrateBaseline := flag.String("migrate-baseline", "", "record server SQL migrations up to and including this file as applied, without running them, and exit")
version := flag.Bool("version", false, "show version information")
@@ -132,6 +133,10 @@ func main() {
// Check if running in setup mode (no connection settings)
if !local.HasSettings() {
if *migrate || *migrateStatus || *migrateBaseline != "" || *migrateRestore != "" {
slog.Error("server migration command requires saved DBA connection settings")
os.Exit(1)
}
slog.Info("no database settings found, starting setup mode")
runSetupMode(local)
return
@@ -174,13 +179,6 @@ func main() {
// Create connection manager. Runtime stays local-first; MariaDB is used on demand by sync/setup only.
connMgr := db.NewConnectionManager(local)
// Keep the online-status cache fresh in the background so request-handling
// goroutines (health checks, middleware, etc.) never block on the MySQL
// dial/read timeout themselves.
connMgrCtx, connMgrCancel := context.WithCancel(context.Background())
defer connMgrCancel()
connMgr.Start(connMgrCtx)
dbUser := local.GetDBUser()
slog.Info("starting QuoteForge server",
@@ -191,19 +189,21 @@ func main() {
"online", false,
)
if *migrate || *migrateStatus || *migrateBaseline != "" {
if *migrate || *migrateStatus || *migrateBaseline != "" || *migrateRestore != "" {
mariaDB, err := openMigrationDB(local)
if err != nil {
slog.Error("cannot run migrations: database not available", "error", err)
os.Exit(1)
}
switch {
case *migrateRestore != "":
err = models.RestoreSQLMigrationSession(mariaDB, *migrateRestore)
case *migrateStatus:
err = runMigrateStatus(mariaDB)
case *migrateBaseline != "":
err = runMigrateBaseline(mariaDB, *migrateBaseline)
default:
err = runMigrateApply(mariaDB)
err = runMigrateApply(mariaDB, resolvedLocalDBPath)
}
if err != nil {
slog.Error("migration command failed", "error", err)
@@ -212,6 +212,13 @@ func main() {
os.Exit(0)
}
// Keep the online-status cache fresh in the background so request-handling
// goroutines (health checks, middleware, etc.) never block on the MySQL
// dial/read timeout themselves.
connMgrCtx, connMgrCancel := context.WithCancel(context.Background())
defer connMgrCancel()
connMgr.Start(connMgrCtx)
gin.SetMode(cfg.Server.Mode)
restartSig := make(chan struct{}, 1)
@@ -542,47 +549,12 @@ func nextBackupTime(now time.Time, hour, minute int) time.Time {
func runSetupMode(local *localdb.LocalDB) {
restartSig := make(chan struct{}, 1)
// In setup mode, we don't have a connection manager yet (will restart after setup)
templatesPath := filepath.Join("web", "templates")
setupHandler, err := handlers.NewSetupHandler(local, nil, templatesPath, restartSig)
router, err := setupSetupRouter(local, restartSig)
if err != nil {
slog.Error("failed to create setup handler", "error", err)
slog.Error("failed to create setup router", "error", err)
os.Exit(1)
}
gin.SetMode(gin.ReleaseMode)
router := gin.New()
router.Use(gin.Recovery())
router.Use(middleware.CORS())
if staticFS, err := qfassets.StaticFS(); err == nil {
router.StaticFS("/static", http.FS(staticFS))
} else {
slog.Error("failed to load embedded static assets", "error", err)
os.Exit(1)
}
// Setup routes only
router.GET("/", func(c *gin.Context) {
c.Redirect(http.StatusFound, "/setup")
})
router.GET("/setup", setupHandler.ShowSetup)
router.POST("/setup", setupHandler.SaveConnection)
router.POST("/setup/test", setupHandler.TestConnection)
router.GET("/setup/status", setupHandler.GetStatus)
settingsHandler := handlers.NewSettingsHandler(local)
router.GET("/api/settings/ui", settingsHandler.GetUI)
router.PUT("/api/settings/ui", settingsHandler.PutUI)
// Health check
router.GET("/health", func(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{
"status": "setup_required",
"time": time.Now().UTC().Format(time.RFC3339),
})
})
addr := "127.0.0.1:8080"
slog.Info("starting setup mode server", "address", addr, "version", Version)
+13 -2
View File
@@ -6,6 +6,7 @@ import (
"time"
qfassets "git.mchus.pro/mchus/quoteforge"
"git.mchus.pro/mchus/quoteforge/internal/appstate"
"git.mchus.pro/mchus/quoteforge/internal/localdb"
"git.mchus.pro/mchus/quoteforge/internal/models"
mysqlDriver "github.com/go-sql-driver/mysql"
@@ -39,6 +40,12 @@ func openMigrationDB(local *localdb.LocalDB) (*gorm.DB, error) {
if err != nil {
return nil, fmt.Errorf("connect to server: %w", err)
}
sqlDB, err := db.DB()
if err != nil {
return nil, fmt.Errorf("migration pool: %w", err)
}
sqlDB.SetMaxOpenConns(1)
sqlDB.SetMaxIdleConns(1)
return db, nil
}
@@ -68,7 +75,7 @@ func runMigrateStatus(db *gorm.DB) error {
// runMigrateApply applies pending server migrations. Refuses on a server with no
// migration history at all: its schema was built by other means, so every file would be
// re-run — record a baseline with -migrate-baseline first.
func runMigrateApply(db *gorm.DB) error {
func runMigrateApply(db *gorm.DB, localDBPath string) error {
migrations, err := qfassets.MigrationsFS()
if err != nil {
return err
@@ -76,9 +83,13 @@ func runMigrateApply(db *gorm.DB) error {
if err := requireMigrationHistory(db, migrations); err != nil {
return err
}
backupRoot, err := appstate.ServerMigrationBackupDir(localDBPath)
if err != nil {
return err
}
applied, err := models.RunSQLMigrations(db, migrations, func(filename string) {
fmt.Printf(" applying %s ...\n", filename)
})
}, backupRoot)
if err != nil {
return err
}
+72
View File
@@ -0,0 +1,72 @@
package main
import (
"net/http/httptest"
"reflect"
"strings"
"testing"
"git.mchus.pro/mchus/quoteforge/internal/config"
"github.com/gin-gonic/gin"
)
func TestRequestBoundaryOnApplicationRouters(t *testing.T) {
moveToRepoRoot(t)
local, connMgr, _ := newAPITestStack(t)
if err := local.SaveSettings("db.example", 3306, "example", "example", "example"); err != nil {
t.Fatal(err)
}
cfg := &config.Config{}
setConfigDefaults(cfg)
runtimeRouter, _, err := setupRouter(cfg, local, connMgr, "example", nil)
if err != nil {
t.Fatal(err)
}
setupRouter, err := setupSetupRouter(local, nil)
if err != nil {
t.Fatal(err)
}
for name, router := range map[string]*gin.Engine{"setup": setupRouter, "runtime": runtimeRouter} {
t.Run(name, func(t *testing.T) {
before, err := local.GetSettings()
if err != nil {
t.Fatal(err)
}
projects, configs, pending := local.CountProjects(), local.CountConfigurations(), local.CountPendingChanges()
for _, tt := range []struct {
name, method, path, host, origin, contentType, body string
want int
}{
{"foreign setup save", "POST", "/setup", "", "https://attacker.example", "application/x-www-form-urlencoded", "host=attacker.example&user=example", 403},
{"foreign setup test", "POST", "/setup/test", "", "https://attacker.example", "application/x-www-form-urlencoded", "host=attacker.example&user=example", 403},
{"rebind settings read", "GET", "/setup/status", "attacker.example:8080", "", "", "", 403},
{"opaque origin", "PUT", "/api/settings/ui", "", "null", "application/json", `{"indicator_mode":"accessible"}`, 403},
{"plain JSON", "PUT", "/api/settings/ui", "", "", "text/plain", `{"indicator_mode":"accessible"}`, 415},
{"saved password destination save", "POST", "/setup", "", "", "application/x-www-form-urlencoded", "host=attacker.example&user=example", 422},
{"saved password destination test", "POST", "/setup/test", "", "", "application/x-www-form-urlencoded", "host=attacker.example&user=example", 422},
{"local read", "GET", "/api/settings/ui", "", "http://127.0.0.1:8080", "", "", 200},
} {
t.Run(tt.name, func(t *testing.T) {
req := httptest.NewRequest(tt.method, "http://127.0.0.1:8080"+tt.path, strings.NewReader(tt.body))
if tt.host != "" {
req.Host = tt.host
}
req.Header.Set("Origin", tt.origin)
req.Header.Set("Content-Type", tt.contentType)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != tt.want {
t.Fatalf("status=%d want=%d body=%s", w.Code, tt.want, w.Body.String())
}
})
}
after, err := local.GetSettings()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(before, after) || projects != local.CountProjects() || configs != local.CountConfigurations() || pending != local.CountPendingChanges() {
t.Fatal("rejected requests changed local state")
}
})
}
}
+130
View File
@@ -0,0 +1,130 @@
package main
import (
"archive/zip"
"context"
"flag"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"git.mchus.pro/mchus/quoteforge/internal/localdb"
)
func TestResetCLIHelper(t *testing.T) {
if os.Getenv("QFS_RESET_CLI_HELPER") != "1" {
return
}
for i, arg := range os.Args {
if arg == "--" {
os.Args = append([]string{"qfs"}, os.Args[i+1:]...)
flag.CommandLine = flag.NewFlagSet("qfs", flag.ExitOnError)
main()
return
}
}
t.Fatal("missing CLI arguments")
}
func TestResetCLIAndRestore(t *testing.T) {
dir := t.TempDir()
t.Setenv("QFS_STATE_DIR", dir)
t.Setenv("QFS_BACKUP_DISABLE", "1")
t.Setenv("QFS_RESET_LOCAL_DB", "yes")
t.Setenv("QFS_BACKUP_DIR", filepath.Join(dir, "backups"))
dbPath := filepath.Join(dir, "qfs.db")
configPath := filepath.Join(dir, "config.yaml")
if err := os.WriteFile(configPath, []byte("server:\n host: 127.0.0.1\n"), 0600); err != nil {
t.Fatal(err)
}
local, err := localdb.New(dbPath)
if err != nil {
t.Fatal(err)
}
if err := local.SaveSettings("127.0.0.1", 1, "example", "example", "example"); err != nil {
t.Fatal(err)
}
if err := local.AddPendingChange("configuration", "example", "update", "{}"); err != nil {
t.Fatal(err)
}
if err := local.Close(); err != nil {
t.Fatal(err)
}
run := func(args ...string) string {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
argv := append([]string{"-test.run=^TestResetCLIHelper$", "--", "-localdb", dbPath, "-config", configPath, "-migrate-status"}, args...)
cmd := exec.CommandContext(ctx, os.Args[0], argv...)
cmd.Env = append(os.Environ(), "QFS_RESET_CLI_HELPER=1")
output, err := cmd.CombinedOutput()
if ctx.Err() != nil {
t.Fatalf("CLI timed out: %s", output)
}
if err == nil {
t.Fatalf("expected reset refusal or offline migration-status exit: %s", output)
}
return string(output)
}
assertPending := func(want int64) {
t.Helper()
local, err := localdb.New(dbPath)
if err != nil {
t.Fatal(err)
}
defer local.Close()
if got := local.CountPendingChanges(); got != want {
t.Fatalf("pending=%d want=%d", got, want)
}
if !local.HasSettings() {
t.Fatal("connection settings lost")
}
}
if output := run(); !strings.Contains(output, "QFS_RESET_LOCAL_DB is ignored") {
t.Fatalf("environment reset not diagnosed: %s", output)
}
assertPending(1)
if output := run("-reset-localdb"); !strings.Contains(output, "reset refused") {
t.Fatalf("pending reset not refused: %s", output)
}
assertPending(1)
if output := run("-reset-localdb", "-force-reset-localdb"); !strings.Contains(output, "verified pre-reset backup") {
t.Fatalf("missing safety backup: %s", output)
}
assertPending(0)
archives, err := filepath.Glob(filepath.Join(dir, "backups", "pre-reset", "*.zip"))
if err != nil || len(archives) != 1 {
t.Fatalf("archives=%v err=%v", archives, err)
}
archive, err := zip.OpenReader(archives[0])
if err != nil {
t.Fatal(err)
}
defer archive.Close()
for _, entry := range archive.File {
if entry.Name != "qfs.db" && entry.Name != "config.yaml" {
t.Fatalf("unexpected archive entry: %s", entry.Name)
}
in, err := entry.Open()
if err != nil {
t.Fatal(err)
}
data, readErr := io.ReadAll(in)
closeErr := in.Close()
if readErr != nil || closeErr != nil {
t.Fatalf("restore entry: read=%v close=%v", readErr, closeErr)
}
if err := os.WriteFile(filepath.Join(dir, entry.Name), data, 0600); err != nil {
t.Fatal(err)
}
}
assertPending(1)
if output := run(); !strings.Contains(output, "cannot run migrations: database not available") {
t.Fatalf("restored startup did not reach migration status: %s", output)
}
assertPending(1)
}
+57
View File
@@ -0,0 +1,57 @@
package main
import (
"fmt"
"net/http"
"path/filepath"
"time"
qfassets "git.mchus.pro/mchus/quoteforge"
"git.mchus.pro/mchus/quoteforge/internal/handlers"
"git.mchus.pro/mchus/quoteforge/internal/localdb"
"git.mchus.pro/mchus/quoteforge/internal/middleware"
"github.com/gin-gonic/gin"
)
func setupSetupRouter(local *localdb.LocalDB, restartSig chan struct{}) (*gin.Engine, error) {
// In setup mode, we don't have a connection manager yet (will restart after setup)
templatesPath := filepath.Join("web", "templates")
setupHandler, err := handlers.NewSetupHandler(local, nil, templatesPath, restartSig)
if err != nil {
return nil, fmt.Errorf("create setup handler: %w", err)
}
gin.SetMode(gin.ReleaseMode)
router := gin.New()
router.Use(gin.Recovery())
router.Use(middleware.CORS())
if staticFS, err := qfassets.StaticFS(); err == nil {
router.StaticFS("/static", http.FS(staticFS))
} else {
return nil, fmt.Errorf("load embedded static assets: %w", err)
}
// Setup routes only
router.GET("/", func(c *gin.Context) {
c.Redirect(http.StatusFound, "/setup")
})
router.GET("/setup", setupHandler.ShowSetup)
router.POST("/setup", setupHandler.SaveConnection)
router.POST("/setup/test", setupHandler.TestConnection)
router.GET("/setup/status", setupHandler.GetStatus)
settingsHandler := handlers.NewSettingsHandler(local)
router.GET("/api/settings/ui", settingsHandler.GetUI)
router.PUT("/api/settings/ui", settingsHandler.PutUI)
// Health check
router.GET("/health", func(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{
"status": "setup_required",
"time": time.Now().UTC().Format(time.RFC3339),
})
})
return router, nil
}
+51
View File
@@ -0,0 +1,51 @@
package appstate
import (
"fmt"
"os"
"path/filepath"
)
// ServerMigrationBackupDir is separate from local backup retention.
func ServerMigrationBackupDir(dbPath string) (string, error) {
root := filepath.Join(resolveBackupRoot(dbPath), "server-migrations")
if err := ValidateBackupLocation(root); err != nil {
return "", err
}
if err := os.MkdirAll(root, 0700); err != nil {
return "", fmt.Errorf("create server backup directory: %w", err)
}
resolved, err := filepath.EvalSymlinks(root)
if err != nil {
return "", fmt.Errorf("resolve server backup directory: %w", err)
}
if err := ValidateBackupLocation(resolved); err != nil {
return "", err
}
return resolved, nil
}
// ValidateBackupLocation checks the destination, including existing symlinks.
func ValidateBackupLocation(path string) error {
if err := validateBackupRoot(path); err != nil {
return err
}
existing := path
for {
if _, err := os.Lstat(existing); err == nil {
break
} else if !os.IsNotExist(err) {
return fmt.Errorf("inspect backup path: %w", err)
}
parent := filepath.Dir(existing)
if parent == existing {
return fmt.Errorf("backup path has no existing parent")
}
existing = parent
}
resolved, err := filepath.EvalSymlinks(existing)
if err != nil {
return fmt.Errorf("resolve backup path: %w", err)
}
return validateBackupRoot(resolved)
}
@@ -0,0 +1,29 @@
package appstate
import (
"os"
"path/filepath"
"testing"
)
func TestServerBackupRejectsGitAndSymlinks(t *testing.T) {
root := t.TempDir()
repo := filepath.Join(root, "repo")
if err := os.MkdirAll(filepath.Join(repo, ".git"), 0700); err != nil {
t.Fatal(err)
}
link := filepath.Join(root, "link")
if err := os.Symlink(repo, link); err != nil {
t.Fatal(err)
}
for _, path := range []string{filepath.Join(repo, "backups"), filepath.Join(link, "backups")} {
t.Setenv("QFS_BACKUP_DIR", path)
if _, err := ServerMigrationBackupDir(filepath.Join(root, "state", "qfs.db")); err == nil {
t.Fatalf("accepted repository backup: %s", path)
}
}
t.Setenv("QFS_BACKUP_DIR", filepath.Join(root, "backups"))
if _, err := ServerMigrationBackupDir(filepath.Join(root, "state", "qfs.db")); err != nil {
t.Fatal(err)
}
}
+67 -18
View File
@@ -22,9 +22,10 @@ const (
// goroutines must never pay the MySQL dial/read timeout themselves.
defaultStatusCheckInterval = 15 * time.Second
maxOpenConns = 10
maxOpenConns = 2
maxIdleConns = 2
connMaxLifetime = 5 * time.Minute
connMaxIdleTime = time.Minute
)
// ConnectionStatus represents the current status of the database connection
@@ -49,11 +50,13 @@ type ConnectionManager struct {
statusCheckInterval time.Duration // background prober cadence (default: 15s)
stopStatusLoop chan struct{} // closed by Stop() to end the background loop
dialFn func() (*gorm.DB, error) // opens a new pool; cm.dial unless overridden in tests
}
// NewConnectionManager creates a new ConnectionManager instance
func NewConnectionManager(localDB *localdb.LocalDB) *ConnectionManager {
return &ConnectionManager{
cm := &ConnectionManager{
localDB: localDB,
connectTimeout: defaultConnectTimeout,
pingInterval: defaultPingInterval,
@@ -63,6 +66,8 @@ func NewConnectionManager(localDB *localdb.LocalDB) *ConnectionManager {
lastError: nil,
lastCheck: time.Time{},
}
cm.dialFn = cm.dial
return cm
}
// Start launches a background goroutine that keeps the online-status cache
@@ -137,25 +142,66 @@ func (cm *ConnectionManager) GetDB() (*gorm.DB, error) {
return db, err
}
newDB, err := cm.dial()
// The cached pool is only stale, not broken: ping it instead of dialing a
// new one. Re-dialing here used to orphan the old *sql.DB together with its
// idle connections, leaking a pool every pingInterval during sync.
cm.mu.RLock()
existing := cm.db
cm.mu.RUnlock()
if existing != nil {
if err := cm.ping(existing); err == nil {
cm.mu.Lock()
cm.lastError = nil
cm.lastCheck = time.Now()
cm.mu.Unlock()
return existing, nil
}
}
newDB, err := cm.dialFn()
cm.mu.Lock()
old := cm.db
if err != nil {
// Drop stale handle so callers don't treat it as an active connection.
cm.db = nil
cm.lastError = err
cm.lastCheck = time.Now()
cm.mu.Unlock()
closeDB(old)
return nil, err
}
cm.db = newDB
cm.lastError = nil
cm.lastCheck = time.Now()
cm.mu.Unlock()
closeDB(old)
return newDB, nil
}
// ping checks an existing pool with the connect timeout. No locks held.
func (cm *ConnectionManager) ping(db *gorm.DB) error {
ctx, cancel := context.WithTimeout(context.Background(), cm.connectTimeout)
defer cancel()
sqlDB, err := db.DB()
if err != nil {
return err
}
return sqlDB.PingContext(ctx)
}
// closeDB releases a pool that is no longer referenced by the manager.
func closeDB(db *gorm.DB) {
if db == nil {
return
}
if sqlDB, err := db.DB(); err == nil {
sqlDB.Close()
}
}
// cachedResult returns (db, err, true) if the cached state is still fresh
// enough to answer without a new network round-trip: either a live
// connection within pingInterval, or a recent failure still within
@@ -200,15 +246,17 @@ func (cm *ConnectionManager) dial() (*gorm.DB, error) {
return nil, fmt.Errorf("getting sql.DB: %w", err)
}
// Ping with timeout
if err = sqlDB.PingContext(ctx); err != nil {
return nil, fmt.Errorf("pinging database: %w", err)
}
// Set connection pool settings
sqlDB.SetMaxOpenConns(maxOpenConns)
sqlDB.SetMaxIdleConns(maxIdleConns)
sqlDB.SetConnMaxLifetime(connMaxLifetime)
sqlDB.SetConnMaxIdleTime(connMaxIdleTime)
// Ping with timeout
if err = sqlDB.PingContext(ctx); err != nil {
sqlDB.Close()
return nil, fmt.Errorf("pinging database: %w", err)
}
return db, nil
}
@@ -264,20 +312,17 @@ func (cm *ConnectionManager) checkOnlineNow() bool {
}
// Perform ping with timeout — no locks held here.
ctx, cancel := context.WithTimeout(context.Background(), cm.connectTimeout)
defer cancel()
sqlDB, err := db.DB()
if err == nil {
err = sqlDB.PingContext(ctx)
}
err := cm.ping(db)
cm.mu.Lock()
defer cm.mu.Unlock()
if err != nil {
cm.lastError = err
cm.lastCheck = time.Now()
cm.db = nil
if cm.db == db {
cm.db = nil
}
closeDB(db)
return false
}
cm.lastCheck = time.Now()
@@ -292,19 +337,23 @@ func (cm *ConnectionManager) TryConnect() error {
cm.connMu.Lock()
defer cm.connMu.Unlock()
newDB, err := cm.dial()
newDB, err := cm.dialFn()
cm.mu.Lock()
defer cm.mu.Unlock()
old := cm.db
if err != nil {
cm.db = nil
cm.lastError = err
cm.lastCheck = time.Now()
cm.mu.Unlock()
closeDB(old)
return err
}
cm.db = newDB
cm.lastError = nil
cm.lastCheck = time.Now()
cm.mu.Unlock()
closeDB(old)
return nil
}
+74
View File
@@ -0,0 +1,74 @@
package db
import (
"path/filepath"
"testing"
"time"
"git.mchus.pro/mchus/quoteforge/internal/localdb"
"github.com/glebarez/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/logger"
)
func newTestConnectionManager(t *testing.T) (*ConnectionManager, *int) {
t.Helper()
local, err := localdb.New(filepath.Join(t.TempDir(), "qfs.db"))
if err != nil {
t.Fatalf("open local db: %v", err)
}
cm := NewConnectionManager(local)
dials := 0
cm.dialFn = func() (*gorm.DB, error) {
dials++
return gorm.Open(sqlite.Open(filepath.Join(t.TempDir(), "remote.db")), &gorm.Config{
Logger: logger.Default.LogMode(logger.Silent),
})
}
return cm, &dials
}
func TestGetDBReusesStalePoolInsteadOfRedialing(t *testing.T) {
cm, dials := newTestConnectionManager(t)
first, err := cm.GetDB()
if err != nil {
t.Fatalf("first GetDB: %v", err)
}
// Simulate pingInterval elapsing between sync steps.
cm.mu.Lock()
cm.lastCheck = time.Now().Add(-2 * cm.pingInterval)
cm.mu.Unlock()
second, err := cm.GetDB()
if err != nil {
t.Fatalf("second GetDB: %v", err)
}
if second != first {
t.Fatalf("expected stale but healthy pool to be reused")
}
if *dials != 1 {
t.Fatalf("expected 1 dial, got %d", *dials)
}
}
func TestTryConnectClosesReplacedPool(t *testing.T) {
cm, _ := newTestConnectionManager(t)
first, err := cm.GetDB()
if err != nil {
t.Fatalf("GetDB: %v", err)
}
if err := cm.TryConnect(); err != nil {
t.Fatalf("TryConnect: %v", err)
}
sqlDB, err := first.DB()
if err != nil {
t.Fatalf("sql.DB: %v", err)
}
if err := sqlDB.Ping(); err == nil {
t.Fatalf("expected replaced pool to be closed")
}
}
+1 -1
View File
@@ -43,7 +43,7 @@ func NewSyncHandler(localDB *localdb.LocalDB, syncService *sync.Service, connMgr
syncService: syncService,
connMgr: connMgr,
autoSyncInterval: autoSyncInterval,
onlineGraceFactor: 1.10,
onlineGraceFactor: 1 + sync.WorkerJitter, // longest gap between jittered sync cycles
tmpl: tmpl,
}, nil
}
+51
View File
@@ -0,0 +1,51 @@
package localdb
import (
"encoding/json"
"fmt"
"time"
)
// serverPullCursor is the server-side NOW(6) taken before the last successful
// incremental pull of a table, bound to the server it was taken from.
type serverPullCursor struct {
Scope string `json:"scope"`
ServerTime string `json:"server_time"`
}
func serverPullCursorKey(table string) string {
return "server_pull_cursor:" + table
}
// ServerPullScope identifies the MariaDB server+database a pull cursor belongs to,
// so a cursor never survives a switch to another server.
func (l *LocalDB) ServerPullScope() (string, error) {
settings, err := l.GetSettings()
if err != nil {
return "", err
}
return fmt.Sprintf("%s:%d/%s", settings.Host, settings.Port, settings.Database), nil
}
// GetServerPullCursor returns the stored server time for table if it was taken
// on the server identified by scope.
func (l *LocalDB) GetServerPullCursor(table, scope string) (string, bool) {
raw, ok := l.getAppSettingValue(serverPullCursorKey(table))
if !ok {
return "", false
}
var cursor serverPullCursor
if err := json.Unmarshal([]byte(raw), &cursor); err != nil || cursor.Scope != scope || cursor.ServerTime == "" {
return "", false
}
return cursor.ServerTime, true
}
// SetServerPullCursor stores the server time a pull of table started at.
func (l *LocalDB) SetServerPullCursor(table, scope, serverTime string) error {
raw, err := json.Marshal(serverPullCursor{Scope: scope, ServerTime: serverTime})
if err != nil {
return err
}
return l.upsertAppSetting(l.db, serverPullCursorKey(table), string(raw), time.Now())
}
+21
View File
@@ -72,6 +72,11 @@ func (l *LocalDB) SyncQtSettings(mariaDB *gorm.DB) error {
return nil
}
var current []LocalQtSetting
if err := l.db.Select("name, value").Find(&current).Error; err == nil && sameQtSettings(current, rows) {
return nil
}
return l.db.Transaction(func(tx *gorm.DB) error {
if err := tx.Exec("DELETE FROM local_qt_settings").Error; err != nil {
return fmt.Errorf("clearing local_qt_settings: %w", err)
@@ -142,3 +147,19 @@ func (l *LocalDB) GetConfiguratorSettings() (*ConfiguratorSettings, error) {
return out, nil
}
func sameQtSettings(a, b []LocalQtSetting) bool {
if len(a) != len(b) {
return false
}
values := make(map[string]string, len(a))
for _, row := range a {
values[row.Name] = row.Value
}
for _, row := range b {
if v, ok := values[row.Name]; !ok || v != row.Value {
return false
}
}
return true
}
+185
View File
@@ -0,0 +1,185 @@
package mariadb
import (
"archive/zip"
"crypto/sha256"
"database/sql"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"
"git.mchus.pro/mchus/quoteforge/internal/appstate"
"gorm.io/gorm"
)
type snapshotTable struct {
Name, Create, Entry, Digest string
Columns []string
Rows int64
}
type snapshotManifest struct {
Format int
Schema, Collation string
Tables []snapshotTable
Triggers []snapshotTrigger
}
func freshDB(db *gorm.DB) *gorm.DB { return db.Session(&gorm.Session{NewDB: true}) }
func quoteIdentifier(s string) string { return "`" + strings.ReplaceAll(s, "`", "``") + "`" }
// CreateSnapshot writes a consistent InnoDB snapshot, without external DB tools.
// The caller supplies a dedicated connection and excludes concurrent writers
// for the entire migration session, including any recovery.
func CreateSnapshot(db *gorm.DB, root, label string) (string, error) {
if err := appstate.ValidateBackupLocation(root); err != nil {
return "", err
}
if filepath.Base(label) != label {
return "", fmt.Errorf("invalid snapshot label")
}
if err := os.MkdirAll(root, 0700); err != nil {
return "", fmt.Errorf("create snapshot directory: %w", err)
}
if err := appstate.ValidateBackupLocation(root); err != nil {
return "", err
}
file, err := os.CreateTemp(root, "qfs-mariadb-v1-"+label+"-"+time.Now().UTC().Format("20060102T150405Z")+"-*.zip")
if err != nil {
return "", fmt.Errorf("create snapshot: %w", err)
}
path := file.Name()
archive := zip.NewWriter(file)
err = freshDB(db).Transaction(func(tx *gorm.DB) error {
manifest, err := snapshotInventory(tx)
if err != nil {
return err
}
for i := range manifest.Tables {
table := &manifest.Tables[i]
table.Entry = fmt.Sprintf("tables/%06d.jsonl", i)
out, err := archive.Create(table.Entry)
if err != nil {
return fmt.Errorf("create table entry %s: %w", table.Name, err)
}
if err := snapshotRows(tx, table, out); err != nil {
return err
}
}
out, err := archive.Create("manifest.json")
if err != nil {
return fmt.Errorf("create snapshot manifest: %w", err)
}
if err := json.NewEncoder(out).Encode(manifest); err != nil {
return fmt.Errorf("write snapshot manifest: %w", err)
}
return nil
}, &sql.TxOptions{Isolation: sql.LevelRepeatableRead, ReadOnly: true})
zipErr := archive.Close()
syncErr := file.Sync()
closeErr := file.Close()
for _, candidate := range []error{err, zipErr, syncErr, closeErr} {
if candidate != nil {
return "", fmt.Errorf("write snapshot %s: %w", path, candidate)
}
}
if _, err := readSnapshot(path, nil); err != nil {
return "", err
}
return path, nil
}
func snapshotInventory(db *gorm.DB) (*snapshotManifest, error) {
m := &snapshotManifest{Format: 1}
if err := freshDB(db).Raw("SELECT DATABASE(), @@collation_database").Row().Scan(&m.Schema, &m.Collation); err != nil {
return nil, fmt.Errorf("read snapshot schema: %w", err)
}
var unsupported int64
query := `SELECT
(SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA=? AND (TABLE_TYPE<>'BASE TABLE' OR ENGINE<>'InnoDB')) +
(SELECT COUNT(*) FROM information_schema.ROUTINES WHERE ROUTINE_SCHEMA=?) +
(SELECT COUNT(*) FROM information_schema.EVENTS WHERE EVENT_SCHEMA=?) +
(SELECT COUNT(*) FROM information_schema.KEY_COLUMN_USAGE WHERE
(TABLE_SCHEMA=? AND REFERENCED_TABLE_SCHEMA<>?) OR (REFERENCED_TABLE_SCHEMA=? AND TABLE_SCHEMA<>?))`
if err := freshDB(db).Raw(query, m.Schema, m.Schema, m.Schema, m.Schema, m.Schema, m.Schema, m.Schema).Scan(&unsupported).Error; err != nil {
return nil, fmt.Errorf("check snapshot support: %w", err)
}
if unsupported != 0 {
return nil, fmt.Errorf("snapshot refused: views, non-InnoDB tables, routines, events or cross-schema foreign keys require supported recovery before migration")
}
var names []string
if err := freshDB(db).Raw("SELECT TABLE_NAME FROM information_schema.TABLES WHERE TABLE_SCHEMA=? ORDER BY TABLE_NAME", m.Schema).Scan(&names).Error; err != nil {
return nil, fmt.Errorf("list snapshot tables: %w", err)
}
for _, name := range names {
table := snapshotTable{Name: name}
var ignored string
if err := freshDB(db).Raw("SHOW CREATE TABLE "+quoteIdentifier(name)).Row().Scan(&ignored, &table.Create); err != nil {
return nil, fmt.Errorf("read schema for %s: %w", name, err)
}
if err := freshDB(db).Raw(`SELECT COLUMN_NAME FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=? AND TABLE_NAME=? AND EXTRA NOT LIKE '%VIRTUAL GENERATED%' AND EXTRA NOT LIKE '%STORED GENERATED%' ORDER BY ORDINAL_POSITION`, m.Schema, name).Scan(&table.Columns).Error; err != nil {
return nil, fmt.Errorf("read columns for %s: %w", name, err)
}
if len(table.Columns) == 0 {
return nil, fmt.Errorf("snapshot table %s has no writable columns", name)
}
m.Tables = append(m.Tables, table)
}
if err := readSnapshotTriggers(db, m); err != nil {
return nil, err
}
return m, nil
}
func snapshotRows(db *gorm.DB, table *snapshotTable, out io.Writer) error {
columns := make([]string, len(table.Columns))
for i, col := range table.Columns {
columns[i] = "CAST(" + quoteIdentifier(col) + " AS BINARY)"
}
rows, err := freshDB(db).Raw("SELECT " + strings.Join(columns, ",") + " FROM " + quoteIdentifier(table.Name)).Rows()
if err != nil {
return fmt.Errorf("read snapshot table %s: %w", table.Name, err)
}
defer rows.Close()
var digest [32]byte
encoder := json.NewEncoder(out)
table.Rows = 0
for rows.Next() {
values := make([]sql.RawBytes, len(columns))
args := make([]any, len(columns))
for i := range args {
args[i] = &values[i]
}
if err := rows.Scan(args...); err != nil {
return fmt.Errorf("read row in %s: %w", table.Name, err)
}
if err := encoder.Encode(values); err != nil {
return fmt.Errorf("write row in %s: %w", table.Name, err)
}
addRowDigest(&digest, values)
table.Rows++
}
if err := rows.Err(); err != nil {
return fmt.Errorf("iterate %s: %w", table.Name, err)
}
if err := rows.Close(); err != nil {
return fmt.Errorf("close rows in %s: %w", table.Name, err)
}
table.Digest = hex.EncodeToString(digest[:])
return nil
}
func addRowDigest(sum *[32]byte, row []sql.RawBytes) {
encoded, _ := json.Marshal(row) // Raw bytes and nil always marshal successfully.
hash := sha256.Sum256(encoded)
carry := uint16(0)
for i := len(sum) - 1; i >= 0; i-- {
value := uint16(sum[i]) + uint16(hash[i]) + carry
sum[i], carry = byte(value), value>>8
}
}
+101
View File
@@ -0,0 +1,101 @@
package mariadb
import (
"archive/zip"
"database/sql"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"strings"
"git.mchus.pro/mchus/quoteforge/internal/appstate"
)
func readSnapshot(path string, visit func(snapshotTable, []sql.RawBytes) error) (*snapshotManifest, error) {
if err := appstate.ValidateBackupLocation(path); err != nil {
return nil, err
}
archive, err := zip.OpenReader(path)
if err != nil {
return nil, fmt.Errorf("open server snapshot: %w", err)
}
defer archive.Close()
files := map[string]*zip.File{}
for _, f := range archive.File {
if files[f.Name] != nil {
return nil, fmt.Errorf("duplicate snapshot entry")
}
files[f.Name] = f
}
manifestFile := files["manifest.json"]
if manifestFile == nil || manifestFile.UncompressedSize64 > 16<<20 {
return nil, fmt.Errorf("missing or oversized snapshot manifest")
}
in, err := manifestFile.Open()
if err != nil {
return nil, fmt.Errorf("open server manifest: %w", err)
}
data, readErr := io.ReadAll(in)
closeErr := in.Close()
if readErr != nil || closeErr != nil {
return nil, fmt.Errorf("read server manifest: %w", errors.Join(readErr, closeErr))
}
var m snapshotManifest
if err := json.Unmarshal(data, &m); err != nil {
return nil, fmt.Errorf("parse server manifest: %w", err)
}
if m.Format != 1 || m.Schema == "" || len(files) != len(m.Tables)+1 {
return nil, fmt.Errorf("invalid server snapshot manifest")
}
names, entries := map[string]bool{}, map[string]bool{}
for _, table := range m.Tables {
if table.Name == "" || names[table.Name] || entries[table.Entry] || files[table.Entry] == nil || len(table.Columns) == 0 || table.Rows < 0 || !strings.HasPrefix(table.Create, "CREATE TABLE "+quoteIdentifier(table.Name)+" (") {
return nil, fmt.Errorf("invalid snapshot table metadata")
}
names[table.Name], entries[table.Entry] = true, true
if err := readSnapshotTable(files[table.Entry], table, visit); err != nil {
return nil, err
}
}
return &m, nil
}
func readSnapshotTable(file *zip.File, table snapshotTable, visit func(snapshotTable, []sql.RawBytes) error) error {
in, err := file.Open()
if err != nil {
return fmt.Errorf("open snapshot table %s: %w", table.Name, err)
}
defer in.Close()
decoder := json.NewDecoder(in)
var rows int64
var digest [32]byte
for {
var values []sql.RawBytes
err := decoder.Decode(&values)
if err == io.EOF {
break
}
if err != nil {
return fmt.Errorf("read snapshot row in %s: %w", table.Name, err)
}
if len(values) != len(table.Columns) {
return fmt.Errorf("snapshot column count mismatch in %s", table.Name)
}
addRowDigest(&digest, values)
rows++
if visit != nil {
if err := visit(table, values); err != nil {
return err
}
}
}
if err := in.Close(); err != nil {
return fmt.Errorf("close snapshot table %s: %w", table.Name, err)
}
if rows != table.Rows || hex.EncodeToString(digest[:]) != table.Digest {
return fmt.Errorf("snapshot checksum mismatch in %s", table.Name)
}
return nil
}
+118
View File
@@ -0,0 +1,118 @@
package mariadb
import (
"database/sql"
"errors"
"fmt"
"strings"
"gorm.io/gorm"
)
// RestoreSnapshot restores and checks a snapshot on a dedicated connection.
// The same maintenance exclusion as migration is required throughout recovery.
func RestoreSnapshot(db *gorm.DB, path string) error {
m, err := readSnapshot(path, nil)
if err != nil {
return err
}
var schema string
if err := freshDB(db).Raw("SELECT DATABASE()").Scan(&schema).Error; err != nil {
return fmt.Errorf("read restore destination: %w", err)
}
if schema != m.Schema {
return fmt.Errorf("snapshot schema does not match restore destination")
}
if err := restoreSnapshotTables(db, path, m); err != nil {
return err
}
return compareSnapshot(db, m)
}
func restoreSnapshotTables(db *gorm.DB, path string, m *snapshotManifest) (err error) {
current, err := snapshotInventory(db)
if err != nil {
return err
}
var foreignKeys int
var sqlMode string
if err := freshDB(db).Raw("SELECT @@foreign_key_checks, @@sql_mode").Row().Scan(&foreignKeys, &sqlMode); err != nil {
return fmt.Errorf("read restore session policy: %w", err)
}
if err := freshDB(db).Exec("SET FOREIGN_KEY_CHECKS=0").Error; err != nil {
return fmt.Errorf("disable FK checks for restore: %w", err)
}
defer func() {
if restoreErr := freshDB(db).Exec("SET sql_mode=?", sqlMode).Error; restoreErr != nil {
err = errors.Join(err, fmt.Errorf("restore SQL mode: %w", restoreErr))
}
if restoreErr := freshDB(db).Exec("SET FOREIGN_KEY_CHECKS=?", foreignKeys).Error; restoreErr != nil {
err = errors.Join(err, fmt.Errorf("restore FK policy: %w", restoreErr))
}
}()
// Preserve explicit zero auto-increment IDs and legacy zero dates. Exact
// schema/data comparison detects any coercion before the source is changed.
if err := freshDB(db).Exec("SET sql_mode='NO_AUTO_VALUE_ON_ZERO'").Error; err != nil {
return fmt.Errorf("set snapshot insert mode: %w", err)
}
for i := len(current.Tables) - 1; i >= 0; i-- {
if err := freshDB(db).Exec("DROP TABLE " + quoteIdentifier(current.Tables[i].Name)).Error; err != nil {
return fmt.Errorf("drop restore table %s: %w", current.Tables[i].Name, err)
}
}
for _, table := range m.Tables {
if err := freshDB(db).Exec(table.Create).Error; err != nil {
return fmt.Errorf("restore schema for %s: %w", table.Name, err)
}
}
var prefix, rowSQL, tableName string
var args []any
var groups []string
var bytes int
flush := func() error {
if len(groups) == 0 {
return nil
}
// Bind through database/sql: GORM expands a byte slice after '('.
// Direct execution also keeps restored values out of SQL logs.
if _, err := db.Statement.ConnPool.ExecContext(db.Statement.Context, prefix+strings.Join(groups, ","), args...); err != nil {
return fmt.Errorf("restore rows in %s: %w", tableName, err)
}
args, groups, bytes = nil, nil, 0
return nil
}
_, err = readSnapshot(path, func(table snapshotTable, values []sql.RawBytes) error {
if tableName != table.Name {
if err := flush(); err != nil {
return err
}
tableName = table.Name
columns, placeholders := make([]string, len(values)), make([]string, len(values))
for i := range values {
columns[i], placeholders[i] = quoteIdentifier(table.Columns[i]), "?"
}
prefix = "INSERT INTO " + quoteIdentifier(tableName) + " (" + strings.Join(columns, ",") + ") VALUES "
rowSQL = "(" + strings.Join(placeholders, ",") + ")"
}
for _, value := range values {
if value == nil {
args = append(args, nil)
} else {
args = append(args, []byte(value))
bytes += len(value)
}
}
groups = append(groups, rowSQL)
if len(groups) >= 250 || len(args)+len(values) > 60000 || bytes >= 1<<20 {
return flush()
}
return nil
})
if err == nil {
err = flush()
}
if err != nil {
return err
}
return restoreSnapshotTriggers(db, m)
}
+87
View File
@@ -0,0 +1,87 @@
package mariadb
import (
"archive/zip"
"database/sql"
"encoding/hex"
"encoding/json"
"os"
"path/filepath"
"testing"
)
func TestSnapshotRowsAndValidation(t *testing.T) {
row := []sql.RawBytes{[]byte{0, 1, 255}, {}, nil}
var sum [32]byte
addRowDigest(&sum, row)
for _, tt := range []struct {
name string
format int
rows int64
digest string
width int
valid bool
}{
{"valid", 1, 1, hex.EncodeToString(sum[:]), 3, true},
{"bad format", 2, 1, hex.EncodeToString(sum[:]), 3, false},
{"bad count", 1, 2, hex.EncodeToString(sum[:]), 3, false},
{"bad digest", 1, 1, "invalid", 3, false},
{"bad width", 1, 1, hex.EncodeToString(sum[:]), 2, false},
} {
t.Run(tt.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup.zip")
file, err := os.Create(path)
if err != nil {
t.Fatal(err)
}
archive := zip.NewWriter(file)
entry, err := archive.Create("table.jsonl")
if err != nil {
t.Fatal(err)
}
if err := json.NewEncoder(entry).Encode(row); err != nil {
t.Fatal(err)
}
entry, err = archive.Create("manifest.json")
if err != nil {
t.Fatal(err)
}
table := snapshotTable{Name: "example", Create: "CREATE TABLE `example` (id INT)", Entry: "table.jsonl", Columns: []string{"a", "b", "c"}[:tt.width], Rows: tt.rows, Digest: tt.digest}
if err := json.NewEncoder(entry).Encode(snapshotManifest{Format: tt.format, Schema: "example", Tables: []snapshotTable{table}}); err != nil {
t.Fatal(err)
}
if err := archive.Close(); err != nil {
t.Fatal(err)
}
if err := file.Close(); err != nil {
t.Fatal(err)
}
_, err = readSnapshot(path, nil)
if (err == nil) != tt.valid {
t.Fatalf("valid=%v err=%v", tt.valid, err)
}
if !tt.valid {
if err := RestoreSnapshot(nil, path); err == nil {
t.Fatal("invalid archive reached restore without preflight refusal")
}
}
})
}
}
func TestSnapshotDigestPreservesValues(t *testing.T) {
a, b := []sql.RawBytes{[]byte("a")}, []sql.RawBytes{[]byte("b")}
var forward, reverse, null, empty [32]byte
addRowDigest(&forward, a)
addRowDigest(&forward, b)
addRowDigest(&reverse, b)
addRowDigest(&reverse, a)
if forward != reverse {
t.Fatal("digest depends on row order")
}
addRowDigest(&null, []sql.RawBytes{nil})
addRowDigest(&empty, []sql.RawBytes{{}})
if null == empty {
t.Fatal("digest conflates NULL and empty bytes")
}
}
@@ -0,0 +1,43 @@
package mariadb
import (
"fmt"
"regexp"
"strings"
)
// Recovery covers the selected schema, so external trigger effects are refused.
func validateSnapshotTriggerReferences(trigger snapshotTrigger, schema string) error {
body := trigger.Body
quoted := `'(?:''|\\.|[^'\\])*'`
if strings.Contains(trigger.SQLMode, "NO_BACKSLASH_ESCAPES") {
quoted = `'(?:''|[^'])*'`
}
body = regexp.MustCompile(quoted).ReplaceAllString(body, " ")
if !strings.Contains(trigger.SQLMode, "ANSI_QUOTES") {
double := `"(?:""|\\.|[^"\\])*"`
if strings.Contains(trigger.SQLMode, "NO_BACKSLASH_ESCAPES") {
double = `"(?:""|[^"])*"`
}
body = regexp.MustCompile(double).ReplaceAllString(body, " ")
}
if strings.Contains(body, "/*!") || strings.Contains(body, "/*M!") {
return fmt.Errorf("trigger %s has unsupported executable comments", trigger.Name)
}
body = regexp.MustCompile(`(?s)/\*.*?\*/|--[\t ][^\n]*|#[^\n]*`).ReplaceAllString(body, " ")
if regexp.MustCompile(`(?i)\bCALL\s+`).MatchString(body) {
return fmt.Errorf("trigger %s calls a routine outside supported recovery", trigger.Name)
}
identifier := "(?:`(?:``|[^`])+`|\"(?:\"\"|[^\"])+\"|[\\pL_$][\\pL\\pN_$]*)"
for _, pattern := range []string{
"(?i)\\b(?:FROM|JOIN|UPDATE|INTO)\\s+(?:(?:LOW_PRIORITY|HIGH_PRIORITY|DELAYED|IGNORE)\\s+)*(" + identifier + ")\\s*\\.",
"(?i)(" + identifier + ")\\s*\\.\\s*" + identifier + "\\s*\\(",
} {
for _, match := range regexp.MustCompile(pattern).FindAllStringSubmatch(body, -1) {
if !strings.EqualFold(strings.Trim(match[1], "`\""), schema) {
return fmt.Errorf("trigger %s references another schema outside supported recovery", trigger.Name)
}
}
}
return nil
}
@@ -0,0 +1,29 @@
package mariadb
import "testing"
func TestSnapshotTriggerReferences(t *testing.T) {
for _, tt := range []struct {
name, body, mode string
fail bool
}{
{"local", "INSERT INTO audit VALUES (NEW.id, OLD.value)", "", false},
{"local qualified", "UPDATE `example`.`audit` SET id=NEW.id", "", false},
{"alias", "SELECT 1 FROM audit a WHERE a.id=OLD.id", "", false},
{"literal", "INSERT INTO audit VALUES ('example.com', '$.field', 'UPDATE other.audit')", "", false},
{"comment", "/* UPDATE other.audit */ INSERT INTO audit VALUES (NEW.id)", "", false},
{"external write", "UPDATE other.audit SET id=NEW.id", "", true},
{"external insert", "INSERT INTO `other`.`audit` VALUES (NEW.id)", "", true},
{"ANSI external", `UPDATE "other"."audit" SET id=NEW.id`, "ANSI_QUOTES", true},
{"external function", "SET NEW.value=other.example(NEW.value)", "", true},
{"routine", "CALL example.change_rows()", "", true},
{"executable", "/*!50000 UPDATE other.audit SET id=1 */", "", true},
} {
t.Run(tt.name, func(t *testing.T) {
err := validateSnapshotTriggerReferences(snapshotTrigger{Name: "example", Body: tt.body, SQLMode: tt.mode}, "example")
if (err != nil) != tt.fail {
t.Fatalf("error=%v", err)
}
})
}
}
+83
View File
@@ -0,0 +1,83 @@
package mariadb
import (
"errors"
"fmt"
"strings"
"gorm.io/gorm"
)
type snapshotTrigger struct {
Name string `gorm:"column:TRIGGER_NAME"`
Table string `gorm:"column:EVENT_OBJECT_TABLE"`
Event string `gorm:"column:EVENT_MANIPULATION"`
Timing string `gorm:"column:ACTION_TIMING"`
Order int `gorm:"column:ACTION_ORDER"`
Body string `gorm:"column:ACTION_STATEMENT"`
SQLMode string `gorm:"column:SQL_MODE"`
Definer string `gorm:"column:DEFINER"`
Charset string `gorm:"column:CHARACTER_SET_CLIENT"`
Collation string `gorm:"column:COLLATION_CONNECTION"`
DatabaseCollation string `gorm:"column:DATABASE_COLLATION"`
}
func readSnapshotTriggers(db *gorm.DB, m *snapshotManifest) error {
query := `SELECT TRIGGER_NAME, EVENT_OBJECT_TABLE, EVENT_MANIPULATION, ACTION_TIMING,
ACTION_ORDER, ACTION_STATEMENT, SQL_MODE, DEFINER, CHARACTER_SET_CLIENT,
COLLATION_CONNECTION, DATABASE_COLLATION FROM information_schema.TRIGGERS
WHERE TRIGGER_SCHEMA=? ORDER BY EVENT_OBJECT_TABLE, ACTION_TIMING, EVENT_MANIPULATION, ACTION_ORDER`
if err := freshDB(db).Raw(query, m.Schema).Scan(&m.Triggers).Error; err != nil {
return fmt.Errorf("read snapshot triggers: %w", err)
}
for _, trigger := range m.Triggers {
if err := validateSnapshotTriggerReferences(trigger, m.Schema); err != nil {
return err
}
if trigger.DatabaseCollation != m.Collation {
return fmt.Errorf("trigger %s has a legacy database collation; restore support is required before migration", trigger.Name)
}
}
return nil
}
func restoreSnapshotTriggers(db *gorm.DB, m *snapshotManifest) (err error) {
if len(m.Triggers) == 0 {
return nil
}
var mode, charset, collation, schema string
if err := freshDB(db).Raw("SELECT @@sql_mode, @@character_set_client, @@collation_connection, DATABASE()").Row().Scan(&mode, &charset, &collation, &schema); err != nil {
return fmt.Errorf("read trigger restore session: %w", err)
}
defer func() {
for _, reset := range []struct{ sql, value string }{{"SET sql_mode=?", mode}, {"SET character_set_client=?", charset}, {"SET collation_connection=?", collation}} {
if e := freshDB(db).Exec(reset.sql, reset.value).Error; e != nil {
err = errors.Join(err, fmt.Errorf("reset trigger restore session: %w", e))
}
}
}()
for _, trigger := range m.Triggers {
if trigger.Timing != "BEFORE" && trigger.Timing != "AFTER" {
return fmt.Errorf("invalid trigger timing")
}
if trigger.Event != "INSERT" && trigger.Event != "UPDATE" && trigger.Event != "DELETE" {
return fmt.Errorf("invalid trigger event")
}
at := strings.LastIndex(trigger.Definer, "@")
if at < 1 || at == len(trigger.Definer)-1 {
return fmt.Errorf("invalid trigger definer")
}
for _, setting := range []struct{ sql, value string }{{"SET sql_mode=?", trigger.SQLMode}, {"SET character_set_client=?", trigger.Charset}, {"SET collation_connection=?", trigger.Collation}} {
if err := freshDB(db).Exec(setting.sql, setting.value).Error; err != nil {
return fmt.Errorf("set trigger restore session: %w", err)
}
}
// Explicitly bind name and target table to the destination. Trigger bodies
// retain their original SQL; verification performs no DML after creation.
statement := "CREATE DEFINER=" + quoteIdentifier(trigger.Definer[:at]) + "@" + quoteIdentifier(trigger.Definer[at+1:]) + " TRIGGER " + quoteIdentifier(schema) + "." + quoteIdentifier(trigger.Name) + " " + trigger.Timing + " " + trigger.Event + " ON " + quoteIdentifier(schema) + "." + quoteIdentifier(trigger.Table) + " FOR EACH ROW " + trigger.Body
if _, err := db.Statement.ConnPool.ExecContext(db.Statement.Context, statement); err != nil {
return fmt.Errorf("restore trigger %s: %w", trigger.Name, err)
}
}
return nil
}
+79
View File
@@ -0,0 +1,79 @@
package mariadb
import (
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"io"
"reflect"
"gorm.io/gorm"
)
// CheckSnapshot compares restored schema and all writable row values.
func CheckSnapshot(db *gorm.DB, path string) error {
m, err := readSnapshot(path, nil)
if err != nil {
return err
}
return compareSnapshot(db, m)
}
// VerifySnapshot proves restore readiness in an isolated temporary database.
func VerifySnapshot(db *gorm.DB, path string) (err error) {
m, err := readSnapshot(path, nil)
if err != nil {
return err
}
var random [12]byte
if _, err := rand.Read(random[:]); err != nil {
return fmt.Errorf("name verification schema: %w", err)
}
scratch := "qfs_verify_" + hex.EncodeToString(random[:])
var original string
if err := freshDB(db).Raw("SELECT DATABASE()").Scan(&original).Error; err != nil {
return fmt.Errorf("read original schema: %w", err)
}
if err := freshDB(db).Exec("CREATE DATABASE " + quoteIdentifier(scratch) + " COLLATE " + quoteIdentifier(m.Collation)).Error; err != nil {
return fmt.Errorf("create restore verification schema: %w", err)
}
defer func() {
if e := freshDB(db).Exec("USE " + quoteIdentifier(original)).Error; e != nil {
err = errors.Join(err, fmt.Errorf("return to original schema: %w", e))
}
if e := freshDB(db).Exec("DROP DATABASE " + quoteIdentifier(scratch)).Error; e != nil {
err = errors.Join(err, fmt.Errorf("remove verification schema %s: %w", scratch, e))
}
}()
if err := freshDB(db).Exec("USE " + quoteIdentifier(scratch)).Error; err != nil {
return fmt.Errorf("select verification schema: %w", err)
}
if err := restoreSnapshotTables(db, path, m); err != nil {
return err
}
return compareSnapshot(db, m)
}
func compareSnapshot(db *gorm.DB, m *snapshotManifest) error {
current, err := snapshotInventory(db)
if err != nil {
return err
}
if !reflect.DeepEqual(current.Triggers, m.Triggers) {
return fmt.Errorf("restored trigger metadata mismatch")
}
if len(current.Tables) != len(m.Tables) {
return fmt.Errorf("restored table count mismatch")
}
for i := range m.Tables {
want, got := m.Tables[i], current.Tables[i]
if err := snapshotRows(db, &got, io.Discard); err != nil {
return err
}
if want.Name != got.Name || want.Create != got.Create || want.Rows != got.Rows || want.Digest != got.Digest {
return fmt.Errorf("restored snapshot mismatch in %s (name=%t schema=%t rows=%t digest=%t)", want.Name, want.Name == got.Name, want.Create == got.Create, want.Rows == got.Rows, want.Digest == got.Digest)
}
}
return nil
}
+128
View File
@@ -0,0 +1,128 @@
//go:build integration
package mariadb
import (
"context"
"crypto/rand"
"crypto/rsa"
"crypto/x509"
"crypto/x509/pkix"
"database/sql"
"encoding/pem"
"math/big"
"net"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
mysql "github.com/go-sql-driver/mysql"
)
func TestMariaDBTLSIntegration(t *testing.T) {
dir := t.TempDir()
if err := os.Chmod(dir, 0755); err != nil {
t.Fatal(err)
}
key, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
t.Fatal(err)
}
cert := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "example"},
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour),
IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}}
der, err := x509.CreateCertificate(rand.Reader, cert, cert, &key.PublicKey, key)
if err != nil {
t.Fatal(err)
}
ca := string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}))
for name, data := range map[string][]byte{"server.crt": []byte(ca), "server.key": pem.EncodeToMemory(&pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(key)})} {
// Docker's mysql user must read these disposable test files.
if err := os.WriteFile(filepath.Join(dir, name), data, 0644); err != nil {
t.Fatal(err)
}
}
otherCA, _ := testCertificate(t)
for _, tlsEnabled := range []bool{true, false} {
t.Run(map[bool]string{true: "TLS", false: "plaintext"}[tlsEnabled], func(t *testing.T) {
args := []string{"run", "--detach", "--rm", "--pull=never", "-p", "127.0.0.1::3306",
"-e", "MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1", "-e", "MARIADB_ROOT_HOST=%", "-v", dir + ":/qfs-tls:ro", "mariadb:11.8"}
if tlsEnabled {
args = append(args, "--ssl-cert=/qfs-tls/server.crt", "--ssl-key=/qfs-tls/server.key")
} else {
args = append(args, "--skip-ssl")
}
out, err := exec.Command("docker", args...).CombinedOutput()
if err != nil {
t.Fatalf("start disposable MariaDB: %v: %s", err, out)
}
id := strings.TrimSpace(string(out))
t.Cleanup(func() {
if out, err := exec.Command("docker", "stop", "-t", "2", id).CombinedOutput(); err != nil {
t.Errorf("stop test MariaDB: %v: %s", err, out)
}
})
out, err = exec.Command("docker", "port", id, "3306/tcp").CombinedOutput()
if err != nil {
t.Fatalf("test MariaDB port: %v: %s", err, out)
}
addr := strings.TrimSpace(string(out))
connect := func(mode, roots, serverName string) (string, error) {
cfg := mysql.NewConfig()
cfg.User, cfg.Net, cfg.Addr, cfg.Timeout = "root", "tcp", addr, time.Second
if err := ConfigureTLS(cfg, mode, roots); err != nil {
return "", err
}
cfg, err = mysql.ParseDSN(cfg.FormatDSN())
if err != nil {
return "", err
}
if serverName != "" {
cfg.TLS.ServerName = serverName
}
connector, err := mysql.NewConnector(cfg)
if err != nil {
return "", err
}
db := sql.OpenDB(connector)
defer db.Close()
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
var name, cipher string
err = db.QueryRowContext(ctx, "SHOW SESSION STATUS LIKE 'Ssl_cipher'").Scan(&name, &cipher)
return cipher, err
}
deadline := time.Now().Add(45 * time.Second)
for {
if _, err := connect("preferred", "", ""); err == nil {
break
}
if time.Now().After(deadline) {
t.Fatal("test MariaDB did not become ready")
}
time.Sleep(100 * time.Millisecond)
}
for _, tt := range []struct {
mode, roots, name string
success bool
}{
{"preferred", "", "", true}, {"required", "", "", tlsEnabled},
{"verify-ca", ca, "", tlsEnabled}, {"verify-full", ca, "", tlsEnabled},
{"verify-ca", otherCA, "", false}, {"verify-full", otherCA, "", false},
{"verify-full", ca, "wrong.example", false},
} {
cipher, err := connect(tt.mode, tt.roots, tt.name)
if (err == nil) != tt.success {
t.Errorf("mode=%s name=%s success=%v error=%v", tt.mode, tt.name, tt.success, err)
}
if err == nil && (cipher != "") != tlsEnabled {
t.Errorf("mode=%s cipher=%q TLS=%v", tt.mode, cipher, tlsEnabled)
}
}
})
}
}
@@ -0,0 +1,162 @@
//go:build integration
package models
import (
"context"
"os"
"os/exec"
"regexp"
"strings"
"testing"
"testing/fstest"
"time"
qfassets "git.mchus.pro/mchus/quoteforge"
"git.mchus.pro/mchus/quoteforge/internal/mariadb"
"gorm.io/driver/mysql"
"gorm.io/gorm"
"gorm.io/gorm/logger"
)
// Input is read-only. All SQL runs in a new, disposable local container.
func TestProductionDumpMigrationMariaDB(t *testing.T) {
dump := os.Getenv("QFS_TEST_PRODUCTION_DUMP")
if dump == "" {
t.Skip("QFS_TEST_PRODUCTION_DUMP is not set")
}
input, err := os.Open(dump)
if err != nil {
t.Fatal(err)
}
defer input.Close()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
image := os.Getenv("QFS_TEST_MARIADB_IMAGE")
if image == "" {
image = "mariadb:11.8.3"
}
out, err := exec.CommandContext(ctx, "docker", "run", "--detach", "--rm", "--pull=never", "-p", "127.0.0.1::3306",
"-e", "MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1", "-e", "MARIADB_ROOT_HOST=%", "-e", "MARIADB_DATABASE=RFQ_LOG", image,
"--skip-ssl", "--character-set-server=utf8mb4", "--collation-server=utf8mb4_uca1400_ai_ci", "--max-allowed-packet=256M").CombinedOutput()
if err != nil {
t.Fatalf("start dump rehearsal MariaDB: %v", err)
}
id := strings.TrimSpace(string(out))
t.Cleanup(func() {
if _, err := exec.Command("docker", "stop", "-t", "2", id).CombinedOutput(); err != nil {
t.Errorf("stop dump rehearsal MariaDB: %v", err)
}
})
out, err = exec.CommandContext(ctx, "docker", "port", id, "3306/tcp").CombinedOutput()
if err != nil {
t.Fatal(err)
}
dsn := "root@tcp(" + strings.TrimSpace(string(out)) + ")/RFQ_LOG?parseTime=true&timeout=1s&tls=false"
var db *gorm.DB
deadline := time.Now().Add(45 * time.Second)
for {
db, err = gorm.Open(mysql.Open(dsn), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
if err == nil {
break
}
if time.Now().After(deadline) {
t.Fatal("dump rehearsal database did not become ready")
}
time.Sleep(150 * time.Millisecond)
}
pool, err := db.DB()
if err != nil {
t.Fatal(err)
}
defer pool.Close()
pool.SetMaxOpenConns(1)
pool.SetMaxIdleConns(1)
importer := exec.CommandContext(ctx, "docker", "exec", "-i", id, "mariadb", "--skip-ssl", "--binary-mode", "--default-character-set=utf8mb4", "RFQ_LOG")
importer.Stdin = input
if output, err := importer.CombinedOutput(); err != nil {
// Import errors can include entire data rows; report only error code/line.
location := regexp.MustCompile(`(?m)^ERROR[^\n]*`).Find(output)
location = regexp.MustCompile(`'[^']*'`).ReplaceAll(location, []byte("'<redacted>'"))
if len(location) > 250 {
location = location[:250]
}
t.Fatalf("load dump into disposable database: %v (%s)", err, location)
}
var version, collation string
if err := db.Raw("SELECT VERSION(), @@collation_database").Row().Scan(&version, &collation); err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(version, "11.8.3-") || collation != "utf8mb4_uca1400_ai_ci" {
t.Fatalf("unexpected rehearsal server: version=%s collation=%s", version, collation)
}
var tables, triggers int64
if err := db.Raw("SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA=DATABASE()").Scan(&tables).Error; err != nil {
t.Fatal(err)
}
if err := db.Raw("SELECT COUNT(*) FROM information_schema.TRIGGERS WHERE TRIGGER_SCHEMA=DATABASE()").Scan(&triggers).Error; err != nil {
t.Fatal(err)
}
t.Logf("production dump imported: MariaDB=%s collation=%s tables=%d triggers=%d", version, collation, tables, triggers)
if err := db.Exec("SET time_zone='+00:00'").Error; err != nil {
t.Fatal(err)
}
root := t.TempDir()
var original string
if err := db.Connection(func(conn *gorm.DB) error {
var err error
original, err = mariadb.CreateSnapshot(conn, root, "production-dump")
if err != nil {
return err
}
return mariadb.VerifySnapshot(conn, original)
}); err != nil {
t.Fatal(err)
}
t.Log("full dump snapshot restore verified")
const filename = "035_add_server_updated_at.sql"
if err := db.Where("filename=?", filename).Delete(&SQLSchemaMigration{}).Error; err != nil {
t.Fatal(err)
}
// Reconstruct the real pre-035 schema only in this disposable clone.
for _, table := range []string{"qt_configurations", "qt_projects"} {
if err := db.Exec("ALTER TABLE " + table + " DROP INDEX IF EXISTS idx_" + table + "_server_updated_at, DROP COLUMN IF EXISTS server_updated_at").Error; err != nil {
t.Fatal(err)
}
}
var before string
if err := db.Connection(func(conn *gorm.DB) error {
var err error
before, err = mariadb.CreateSnapshot(conn, root, "before-rehearsal")
return err
}); err != nil {
t.Fatal(err)
}
content, err := qfassets.MigrationFiles.ReadFile("migrations/" + filename)
if err != nil {
t.Fatal(err)
}
files := fstest.MapFS{filename: &fstest.MapFile{Data: content}, "036_recovery_probe.sql": &fstest.MapFile{Data: []byte("-- Tables affected: qt_configurations\n-- verify: deliberate rehearsal failure | SELECT 1\nALTER TABLE qt_configurations ADD COLUMN qfs_recovery_probe INT;")}}
if _, err := RunSQLMigrations(db, files, nil, root); err == nil || !strings.Contains(err.Error(), "all session changes restored") {
t.Fatalf("dump rollback rehearsal: %v", err)
}
if err := mariadb.CheckSnapshot(db, before); err != nil {
t.Fatalf("dump changed after rollback: %v", err)
}
t.Log("real migration plus later failed DDL/verify restored the complete dump")
delete(files, "036_recovery_probe.sql")
if applied, err := RunSQLMigrations(db, files, nil, root); err != nil || len(applied) != 1 {
t.Fatalf("real migration rehearsal: applied=%v error=%v", applied, err)
}
states, err := SQLMigrationStatus(db, files)
if err != nil || len(states) != 1 || states[0].AppliedAt == nil {
t.Fatalf("rehearsal history invalid: %v", err)
}
if err := RestoreSQLMigrationSession(db, original); err != nil {
t.Fatal(err)
}
if err := mariadb.CheckSnapshot(db, original); err != nil {
t.Fatal(err)
}
t.Logf("production dump rehearsal complete: %s, exact original restored", filename)
}
@@ -0,0 +1,44 @@
package models
import (
"crypto/sha256"
"fmt"
"strings"
"gorm.io/driver/mysql"
"gorm.io/gorm"
)
// migrationJournalName uses the configured endpoint, which survives server restarts.
func migrationJournalName(db *gorm.DB, schema string) (string, error) {
dialect, ok := db.Dialector.(*mysql.Dialector)
if !ok || dialect.DSNConfig == nil {
return "", fmt.Errorf("migration requires a MariaDB DSN")
}
identity := dialect.DSNConfig.Net + "/" + dialect.DSNConfig.Addr + "/" + schema
return fmt.Sprintf("qfs-migration-v1-%x.pending", sha256.Sum256([]byte(identity))), nil
}
func requireExclusiveMigrationSession(db *gorm.DB) error {
var grants []string
if err := db.Session(&gorm.Session{NewDB: true}).Raw("SHOW GRANTS FOR CURRENT_USER").Scan(&grants).Error; err != nil {
return fmt.Errorf("read maintenance privileges: %w", err)
}
allowed := false
for _, grant := range grants {
if strings.Contains(grant, " ON *.* ") && strings.Contains(grant, "ALL PRIVILEGES") {
allowed = true
}
}
if !allowed {
return fmt.Errorf("migration recovery requires a DBA with directly granted global ALL PRIVILEGES to inspect all objects and clients")
}
var clients int64
if err := db.Session(&gorm.Session{NewDB: true}).Raw("SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE ID<>CONNECTION_ID() AND DB=DATABASE()").Scan(&clients).Error; err != nil {
return fmt.Errorf("check maintenance clients: %w", err)
}
if clients != 0 {
return fmt.Errorf("migration requires maintenance window: %d other connections use this schema", clients)
}
return nil
}
+45
View File
@@ -0,0 +1,45 @@
package models
import (
"fmt"
"io/fs"
"regexp"
"gorm.io/gorm"
)
func planSQLMigrations(db *gorm.DB, migrations fs.FS) ([]plannedSQLMigration, error) {
states, err := SQLMigrationStatus(db, migrations)
if err != nil {
return nil, err
}
var plans []plannedSQLMigration
unsupported := regexp.MustCompile(`(?i)\b(?:CREATE|DROP|ALTER)\s+(?:DATABASE|SCHEMA|VIEW|TRIGGER|PROCEDURE|FUNCTION|EVENT)\b|\b(?:USE|DELIMITER|GRANT|REVOKE)\s|\bSET\s+(?:GLOBAL|SQL_MODE)\b|\bENGINE\s*=\s*(?:MyISAM|Aria|MEMORY|BLACKHOLE)\b`)
for _, state := range states {
if state.AppliedAt != nil {
continue
}
content, err := fs.ReadFile(migrations, state.Filename)
if err != nil {
return nil, fmt.Errorf("read migration %s: %w", state.Filename, err)
}
statements, err := splitSQLStatements(string(content))
if err != nil {
return nil, fmt.Errorf("parse migration %s: %w", state.Filename, err)
}
checks, err := parseMigrationVerifications(string(content))
if err != nil {
return nil, fmt.Errorf("migration %s verify headers: %w", state.Filename, err)
}
if len(checks) == 0 {
return nil, fmt.Errorf("migration %s needs verify headers", state.Filename)
}
for _, stmt := range statements {
if unsupported.MatchString(stmt) {
return nil, fmt.Errorf("migration %s contains SQL outside supported table recovery", state.Filename)
}
}
plans = append(plans, plannedSQLMigration{state.Filename, statements, checks})
}
return plans, nil
}
+67
View File
@@ -0,0 +1,67 @@
package models
import (
"crypto/sha256"
"errors"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"git.mchus.pro/mchus/quoteforge/internal/mariadb"
"gorm.io/gorm"
)
// RestoreSQLMigrationSession is the shipped recovery path after interruption.
func RestoreSQLMigrationSession(db *gorm.DB, snapshot string) error {
return db.Connection(func(conn *gorm.DB) (err error) {
fresh := func() *gorm.DB { return conn.Session(&gorm.Session{NewDB: true}) }
var schema, zone string
var acquired int
if err := fresh().Raw("SELECT DATABASE(), @@time_zone").Row().Scan(&schema, &zone); err != nil {
return fmt.Errorf("read recovery session: %w", err)
}
lock := fmt.Sprintf("qfs-migrate-%x", sha256.Sum256([]byte(schema)))
if err := fresh().Raw("SELECT GET_LOCK(?,0)", lock).Scan(&acquired).Error; err != nil || acquired != 1 {
return fmt.Errorf("migration lock unavailable for recovery")
}
defer func() {
if e := fresh().Exec("SET time_zone=?", zone).Error; e != nil {
err = errors.Join(err, fmt.Errorf("reset recovery time zone: %w", e))
}
if e := fresh().Raw("SELECT RELEASE_LOCK(?)", lock).Scan(&acquired).Error; e != nil {
err = errors.Join(err, fmt.Errorf("release recovery lock: %w", e))
}
}()
if err := requireExclusiveMigrationSession(fresh()); err != nil {
return err
}
journalName, err := migrationJournalName(conn, schema)
if err != nil {
return err
}
journal := filepath.Join(filepath.Dir(snapshot), journalName)
if content, e := os.ReadFile(journal); e == nil {
if strings.TrimSpace(string(content)) != snapshot {
return fmt.Errorf("pending recovery requires its original snapshot: %s", strings.TrimSpace(string(content)))
}
} else if !os.IsNotExist(e) {
return fmt.Errorf("read recovery marker: %w", e)
}
if err := fresh().Exec("SET time_zone='+00:00'").Error; err != nil {
return fmt.Errorf("set restore time zone: %w", err)
}
if err := mariadb.VerifySnapshot(fresh(), snapshot); err != nil {
return fmt.Errorf("verify recovery snapshot: %w", err)
}
if err := mariadb.RestoreSnapshot(fresh(), snapshot); err != nil {
return fmt.Errorf("restore migration session: %w", err)
}
if err := os.Remove(journal); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("clear recovery marker: %w", err)
}
slog.Info("server migration snapshot restored", "schema", schema, "backup", snapshot, "state", "verified")
return nil
})
}
+165
View File
@@ -0,0 +1,165 @@
package models
import (
"crypto/sha256"
"errors"
"fmt"
"io/fs"
"log/slog"
"os"
"path/filepath"
"strings"
"git.mchus.pro/mchus/quoteforge/internal/appstate"
"git.mchus.pro/mchus/quoteforge/internal/mariadb"
"gorm.io/gorm"
)
type plannedSQLMigration struct {
filename string
statements []string
checks []migrationVerification
}
// RunSQLMigrations requires a DBA maintenance window: stop all other writers.
// Backups are mandatory; each file gets per-table snapshots before any DDL.
func RunSQLMigrations(db *gorm.DB, migrations fs.FS, onApply func(string), backupRoot string) ([]string, error) {
var applied []string
err := db.Connection(func(conn *gorm.DB) (err error) {
fresh := func() *gorm.DB { return conn.Session(&gorm.Session{NewDB: true}) }
var schema, mode, zone, engine string
var foreignKeys int
if err := fresh().Raw("SELECT DATABASE(), @@sql_mode, @@time_zone, @@foreign_key_checks, @@default_storage_engine").Row().Scan(&schema, &mode, &zone, &foreignKeys, &engine); err != nil {
return fmt.Errorf("read migration session: %w", err)
}
if schema == "" || strings.Contains(mode, "NO_BACKSLASH_ESCAPES") || engine != "InnoDB" {
return fmt.Errorf("migration requires a selected InnoDB schema and backslash escape SQL mode")
}
lock := fmt.Sprintf("qfs-migrate-%x", sha256.Sum256([]byte(schema)))
var acquired int
if err := fresh().Raw("SELECT GET_LOCK(?,0)", lock).Scan(&acquired).Error; err != nil || acquired != 1 {
return fmt.Errorf("migration lock unavailable for %s", schema)
}
defer func() {
for _, reset := range []struct {
sql string
value any
}{{"SET time_zone=?", zone}, {"SET sql_mode=?", mode}, {"SET foreign_key_checks=?", foreignKeys}} {
if e := fresh().Exec(reset.sql, reset.value).Error; e != nil {
err = errors.Join(err, fmt.Errorf("reset migration session: %w", e))
}
}
if e := fresh().Raw("SELECT RELEASE_LOCK(?)", lock).Scan(&acquired).Error; e != nil {
err = errors.Join(err, fmt.Errorf("release migration lock: %w", e))
}
}()
plans, err := planSQLMigrations(fresh(), migrations)
if err != nil {
return err
}
if err := appstate.ValidateBackupLocation(backupRoot); err != nil {
return err
}
if err := os.MkdirAll(backupRoot, 0700); err != nil {
return fmt.Errorf("create migration backup root: %w", err)
}
journalName, err := migrationJournalName(conn, schema)
if err != nil {
return err
}
journal := filepath.Join(backupRoot, journalName)
if content, e := os.ReadFile(journal); e == nil {
return fmt.Errorf("interrupted migration requires restore before retry; safety snapshot: %s", strings.TrimSpace(string(content)))
} else if !os.IsNotExist(e) {
return fmt.Errorf("read migration recovery marker: %w", e)
}
if len(plans) == 0 {
return nil
}
if err := requireExclusiveMigrationSession(fresh()); err != nil {
return err
}
if err := fresh().Exec("SET time_zone='+00:00'").Error; err != nil {
return fmt.Errorf("set snapshot time zone: %w", err)
}
full, err := mariadb.CreateSnapshot(fresh(), backupRoot, "session")
if err != nil {
return fmt.Errorf("pre-migration snapshot: %w", err)
}
if err := mariadb.VerifySnapshot(fresh(), full); err != nil {
return fmt.Errorf("pre-migration restore verification: %w", err)
}
marker, err := os.OpenFile(journal, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
if err != nil {
return fmt.Errorf("create migration recovery marker: %w", err)
}
_, writeErr := fmt.Fprintln(marker, full)
syncErr, closeErr := marker.Sync(), marker.Close()
if e := errors.Join(writeErr, syncErr, closeErr); e != nil {
return fmt.Errorf("persist migration recovery marker: %w", e)
}
slog.Info("verified server migration backup", "schema", schema, "path", full)
var steps []string
for _, plan := range plans {
if err := requireExclusiveMigrationSession(fresh()); err != nil {
return recoverSQLMigration(fresh(), journal, full, steps, plan.filename, err)
}
step, err := mariadb.CreateSnapshot(fresh(), backupRoot, plan.filename)
if err == nil {
err = mariadb.VerifySnapshot(fresh(), step)
}
if err != nil {
return recoverSQLMigration(fresh(), journal, full, steps, plan.filename, err)
}
steps = append(steps, step)
if onApply != nil {
onApply(plan.filename)
}
if err := ensureSQLMigrationsTable(fresh()); err != nil {
return recoverSQLMigration(fresh(), journal, full, steps, plan.filename, err)
}
if err := applySQLMigrationOnSession(fresh(), plan.filename, plan.statements, plan.checks); err != nil {
return recoverSQLMigration(fresh(), journal, full, steps, plan.filename, err)
}
applied = append(applied, plan.filename)
}
if err := os.Remove(journal); err != nil {
return fmt.Errorf("remove completed migration marker: %w", err)
}
return nil
})
if err != nil {
return nil, err
}
return applied, nil
}
func recoverSQLMigration(db *gorm.DB, journal, full string, steps []string, filename string, cause error) error {
slog.Error("server migration failed", "step", filename, "error", cause, "backup", full)
if err := requireExclusiveMigrationSession(db); err != nil {
return errors.Join(cause, fmt.Errorf("recovery refused with active clients; keep maintenance window and restore %s: %w", full, err))
}
var recovery error
for i := len(steps) - 1; i >= 0; i-- {
slog.Warn("restoring migration step", "step", filename, "backup", steps[i])
if err := mariadb.RestoreSnapshot(db, steps[i]); err != nil {
recovery = err
break
}
}
if recovery == nil {
recovery = mariadb.CheckSnapshot(db, full)
}
if recovery != nil {
slog.Warn("restoring full migration session", "backup", full, "error", recovery)
recovery = mariadb.RestoreSnapshot(db, full)
}
if recovery != nil {
return errors.Join(cause, fmt.Errorf("recovery failed; database requires restore from %s: %w", full, recovery))
}
if err := os.Remove(journal); err != nil {
return errors.Join(cause, fmt.Errorf("database restored but recovery marker remains: %w", err))
}
slog.Warn("server migration session restored", "step", filename, "state", "before-session", "backup", full)
return fmt.Errorf("migration %s failed; all session changes restored: %w", filename, cause)
}
@@ -0,0 +1,227 @@
//go:build integration
package models
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"testing/fstest"
"time"
"git.mchus.pro/mchus/quoteforge/internal/mariadb"
"gorm.io/driver/mysql"
"gorm.io/gorm"
"gorm.io/gorm/logger"
)
func TestSQLMigrationSafetyMariaDB(t *testing.T) {
out, err := exec.Command("docker", "run", "--detach", "--rm", "--pull=never", "-p", "127.0.0.1::3306", "-e", "MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1", "-e", "MARIADB_ROOT_HOST=%", "mariadb:11.8", "--skip-ssl").CombinedOutput()
if err != nil {
t.Fatalf("start test MariaDB: %v: %s", err, out)
}
id := strings.TrimSpace(string(out))
t.Cleanup(func() {
if out, err := exec.Command("docker", "stop", "-t", "2", id).CombinedOutput(); err != nil {
t.Errorf("stop test MariaDB: %v: %s", err, out)
}
})
out, err = exec.Command("docker", "port", id, "3306/tcp").CombinedOutput()
if err != nil {
t.Fatalf("get test MariaDB port: %v: %s", err, out)
}
addr := strings.TrimSpace(string(out))
dsn := "root@tcp(" + addr + ")/?parseTime=true&timeout=1s&tls=false"
var admin *gorm.DB
deadline := time.Now().Add(45 * time.Second)
for {
admin, err = gorm.Open(mysql.Open(dsn), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
if err == nil {
break
}
if time.Now().After(deadline) {
t.Fatalf("test MariaDB unavailable: %v", err)
}
time.Sleep(150 * time.Millisecond)
}
pool, err := admin.DB()
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { pool.Close() })
newDB := func(t *testing.T) *gorm.DB {
t.Helper()
schema := fmt.Sprintf("qfs_test_%d", time.Now().UnixNano())
if err := admin.Exec("CREATE DATABASE " + schema + " COLLATE utf8mb4_unicode_ci").Error; err != nil {
t.Fatal(err)
}
db, err := gorm.Open(mysql.Open(strings.Replace(dsn, ")/?", ")/"+schema+"?", 1)), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
if err != nil {
t.Fatal(err)
}
p, err := db.DB()
if err != nil {
t.Fatal(err)
}
p.SetMaxOpenConns(1)
p.SetMaxIdleConns(1)
t.Cleanup(func() {
p.Close()
if err := admin.Exec("DROP DATABASE " + schema).Error; err != nil {
t.Error(err)
}
})
for _, sql := range []string{
"SET time_zone='+00:00'",
"CREATE TABLE parent (id INT PRIMARY KEY AUTO_INCREMENT, value TEXT, empty_value TEXT, nullable TEXT, raw VARBINARY(32), amount DECIMAL(20,6), stamp TIMESTAMP(6), generated INT AS (id+1) VIRTUAL) ENGINE=InnoDB",
"INSERT INTO parent (id,value,empty_value,nullable,raw,amount,stamp) VALUES (1,'quote; and ''text''','',NULL,X'0001FF',123.456789,'2026-10-26 12:34:56.123456')",
"SET @saved_test_mode=@@sql_mode",
"SET sql_mode='NO_AUTO_VALUE_ON_ZERO'",
"INSERT INTO parent (id,value,stamp) VALUES (0,'legacy','0000-00-00 00:00:00')",
"SET sql_mode=@saved_test_mode",
"CREATE TABLE child (id INT PRIMARY KEY, parent_id INT, CONSTRAINT child_parent FOREIGN KEY (parent_id) REFERENCES parent(id)) ENGINE=InnoDB",
"INSERT INTO child VALUES (1,1)",
"CREATE TABLE audit (id INT PRIMARY KEY AUTO_INCREMENT, note TEXT) ENGINE=InnoDB",
"CREATE TRIGGER insert_value BEFORE INSERT ON parent FOR EACH ROW SET NEW.value=CONCAT(NEW.value,'!')",
"CREATE TRIGGER audit_z AFTER UPDATE ON parent FOR EACH ROW INSERT INTO audit (note) VALUES ('first')",
"CREATE TRIGGER audit_a AFTER UPDATE ON parent FOR EACH ROW INSERT INTO audit (note) VALUES ('second')",
} {
if err := db.Exec(sql).Error; err != nil {
t.Fatal(err)
}
}
return db
}
migration := func(sql, checks string) *fstest.MapFile {
return &fstest.MapFile{Data: []byte("-- Tables affected: parent, child\n" + checks + "\n" + sql)}
}
for _, failure := range []string{"none", "verify", "FK"} {
t.Run(failure, func(t *testing.T) {
db, root := newDB(t), t.TempDir()
var original string
err := db.Connection(func(conn *gorm.DB) error {
var e error
original, e = mariadb.CreateSnapshot(conn, root, "original")
return e
})
if err != nil {
t.Fatal(err)
}
files := fstest.MapFS{"001_a.sql": migration("SET @example='value; quoted'; PREPARE q FROM 'SELECT 1'; EXECUTE q; DEALLOCATE PREPARE q; ALTER TABLE parent ADD COLUMN added INT; UPDATE parent SET value=@example;", "-- verify: added missing | SELECT 1 FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=DATABASE() AND TABLE_NAME='parent' AND COLUMN_NAME='added' HAVING COUNT(*)=0\n-- verify: wrong value | SELECT 1 FROM parent WHERE value<>'value; quoted'")}
if failure == "verify" {
files["002_b.sql"] = migration("CREATE TABLE new_table (id INT); UPDATE parent SET value='changed';", "-- verify: deliberate failure | SELECT 1")
}
if failure == "FK" {
files["002_b.sql"] = migration("ALTER TABLE child ADD COLUMN added INT; ALTER TABLE child ADD CONSTRAINT broken_fk FOREIGN KEY (added) REFERENCES parent(missing);", "-- verify: should never run | SELECT 1 WHERE 0")
}
applied, err := RunSQLMigrations(db, files, nil, root)
if failure == "none" {
if err != nil || len(applied) != 1 {
t.Fatalf("apply=%v error=%v", applied, err)
}
var count int64
if err := db.Model(&SQLSchemaMigration{}).Count(&count).Error; err != nil || count != 1 {
t.Fatalf("history=%d err=%v", count, err)
}
} else {
if err == nil || !strings.Contains(err.Error(), "all session changes restored") || len(applied) != 0 {
t.Fatalf("apply=%v error=%v", applied, err)
}
if err := mariadb.CheckSnapshot(db, original); err != nil {
t.Fatalf("recovery changed original state: %v", err)
}
}
markers, err := filepath.Glob(filepath.Join(root, "*.pending"))
if err != nil || len(markers) != 0 {
t.Fatalf("markers=%v error=%v", markers, err)
}
})
}
t.Run("interrupted session", func(t *testing.T) {
db, root := newDB(t), t.TempDir()
files := fstest.MapFS{"001_a.sql": migration("ALTER TABLE parent ADD COLUMN added INT", "-- verify: ok | SELECT 1 WHERE 0")}
func() {
defer func() {
if recover() == nil {
t.Error("expected interruption")
}
}()
RunSQLMigrations(db, files, func(string) { panic("interruption") }, root)
}()
if err := db.Exec("ALTER TABLE parent ADD COLUMN added INT").Error; err != nil {
t.Fatal(err)
}
if _, err := RunSQLMigrations(db, files, nil, root); err == nil || !strings.Contains(err.Error(), "interrupted migration") {
t.Fatalf("retry not refused: %v", err)
}
markers, err := filepath.Glob(filepath.Join(root, "*.pending"))
if err != nil || len(markers) != 1 {
t.Fatalf("markers=%v err=%v", markers, err)
}
data, err := os.ReadFile(markers[0])
if err != nil {
t.Fatal(err)
}
if err := RestoreSQLMigrationSession(db, strings.TrimSpace(string(data))); err != nil {
t.Fatal(err)
}
if db.Table("parent").Migrator().HasColumn(&struct{ Added int }{}, "added") {
t.Fatal("interrupted DDL not restored")
}
if _, err := RunSQLMigrations(db, files, nil, root); err != nil {
t.Fatalf("retry after restore: %v", err)
}
})
t.Run("unsupported object", func(t *testing.T) {
db := newDB(t)
if err := db.Exec("CREATE VIEW example_view AS SELECT id FROM parent").Error; err != nil {
t.Fatal(err)
}
files := fstest.MapFS{"001_a.sql": migration("ALTER TABLE parent ADD COLUMN added INT", "-- verify: ok | SELECT 1 WHERE 0")}
if _, err := RunSQLMigrations(db, files, nil, t.TempDir()); err == nil || !strings.Contains(err.Error(), "snapshot refused") {
t.Fatalf("unsupported schema accepted: %v", err)
}
if db.Table("parent").Migrator().HasColumn(&struct{ Added int }{}, "added") {
t.Fatal("unsupported schema was changed")
}
})
t.Run("other client", func(t *testing.T) {
db := newDB(t)
p, err := db.DB()
if err != nil {
t.Fatal(err)
}
p.SetMaxOpenConns(2)
other, err := p.Conn(context.Background())
if err != nil {
t.Fatal(err)
}
defer other.Close()
files := fstest.MapFS{"001_a.sql": migration("ALTER TABLE parent ADD COLUMN added INT", "-- verify: ok | SELECT 1 WHERE 0")}
if _, err := RunSQLMigrations(db, files, nil, t.TempDir()); err == nil || !strings.Contains(err.Error(), "other connections") {
t.Fatalf("active client accepted: %v", err)
}
if db.Table("parent").Migrator().HasColumn(&struct{ Added int }{}, "added") {
t.Fatal("active schema changed")
}
})
t.Run("backup failure", func(t *testing.T) {
db := newDB(t)
path := filepath.Join(t.TempDir(), "not-directory")
if err := os.WriteFile(path, []byte("example"), 0600); err != nil {
t.Fatal(err)
}
files := fstest.MapFS{"001_a.sql": migration("ALTER TABLE parent ADD COLUMN added INT", "-- verify: ok | SELECT 1 WHERE 0")}
if _, err := RunSQLMigrations(db, files, nil, path); err == nil {
t.Fatal("backup failure ignored")
}
if db.Table("parent").Migrator().HasColumn(&struct{ Added int }{}, "added") {
t.Fatal("schema changed without backup")
}
})
}
+37 -25
View File
@@ -34,31 +34,43 @@ func applySQLMigrationFile(db *gorm.DB, filename, content string) error {
if err != nil {
return fmt.Errorf("parse migration %s: %w", filename, err)
}
statements, err := splitSQLStatements(content)
if err != nil {
return fmt.Errorf("parse migration %s SQL: %w", filename, err)
}
return db.Connection(func(conn *gorm.DB) error {
if err := executeMigrationStatements(conn, filename, splitSQLStatements(content)); err != nil {
return err
}
for _, check := range checks {
rows, err := conn.Raw(check.query).Rows()
if err != nil {
return fmt.Errorf("verify migration %s (%s): %w", filename, check.description, err)
}
failed := rows.Next()
rowErr := rows.Err()
closeErr := rows.Close()
if rowErr != nil {
return fmt.Errorf("read migration verification %s (%s): %w", filename, check.description, rowErr)
}
if closeErr != nil {
return fmt.Errorf("close migration verification %s (%s): %w", filename, check.description, closeErr)
}
if failed {
return fmt.Errorf("verify migration %s failed: %s", filename, check.description)
}
}
if err := conn.Create(&SQLSchemaMigration{Filename: filename}).Error; err != nil {
return fmt.Errorf("record migration %s: %w", filename, err)
}
return nil
return applySQLMigrationOnSession(conn, filename, statements, checks)
})
}
func applySQLMigrationOnSession(conn *gorm.DB, filename string, statements []string, checks []migrationVerification) error {
if err := executeMigrationStatements(conn, filename, statements); err != nil {
return err
}
// Each query gets a fresh statement on the pinned connection: chaining
// Raw/Create on conn itself leaks the previous query's state (with two
// verify checks the history insert targeted a table named "rows").
fresh := func() *gorm.DB { return conn.Session(&gorm.Session{NewDB: true}) }
for _, check := range checks {
rows, err := fresh().Raw(check.query).Rows()
if err != nil {
return fmt.Errorf("verify migration %s (%s): %w", filename, check.description, err)
}
failed := rows.Next()
rowErr := rows.Err()
closeErr := rows.Close()
if rowErr != nil {
return fmt.Errorf("read migration verification %s (%s): %w", filename, check.description, rowErr)
}
if closeErr != nil {
return fmt.Errorf("close migration verification %s (%s): %w", filename, check.description, closeErr)
}
if failed {
return fmt.Errorf("verify migration %s failed: %s", filename, check.description)
}
}
if err := fresh().Create(&SQLSchemaMigration{Filename: filename}).Error; err != nil {
return fmt.Errorf("record migration %s: %w", filename, err)
}
return nil
}
@@ -15,6 +15,7 @@ func TestMigrationVerification(t *testing.T) {
{"failed", "-- verify: bad value | SELECT 1 FROM example WHERE value <> 1\nCREATE TEMP TABLE example (value INT); INSERT INTO example VALUES (2);", "bad value"},
{"query error", "-- verify: missing table | SELECT 1 FROM missing\nCREATE TEMP TABLE example (value INT);", "missing table"},
{"malformed", "-- verify: no query\nCREATE TABLE example (value INT);", "invalid migration verify header"},
{"two checks pass", "-- verify: first | SELECT 1 WHERE 0\n-- verify: second | SELECT 1 WHERE 0\nCREATE TEMP TABLE example (value INT);", ""},
{"second check fails", "-- verify: first | SELECT 1 WHERE 0\n-- verify: second | SELECT 1\nCREATE TEMP TABLE example (value INT);", "second"},
{"execution failure", "-- verify: none | SELECT 1 WHERE 0\nINVALID SQL;", "exec migration"},
} {
-72
View File
@@ -1,7 +1,6 @@
package models
import (
"bufio"
"errors"
"fmt"
"io/fs"
@@ -124,48 +123,6 @@ func NeedsSQLMigrations(db *gorm.DB, migrations fs.FS) (bool, error) {
return count < int64(len(files)), nil
}
// RunSQLMigrations applies pending SQL files from migrations once each, in filename
// order, and records them in qt_schema_migrations. onApply (optional) is called with
// each filename before it runs. Local SQLite-only scripts are skipped automatically.
// Returns the filenames applied.
func RunSQLMigrations(db *gorm.DB, migrations fs.FS, onApply func(filename string)) ([]string, error) {
if err := ensureSQLMigrationsTable(db); err != nil {
return nil, fmt.Errorf("migrate qt_schema_migrations table: %w", err)
}
files, err := listSQLMigrationFiles(migrations)
if err != nil {
return nil, err
}
var applied []string
for _, filename := range files {
var count int64
if err := db.Model(&SQLSchemaMigration{}).Where("filename = ?", filename).Count(&count).Error; err != nil {
return applied, fmt.Errorf("check migration %s: %w", filename, err)
}
if count > 0 {
continue
}
if onApply != nil {
onApply(filename)
}
content, err := fs.ReadFile(migrations, filename)
if err != nil {
return applied, fmt.Errorf("read migration %s: %w", filename, err)
}
if err := applySQLMigrationFile(db, filename, string(content)); err != nil {
return applied, err
}
applied = append(applied, filename)
}
return applied, nil
}
// IsMigrationPermissionError returns true if err indicates insufficient privileges
// to create/alter/read migration metadata or target schema objects.
func IsMigrationPermissionError(err error) bool {
@@ -239,35 +196,6 @@ func isIgnorableMigrationError(message string) bool {
return false
}
func splitSQLStatements(script string) []string {
scanner := bufio.NewScanner(strings.NewReader(script))
scanner.Buffer(make([]byte, 1024), 1024*1024)
lines := make([]string, 0, 128)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" {
continue
}
if strings.HasPrefix(line, "--") {
continue
}
lines = append(lines, scanner.Text())
}
combined := strings.Join(lines, "\n")
raw := strings.Split(combined, ";")
stmts := make([]string, 0, len(raw))
for _, stmt := range raw {
trimmed := strings.TrimSpace(stmt)
if trimmed == "" {
continue
}
stmts = append(stmts, trimmed)
}
return stmts
}
func listSQLMigrationFiles(migrations fs.FS) ([]string, error) {
entries, err := fs.ReadDir(migrations, ".")
if err != nil {
+4 -1
View File
@@ -46,7 +46,10 @@ func TestSQLMigrationStatusAndListing(t *testing.T) {
}
func TestSplitSQLStatementsSkipsComments(t *testing.T) {
got := splitSQLStatements("-- header; with semicolon\nALTER TABLE a ADD COLUMN x INT;\n\nALTER TABLE b\n ADD COLUMN y INT;\n")
got, err := splitSQLStatements("-- header; with semicolon\nALTER TABLE a ADD COLUMN x INT;\n\nALTER TABLE b\n ADD COLUMN y INT;\n")
if err != nil {
t.Fatal(err)
}
if len(got) != 2 || got[1] != "ALTER TABLE b\n ADD COLUMN y INT" {
t.Fatalf("unexpected statements %q", got)
}
+70
View File
@@ -0,0 +1,70 @@
package models
import (
"fmt"
"strings"
)
// splitSQLStatements supports MariaDB literals, quoted identifiers and comments.
func splitSQLStatements(script string) ([]string, error) {
var statements []string
var current strings.Builder
var quote byte
for i := 0; i < len(script); i++ {
ch := script[i]
if quote != 0 {
current.WriteByte(ch)
if ch == '\\' && quote != '`' && i+1 < len(script) {
i++
current.WriteByte(script[i])
} else if ch == quote {
if i+1 < len(script) && script[i+1] == quote {
i++
current.WriteByte(script[i])
} else {
quote = 0
}
}
continue
}
switch {
case ch == '\'' || ch == '"' || ch == '`':
quote = ch
current.WriteByte(ch)
case ch == '#' || (ch == '-' && i+2 < len(script) && script[i+1] == '-' && script[i+2] <= ' '):
for i < len(script) && script[i] != '\n' {
i++
}
current.WriteByte('\n')
case ch == '/' && i+1 < len(script) && script[i+1] == '*':
if strings.HasPrefix(script[i:], "/*!") || strings.HasPrefix(script[i:], "/*M!") {
return nil, fmt.Errorf("executable SQL comments are unsupported")
}
end := strings.Index(script[i+2:], "*/")
if end < 0 {
return nil, fmt.Errorf("unterminated SQL comment")
}
i += end + 3
current.WriteByte(' ')
case ch == ';':
if stmt := strings.TrimSpace(current.String()); stmt != "" {
statements = append(statements, stmt)
}
current.Reset()
default:
current.WriteByte(ch)
}
}
if quote != 0 {
return nil, fmt.Errorf("unterminated SQL quote")
}
if stmt := strings.TrimSpace(current.String()); stmt != "" {
statements = append(statements, stmt)
}
for _, stmt := range statements {
if strings.HasPrefix(strings.ToUpper(stmt), "DELIMITER ") {
return nil, fmt.Errorf("DELIMITER directives are unsupported")
}
}
return statements, nil
}
@@ -0,0 +1,70 @@
package models
import (
qfassets "git.mchus.pro/mchus/quoteforge"
"io/fs"
"reflect"
"strings"
"testing"
)
func TestSQLStatementParser(t *testing.T) {
for _, tt := range []struct {
name, sql string
want []string
fail bool
}{
{"empty", "-- comment;\n# more;", nil, false},
{"literal separators", "SELECT 'a;b', \"c;d\", `e;f`; SELECT 2;", []string{"SELECT 'a;b', \"c;d\", `e;f`", "SELECT 2"}, false},
{"doubled quotes", "SELECT 'a'';b', `a``;b`;", []string{"SELECT 'a'';b', `a``;b`"}, false},
{"backslash quote", "SELECT 'a\\';b'; SELECT 2", []string{"SELECT 'a\\';b'", "SELECT 2"}, false},
{"inline comments", "SELECT/* ; */ 1; -- end;\nSELECT 2# end;\n;", []string{"SELECT 1", "SELECT 2"}, false},
{"not a comment", "SELECT 4--2;", []string{"SELECT 4--2"}, false},
{"unclosed quote", "SELECT 'a;", nil, true},
{"unclosed block", "SELECT 1; /*", nil, true},
{"executable comment", "/*! SELECT 1 */", nil, true},
{"MariaDB comment", "/*M! SELECT 1 */", nil, true},
{"delimiter", "DELIMITER $$\nSELECT 1", nil, true},
} {
t.Run(tt.name, func(t *testing.T) {
got, err := splitSQLStatements(tt.sql)
if (err != nil) != tt.fail {
t.Fatalf("error=%v", err)
}
if !tt.fail && !reflect.DeepEqual(got, tt.want) {
t.Fatalf("got=%q want=%q", got, tt.want)
}
})
}
long := "SELECT '" + strings.Repeat("x", 2<<20) + ";y';"
if got, err := splitSQLStatements(long); err != nil || len(got) != 1 {
t.Fatalf("large statement: count=%d err=%v", len(got), err)
}
}
func TestEmbeddedServerMigrationSQL(t *testing.T) {
migrations, err := qfassets.MigrationsFS()
if err != nil {
t.Fatal(err)
}
files, err := listSQLMigrationFiles(migrations)
if err != nil {
t.Fatal(err)
}
for _, name := range files {
t.Run(name, func(t *testing.T) {
content, err := fs.ReadFile(migrations, name)
if err != nil {
t.Fatal(err)
}
statements, err := splitSQLStatements(string(content))
if err != nil || len(statements) == 0 {
t.Fatalf("count=%d err=%v", len(statements), err)
}
checks, err := parseMigrationVerifications(string(content))
if err != nil || len(checks) == 0 {
t.Fatalf("checks=%d err=%v", len(checks), err)
}
})
}
}
+25
View File
@@ -141,6 +141,31 @@ func (r *PartnumberBookRepository) HasAllBookItems(bookID uint) bool {
return count == int64(len(book.PartnumbersJSON))
}
// MissingBookItems returns the book's partnumbers that have no local catalog row.
func (r *PartnumberBookRepository) MissingBookItems(bookID uint) []string {
book, err := r.getBook(bookID)
if err != nil || len(book.PartnumbersJSON) == 0 {
return nil
}
var present []string
if err := r.db.Model(&localdb.LocalPartnumberBookItem{}).
Where("partnumber IN ?", []string(book.PartnumbersJSON)).
Pluck("partnumber", &present).Error; err != nil {
return nil
}
have := make(map[string]struct{}, len(present))
for _, pn := range present {
have[pn] = struct{}{}
}
var missing []string
for _, pn := range book.PartnumbersJSON {
if _, ok := have[pn]; !ok {
missing = append(missing, pn)
}
}
return missing
}
func (r *PartnumberBookRepository) getBook(bookID uint) (*localdb.LocalPartnumberBook, error) {
var book localdb.LocalPartnumberBook
if err := r.db.First(&book, bookID).Error; err != nil {
+29
View File
@@ -156,6 +156,35 @@ func (r *PricelistRepository) GetLatestActiveBySource(source string) (*models.Pr
return &pricelist, nil
}
// GetLatestActiveIDsBySources returns, per source, the ID GetLatestActiveBySource
// would return, in one round trip. Sources with no such pricelist are absent.
func (r *PricelistRepository) GetLatestActiveIDsBySources(sources []string) (map[string]uint, error) {
result := make(map[string]uint, len(sources))
if len(sources) == 0 {
return result, nil
}
parts := make([]string, 0, len(sources))
args := make([]any, 0, len(sources))
for _, source := range sources {
parts = append(parts, `(SELECT id, source FROM qt_pricelists
WHERE is_active = 1 AND source = ?
AND EXISTS (SELECT 1 FROM qt_pricelist_items WHERE qt_pricelist_items.pricelist_id = qt_pricelists.id)
ORDER BY created_at DESC, id DESC LIMIT 1)`)
args = append(args, source)
}
var rows []struct {
ID uint
Source string
}
if err := r.db.Raw(strings.Join(parts, " UNION ALL "), args...).Scan(&rows).Error; err != nil {
return nil, fmt.Errorf("getting latest pricelists: %w", err)
}
for _, row := range rows {
result[row.Source] = row.ID
}
return result, nil
}
// Create creates a new pricelist
func (r *PricelistRepository) Create(pricelist *models.Pricelist) error {
if err := r.db.Create(pricelist).Error; err != nil {
@@ -0,0 +1,203 @@
package sync_test
import (
"os"
"strings"
"testing"
"git.mchus.pro/mchus/quoteforge/internal/models"
syncsvc "git.mchus.pro/mchus/quoteforge/internal/services/sync"
"gorm.io/driver/mysql"
"gorm.io/gorm"
"gorm.io/gorm/logger"
)
// Runs only against a disposable MariaDB that already has the qt_projects and
// qt_configurations tables (e.g. a container loaded with SHOW CREATE TABLE from
// production). The test truncates them. Example:
//
// QF_TEST_MARIADB_DSN='root:test@tcp(127.0.0.1:33306)/qf?parseTime=true' go test ./internal/services/sync -run MariaDB
func newMariaDBForPullTest(t *testing.T) *gorm.DB {
t.Helper()
dsn := os.Getenv("QF_TEST_MARIADB_DSN")
if dsn == "" {
t.Skip("QF_TEST_MARIADB_DSN not set")
}
db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
if err != nil {
t.Fatalf("open mariadb: %v", err)
}
mustExec(t, db, "SET FOREIGN_KEY_CHECKS=0")
mustExec(t, db, "TRUNCATE TABLE qt_configurations")
mustExec(t, db, "TRUNCATE TABLE qt_projects")
mustExec(t, db, "SET FOREIGN_KEY_CHECKS=1")
for _, table := range []string{"qt_configurations", "qt_projects"} {
mustExec(t, db, "ALTER TABLE "+table+" DROP INDEX IF EXISTS idx_"+table+"_server_updated_at")
mustExec(t, db, "ALTER TABLE "+table+" DROP COLUMN IF EXISTS server_updated_at")
}
return db
}
func mustExec(t *testing.T, db *gorm.DB, sql string, args ...any) {
t.Helper()
if err := db.Exec(sql, args...).Error; err != nil {
t.Fatalf("%s: %v", sql, err)
}
}
func applyServerUpdatedAtMigration(t *testing.T, db *gorm.DB) {
t.Helper()
raw, err := os.ReadFile("../../../migrations/035_add_server_updated_at.sql")
if err != nil {
t.Fatalf("read migration: %v", err)
}
var body []string
for _, line := range strings.Split(string(raw), "\n") {
if !strings.HasPrefix(strings.TrimSpace(line), "--") {
body = append(body, line)
}
}
for _, stmt := range strings.Split(strings.Join(body, "\n"), ";") {
if strings.TrimSpace(stmt) != "" {
mustExec(t, db, stmt)
}
}
}
// seedServerConfig writes the way any client version does: through the
// models.Configuration struct, which knows nothing about server_updated_at.
func seedServerConfig(t *testing.T, db *gorm.DB, uuid, name string) *models.Configuration {
t.Helper()
cfg := &models.Configuration{
UUID: uuid,
OwnerUsername: "tester",
Name: name,
Items: models.ConfigItems{{LotName: "CPU_PULL", Quantity: 1, UnitPrice: 100}},
ServerCount: 1,
Line: 10,
}
if err := db.Create(cfg).Error; err != nil {
t.Fatalf("seed config %s: %v", uuid, err)
}
return cfg
}
func TestIncrementalPullMariaDB(t *testing.T) {
serverDB := newMariaDBForPullTest(t)
local := newLocalDBForSyncTest(t)
if err := local.SaveSettings("127.0.0.1", 33306, "qf", "root", "test"); err != nil {
t.Fatalf("save settings: %v", err)
}
// Before migration 035: every pull is full, exactly as before.
seedServerConfig(t, serverDB, "cfg-a", "A")
seedServerConfig(t, serverDB, "cfg-b", "B")
svc := syncsvc.NewServiceWithDB(serverDB, local)
for i := 0; i < 2; i++ {
res, err := svc.ImportConfigurationsToLocal()
if err != nil {
t.Fatalf("pre-migration import: %v", err)
}
if res.Imported+res.Updated != 2 {
t.Fatalf("pre-migration pull %d: expected full pull of 2, got %+v", i, res)
}
}
applyServerUpdatedAtMigration(t, serverDB)
applyServerUpdatedAtMigration(t, serverDB) // re-runnable
// A client without the column in its model still inserts and updates fine.
seedServerConfig(t, serverDB, "cfg-c", "C")
// Age all rows past the overlap window.
mustExec(t, serverDB, "UPDATE qt_configurations SET server_updated_at = NOW(6) - INTERVAL 1 HOUR")
// First pull after the migration in this process: full, sets the cursor.
res, err := svc.ImportConfigurationsToLocal()
if err != nil {
t.Fatalf("first post-migration import: %v", err)
}
if res.Imported+res.Updated != 3 {
t.Fatalf("expected full pull of 3, got %+v", res)
}
// Nothing changed on the server: nothing is pulled.
res, err = svc.ImportConfigurationsToLocal()
if err != nil {
t.Fatalf("idle incremental import: %v", err)
}
if res.Imported+res.Updated != 0 {
t.Fatalf("expected empty incremental pull, got %+v", res)
}
// An old-style client update (gorm Save of the whole struct) bumps the marker.
var b models.Configuration
if err := serverDB.Where("uuid = ?", "cfg-b").First(&b).Error; err != nil {
t.Fatalf("load cfg-b: %v", err)
}
b.Name = "B renamed"
if err := serverDB.Save(&b).Error; err != nil {
t.Fatalf("save cfg-b: %v", err)
}
seedServerConfig(t, serverDB, "cfg-d", "D")
res, err = svc.ImportConfigurationsToLocal()
if err != nil {
t.Fatalf("incremental import: %v", err)
}
if res.Updated != 1 || res.Imported != 1 {
t.Fatalf("expected 1 updated + 1 imported, got %+v", res)
}
localB, err := local.GetConfigurationByUUID("cfg-b")
if err != nil || localB.Name != "B renamed" {
t.Fatalf("expected cfg-b renamed locally, got %+v err=%v", localB, err)
}
// A new process does one full pull again.
restarted := syncsvc.NewServiceWithDB(serverDB, local)
res, err = restarted.ImportConfigurationsToLocal()
if err != nil {
t.Fatalf("restart import: %v", err)
}
if res.Imported+res.Updated != 4 {
t.Fatalf("expected full pull of 4 after restart, got %+v", res)
}
// Switching to another server invalidates the cursor.
if err := local.SaveSettings("other-host", 3306, "qf", "root", "test"); err != nil {
t.Fatalf("save settings: %v", err)
}
res, err = restarted.ImportConfigurationsToLocal()
if err != nil {
t.Fatalf("import after server switch: %v", err)
}
if res.Imported+res.Updated != 4 {
t.Fatalf("expected full pull of 4 after server switch, got %+v", res)
}
}
func TestIncrementalProjectPullMariaDB(t *testing.T) {
serverDB := newMariaDBForPullTest(t)
local := newLocalDBForSyncTest(t)
if err := local.SaveSettings("127.0.0.1", 33306, "qf", "root", "test"); err != nil {
t.Fatalf("save settings: %v", err)
}
applyServerUpdatedAtMigration(t, serverDB)
code := "P-1"
project := &models.Project{UUID: "prj-1", OwnerUsername: "tester", Code: code, IsActive: true}
if err := serverDB.Create(project).Error; err != nil {
t.Fatalf("seed project: %v", err)
}
mustExec(t, serverDB, "UPDATE qt_projects SET server_updated_at = NOW(6) - INTERVAL 1 HOUR")
svc := syncsvc.NewServiceWithDB(serverDB, local)
if res, err := svc.ImportProjectsToLocal(); err != nil || res.Imported != 1 {
t.Fatalf("first project pull: %+v err=%v", res, err)
}
if res, err := svc.ImportProjectsToLocal(); err != nil || res.Imported+res.Updated != 0 {
t.Fatalf("idle project pull: %+v err=%v", res, err)
}
mustExec(t, serverDB, "UPDATE qt_projects SET name = 'renamed' WHERE uuid = 'prj-1'")
if res, err := svc.ImportProjectsToLocal(); err != nil || res.Updated != 1 {
t.Fatalf("incremental project pull: %+v err=%v", res, err)
}
}
+65 -21
View File
@@ -23,15 +23,16 @@ func (s *Service) PullPartnumberBooks() (int, error) {
localBookRepo := repository.NewPartnumberBookRepository(s.localDB.DB())
// Headers only: partnumbers_json is several MB in total and the worker runs
// every minute, so it is fetched per book only when that book needs items.
type serverBook struct {
ID int `gorm:"column:id"`
Version string `gorm:"column:version"`
CreatedAt time.Time `gorm:"column:created_at"`
IsActive bool `gorm:"column:is_active"`
PartnumbersJSON string `gorm:"column:partnumbers_json"`
ID int `gorm:"column:id"`
Version string `gorm:"column:version"`
CreatedAt time.Time `gorm:"column:created_at"`
IsActive bool `gorm:"column:is_active"`
}
var serverBooks []serverBook
if err := mariaDB.Raw("SELECT id, version, created_at, is_active, partnumbers_json FROM qt_partnumber_books ORDER BY created_at DESC, id DESC").Scan(&serverBooks).Error; err != nil {
if err := mariaDB.Raw("SELECT id, version, created_at, is_active FROM qt_partnumber_books ORDER BY created_at DESC, id DESC").Scan(&serverBooks).Error; err != nil {
return 0, fmt.Errorf("querying server partnumber books: %w", err)
}
slog.Info("partnumber books found on server", "count", len(serverBooks))
@@ -40,39 +41,58 @@ func (s *Service) PullPartnumberBooks() (int, error) {
for _, sb := range serverBooks {
var existing localdb.LocalPartnumberBook
err := s.localDB.DB().Where("server_id = ?", sb.ID).First(&existing).Error
partnumbers, errPartnumbers := decodeServerPartnumbers(sb.PartnumbersJSON)
if errPartnumbers != nil {
slog.Error("failed to decode server partnumbers_json", "server_id", sb.ID, "error", errPartnumbers)
continue
}
if err == nil {
existing.Version = sb.Version
existing.CreatedAt = sb.CreatedAt
existing.IsActive = sb.IsActive
if existing.Version != sb.Version || existing.IsActive != sb.IsActive {
existing.Version = sb.Version
existing.CreatedAt = sb.CreatedAt
existing.IsActive = sb.IsActive
if err := localBookRepo.SaveBook(&existing); err != nil {
slog.Error("failed to update local partnumber book header", "server_id", sb.ID, "error", err)
continue
}
}
if localBookRepo.HasAllBookItems(existing.ID) {
slog.Debug("partnumber book already synced, skipping", "server_id", sb.ID, "version", sb.Version)
continue
}
// A server book may list partnumbers that qt_partnumber_book_items
// lacks; re-pulling cannot fix that, so try once per process.
if !s.markPartnumberBookRepair(sb.ID, sb.Version) {
slog.Debug("partnumber book still incomplete after re-pull this session, skipping", "server_id", sb.ID, "version", sb.Version)
continue
}
slog.Info("partnumber book header exists but catalog items are missing, re-pulling items", "server_id", sb.ID, "version", sb.Version)
partnumbers, err := fetchServerBookPartnumbers(mariaDB, sb.ID)
if err != nil {
slog.Error("failed to fetch server partnumbers_json", "server_id", sb.ID, "error", err)
continue
}
existing.PartnumbersJSON = partnumbers
if err := localBookRepo.SaveBook(&existing); err != nil {
slog.Error("failed to update local partnumber book header", "server_id", sb.ID, "error", err)
continue
}
localItemCount := localBookRepo.CountBookItems(existing.ID)
if localItemCount > 0 && localBookRepo.HasAllBookItems(existing.ID) {
slog.Debug("partnumber book already synced, skipping", "server_id", sb.ID, "version", sb.Version, "items", localItemCount)
continue
}
slog.Info("partnumber book header exists but catalog items are missing, re-pulling items", "server_id", sb.ID, "version", sb.Version)
n, err := pullBookItems(mariaDB, localBookRepo, existing.PartnumbersJSON)
if err != nil {
slog.Error("failed to re-pull items for existing book", "server_id", sb.ID, "error", err)
} else {
slog.Info("re-pulled items for existing book", "server_id", sb.ID, "version", sb.Version, "items_saved", n)
pulled++
if missing := localBookRepo.MissingBookItems(existing.ID); len(missing) > 0 {
slog.Warn("partnumber book lists partnumbers absent from server qt_partnumber_book_items", "server_id", sb.ID, "version", sb.Version, "missing", missing)
}
}
continue
}
slog.Info("pulling new partnumber book", "server_id", sb.ID, "version", sb.Version, "is_active", sb.IsActive)
partnumbers, err := fetchServerBookPartnumbers(mariaDB, sb.ID)
if err != nil {
slog.Error("failed to fetch server partnumbers_json", "server_id", sb.ID, "error", err)
continue
}
localBook := &localdb.LocalPartnumberBook{
ServerID: sb.ID,
Version: sb.Version,
@@ -99,6 +119,30 @@ func (s *Service) PullPartnumberBooks() (int, error) {
return pulled, nil
}
// markPartnumberBookRepair records a re-pull attempt for a book version and
// reports whether this is the first attempt in the current process.
func (s *Service) markPartnumberBookRepair(serverID int, version string) bool {
s.pnRepairMu.Lock()
defer s.pnRepairMu.Unlock()
if s.pnRepairAttempts == nil {
s.pnRepairAttempts = map[int]string{}
}
if v, ok := s.pnRepairAttempts[serverID]; ok && v == version {
return false
}
s.pnRepairAttempts[serverID] = version
return true
}
// fetchServerBookPartnumbers loads and decodes partnumbers_json of one server book.
func fetchServerBookPartnumbers(mariaDB *gorm.DB, serverID int) (localdb.LocalStringList, error) {
var raw string
if err := mariaDB.Raw("SELECT partnumbers_json FROM qt_partnumber_books WHERE id = ?", serverID).Scan(&raw).Error; err != nil {
return nil, err
}
return decodeServerPartnumbers(raw)
}
// pullBookItems fetches catalog items for a partnumber list from MariaDB and saves them to SQLite.
// Returns the number of items saved.
func pullBookItems(mariaDB *gorm.DB, repo *repository.PartnumberBookRepository, partnumbers localdb.LocalStringList) (int, error) {
+105
View File
@@ -0,0 +1,105 @@
package sync
import (
"log/slog"
"gorm.io/gorm"
)
// serverPullOverlap re-reads rows changed shortly before the cursor, covering
// transactions that stamped server_updated_at before the cursor but committed
// after the previous pull read it.
const serverPullOverlap = "INTERVAL 120 SECOND"
// serverPull describes how one pull of a server table should run. See
// bible-local/decisions/2026-10-26-incremental-config-project-pull.md.
type serverPull struct {
table string
scope string
since string // cursor; empty = full pull
startedAt string // server NOW(6) before reading; becomes the next cursor
trackable bool // server has server_updated_at (migration 035)
}
// beginServerPull decides between a full and an incremental pull of table.
// The first pull of each table per process is full, so rows lost locally are
// restored at least once per app start; later pulls read only rows whose
// server_updated_at moved since the stored cursor.
func (s *Service) beginServerPull(mariaDB *gorm.DB, table string) serverPull {
pull := serverPull{table: table}
if !s.serverHasChangeMarker(mariaDB, table) {
return pull
}
if err := mariaDB.Raw("SELECT DATE_FORMAT(NOW(6), '%Y-%m-%d %H:%i:%s.%f')").Scan(&pull.startedAt).Error; err != nil || pull.startedAt == "" {
slog.Warn("server pull: failed to read server time, falling back to full pull", "table", table, "error", err)
return serverPull{table: table}
}
scope, err := s.localDB.ServerPullScope()
if err != nil {
return serverPull{table: table}
}
pull.scope = scope
pull.trackable = true
s.pullMu.Lock()
fullDone := s.fullPullDone[table]
s.pullMu.Unlock()
if fullDone {
if since, ok := s.localDB.GetServerPullCursor(table, scope); ok {
pull.since = since
}
}
return pull
}
// incrementalScope narrows query to rows changed since the cursor.
func (p serverPull) incrementalScope(query *gorm.DB) *gorm.DB {
return query.Where("server_updated_at >= CAST(? AS DATETIME(6)) - "+serverPullOverlap, p.since)
}
// finishServerPull records a successful pull: the cursor moves to the server
// time taken before reading, and a full pull marks the table as verified for
// this process.
func (s *Service) finishServerPull(pull serverPull) {
if !pull.trackable {
return
}
if err := s.localDB.SetServerPullCursor(pull.table, pull.scope, pull.startedAt); err != nil {
slog.Warn("server pull: failed to store cursor", "table", pull.table, "error", err)
return
}
if pull.since == "" {
s.pullMu.Lock()
if s.fullPullDone == nil {
s.fullPullDone = map[string]bool{}
}
s.fullPullDone[pull.table] = true
s.pullMu.Unlock()
}
}
// serverHasChangeMarker reports whether table has server_updated_at. A positive
// answer is cached for the process; a negative one is re-checked next time so a
// migration applied while the client runs is picked up.
func (s *Service) serverHasChangeMarker(mariaDB *gorm.DB, table string) bool {
s.pullMu.Lock()
known := s.changeMarker[table]
s.pullMu.Unlock()
if known {
return true
}
var n int64
if err := mariaDB.Raw(`
SELECT COUNT(*) FROM information_schema.COLUMNS
WHERE table_schema = DATABASE() AND table_name = ? AND column_name = 'server_updated_at'
`, table).Scan(&n).Error; err != nil || n == 0 {
return false
}
s.pullMu.Lock()
if s.changeMarker == nil {
s.changeMarker = map[string]bool{}
}
s.changeMarker[table] = true
s.pullMu.Unlock()
return true
}
+70 -25
View File
@@ -30,6 +30,13 @@ type Service struct {
directDB *gorm.DB
pricelistMu sync.Mutex // prevents concurrent pricelist syncs
schemaOnce sync.Once // ensures ensureClientSchemaStateTable runs at most once per process
pnRepairMu sync.Mutex
pnRepairAttempts map[int]string // server book id -> version already re-pulled this process
pullMu sync.Mutex
changeMarker map[string]bool // server table -> has server_updated_at
fullPullDone map[string]bool // server table -> full pull succeeded this process
}
// NewService creates a new sync service
@@ -118,13 +125,27 @@ func (s *Service) ImportConfigurationsToLocal() (*ConfigImportResult, error) {
configRepo := repository.NewConfigurationRepository(mariaDB)
result := &ConfigImportResult{}
pull := s.beginServerPull(mariaDB, "qt_configurations")
offset := 0
const limit = 200
for {
serverConfigs, _, err := configRepo.ListAll(offset, limit)
if err != nil {
return nil, fmt.Errorf("listing server configurations: %w", err)
var serverConfigs []models.Configuration
if pull.since != "" {
// Incremental: the changed set is small, read it in one go.
if offset > 0 {
break
}
if err := pull.incrementalScope(mariaDB.Model(&models.Configuration{})).
Order("created_at DESC, id DESC").
Find(&serverConfigs).Error; err != nil {
return nil, fmt.Errorf("listing changed server configurations: %w", err)
}
} else {
serverConfigs, _, err = configRepo.ListAll(offset, limit)
if err != nil {
return nil, fmt.Errorf("listing server configurations: %w", err)
}
}
if len(serverConfigs) == 0 {
break
@@ -171,6 +192,7 @@ func (s *Service) ImportConfigurationsToLocal() (*ConfigImportResult, error) {
offset += len(serverConfigs)
}
s.finishServerPull(pull)
return result, nil
}
@@ -184,13 +206,27 @@ func (s *Service) ImportProjectsToLocal() (*ProjectImportResult, error) {
projectRepo := repository.NewProjectRepository(mariaDB)
result := &ProjectImportResult{}
pull := s.beginServerPull(mariaDB, "qt_projects")
offset := 0
const limit = 200
for {
serverProjects, _, err := projectRepo.List(offset, limit, true)
if err != nil {
return nil, fmt.Errorf("listing server projects: %w", err)
var serverProjects []models.Project
if pull.since != "" {
// Incremental: the changed set is small, read it in one go.
if offset > 0 {
break
}
if err := pull.incrementalScope(mariaDB.Model(&models.Project{})).
Order("created_at DESC").
Find(&serverProjects).Error; err != nil {
return nil, fmt.Errorf("listing changed server projects: %w", err)
}
} else {
serverProjects, _, err = projectRepo.List(offset, limit, true)
if err != nil {
return nil, fmt.Errorf("listing server projects: %w", err)
}
}
if len(serverProjects) == 0 {
break
@@ -247,6 +283,7 @@ func (s *Service) ImportProjectsToLocal() (*ProjectImportResult, error) {
offset += len(serverProjects)
}
s.finishServerPull(pull)
return result, nil
}
@@ -291,28 +328,30 @@ func (s *Service) NeedSync() (bool, error) {
mariaDB, err := s.getDB()
if err == nil {
pricelistRepo := repository.NewPricelistRepository(mariaDB)
sources := []models.PricelistSource{
models.PricelistSourceEstimate,
models.PricelistSourceWarehouse,
models.PricelistSourceCompetitor,
sources := []string{
string(models.PricelistSourceEstimate),
string(models.PricelistSourceWarehouse),
string(models.PricelistSourceCompetitor),
}
allMatch := true
for _, source := range sources {
latestServer, err := pricelistRepo.GetLatestActiveBySource(string(source))
if err != nil {
continue
}
latestLocal, err := s.localDB.GetLatestLocalPricelistBySource(string(source))
if err != nil {
return true, nil
}
if latestServer.ID != latestLocal.ServerID {
return true, nil
}
}
if allMatch {
latestServer, err := pricelistRepo.GetLatestActiveIDsBySources(sources)
if err != nil {
slog.Warn("failed to check latest server pricelists", "error", err)
return false, nil
}
for _, source := range sources {
serverID, ok := latestServer[source]
if !ok {
continue
}
latestLocal, err := s.localDB.GetLatestLocalPricelistBySource(source)
if err != nil {
return true, nil
}
if serverID != latestLocal.ServerID {
return true, nil
}
}
return false, nil
}
}
@@ -941,6 +980,12 @@ func (s *Service) PushPendingChanges() (int, error) {
if _, err := s.EnsureReadinessForSync(); err != nil {
return 0, err
}
return s.pushPendingChanges()
}
// pushPendingChanges pushes without its own readiness check; the caller has
// just run EnsureReadinessForSync (and with it the client heartbeat).
func (s *Service) pushPendingChanges() (int, error) {
removed, err := s.localDB.PurgeOrphanConfigurationPendingChanges()
if err != nil {
+28 -7
View File
@@ -3,12 +3,29 @@ package sync
import (
"context"
"log/slog"
"math/rand/v2"
"time"
"git.mchus.pro/mchus/quoteforge/internal/db"
)
// Worker performs background synchronization at regular intervals
// WorkerJitter spreads background sync cycles by ±30% of the interval so that
// clients started at the same time do not hit MariaDB in lockstep.
const WorkerJitter = 0.3
// jitteredInterval returns base scaled by a random factor in [1-WorkerJitter, 1+WorkerJitter].
func jitteredInterval(base time.Duration) time.Duration {
factor := 1 - WorkerJitter + rand.Float64()*2*WorkerJitter
return time.Duration(float64(base) * factor)
}
// nextCycleDelay returns how long to wait so the next cycle starts a jittered
// interval after started; zero if the cycle itself took longer than that.
func nextCycleDelay(started time.Time, base time.Duration) time.Duration {
return max(jitteredInterval(base)-time.Since(started), 0)
}
// Worker performs background synchronization at jittered intervals
type Worker struct {
service *Service
connMgr *db.ConnectionManager
@@ -37,12 +54,14 @@ func (w *Worker) isOnline() bool {
func (w *Worker) Start(ctx context.Context) {
w.logger.Info("starting background sync worker", "interval", w.interval)
ticker := time.NewTicker(w.interval)
defer ticker.Stop()
// Run once immediately
// Run once immediately. The next cycle is scheduled from the start of the
// previous one, so heartbeats are at most (1+WorkerJitter)*interval apart.
started := time.Now()
w.runSync()
timer := time.NewTimer(nextCycleDelay(started, w.interval))
defer timer.Stop()
for {
select {
case <-ctx.Done():
@@ -51,8 +70,10 @@ func (w *Worker) Start(ctx context.Context) {
case <-w.stopCh:
w.logger.Info("background sync worker stopped")
return
case <-ticker.C:
case <-timer.C:
started = time.Now()
w.runSync()
timer.Reset(nextCycleDelay(started, w.interval))
}
}
}
@@ -79,7 +100,7 @@ func (w *Worker) runSync() {
"reason_code", readiness.ReasonCode,
"reason_text", readiness.ReasonText,
)
} else if pushed, err := w.service.PushPendingChanges(); err != nil {
} else if pushed, err := w.service.pushPendingChanges(); err != nil {
w.logger.Warn("background sync: failed to push pending changes", "error", err)
} else if pushed > 0 {
w.logger.Info("background sync: pushed pending changes", "count", pushed)
@@ -0,0 +1,23 @@
package sync
import (
"testing"
"time"
)
func TestJitteredIntervalStaysWithinBounds(t *testing.T) {
base := time.Minute
lo := time.Duration(float64(base) * (1 - WorkerJitter))
hi := time.Duration(float64(base) * (1 + WorkerJitter))
seen := map[time.Duration]struct{}{}
for i := 0; i < 1000; i++ {
d := jitteredInterval(base)
if d < lo || d > hi {
t.Fatalf("interval %v outside [%v, %v]", d, lo, hi)
}
seen[d] = struct{}{}
}
if len(seen) < 100 {
t.Fatalf("expected spread of intervals, got %d distinct values", len(seen))
}
}
+20
View File
@@ -0,0 +1,20 @@
-- Tables affected: qt_configurations, qt_projects
-- recovery.not-started: safe to re-run; ADD COLUMN IF NOT EXISTS / ADD INDEX IF NOT EXISTS
-- recovery.partial: ALTER TABLE qt_configurations DROP COLUMN server_updated_at; ALTER TABLE qt_projects DROP COLUMN server_updated_at;
-- recovery.completed: no action needed
-- verify: qt_configurations.server_updated_at column missing | SELECT 1 FROM information_schema.COLUMNS WHERE table_schema=DATABASE() AND table_name='qt_configurations' AND column_name='server_updated_at' HAVING COUNT(*)=0
-- verify: qt_projects.server_updated_at column missing | SELECT 1 FROM information_schema.COLUMNS WHERE table_schema=DATABASE() AND table_name='qt_projects' AND column_name='server_updated_at' HAVING COUNT(*)=0
-- Server-maintained change marker for incremental client pulls. Never written by
-- clients: DEFAULT on INSERT and ON UPDATE on any row change, so clients that do not
-- know the column keep working unchanged. See
-- bible-local/decisions/2026-10-26-incremental-config-project-pull.md.
ALTER TABLE qt_configurations
ADD COLUMN IF NOT EXISTS server_updated_at TIMESTAMP(6) NOT NULL
DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
ADD INDEX IF NOT EXISTS idx_qt_configurations_server_updated_at (server_updated_at, id);
ALTER TABLE qt_projects
ADD COLUMN IF NOT EXISTS server_updated_at TIMESTAMP(6) NOT NULL
DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
ADD INDEX IF NOT EXISTS idx_qt_projects_server_updated_at (server_updated_at, id);
+26
View File
@@ -0,0 +1,26 @@
# QuoteForge v3.50
Дата релиза: 2026-10-26
Тег: `v3.50`
Предыдущий релиз: `v3.49`
## Ключевые изменения
- локальный API и первоначальная настройка защищены от CSRF и DNS rebinding; сохранённый пароль не отправляется при изменении хоста, порта или пользователя без явного ввода пароля (#2);
- устранены XSS через LOT, BOM и сообщения синхронизации; обработчики корзины и аренды получают LOT через data-атрибуты (#4);
- перед сбросом локальной базы создаётся отдельный проверенный архив; несинхронизированные данные защищены отказом без явного принудительного сброса, переменная QFS_RESET_LOCAL_DB больше не запускает сброс (#7);
- в настройках подключения MariaDB доступны TLS preferred, required, verify-ca и verify-full, а также CA-сертификат; preferred допускает соединение без TLS, verify-full проверяет сертификат и имя сервера (#5);
- SQL-миграции выполняются по одной сессии на файл, запускают verify-проверки до записи в историю и не скрывают ошибки внешних ключей (#3);
- минимальная версия Go повышена до 1.27.3, обновлены зависимости; релизная сборка блокируется при достижимых уязвимостях по govulncheck (#6).
## Проверки
Пройдены Go-тесты, go vet, проверка XSS и сборки macOS arm64 / Windows amd64.
Govulncheck: 0 достижимых уязвимостей; остаётся одна запись на уровне модуля,
уязвимый код которого приложение не вызывает.
## Запуск на macOS
Снимите карантинный атрибут через терминал: `xattr -d com.apple.quarantine /path/to/qfs-darwin-arm64`
После этого бинарник запустится без предупреждения Gatekeeper.
+26
View File
@@ -0,0 +1,26 @@
# QuoteForge v3.51
Дата релиза: 2026-10-26
Тег: `v3.51`
Предыдущий релиз: `v3.50`
## Ключевые изменения
- устранена утечка подключений к MariaDB: при синхронизации клиент каждые 30 секунд открывал новый пул, не закрывая старый, и держал до ~30 соединений; теперь не больше 2, простаивающие закрываются через минуту;
- фоновая синхронизация раз в минуту со случайным разбросом ±30%, чтобы клиенты не обращались к серверу одновременно; статус «онлайн» в списке пользователей считается по интервалу +30%;
- списки проектов и конфигураций подгружают с сервера только изменённые записи (миграция `035`, колонка `server_updated_at`): ~0,35 с вместо ~27 с и 3 МБ; полная загрузка — один раз при запуске приложения; старые клиенты и сервер без миграции работают как раньше;
- PN-книги больше не скачиваются целиком на каждом цикле: забираются только заголовки, содержимое — для новых или неполных книг; неполная книга докачивается один раз за запуск с предупреждением в логе;
- проверка новых прайслистов одним запросом вместо трёх, одна отметка о клиенте на цикл синхронизации, настройки конфигуратора не перезаписываются без изменений;
- исправлен мигратор: файл с двумя и более `verify`-проверками применялся, но не записывался в историю миграций.
## Проверки
Пройдены Go-тесты и go vet. Инкрементальная загрузка проверена интеграционным тестом
на MariaDB 11.8 со схемой продакшена и на продакшен-сервере после миграции `035`.
Govulncheck: 0 достижимых уязвимостей.
## Запуск на macOS
Снимите карантинный атрибут через терминал: `xattr -d com.apple.quarantine /path/to/qfs-darwin-arm64`
После этого бинарник запустится без предупреждения Gatekeeper.
+35
View File
@@ -0,0 +1,35 @@
# QuoteForge v3.52
Дата релиза: 2026-10-26
Тег: `v3.52`
Предыдущий релиз: `v3.51`
## Ключевые изменения
- серверный мигратор автоматически создаёт проверяемые архивы перед изменениями и перед каждым SQL-файлом; при ошибке восстанавливает всю сессию, включая данные, триггеры и историю миграций;
- добавлен `-migrate-restore <archive.zip>` для восстановления после прерывания процесса; незавершённая сессия блокирует повторный запуск до восстановления;
- SQL-парсер учитывает кавычки, комментарии и точки с запятой внутри литералов; некорректный ввод отклоняется до выполнения;
- миграционные команды используют выделенное соединение без фоновых подключений runtime; требуют окна обслуживания и существующей DBA-учётной записи, права обычного пользователя приложения не расширяются;
- завершена приёмка issues #2–#7: настоящие setup/runtime-роутеры, XSS в браузерном DOM, CLI сброса и восстановления, TLS на MariaDB и остановка релизной сборки при ошибке сканера уязвимостей.
## База данных
Новых SQL-миграций нет. Для обновления с v3.51 изменять схему БД не требуется.
В этом релизе изменён движок выполнения существующих миграций.
## Проверки
Пройдены `go test ./...`, `go vet ./...`, JavaScript-регрессии, headless Chrome,
интеграционные проверки TLS и миграций, сборки macOS arm64 и Windows amd64.
Govulncheck: 0 достижимых уязвимостей; одна запись на уровне невызываемой зависимости.
На предоставленном дампе от 2026-10-26 проверены 45 таблиц и 27 триггеров
в отдельном контейнере MariaDB 11.8.3 с utf8mb4_uca1400_ai_ci: полный restore,
миграция 035 на копии pre-035 схемы, намеренный последующий сбой, откат всей
сессии, успешный повтор и восстановление исходного состояния. Рабочая БД
и исходный дамп не изменялись.
## Запуск на macOS
Снимите карантинный атрибут через терминал: `xattr -d com.apple.quarantine /path/to/qfs-darwin-arm64`
+98
View File
@@ -0,0 +1,98 @@
const assert = require('node:assert/strict');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const {spawnSync} = require('node:child_process');
const index = fs.readFileSync('web/templates/index.html', 'utf8');
const base = fs.readFileSync('web/templates/base.html', 'utf8');
function extract(source, name) {
const match = source.match(new RegExp(`^( *)(?:async )?function ${name}\\(`, 'm'));
assert(match, `Missing function ${name}`);
const end = source.indexOf(`\n${match[1]}}`, match.index);
assert(end > match.index);
return source.slice(match.index, end + match[1].length + 2);
}
const functions = [
...['renderMultiSelectTab', 'renderMultiSelectTabWithSections', 'renderRentalTab', '_renderBOMParsedTable'].map(n => extract(index, n)),
...['showToast', 'loadSyncInfo', 'repairPendingChanges'].map(n => extract(base, n)),
].join('\n');
const ids = new Set([...functions.matchAll(/getElementById\('([^']+)'\)/g)].map(m => m[1]));
const fixtures = [...ids].map(id => id.endsWith('-body') ? `<table><tbody id="${id}"></tbody></table>` : `<div id="${id}"></div>`).join('');
const listeners = index.split('<script>')[1].split('// Tab configuration')[0];
const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'qfs-browser-security-'));
const target = path.join(directory, 'test.html');
const test = `
let cart = [], allComponents = [], currentTab = 'cpu', configUUID = 'example';
let rentalUpliftPercent = 0, bomRows = [], reply = {}, calls = [];
window.__injected = 0;
const getTabForCategory = () => 'cpu', getDisplayPrice = () => 1;
const qualityDotHtml = () => '', formatMoney = String, formatPriceOrNA = String, formatCurrency = String;
const ciStr = s => s.toLowerCase(), getComponentCategory = c => c.category;
const _setBomUIError = () => {}, _ensureBomDatalist = () => {}, _getRowLotQtyPerPN = () => 1;
const renderBOMLotAllocationsEditor = () => '', buildRentalItemsPayload = () => [];
const removeFromCart = lot => calls.push(['remove', lot]);
const updateMultiQuantity = (lot, value) => calls.push(['quantity', lot, value]);
const onRentalConditionChange = (lot, value) => calls.push(['condition', lot, value]);
const showAutocompleteEditItem = lot => calls.push(['edit', lot]);
window.fetch = async () => ({ok: true, json: async () => reply});
function check(ok, message) { if (!ok) throw new Error(message); }
function inert(id, payload) {
const el = document.getElementById(id);
check(el.textContent.includes(payload), id + ': text did not survive');
check(!el.querySelector('img,script,iframe'), id + ': injected element');
}
(async () => {
for (const lot of [
"LOT'\\\"><img src=x onerror=window.__injected++>",
'LOT&#X61;&#39;);window.__injected++;//',
'LOT\\\\&<>"',
]) {
cart = [{lot_name: lot, description: lot, category: 'cpu', quantity: 1}];
for (const render of [() => renderMultiSelectTab([]), () => renderMultiSelectTabWithSections([{title: 'CPU', categories: ['cpu']}])]) {
render();
const root = document.getElementById('tab-content');
check(!root.querySelector('img,script'), 'cart: injected element');
const quantity = root.querySelector('[data-cart-action="quantity"]');
check(quantity.dataset.lot === lot, 'cart: dataset changed');
quantity.value = '2'; quantity.dispatchEvent(new Event('change', {bubbles: true}));
root.querySelector('[data-cart-action="edit"]').dispatchEvent(new FocusEvent('focusin', {bubbles: true}));
root.querySelector('[data-cart-action="remove"] svg').dispatchEvent(new MouseEvent('click', {bubbles: true}));
check(calls.length === 3 && calls.every(c => c[1] === lot), 'cart: delegated action changed LOT');
calls = [];
}
reply = {items: [{...cart[0], condition: 'new', buy_price: 1, price: 1, one_time: 1, weekly: 1}]};
await renderRentalTab();
inert('rental-body', lot);
const condition = document.querySelector('[data-cart-action="condition"]');
condition.dispatchEvent(new Event('change', {bubbles: true}));
check(calls.length === 1 && calls[0][1] === lot, 'rental: delegated action changed LOT'); calls = [];
bomRows = [{resolved_lot: lot, vendor_pn: lot, description: lot, quantity: 1, resolution_source: 'manual'}];
_renderBOMParsedTable(); inert('bom-table-body', lot);
showToast(lot, 'error'); inert('toast', lot);
reply = {is_online: false, config_count: 0, project_count: 0,
pending_changes: [{entity_uuid: 'example', operation: lot, entity_type: lot, last_error: lot}],
errors: [{message: lot}]};
await loadSyncInfo(); inert('modal-pending-list', lot); inert('modal-errors-list', lot);
reply = {success: true, repaired: 0, remaining_errors: [lot]};
await repairPendingChanges(); inert('repair-result', lot);
}
await new Promise(resolve => setTimeout(resolve, 50));
check(window.__injected === 0, 'payload executed');
document.getElementById('result').textContent = 'PASS';
})().catch(error => {document.getElementById('result').textContent = 'FAIL: ' + error.stack;});
`;
fs.writeFileSync(target, `<!doctype html><meta charset="utf-8">${fixtures}<pre id="result">PENDING</pre><script>${fs.readFileSync('web/static/html-escape.js', 'utf8')}\n${functions}\n${listeners}\n${test}</script>`);
const chrome = process.env.CHROME_BIN || '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome';
try {
const result = spawnSync(chrome, ['--headless', '--disable-gpu', '--no-first-run', '--disable-background-networking',
'--user-data-dir=' + path.join(directory, 'profile'), '--virtual-time-budget=1000', '--dump-dom', 'file://' + target], {encoding: 'utf8', timeout: 30000});
assert.ifError(result.error);
assert.equal(result.status, 0, result.stderr);
const outcome = result.stdout.match(/<pre id="result">([\s\S]*?)<\/pre>/)?.[1];
assert.equal(outcome, 'PASS', outcome || 'Browser did not finish');
console.log('Browser HTML security regressions passed');
} finally {
fs.rmSync(directory, {recursive: true, force: true});
}
+33
View File
@@ -0,0 +1,33 @@
const assert = require('node:assert/strict');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const {spawnSync} = require('node:child_process');
const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'qfs-release-gate-'));
try {
fs.copyFileSync('Makefile', path.join(directory, 'Makefile'));
fs.copyFileSync('scripts/release.sh', path.join(directory, 'release.sh'));
const tools = path.join(directory, 'tools');
fs.mkdirSync(tools);
fs.writeFileSync(path.join(tools, 'go'), '#!/bin/sh\nprintf "%s\\n" "$*" >> "$QFS_TEST_GO_CALLS"\nexit 23\n', {mode: 0o755});
const callsPath = path.join(directory, 'calls');
const env = {...process.env, PATH: tools + path.delimiter + process.env.PATH, QFS_TEST_GO_CALLS: callsPath};
const cases = [
...['build-release', 'build-linux', 'build-macos', 'build-windows', 'build-all'].map(target => ['make', [target]]),
['bash', ['release.sh']],
];
for (const [cmd, args] of cases) {
fs.writeFileSync(callsPath, '');
const result = spawnSync(cmd, args, {cwd: directory, env, encoding: 'utf8', timeout: 10000});
assert.ifError(result.error);
assert.notEqual(result.status, 0, `${cmd} ${args}: scanner failure was ignored`);
const calls = fs.readFileSync(callsPath, 'utf8').trim().split('\n');
assert.deepEqual(calls, ['run golang.org/x/vuln/cmd/govulncheck@v1.9.0 ./...']);
assert(!fs.existsSync(path.join(directory, 'bin')), 'Build artifacts created after failed scan');
assert(!fs.existsSync(path.join(directory, 'releases')), 'Release artifacts created after failed scan');
}
console.log('Release security failure checks passed');
} finally {
fs.rmSync(directory, {recursive: true, force: true});
}