Compare commits
13
Commits
d6c498f0cf
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
52720d8e38 | ||
|
|
0ca30afba3 | ||
|
|
60b6047279 | ||
|
|
8569871a17 | ||
|
|
3d4101ecc0 | ||
|
|
46d7ccda4d | ||
|
|
2d60fcfb39 | ||
|
|
ffdbe4fbd0 | ||
|
|
e8dd03c4f5 | ||
|
|
fa09513e26 | ||
|
|
44a09d4420 | ||
|
|
8a08270b79 | ||
|
|
cab5d13f16 |
No files matched your search
@@ -349,4 +349,7 @@ Rules:
|
||||
quoted attributes. Never interpolate data into JavaScript event attributes.
|
||||
Cart and rental controls carry escaped LOT values in `data-lot` and dispatch
|
||||
through delegated listeners. Sync errors are escaped as text; toast messages use
|
||||
`textContent`. Run `node scripts/tests/html-security.cjs` for the regression check.
|
||||
`textContent`. Run `node scripts/tests/html-security.cjs` for the fast regression check and
|
||||
`node scripts/tests/html-security-browser.cjs` for real DOM parsing and delegated
|
||||
events in headless Chrome (`CHROME_BIN` overrides the executable). The browser
|
||||
check uses synthetic data, controlled API responses and a temporary profile.
|
||||
@@ -196,6 +196,7 @@ PK: lot_name
|
||||
| rental_items | JSON | per-lot New/БУ condition for the Аренда tab; see [02-architecture.md](02-architecture.md#rental--paid-testing-pricing-contract) |
|
||||
| rental_uplift_percent | decimal(8,2) DEFAULT 0 | uplift applied to Estimate buy price for rental "Цена" |
|
||||
| feature_flags | JSON NULL | added by migration 034. Data features the configuration uses (`["sale_row_markups"]`); derived on save, a client refuses to write a config with a flag it doesn't know. See [decisions/2026-09-24-configuration-feature-flags.md](decisions/2026-09-24-configuration-feature-flags.md) |
|
||||
| server_updated_at | timestamp(6) NOT NULL | added by migration 035. Server-maintained change marker (`DEFAULT`/`ON UPDATE CURRENT_TIMESTAMP(6)`, index `(server_updated_at, id)`); never written by clients, used for incremental pulls. See [decisions/2026-10-26-incremental-config-project-pull.md](decisions/2026-10-26-incremental-config-project-pull.md) |
|
||||
|
||||
### qt_lot_metadata
|
||||
PK: lot_name
|
||||
@@ -306,6 +307,7 @@ PK: username
|
||||
| rental_enabled | tinyint(1) DEFAULT 0 | shows the Аренда tab on this project's configurations |
|
||||
| created_at | timestamp | |
|
||||
| updated_at | timestamp ON UPDATE | |
|
||||
| server_updated_at | timestamp(6) NOT NULL | added by migration 035. Server-maintained change marker (`DEFAULT`/`ON UPDATE CURRENT_TIMESTAMP(6)`, index `(server_updated_at, id)`); never written by clients, used for incremental pulls. See [decisions/2026-10-26-incremental-config-project-pull.md](decisions/2026-10-26-incremental-config-project-pull.md) |
|
||||
|
||||
### qt_schema_migrations
|
||||
| Column | Type | Notes |
|
||||
@@ -439,16 +441,39 @@ MariaDB:
|
||||
`UPTO` as applied **without running them**, for a server whose schema is already current;
|
||||
- migrations use their own connection without read/write timeouts (`openMigrationDB`),
|
||||
not the app's 3s-timeout sync connection — a long backfill must not be cut mid-file;
|
||||
- server migration commands finish before runtime connection workers start;
|
||||
- `-migrate` requires a DBA maintenance window with every other writer stopped,
|
||||
directly granted global ALL PRIVILEGES, and no other connection using the schema.
|
||||
Do not expand runtime-user privileges; use the existing developer DBA workflow;
|
||||
- the runner holds a schema advisory lock and one connection for the entire session.
|
||||
It automatically writes a full consistent snapshot before changes and per-table
|
||||
snapshots before every file; each archive is restored into a temporary schema
|
||||
and compared before any corresponding original-schema change. Snapshot or
|
||||
verification failure stops the run. See [06-backup.md](06-backup.md);
|
||||
- on a statement/verify failure, per-step snapshots are restored in reverse order,
|
||||
including migration history; a full-session snapshot is the fallback. A persistent
|
||||
marker refuses retry after interruption until `qfs -migrate-restore <archive.zip>`
|
||||
restores and verifies the original state. Connection/recovery failure retains the
|
||||
marker and archives and emits an explicit restore diagnostic;
|
||||
- recovery supports InnoDB base tables and their triggers. Views, routines,
|
||||
events and cross-schema FKs block migrations before changes; new migrations must
|
||||
stay within supported table SQL. Existing triggers are captured with their
|
||||
definer, body, order and SQL/charset context, then restored after all data; Verification requires CREATE/DROP DATABASE.
|
||||
The runner refuses NO_BACKSLASH_ESCAPES sessions;
|
||||
- each file, its `-- verify: <description> | <SQL>` queries and history insertion run
|
||||
on one database session, preserving session variables;
|
||||
- every verify query and the history insert use a fresh GORM statement
|
||||
(`Session{NewDB: true}`) on that session; chaining them on one statement leaked state
|
||||
and broke files with two or more verify checks (fixed 2026-10-26 with migration 035);
|
||||
- a file is recorded only after statements and all verify checks succeed. Verify
|
||||
queries return rows only for broken invariants; a row or SQL error stops the run
|
||||
without marking that file applied. Malformed verify headers fail before execution;
|
||||
- only duplicate column/key names and already-existing objects are tolerated.
|
||||
FK creation errors (including errno 150/121) remain fatal;
|
||||
- each file must be safe to re-run after a partial failure (`ADD COLUMN IF NOT EXISTS`,
|
||||
idempotent `UPDATE ... WHERE <not yet done>`). The current statement splitter assumes
|
||||
semicolons occur only between statements; do not add semicolons inside literals;
|
||||
idempotent `UPDATE ... WHERE <not yet done>`). The statement parser preserves
|
||||
semicolons in literals/quoted identifiers and strips ordinary comments. Malformed
|
||||
quotes, executable comments and DELIMITER directives are rejected before execution;
|
||||
- there is no GORM `AutoMigrate` against MariaDB: it retypes production columns to match the
|
||||
running binary's models. On 2026-09-24 a stale v1.18 binary run with the old AutoMigrate
|
||||
`-migrate` did exactly that, which is why it was removed.
|
||||
|
||||
@@ -65,6 +65,7 @@ They are stored in SQLite and encrypted with `local_encryption.key` unless `QUOT
|
||||
| `-reset-localdb` | reset after mandatory verified backup; refuses unsynced work |
|
||||
| `-force-reset-localdb` | with `-reset-localdb`, allow clearing unsynced work; backup remains mandatory |
|
||||
| `-migrate` | apply pending server SQL migrations (`migrations/*.sql`) and exit |
|
||||
| `-migrate-restore <archive.zip>` | restore and verify a server migration snapshot during DBA maintenance |
|
||||
| `-migrate-status` | list server SQL migrations with applied/pending state and exit |
|
||||
| `-migrate-baseline <file>` | record server migrations up to `<file>` as applied without running them, and exit |
|
||||
| `-version` | print app version and exit |
|
||||
|
||||
@@ -71,3 +71,67 @@ in one transaction and preserve connection settings. Missing legacy tables are
|
||||
skipped. Use the restore procedure above for the pre-reset archive; retain it
|
||||
until the reset has been verified. The environment reset switch was removed to
|
||||
prevent repeated destructive startup.
|
||||
|
||||
## Server migration snapshots
|
||||
|
||||
`qfs -migrate` writes mandatory archives under
|
||||
`<backup root>/server-migrations/qfs-mariadb-v1-<label>-<UTC timestamp>-<unique>.zip`.
|
||||
`QFS_BACKUP_DIR` overrides the root; `QFS_BACKUP_DISABLE` never disables safety
|
||||
snapshots. Paths are checked outside git, including symlink targets, before
|
||||
creation and restore. Archives have private filesystem permissions and contain
|
||||
server table schemas and data, which may be sensitive; never put them in git.
|
||||
Connection passwords and the local encryption key are not copied into metadata.
|
||||
|
||||
One full snapshot precedes the session. Before each file, a separate snapshot
|
||||
contains a schema definition and row stream for **every** table: this includes
|
||||
all affected tables even when a legacy Tables-affected header is incomplete.
|
||||
This intentionally favors recovery over storage/time efficiency for the manual
|
||||
DBA tool. Retain safety snapshots outside normal rotation until the migration
|
||||
has been verified; remove verified-session artifacts as operational cleanup.
|
||||
|
||||
The versioned ZIP manifest records table schemas, writable column order, row
|
||||
counts and order-independent row digests. Row streams preserve NULL, empty
|
||||
strings, binary values and fractional timestamps. Restore temporarily uses
|
||||
NO_AUTO_VALUE_ON_ZERO to preserve explicit zero IDs and legacy zero dates;
|
||||
row digests detect coercion and the original session mode is restored. Generated columns are rebuilt
|
||||
from schema, not inserted. Trigger bodies, definers, firing order and creation
|
||||
SQL/charset context are preserved. Triggers are recreated after row loading so
|
||||
restore does not duplicate audit data or alter restored values. CRC/format/digests are checked, then each archive is
|
||||
restored into a unique temporary schema on the same server and compared for
|
||||
schema and data equality. The temporary schema is dropped afterward. A failed
|
||||
or interrupted check must be investigated before retry; abandoned `qfs_verify_`
|
||||
schemas are DBA cleanup artifacts, never application data.
|
||||
|
||||
Stop every other writer for the entire migration/recovery window, including
|
||||
clients that could reconnect. The runner detects other current schema connections
|
||||
and uses an advisory lock to exclude a second runner; the lock does not stop
|
||||
unrelated SQL clients. An active client appearing during failure recovery causes
|
||||
recovery refusal rather than overwriting its work. InnoDB base tables and triggers are supported; views, routines, events and
|
||||
cross-schema FKs require explicit recovery support before migration. Triggers
|
||||
whose recorded database collation differs from the current database collation
|
||||
also cause refusal until that legacy state has supported recovery. Trigger
|
||||
routine calls and qualified references to other schemas are refused; source
|
||||
SQL must have no effects outside the captured schema.
|
||||
Unsupported schemas stop before changes. Runtime database-user grants stay as
|
||||
listed in [03-database.md](03-database.md); the tool requires an existing DBA account.
|
||||
|
||||
Failure restores step snapshots in reverse order; the full-session archive is
|
||||
used if step recovery is insufficient. Schema, writable data and migration
|
||||
history must match the session start before recovery is reported successful.
|
||||
The persistent `.pending` marker is keyed by configured endpoint and schema, so
|
||||
server restarts do not hide an interrupted session. Backups survive connection
|
||||
loss; a failed recovery retains its marker and reports the full snapshot path.
|
||||
|
||||
After an interruption, keep clients stopped and use the snapshot path reported
|
||||
by the runner:
|
||||
|
||||
```bash
|
||||
go run ./cmd/qfs -migrate-restore /outside/repository/server-migrations/qfs-mariadb-v1-session-<timestamp>-<unique>.zip
|
||||
make migrate-status
|
||||
make migrate
|
||||
```
|
||||
|
||||
Restore uses the currently saved DBA connection, requires the same database name,
|
||||
checks the archive by restoring it into a temporary schema first, restores the
|
||||
original schema, verifies equality and clears the matching pending marker. It
|
||||
does not require mysql/mysqldump or any operator-installed database utility.
|
||||
@@ -10,6 +10,12 @@ go run ./cmd/migrate_project_updated_at
|
||||
go test ./...
|
||||
go vet ./...
|
||||
make security-check # pinned govulncheck, blocks reachable vulnerabilities
|
||||
# Optional live transport checks; owns disposable Docker containers:
|
||||
go test -tags integration ./internal/mariadb -run TestMariaDBTLSIntegration -count=1
|
||||
# Snapshot/DDL recovery checks; owns a disposable MariaDB 11.8 container:
|
||||
go test -tags integration ./internal/models -run TestSQLMigrationSafetyMariaDB -count=1
|
||||
# Production-dump rehearsal; source path via QFS_TEST_PRODUCTION_DUMP:
|
||||
go test -tags integration ./internal/models -run TestProductionDumpMigrationMariaDB -count=1 -v
|
||||
make build-release
|
||||
make install-hooks
|
||||
```
|
||||
@@ -45,6 +51,8 @@ normal Go toolchain selection downloads a suitable compiler when necessary.
|
||||
The scanner version is pinned in the Makefile. All release build targets and
|
||||
`scripts/release.sh` require this check to succeed before producing artifacts.
|
||||
A scanner/database/download failure also stops release.
|
||||
`node scripts/tests/release-security.cjs` verifies failure stops all release
|
||||
entry points before artifact creation using an isolated failing scanner.
|
||||
|
||||
When updating dependencies: run `go mod tidy`, `go test ./...`, `go vet ./...`,
|
||||
`make security-check` and the JavaScript regression check documented in
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# Decision: incremental pull of configurations and projects via server_updated_at
|
||||
|
||||
**Date:** 2026-10-26
|
||||
**Status:** active
|
||||
|
||||
## Context
|
||||
|
||||
Opening the configurations or projects list (`GET /api/configs`, `/api/projects`,
|
||||
`/api/projects/:uuid/configs`, on-demand with a 30s cooldown) and the manual sync ran
|
||||
`ImportConfigurationsToLocal` / `ImportProjectsToLocal`, which read **every** server row
|
||||
and rewrote it locally. Measured on production (2026-10-26): configurations 22.2s
|
||||
(4158 rows, ~3.2 MB, 4078 rewritten for one actual change), projects 4.6s (1692 rows),
|
||||
holding a pool connection the whole time. `qt_configurations` had no change column;
|
||||
`qt_projects.updated_at` is set by clients (`SaveProjectPreservingUpdatedAt`), so a change
|
||||
made offline and pushed later carries an old timestamp and cannot drive a filter.
|
||||
|
||||
## Decision
|
||||
|
||||
- Server migration `035_add_server_updated_at.sql` adds
|
||||
`server_updated_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6)`
|
||||
plus index `(server_updated_at, id)` to `qt_configurations` and `qt_projects`. MariaDB
|
||||
maintains it; no client writes it and it is **not** in `models.Configuration` /
|
||||
`models.Project`, so writes from any client version (GORM `Create`/`Save`, raw
|
||||
`UPDATE` with column lists) keep working and bump the marker on every real change.
|
||||
- Client (`internal/services/sync/pull_cursor.go`):
|
||||
- if the column is missing (server not migrated yet) every pull is full, exactly as
|
||||
before; presence is re-checked until found, then cached for the process;
|
||||
- the first pull of each table per process is full (restores rows lost locally once per
|
||||
app start), later pulls read `server_updated_at >= cursor - 120s` in one query;
|
||||
- the cursor is the server's `NOW(6)` read *before* the pull, stored in `app_settings`
|
||||
(`server_pull_cursor:<table>`) together with `host:port/database`; a cursor from a
|
||||
different server is ignored;
|
||||
- the per-row merge (skip `pending`, keep local deactivation sticky, keep `line`) is
|
||||
unchanged; only the set of rows read differs.
|
||||
- Old clients keep doing full pulls and are unaffected.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Idle pulls go from ~27s / 3.2 MB to one indexed range query.
|
||||
- Hard deletes on the server are not propagated — same as the full pull before.
|
||||
- Changes applied by InnoDB `ON UPDATE CASCADE` (e.g. a `qt_projects.uuid` change
|
||||
cascading to `qt_configurations.project_uuid`) do not bump `server_updated_at`; they are
|
||||
picked up by the next full pull (app start).
|
||||
- Cursor comparison happens in the server session time zone. Production is `SYSTEM`/MSK
|
||||
(no DST); on a server with DST, rows changed around the autumn transition could be
|
||||
missed until the next app start.
|
||||
- Applying 035 adds the column and builds an index on both tables (~4k and ~2k rows on
|
||||
production, seconds); `qfs -migrate` is run once by the developer as usual.
|
||||
- Tests: `internal/services/sync/incremental_pull_mariadb_test.go` (needs
|
||||
`QF_TEST_MARIADB_DSN`, a disposable MariaDB with the production table DDL; verified on
|
||||
MariaDB 11.8, same as production), migrator fix in
|
||||
`internal/models/sql_migration_verify_test.go` (`two checks pass`).
|
||||
@@ -0,0 +1,54 @@
|
||||
# Decision: one MariaDB pool per client — reuse, never orphan
|
||||
|
||||
**Date:** 2026-10-26
|
||||
**Status:** active
|
||||
|
||||
## Context
|
||||
|
||||
A single client held ~30 MariaDB connections during sync. `ConnectionManager.GetDB`
|
||||
treated a pool older than `pingInterval` (30s) as unusable and dialed a brand-new
|
||||
`*gorm.DB` without closing the previous one. Sync calls `getDB()` per step, so every
|
||||
~30s a new pool appeared while the old one kept its idle connections until
|
||||
`ConnMaxLifetime` (5 min): ~10 orphaned pools × 2 idle + up to 10 active ≈ 30. Each
|
||||
orphaned `*sql.DB` also leaked its opener goroutine. `TryConnect`, the background
|
||||
ping-failure path and a failed ping inside `dial` dropped pools the same way.
|
||||
|
||||
## Decision
|
||||
|
||||
- `GetDB` pings the existing pool when it is merely stale and returns it; it dials only
|
||||
when there is no pool or the ping fails.
|
||||
- Whenever the manager replaces or drops a pool (`GetDB`, `TryConnect`,
|
||||
`checkOnlineNow`, `MarkOffline`, `Disconnect`, failed ping in `dial`) the old
|
||||
`*sql.DB` is closed.
|
||||
- Pool limits: `MaxOpenConns=2`, `MaxIdleConns=2`, `ConnMaxLifetime=5m`,
|
||||
`ConnMaxIdleTime=1m` — one client holds at most 2 connections, idle ones are released
|
||||
within a minute. Sync is sequential (one query at a time); the second connection lets
|
||||
UI server queries (active viewers, users status) run alongside a long sync step.
|
||||
- Background sync interval (`backgroundSyncInterval` in `cmd/qfs/main.go`) is 1 minute
|
||||
with ±30% random jitter (`sync.WorkerJitter`), so clients do not hit MariaDB in
|
||||
lockstep. The next cycle is scheduled from the *start* of the previous one, so
|
||||
heartbeats (`qt_client_schema_state.last_checked_at`) are at most 1.3 × interval apart
|
||||
while a cycle is shorter than that.
|
||||
- `/api/sync/users-status` online threshold = interval × (1 + `WorkerJitter`).
|
||||
- `PullPartnumberBooks` reads only book headers (`id, version, created_at, is_active`)
|
||||
each cycle; `partnumbers_json` (~5 MB for all books) is fetched per book only when the
|
||||
book is new or its local items are incomplete. An empty book counts as complete. An
|
||||
incomplete book is re-pulled at most once per process per version: a server book can
|
||||
list partnumbers missing from `qt_partnumber_book_items` (seen: `BOT_AGIBOT_G2_PERF`
|
||||
in `PNBOOK-2026-10-21-001`), which a re-pull cannot fix; this is logged as `WARN` with
|
||||
the missing partnumbers.
|
||||
- A worker cycle sends one heartbeat: the worker calls `EnsureReadinessForSync` once and
|
||||
then the unexported `pushPendingChanges` (the exported `PushPendingChanges` still checks
|
||||
readiness itself for the manual-sync handlers).
|
||||
- `NeedSync` asks for the latest active pricelist of all three sources in one
|
||||
`UNION ALL` query (`GetLatestActiveIDsBySources`): 0.71s → 0.24s.
|
||||
- `SyncQtSettings` skips the local `DELETE` + `INSERT` when the server rows equal the cache.
|
||||
- Measured against the production server: a cold sync took 171s (pricelists 105s,
|
||||
partnumber books 38s, configurations 23s) and is bound by server reads, not pool
|
||||
size. A warm cycle takes ~1.4s (was ~2.5s plus a 5 MB download).
|
||||
|
||||
## Consequences
|
||||
|
||||
- Code that keeps a `*gorm.DB` across a reconnect gets `sql: database is closed`; sync
|
||||
steps must call `getDB()` per operation instead of caching the handle.
|
||||
- Tests: `internal/db/connection_test.go`, `internal/services/sync/worker_jitter_test.go`.
|
||||
@@ -0,0 +1,62 @@
|
||||
# Decision: verified snapshots and whole-session recovery for server migrations
|
||||
|
||||
**Date:** 2026-10-26
|
||||
**Status:** active
|
||||
|
||||
## Context
|
||||
|
||||
Issue #3 fixes FK error propagation, session variables and verify execution.
|
||||
Its proposed manual-backup exception conflicts with the shared go-database
|
||||
contract. MariaDB DDL cannot be made rollback-safe by wrapping the migration
|
||||
files in an ordinary transaction: see [MariaDB implicit commits](https://mariadb.com/docs/server/reference/sql-statements/transactions/sql-statements-that-cause-an-implicit-commit).
|
||||
|
||||
## Decision
|
||||
|
||||
The migration engine, rather than the operator, owns full-session and pre-file
|
||||
per-table backups. A Go ZIP/JSON snapshot format avoids external database-client
|
||||
dependencies. Every archive is restored into a temporary schema and compared
|
||||
before the corresponding original-schema changes. Per-file archives include all
|
||||
tables to cover incomplete legacy affected-table headers and dynamic SQL.
|
||||
|
||||
On failure, restore each step snapshot in reverse order; use the full-session
|
||||
snapshot as fallback. Include migration history in recovery. Persist a recovery
|
||||
marker before applying any changes, refuse an interrupted session on retry and
|
||||
ship an explicit `-migrate-restore` path. Restore failures retain archives and
|
||||
produce diagnostics instead of claiming success.
|
||||
|
||||
The production dump contains 27 triggers, so the implementation preserves
|
||||
InnoDB tables and trigger bodies, definers, order and SQL/charset context.
|
||||
Triggers are recreated after row loading to avoid firing them during restore;
|
||||
verification binds them to the temporary schema and performs no later DML.
|
||||
Views, routines, events, cross-schema FKs and legacy trigger/database collation
|
||||
mismatches are refused before changes. It requires an existing
|
||||
DBA account with directly granted global ALL PRIVILEGES so object/client
|
||||
visibility cannot silently hide unsupported state, plus a maintenance window
|
||||
without other writers. Do not grant these privileges to the runtime user. The
|
||||
advisory migration lock excludes other runners, not arbitrary SQL clients.
|
||||
|
||||
The parser supports SQL literals and quoted identifiers containing semicolons,
|
||||
escaped/doubled quotes and ordinary comments. It rejects malformed input,
|
||||
executable comments and DELIMITER rather than silently misinterpreting them.
|
||||
Existing merged SQL files are unchanged.
|
||||
|
||||
## Consequences
|
||||
|
||||
Safety snapshots and restore verification add disk usage, temporary server schema
|
||||
creation and runtime to the developer-only migration command. Normal local-first
|
||||
CRUD and sync gain no MariaDB dependency. Migration commands start no background
|
||||
connection worker. Before production use, rehearse on a current production dump
|
||||
in MariaDB matching production version and collation; synthetic tests alone do
|
||||
not satisfy that gate. Unsupported schema features need explicit recovery support
|
||||
before such schemas may be migrated by this tool.
|
||||
|
||||
## Acceptance
|
||||
|
||||
Rehearsed on the supplied 2026-10-26 production dump (45 tables, 27 triggers),
|
||||
using an isolated MariaDB 11.8.3 container and utf8mb4_uca1400_ai_ci. The test
|
||||
verifies full restore, actual migration 035 on the reconstructed pre-035 clone,
|
||||
reverse recovery after a later DDL/verify failure, successful retry and exact
|
||||
restoration of the original dump state. Synthetic tests additionally exercise
|
||||
FK errno 150, interrupted-session recovery, NULL/empty/binary/decimal/timestamp
|
||||
values, zero auto-increment IDs and legacy zero dates. No production changes
|
||||
are performed by these checks.
|
||||
@@ -2,19 +2,19 @@
|
||||
|
||||
Repository: https://git.mchus.pro/mchus/QuoteForge
|
||||
|
||||
All issue pages were retrieved, including closed issues: seven total, six open.
|
||||
Issue [#1](https://git.mchus.pro/mchus/QuoteForge/issues/1) was already closed and
|
||||
is excluded from implementation. Work is sequential on
|
||||
`fix/open-issues-security-recovery`, with one implementation commit per open issue.
|
||||
Scope: issues #2-#7. Initial fixes were merged into `main` in
|
||||
[PR #8](https://git.mchus.pro/mchus/QuoteForge/pulls/8), with release notes in v3.50.
|
||||
The follow-up adds completed acceptance checks and server migration recovery
|
||||
required by the shared rules. Each issue has its own implementation/acceptance commit.
|
||||
|
||||
| Order | Issue | Implementation and acceptance | Status |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | [#2](https://git.mchus.pro/mchus/QuoteForge/issues/2) | Block foreign Host/Origin/Referer and cross-site browser requests in both routers; validate API body types; refuse saved-password reuse for a changed host/port/user. Middleware attack cases and setup credential tests must pass. | Implemented |
|
||||
| 2 | [#4](https://git.mchus.pro/mchus/QuoteForge/issues/4) | Move LOT-bearing cart/rental events to delegated listeners and escaped data attributes; escape BOM and sync output; share HTML escaping; render toast as text. Malicious LOT regression checks and template tests must pass. | Implemented |
|
||||
| 3 | [#7](https://git.mchus.pro/mchus/QuoteForge/issues/7) | Require a fresh verified archive outside rotation before reset, refuse unsynced/legacy states without explicit force, use transactional deletion, remove environment-triggered reset. Verify refusal, ZIP restore, backup failure and rollback. | Implemented |
|
||||
| 4 | [#5](https://git.mchus.pro/mchus/QuoteForge/issues/5) | Persist TLS mode/CA in SQLite, expose setup controls, share policy across setup/sync/migration DSNs. Verify certificate trust, hostname checks, plaintext fallback policy and settings round-trip. | Implemented |
|
||||
| 5 | [#3](https://git.mchus.pro/mchus/QuoteForge/issues/3) | Keep each migration file in one session, execute verify headers before history insertion, propagate FK failures. Verify failed queries/checks never enter history. | Implemented |
|
||||
| 6 | [#6](https://git.mchus.pro/mchus/QuoteForge/issues/6) | Require Go 1.27.3, update vulnerable dependencies, block release on pinned govulncheck. Full tests, vet, security scan and macOS/Windows builds must pass. | Implemented |
|
||||
| 1 | [#2](https://git.mchus.pro/mchus/QuoteForge/issues/2) | Block foreign Host/Origin/Referer and cross-site browser requests in both routers; validate API body types; refuse saved-password reuse for a changed host/port/user. Middleware and real setup/runtime router tests reject attacks without changing saved credentials or user data. | Acceptance passed |
|
||||
| 2 | [#4](https://git.mchus.pro/mchus/QuoteForge/issues/4) | Move LOT-bearing cart/rental events to delegated listeners and escaped data attributes; escape BOM and sync output; share HTML escaping; render toast as text. Malicious LOT regression checks and headless Chrome DOM checks pass. | Acceptance passed |
|
||||
| 3 | [#7](https://git.mchus.pro/mchus/QuoteForge/issues/7) | Require a fresh verified archive outside rotation before reset, refuse unsynced/legacy states without explicit force, use transactional deletion, remove environment-triggered reset. Verified refusal, ZIP restore, backup failure, rollback and CLI startup after restore. | Acceptance passed |
|
||||
| 4 | [#5](https://git.mchus.pro/mchus/QuoteForge/issues/5) | Persist TLS mode/CA in SQLite, expose setup controls, share policy across setup/sync/migration DSNs. Verified certificate trust, hostname checks, fallback policy, settings round-trip and live MariaDB 11.8 transport. | Acceptance passed |
|
||||
| 5 | [#3](https://git.mchus.pro/mchus/QuoteForge/issues/3) | Keep each migration file in one session, execute verify headers before history insertion, propagate FK failures. Failed queries/checks restore the entire session. Verified snapshots, literal-aware SQL parsing, trigger preservation and crash recovery pass synthetic tests and production-dump rehearsal on MariaDB 11.8.3. | Acceptance passed |
|
||||
| 6 | [#6](https://git.mchus.pro/mchus/QuoteForge/issues/6) | Require Go 1.27.3, update vulnerable dependencies, block release on pinned govulncheck. Tests, vet, security scan, cross-platform builds and release failure checks pass. | Acceptance passed |
|
||||
|
||||
## Decisions
|
||||
|
||||
@@ -28,19 +28,103 @@ is excluded from implementation. Work is sequential on
|
||||
encryption. The setup form explains this distinction.
|
||||
- `go 1.27.3` enforces the minimum compiler, including with `GOTOOLCHAIN=local`.
|
||||
A redundant `toolchain go1.27.3` is removed by Go tooling and is unnecessary.
|
||||
- The local ignored `.claude/skills/release/SKILL.md` was updated with the security
|
||||
prerequisite; the shared Makefile and release script enforce it in git.
|
||||
|
||||
## Validation and handoff
|
||||
|
||||
Passed: `go test ./...`, `go vet ./...`, `node scripts/tests/html-security.cjs`,
|
||||
`make security-check`, shell syntax and diff checks, and CGO-free builds for
|
||||
macOS arm64 and Windows amd64.
|
||||
The original implementation PR is merged. Current acceptance results are listed
|
||||
below. Govulncheck reports zero reachable vulnerabilities and one module-level
|
||||
advisory whose affected code is not called. Initial cross-platform builds and
|
||||
all Go/JavaScript checks pass; final verification also covers the recovery change.
|
||||
Production-dump rehearsal also passes on MariaDB 11.8.3 with the production
|
||||
collation. No production migration or new release is performed by this work.
|
||||
|
||||
Govulncheck reports zero reachable vulnerabilities and one module-level advisory
|
||||
whose affected code is not called. Gin's transitive quic-go was explicitly raised
|
||||
to v0.59.1 after the first scan identified a reachable advisory in v0.59.0.
|
||||
## Acceptance follow-up
|
||||
|
||||
The checks use temporary SQLite databases and synthetic certificates. They do
|
||||
not exercise a live MariaDB server or browser automation. Review the draft PR
|
||||
before merge; issue state changes and a production release remain separate steps.
|
||||
### Issue #2
|
||||
|
||||
`go test ./cmd/qfs -run TestRequestBoundaryOnApplicationRouters -count=1`
|
||||
exercises the production setup and runtime route constructors with temporary
|
||||
SQLite state. Foreign setup forms, opaque origins and rebinding hosts are
|
||||
rejected, text/plain JSON receives 415, and both setup handlers reject saved
|
||||
password reuse for a changed destination. Local reads still succeed. Saved
|
||||
credentials, project/configuration counts and the pending queue remain unchanged.
|
||||
The setup router construction was extracted without changing its routes so
|
||||
this regression exercises the same middleware registration as startup.
|
||||
|
||||
### Issue #4
|
||||
|
||||
`node scripts/tests/html-security-browser.cjs` passes in headless Chrome with
|
||||
an isolated temporary profile. It executes the actual template rendering
|
||||
functions in a browser DOM with controlled API responses. Both cart layouts,
|
||||
rental, parsed BOM, toast, pending sync errors, error history and repair errors
|
||||
preserve malicious strings as text. Real click/change/focus events preserve
|
||||
LOT values through delegated handlers; no injected elements or payload execution
|
||||
are observed. This is a rendering integration check, not a full application
|
||||
navigation test. Set `CHROME_BIN` outside macOS to select a Chrome executable.
|
||||
|
||||
### Issue #7
|
||||
|
||||
`go test ./cmd/qfs ./internal/localdb -run TestReset -count=1` covers reset
|
||||
safety and a subprocess executing the real `main` flag/startup path. With
|
||||
rotating backups disabled, the environment switch does not reset pending work,
|
||||
normal reset refuses it, and explicit forced reset creates a verified archive.
|
||||
The test restores ZIP contents and reopens the database through normal startup:
|
||||
connection settings and the pending queue survive. It deliberately selects
|
||||
`-migrate-status` with an unavailable loopback database to stop startup before
|
||||
HTTP/browser launch; no production database or user state is touched.
|
||||
|
||||
### Issue #5
|
||||
|
||||
`go test -tags integration ./internal/mariadb -run TestMariaDBTLSIntegration -count=1`
|
||||
starts its own disposable MariaDB 11.8 containers on random loopback ports, with
|
||||
TLS enabled and disabled, and removes both afterward. The test verifies the
|
||||
actual session cipher: preferred encrypts with a TLS server and falls back only
|
||||
with a plaintext server; required and verification modes reject plaintext.
|
||||
Correct trust succeeds, unrelated roots and wrong hostname fail. Certificates,
|
||||
private keys and database storage are temporary; no saved user DSN is used.
|
||||
The matching image must already exist locally (`docker pull mariadb:11.8`).
|
||||
Setup policy, local settings and synthetic TLS tests also pass.
|
||||
|
||||
### Issue #6
|
||||
|
||||
Go 1.27.3: `go test ./...`, `go vet ./...` and `make security-check` pass.
|
||||
CGO-free builds produce macOS arm64 Mach-O and Windows amd64 PE executables
|
||||
outside the repository. `node scripts/tests/release-security.cjs` substitutes a
|
||||
failing Go scanner in an isolated directory and confirms all five release build
|
||||
targets and `scripts/release.sh` stop before compilation or artifact creation.
|
||||
The scanner reports zero reachable vulnerabilities and one uncalled module-level
|
||||
advisory. No release is published by these acceptance checks.
|
||||
|
||||
### Issue #3
|
||||
|
||||
The parser, mandatory server snapshots, restore-readiness checks, reverse step
|
||||
recovery and explicit crash restore are implemented; see
|
||||
[server migration recovery decision](decisions/2026-10-26-server-migration-recovery.md).
|
||||
`go test -tags integration ./internal/models -run TestSQLMigrationSafetyMariaDB -count=1`
|
||||
passes on a disposable MariaDB 11.8: session variables and two verify checks,
|
||||
semicolon literals, a later failed verify, a real FK errno 150 after successful
|
||||
DDL, reverse recovery of prior files, interrupted-session refusal/restore/retry,
|
||||
unsupported-schema refusal, active-client refusal and backup failure.
|
||||
Restored schema, binary/NULL/empty/decimal/timestamp data and history are compared
|
||||
to the original snapshot. Unit tests cover snapshot format/digests, symlink path
|
||||
refusal and all embedded migration SQL syntax.
|
||||
|
||||
The supplied `2026-10-26_RFQ_LOG.sql` dump (45 tables, 27 triggers) is loaded
|
||||
read-only into a new disposable MariaDB **11.8.3** container with
|
||||
`utf8mb4_uca1400_ai_ci`, matching the production version and collation.
|
||||
`TestProductionDumpMigrationMariaDB` verifies a full snapshot restore, rebuilds
|
||||
the pre-035 schema only in that clone, applies the real embedded migration 035,
|
||||
forces a subsequent DDL/verify failure, verifies reverse recovery against the
|
||||
complete pre-session snapshot, then applies 035 successfully and restores the
|
||||
exact original dump state. Trigger bodies, definers and order are preserved;
|
||||
triggers never fire during row loading. No production schema or dump file is changed.
|
||||
|
||||
Run this optional, destructive-to-its-own-container rehearsal with an existing
|
||||
`mariadb:11.8.3` image and `QFS_TEST_PRODUCTION_DUMP=/outside/repository/dump.sql`:
|
||||
|
||||
```bash
|
||||
go test -tags integration ./internal/models -run TestProductionDumpMigrationMariaDB -count=1 -v
|
||||
```
|
||||
|
||||
The environment variable supplies the source file only; no externally supplied
|
||||
DSN is accepted. The test owns and removes its container and temporary archives.
|
||||
+19
-47
@@ -46,7 +46,7 @@ import (
|
||||
var Version = "dev"
|
||||
var errVendorImportTooLarge = errors.New("vendor workspace file exceeds 1 GiB limit")
|
||||
|
||||
const backgroundSyncInterval = 5 * time.Minute
|
||||
const backgroundSyncInterval = time.Minute
|
||||
const onDemandPullCooldown = 30 * time.Second
|
||||
const startupConsoleWarning = "Не закрывайте консоль иначе приложение не будет работать"
|
||||
|
||||
@@ -62,6 +62,7 @@ func main() {
|
||||
resetLocalDB := flag.Bool("reset-localdb", false, "reset local SQLite data after a verified safety backup (keeps connection settings)")
|
||||
forceResetLocalDB := flag.Bool("force-reset-localdb", false, "allow -reset-localdb to clear unsynced work after a verified backup")
|
||||
migrate := flag.Bool("migrate", false, "apply pending server (MariaDB) SQL migrations and exit")
|
||||
migrateRestore := flag.String("migrate-restore", "", "restore a verified server migration snapshot during maintenance and exit")
|
||||
migrateStatus := flag.Bool("migrate-status", false, "list server SQL migrations with applied/pending state and exit")
|
||||
migrateBaseline := flag.String("migrate-baseline", "", "record server SQL migrations up to and including this file as applied, without running them, and exit")
|
||||
version := flag.Bool("version", false, "show version information")
|
||||
@@ -132,6 +133,10 @@ func main() {
|
||||
|
||||
// Check if running in setup mode (no connection settings)
|
||||
if !local.HasSettings() {
|
||||
if *migrate || *migrateStatus || *migrateBaseline != "" || *migrateRestore != "" {
|
||||
slog.Error("server migration command requires saved DBA connection settings")
|
||||
os.Exit(1)
|
||||
}
|
||||
slog.Info("no database settings found, starting setup mode")
|
||||
runSetupMode(local)
|
||||
return
|
||||
@@ -174,13 +179,6 @@ func main() {
|
||||
// Create connection manager. Runtime stays local-first; MariaDB is used on demand by sync/setup only.
|
||||
connMgr := db.NewConnectionManager(local)
|
||||
|
||||
// Keep the online-status cache fresh in the background so request-handling
|
||||
// goroutines (health checks, middleware, etc.) never block on the MySQL
|
||||
// dial/read timeout themselves.
|
||||
connMgrCtx, connMgrCancel := context.WithCancel(context.Background())
|
||||
defer connMgrCancel()
|
||||
connMgr.Start(connMgrCtx)
|
||||
|
||||
dbUser := local.GetDBUser()
|
||||
|
||||
slog.Info("starting QuoteForge server",
|
||||
@@ -191,19 +189,21 @@ func main() {
|
||||
"online", false,
|
||||
)
|
||||
|
||||
if *migrate || *migrateStatus || *migrateBaseline != "" {
|
||||
if *migrate || *migrateStatus || *migrateBaseline != "" || *migrateRestore != "" {
|
||||
mariaDB, err := openMigrationDB(local)
|
||||
if err != nil {
|
||||
slog.Error("cannot run migrations: database not available", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
switch {
|
||||
case *migrateRestore != "":
|
||||
err = models.RestoreSQLMigrationSession(mariaDB, *migrateRestore)
|
||||
case *migrateStatus:
|
||||
err = runMigrateStatus(mariaDB)
|
||||
case *migrateBaseline != "":
|
||||
err = runMigrateBaseline(mariaDB, *migrateBaseline)
|
||||
default:
|
||||
err = runMigrateApply(mariaDB)
|
||||
err = runMigrateApply(mariaDB, resolvedLocalDBPath)
|
||||
}
|
||||
if err != nil {
|
||||
slog.Error("migration command failed", "error", err)
|
||||
@@ -212,6 +212,13 @@ func main() {
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
// Keep the online-status cache fresh in the background so request-handling
|
||||
// goroutines (health checks, middleware, etc.) never block on the MySQL
|
||||
// dial/read timeout themselves.
|
||||
connMgrCtx, connMgrCancel := context.WithCancel(context.Background())
|
||||
defer connMgrCancel()
|
||||
connMgr.Start(connMgrCtx)
|
||||
|
||||
gin.SetMode(cfg.Server.Mode)
|
||||
restartSig := make(chan struct{}, 1)
|
||||
|
||||
@@ -542,47 +549,12 @@ func nextBackupTime(now time.Time, hour, minute int) time.Time {
|
||||
func runSetupMode(local *localdb.LocalDB) {
|
||||
restartSig := make(chan struct{}, 1)
|
||||
|
||||
// In setup mode, we don't have a connection manager yet (will restart after setup)
|
||||
templatesPath := filepath.Join("web", "templates")
|
||||
setupHandler, err := handlers.NewSetupHandler(local, nil, templatesPath, restartSig)
|
||||
router, err := setupSetupRouter(local, restartSig)
|
||||
if err != nil {
|
||||
slog.Error("failed to create setup handler", "error", err)
|
||||
slog.Error("failed to create setup router", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
gin.SetMode(gin.ReleaseMode)
|
||||
router := gin.New()
|
||||
router.Use(gin.Recovery())
|
||||
router.Use(middleware.CORS())
|
||||
|
||||
if staticFS, err := qfassets.StaticFS(); err == nil {
|
||||
router.StaticFS("/static", http.FS(staticFS))
|
||||
} else {
|
||||
slog.Error("failed to load embedded static assets", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Setup routes only
|
||||
router.GET("/", func(c *gin.Context) {
|
||||
c.Redirect(http.StatusFound, "/setup")
|
||||
})
|
||||
router.GET("/setup", setupHandler.ShowSetup)
|
||||
router.POST("/setup", setupHandler.SaveConnection)
|
||||
router.POST("/setup/test", setupHandler.TestConnection)
|
||||
router.GET("/setup/status", setupHandler.GetStatus)
|
||||
|
||||
settingsHandler := handlers.NewSettingsHandler(local)
|
||||
router.GET("/api/settings/ui", settingsHandler.GetUI)
|
||||
router.PUT("/api/settings/ui", settingsHandler.PutUI)
|
||||
|
||||
// Health check
|
||||
router.GET("/health", func(c *gin.Context) {
|
||||
c.JSON(http.StatusOK, gin.H{
|
||||
"status": "setup_required",
|
||||
"time": time.Now().UTC().Format(time.RFC3339),
|
||||
})
|
||||
})
|
||||
|
||||
addr := "127.0.0.1:8080"
|
||||
slog.Info("starting setup mode server", "address", addr, "version", Version)
|
||||
|
||||
|
||||
+13
-2
@@ -6,6 +6,7 @@ import (
|
||||
"time"
|
||||
|
||||
qfassets "git.mchus.pro/mchus/quoteforge"
|
||||
"git.mchus.pro/mchus/quoteforge/internal/appstate"
|
||||
"git.mchus.pro/mchus/quoteforge/internal/localdb"
|
||||
"git.mchus.pro/mchus/quoteforge/internal/models"
|
||||
mysqlDriver "github.com/go-sql-driver/mysql"
|
||||
@@ -39,6 +40,12 @@ func openMigrationDB(local *localdb.LocalDB) (*gorm.DB, error) {
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("connect to server: %w", err)
|
||||
}
|
||||
sqlDB, err := db.DB()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("migration pool: %w", err)
|
||||
}
|
||||
sqlDB.SetMaxOpenConns(1)
|
||||
sqlDB.SetMaxIdleConns(1)
|
||||
return db, nil
|
||||
}
|
||||
|
||||
@@ -68,7 +75,7 @@ func runMigrateStatus(db *gorm.DB) error {
|
||||
// runMigrateApply applies pending server migrations. Refuses on a server with no
|
||||
// migration history at all: its schema was built by other means, so every file would be
|
||||
// re-run — record a baseline with -migrate-baseline first.
|
||||
func runMigrateApply(db *gorm.DB) error {
|
||||
func runMigrateApply(db *gorm.DB, localDBPath string) error {
|
||||
migrations, err := qfassets.MigrationsFS()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -76,9 +83,13 @@ func runMigrateApply(db *gorm.DB) error {
|
||||
if err := requireMigrationHistory(db, migrations); err != nil {
|
||||
return err
|
||||
}
|
||||
backupRoot, err := appstate.ServerMigrationBackupDir(localDBPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
applied, err := models.RunSQLMigrations(db, migrations, func(filename string) {
|
||||
fmt.Printf(" applying %s ...\n", filename)
|
||||
})
|
||||
}, backupRoot)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/config"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
func TestRequestBoundaryOnApplicationRouters(t *testing.T) {
|
||||
moveToRepoRoot(t)
|
||||
local, connMgr, _ := newAPITestStack(t)
|
||||
if err := local.SaveSettings("db.example", 3306, "example", "example", "example"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg := &config.Config{}
|
||||
setConfigDefaults(cfg)
|
||||
runtimeRouter, _, err := setupRouter(cfg, local, connMgr, "example", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
setupRouter, err := setupSetupRouter(local, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for name, router := range map[string]*gin.Engine{"setup": setupRouter, "runtime": runtimeRouter} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
before, err := local.GetSettings()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
projects, configs, pending := local.CountProjects(), local.CountConfigurations(), local.CountPendingChanges()
|
||||
for _, tt := range []struct {
|
||||
name, method, path, host, origin, contentType, body string
|
||||
want int
|
||||
}{
|
||||
{"foreign setup save", "POST", "/setup", "", "https://attacker.example", "application/x-www-form-urlencoded", "host=attacker.example&user=example", 403},
|
||||
{"foreign setup test", "POST", "/setup/test", "", "https://attacker.example", "application/x-www-form-urlencoded", "host=attacker.example&user=example", 403},
|
||||
{"rebind settings read", "GET", "/setup/status", "attacker.example:8080", "", "", "", 403},
|
||||
{"opaque origin", "PUT", "/api/settings/ui", "", "null", "application/json", `{"indicator_mode":"accessible"}`, 403},
|
||||
{"plain JSON", "PUT", "/api/settings/ui", "", "", "text/plain", `{"indicator_mode":"accessible"}`, 415},
|
||||
{"saved password destination save", "POST", "/setup", "", "", "application/x-www-form-urlencoded", "host=attacker.example&user=example", 422},
|
||||
{"saved password destination test", "POST", "/setup/test", "", "", "application/x-www-form-urlencoded", "host=attacker.example&user=example", 422},
|
||||
{"local read", "GET", "/api/settings/ui", "", "http://127.0.0.1:8080", "", "", 200},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
req := httptest.NewRequest(tt.method, "http://127.0.0.1:8080"+tt.path, strings.NewReader(tt.body))
|
||||
if tt.host != "" {
|
||||
req.Host = tt.host
|
||||
}
|
||||
req.Header.Set("Origin", tt.origin)
|
||||
req.Header.Set("Content-Type", tt.contentType)
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
if w.Code != tt.want {
|
||||
t.Fatalf("status=%d want=%d body=%s", w.Code, tt.want, w.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
after, err := local.GetSettings()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(before, after) || projects != local.CountProjects() || configs != local.CountConfigurations() || pending != local.CountPendingChanges() {
|
||||
t.Fatal("rejected requests changed local state")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"context"
|
||||
"flag"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/localdb"
|
||||
)
|
||||
|
||||
func TestResetCLIHelper(t *testing.T) {
|
||||
if os.Getenv("QFS_RESET_CLI_HELPER") != "1" {
|
||||
return
|
||||
}
|
||||
for i, arg := range os.Args {
|
||||
if arg == "--" {
|
||||
os.Args = append([]string{"qfs"}, os.Args[i+1:]...)
|
||||
flag.CommandLine = flag.NewFlagSet("qfs", flag.ExitOnError)
|
||||
main()
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatal("missing CLI arguments")
|
||||
}
|
||||
|
||||
func TestResetCLIAndRestore(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("QFS_STATE_DIR", dir)
|
||||
t.Setenv("QFS_BACKUP_DISABLE", "1")
|
||||
t.Setenv("QFS_RESET_LOCAL_DB", "yes")
|
||||
t.Setenv("QFS_BACKUP_DIR", filepath.Join(dir, "backups"))
|
||||
dbPath := filepath.Join(dir, "qfs.db")
|
||||
configPath := filepath.Join(dir, "config.yaml")
|
||||
if err := os.WriteFile(configPath, []byte("server:\n host: 127.0.0.1\n"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
local, err := localdb.New(dbPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := local.SaveSettings("127.0.0.1", 1, "example", "example", "example"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := local.AddPendingChange("configuration", "example", "update", "{}"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := local.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
run := func(args ...string) string {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
argv := append([]string{"-test.run=^TestResetCLIHelper$", "--", "-localdb", dbPath, "-config", configPath, "-migrate-status"}, args...)
|
||||
cmd := exec.CommandContext(ctx, os.Args[0], argv...)
|
||||
cmd.Env = append(os.Environ(), "QFS_RESET_CLI_HELPER=1")
|
||||
output, err := cmd.CombinedOutput()
|
||||
if ctx.Err() != nil {
|
||||
t.Fatalf("CLI timed out: %s", output)
|
||||
}
|
||||
if err == nil {
|
||||
t.Fatalf("expected reset refusal or offline migration-status exit: %s", output)
|
||||
}
|
||||
return string(output)
|
||||
}
|
||||
assertPending := func(want int64) {
|
||||
t.Helper()
|
||||
local, err := localdb.New(dbPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer local.Close()
|
||||
if got := local.CountPendingChanges(); got != want {
|
||||
t.Fatalf("pending=%d want=%d", got, want)
|
||||
}
|
||||
if !local.HasSettings() {
|
||||
t.Fatal("connection settings lost")
|
||||
}
|
||||
}
|
||||
if output := run(); !strings.Contains(output, "QFS_RESET_LOCAL_DB is ignored") {
|
||||
t.Fatalf("environment reset not diagnosed: %s", output)
|
||||
}
|
||||
assertPending(1)
|
||||
if output := run("-reset-localdb"); !strings.Contains(output, "reset refused") {
|
||||
t.Fatalf("pending reset not refused: %s", output)
|
||||
}
|
||||
assertPending(1)
|
||||
if output := run("-reset-localdb", "-force-reset-localdb"); !strings.Contains(output, "verified pre-reset backup") {
|
||||
t.Fatalf("missing safety backup: %s", output)
|
||||
}
|
||||
assertPending(0)
|
||||
archives, err := filepath.Glob(filepath.Join(dir, "backups", "pre-reset", "*.zip"))
|
||||
if err != nil || len(archives) != 1 {
|
||||
t.Fatalf("archives=%v err=%v", archives, err)
|
||||
}
|
||||
archive, err := zip.OpenReader(archives[0])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer archive.Close()
|
||||
for _, entry := range archive.File {
|
||||
if entry.Name != "qfs.db" && entry.Name != "config.yaml" {
|
||||
t.Fatalf("unexpected archive entry: %s", entry.Name)
|
||||
}
|
||||
in, err := entry.Open()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, readErr := io.ReadAll(in)
|
||||
closeErr := in.Close()
|
||||
if readErr != nil || closeErr != nil {
|
||||
t.Fatalf("restore entry: read=%v close=%v", readErr, closeErr)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, entry.Name), data, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
assertPending(1)
|
||||
if output := run(); !strings.Contains(output, "cannot run migrations: database not available") {
|
||||
t.Fatalf("restored startup did not reach migration status: %s", output)
|
||||
}
|
||||
assertPending(1)
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
qfassets "git.mchus.pro/mchus/quoteforge"
|
||||
"git.mchus.pro/mchus/quoteforge/internal/handlers"
|
||||
"git.mchus.pro/mchus/quoteforge/internal/localdb"
|
||||
"git.mchus.pro/mchus/quoteforge/internal/middleware"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
func setupSetupRouter(local *localdb.LocalDB, restartSig chan struct{}) (*gin.Engine, error) {
|
||||
// In setup mode, we don't have a connection manager yet (will restart after setup)
|
||||
templatesPath := filepath.Join("web", "templates")
|
||||
setupHandler, err := handlers.NewSetupHandler(local, nil, templatesPath, restartSig)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create setup handler: %w", err)
|
||||
}
|
||||
|
||||
gin.SetMode(gin.ReleaseMode)
|
||||
router := gin.New()
|
||||
router.Use(gin.Recovery())
|
||||
router.Use(middleware.CORS())
|
||||
|
||||
if staticFS, err := qfassets.StaticFS(); err == nil {
|
||||
router.StaticFS("/static", http.FS(staticFS))
|
||||
} else {
|
||||
return nil, fmt.Errorf("load embedded static assets: %w", err)
|
||||
}
|
||||
|
||||
// Setup routes only
|
||||
router.GET("/", func(c *gin.Context) {
|
||||
c.Redirect(http.StatusFound, "/setup")
|
||||
})
|
||||
router.GET("/setup", setupHandler.ShowSetup)
|
||||
router.POST("/setup", setupHandler.SaveConnection)
|
||||
router.POST("/setup/test", setupHandler.TestConnection)
|
||||
router.GET("/setup/status", setupHandler.GetStatus)
|
||||
|
||||
settingsHandler := handlers.NewSettingsHandler(local)
|
||||
router.GET("/api/settings/ui", settingsHandler.GetUI)
|
||||
router.PUT("/api/settings/ui", settingsHandler.PutUI)
|
||||
|
||||
// Health check
|
||||
router.GET("/health", func(c *gin.Context) {
|
||||
c.JSON(http.StatusOK, gin.H{
|
||||
"status": "setup_required",
|
||||
"time": time.Now().UTC().Format(time.RFC3339),
|
||||
})
|
||||
})
|
||||
|
||||
return router, nil
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
package appstate
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// ServerMigrationBackupDir is separate from local backup retention.
|
||||
func ServerMigrationBackupDir(dbPath string) (string, error) {
|
||||
root := filepath.Join(resolveBackupRoot(dbPath), "server-migrations")
|
||||
if err := ValidateBackupLocation(root); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := os.MkdirAll(root, 0700); err != nil {
|
||||
return "", fmt.Errorf("create server backup directory: %w", err)
|
||||
}
|
||||
resolved, err := filepath.EvalSymlinks(root)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("resolve server backup directory: %w", err)
|
||||
}
|
||||
if err := ValidateBackupLocation(resolved); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
// ValidateBackupLocation checks the destination, including existing symlinks.
|
||||
func ValidateBackupLocation(path string) error {
|
||||
if err := validateBackupRoot(path); err != nil {
|
||||
return err
|
||||
}
|
||||
existing := path
|
||||
for {
|
||||
if _, err := os.Lstat(existing); err == nil {
|
||||
break
|
||||
} else if !os.IsNotExist(err) {
|
||||
return fmt.Errorf("inspect backup path: %w", err)
|
||||
}
|
||||
parent := filepath.Dir(existing)
|
||||
if parent == existing {
|
||||
return fmt.Errorf("backup path has no existing parent")
|
||||
}
|
||||
existing = parent
|
||||
}
|
||||
resolved, err := filepath.EvalSymlinks(existing)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resolve backup path: %w", err)
|
||||
}
|
||||
return validateBackupRoot(resolved)
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package appstate
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestServerBackupRejectsGitAndSymlinks(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
repo := filepath.Join(root, "repo")
|
||||
if err := os.MkdirAll(filepath.Join(repo, ".git"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
link := filepath.Join(root, "link")
|
||||
if err := os.Symlink(repo, link); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, path := range []string{filepath.Join(repo, "backups"), filepath.Join(link, "backups")} {
|
||||
t.Setenv("QFS_BACKUP_DIR", path)
|
||||
if _, err := ServerMigrationBackupDir(filepath.Join(root, "state", "qfs.db")); err == nil {
|
||||
t.Fatalf("accepted repository backup: %s", path)
|
||||
}
|
||||
}
|
||||
t.Setenv("QFS_BACKUP_DIR", filepath.Join(root, "backups"))
|
||||
if _, err := ServerMigrationBackupDir(filepath.Join(root, "state", "qfs.db")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
+67
-18
@@ -22,9 +22,10 @@ const (
|
||||
// goroutines must never pay the MySQL dial/read timeout themselves.
|
||||
defaultStatusCheckInterval = 15 * time.Second
|
||||
|
||||
maxOpenConns = 10
|
||||
maxOpenConns = 2
|
||||
maxIdleConns = 2
|
||||
connMaxLifetime = 5 * time.Minute
|
||||
connMaxIdleTime = time.Minute
|
||||
)
|
||||
|
||||
// ConnectionStatus represents the current status of the database connection
|
||||
@@ -49,11 +50,13 @@ type ConnectionManager struct {
|
||||
|
||||
statusCheckInterval time.Duration // background prober cadence (default: 15s)
|
||||
stopStatusLoop chan struct{} // closed by Stop() to end the background loop
|
||||
|
||||
dialFn func() (*gorm.DB, error) // opens a new pool; cm.dial unless overridden in tests
|
||||
}
|
||||
|
||||
// NewConnectionManager creates a new ConnectionManager instance
|
||||
func NewConnectionManager(localDB *localdb.LocalDB) *ConnectionManager {
|
||||
return &ConnectionManager{
|
||||
cm := &ConnectionManager{
|
||||
localDB: localDB,
|
||||
connectTimeout: defaultConnectTimeout,
|
||||
pingInterval: defaultPingInterval,
|
||||
@@ -63,6 +66,8 @@ func NewConnectionManager(localDB *localdb.LocalDB) *ConnectionManager {
|
||||
lastError: nil,
|
||||
lastCheck: time.Time{},
|
||||
}
|
||||
cm.dialFn = cm.dial
|
||||
return cm
|
||||
}
|
||||
|
||||
// Start launches a background goroutine that keeps the online-status cache
|
||||
@@ -137,25 +142,66 @@ func (cm *ConnectionManager) GetDB() (*gorm.DB, error) {
|
||||
return db, err
|
||||
}
|
||||
|
||||
newDB, err := cm.dial()
|
||||
// The cached pool is only stale, not broken: ping it instead of dialing a
|
||||
// new one. Re-dialing here used to orphan the old *sql.DB together with its
|
||||
// idle connections, leaking a pool every pingInterval during sync.
|
||||
cm.mu.RLock()
|
||||
existing := cm.db
|
||||
cm.mu.RUnlock()
|
||||
if existing != nil {
|
||||
if err := cm.ping(existing); err == nil {
|
||||
cm.mu.Lock()
|
||||
cm.lastError = nil
|
||||
cm.lastCheck = time.Now()
|
||||
cm.mu.Unlock()
|
||||
return existing, nil
|
||||
}
|
||||
}
|
||||
|
||||
newDB, err := cm.dialFn()
|
||||
|
||||
cm.mu.Lock()
|
||||
old := cm.db
|
||||
if err != nil {
|
||||
// Drop stale handle so callers don't treat it as an active connection.
|
||||
cm.db = nil
|
||||
cm.lastError = err
|
||||
cm.lastCheck = time.Now()
|
||||
cm.mu.Unlock()
|
||||
closeDB(old)
|
||||
return nil, err
|
||||
}
|
||||
cm.db = newDB
|
||||
cm.lastError = nil
|
||||
cm.lastCheck = time.Now()
|
||||
cm.mu.Unlock()
|
||||
closeDB(old)
|
||||
|
||||
return newDB, nil
|
||||
}
|
||||
|
||||
// ping checks an existing pool with the connect timeout. No locks held.
|
||||
func (cm *ConnectionManager) ping(db *gorm.DB) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), cm.connectTimeout)
|
||||
defer cancel()
|
||||
|
||||
sqlDB, err := db.DB()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return sqlDB.PingContext(ctx)
|
||||
}
|
||||
|
||||
// closeDB releases a pool that is no longer referenced by the manager.
|
||||
func closeDB(db *gorm.DB) {
|
||||
if db == nil {
|
||||
return
|
||||
}
|
||||
if sqlDB, err := db.DB(); err == nil {
|
||||
sqlDB.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// cachedResult returns (db, err, true) if the cached state is still fresh
|
||||
// enough to answer without a new network round-trip: either a live
|
||||
// connection within pingInterval, or a recent failure still within
|
||||
@@ -200,15 +246,17 @@ func (cm *ConnectionManager) dial() (*gorm.DB, error) {
|
||||
return nil, fmt.Errorf("getting sql.DB: %w", err)
|
||||
}
|
||||
|
||||
// Ping with timeout
|
||||
if err = sqlDB.PingContext(ctx); err != nil {
|
||||
return nil, fmt.Errorf("pinging database: %w", err)
|
||||
}
|
||||
|
||||
// Set connection pool settings
|
||||
sqlDB.SetMaxOpenConns(maxOpenConns)
|
||||
sqlDB.SetMaxIdleConns(maxIdleConns)
|
||||
sqlDB.SetConnMaxLifetime(connMaxLifetime)
|
||||
sqlDB.SetConnMaxIdleTime(connMaxIdleTime)
|
||||
|
||||
// Ping with timeout
|
||||
if err = sqlDB.PingContext(ctx); err != nil {
|
||||
sqlDB.Close()
|
||||
return nil, fmt.Errorf("pinging database: %w", err)
|
||||
}
|
||||
|
||||
return db, nil
|
||||
}
|
||||
@@ -264,20 +312,17 @@ func (cm *ConnectionManager) checkOnlineNow() bool {
|
||||
}
|
||||
|
||||
// Perform ping with timeout — no locks held here.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), cm.connectTimeout)
|
||||
defer cancel()
|
||||
|
||||
sqlDB, err := db.DB()
|
||||
if err == nil {
|
||||
err = sqlDB.PingContext(ctx)
|
||||
}
|
||||
err := cm.ping(db)
|
||||
|
||||
cm.mu.Lock()
|
||||
defer cm.mu.Unlock()
|
||||
if err != nil {
|
||||
cm.lastError = err
|
||||
cm.lastCheck = time.Now()
|
||||
cm.db = nil
|
||||
if cm.db == db {
|
||||
cm.db = nil
|
||||
}
|
||||
closeDB(db)
|
||||
return false
|
||||
}
|
||||
cm.lastCheck = time.Now()
|
||||
@@ -292,19 +337,23 @@ func (cm *ConnectionManager) TryConnect() error {
|
||||
cm.connMu.Lock()
|
||||
defer cm.connMu.Unlock()
|
||||
|
||||
newDB, err := cm.dial()
|
||||
newDB, err := cm.dialFn()
|
||||
|
||||
cm.mu.Lock()
|
||||
defer cm.mu.Unlock()
|
||||
old := cm.db
|
||||
if err != nil {
|
||||
cm.db = nil
|
||||
cm.lastError = err
|
||||
cm.lastCheck = time.Now()
|
||||
cm.mu.Unlock()
|
||||
closeDB(old)
|
||||
return err
|
||||
}
|
||||
cm.db = newDB
|
||||
cm.lastError = nil
|
||||
cm.lastCheck = time.Now()
|
||||
cm.mu.Unlock()
|
||||
closeDB(old)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/localdb"
|
||||
"github.com/glebarez/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
func newTestConnectionManager(t *testing.T) (*ConnectionManager, *int) {
|
||||
t.Helper()
|
||||
local, err := localdb.New(filepath.Join(t.TempDir(), "qfs.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("open local db: %v", err)
|
||||
}
|
||||
cm := NewConnectionManager(local)
|
||||
dials := 0
|
||||
cm.dialFn = func() (*gorm.DB, error) {
|
||||
dials++
|
||||
return gorm.Open(sqlite.Open(filepath.Join(t.TempDir(), "remote.db")), &gorm.Config{
|
||||
Logger: logger.Default.LogMode(logger.Silent),
|
||||
})
|
||||
}
|
||||
return cm, &dials
|
||||
}
|
||||
|
||||
func TestGetDBReusesStalePoolInsteadOfRedialing(t *testing.T) {
|
||||
cm, dials := newTestConnectionManager(t)
|
||||
|
||||
first, err := cm.GetDB()
|
||||
if err != nil {
|
||||
t.Fatalf("first GetDB: %v", err)
|
||||
}
|
||||
|
||||
// Simulate pingInterval elapsing between sync steps.
|
||||
cm.mu.Lock()
|
||||
cm.lastCheck = time.Now().Add(-2 * cm.pingInterval)
|
||||
cm.mu.Unlock()
|
||||
|
||||
second, err := cm.GetDB()
|
||||
if err != nil {
|
||||
t.Fatalf("second GetDB: %v", err)
|
||||
}
|
||||
if second != first {
|
||||
t.Fatalf("expected stale but healthy pool to be reused")
|
||||
}
|
||||
if *dials != 1 {
|
||||
t.Fatalf("expected 1 dial, got %d", *dials)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTryConnectClosesReplacedPool(t *testing.T) {
|
||||
cm, _ := newTestConnectionManager(t)
|
||||
|
||||
first, err := cm.GetDB()
|
||||
if err != nil {
|
||||
t.Fatalf("GetDB: %v", err)
|
||||
}
|
||||
if err := cm.TryConnect(); err != nil {
|
||||
t.Fatalf("TryConnect: %v", err)
|
||||
}
|
||||
|
||||
sqlDB, err := first.DB()
|
||||
if err != nil {
|
||||
t.Fatalf("sql.DB: %v", err)
|
||||
}
|
||||
if err := sqlDB.Ping(); err == nil {
|
||||
t.Fatalf("expected replaced pool to be closed")
|
||||
}
|
||||
}
|
||||
@@ -43,7 +43,7 @@ func NewSyncHandler(localDB *localdb.LocalDB, syncService *sync.Service, connMgr
|
||||
syncService: syncService,
|
||||
connMgr: connMgr,
|
||||
autoSyncInterval: autoSyncInterval,
|
||||
onlineGraceFactor: 1.10,
|
||||
onlineGraceFactor: 1 + sync.WorkerJitter, // longest gap between jittered sync cycles
|
||||
tmpl: tmpl,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
package localdb
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// serverPullCursor is the server-side NOW(6) taken before the last successful
|
||||
// incremental pull of a table, bound to the server it was taken from.
|
||||
type serverPullCursor struct {
|
||||
Scope string `json:"scope"`
|
||||
ServerTime string `json:"server_time"`
|
||||
}
|
||||
|
||||
func serverPullCursorKey(table string) string {
|
||||
return "server_pull_cursor:" + table
|
||||
}
|
||||
|
||||
// ServerPullScope identifies the MariaDB server+database a pull cursor belongs to,
|
||||
// so a cursor never survives a switch to another server.
|
||||
func (l *LocalDB) ServerPullScope() (string, error) {
|
||||
settings, err := l.GetSettings()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return fmt.Sprintf("%s:%d/%s", settings.Host, settings.Port, settings.Database), nil
|
||||
}
|
||||
|
||||
// GetServerPullCursor returns the stored server time for table if it was taken
|
||||
// on the server identified by scope.
|
||||
func (l *LocalDB) GetServerPullCursor(table, scope string) (string, bool) {
|
||||
raw, ok := l.getAppSettingValue(serverPullCursorKey(table))
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
var cursor serverPullCursor
|
||||
if err := json.Unmarshal([]byte(raw), &cursor); err != nil || cursor.Scope != scope || cursor.ServerTime == "" {
|
||||
return "", false
|
||||
}
|
||||
return cursor.ServerTime, true
|
||||
}
|
||||
|
||||
// SetServerPullCursor stores the server time a pull of table started at.
|
||||
func (l *LocalDB) SetServerPullCursor(table, scope, serverTime string) error {
|
||||
raw, err := json.Marshal(serverPullCursor{Scope: scope, ServerTime: serverTime})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return l.upsertAppSetting(l.db, serverPullCursorKey(table), string(raw), time.Now())
|
||||
}
|
||||
@@ -72,6 +72,11 @@ func (l *LocalDB) SyncQtSettings(mariaDB *gorm.DB) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
var current []LocalQtSetting
|
||||
if err := l.db.Select("name, value").Find(¤t).Error; err == nil && sameQtSettings(current, rows) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return l.db.Transaction(func(tx *gorm.DB) error {
|
||||
if err := tx.Exec("DELETE FROM local_qt_settings").Error; err != nil {
|
||||
return fmt.Errorf("clearing local_qt_settings: %w", err)
|
||||
@@ -142,3 +147,19 @@ func (l *LocalDB) GetConfiguratorSettings() (*ConfiguratorSettings, error) {
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func sameQtSettings(a, b []LocalQtSetting) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
values := make(map[string]string, len(a))
|
||||
for _, row := range a {
|
||||
values[row.Name] = row.Value
|
||||
}
|
||||
for _, row := range b {
|
||||
if v, ok := values[row.Name]; !ok || v != row.Value {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,185 @@
|
||||
package mariadb
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/appstate"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
type snapshotTable struct {
|
||||
Name, Create, Entry, Digest string
|
||||
Columns []string
|
||||
Rows int64
|
||||
}
|
||||
|
||||
type snapshotManifest struct {
|
||||
Format int
|
||||
Schema, Collation string
|
||||
Tables []snapshotTable
|
||||
Triggers []snapshotTrigger
|
||||
}
|
||||
|
||||
func freshDB(db *gorm.DB) *gorm.DB { return db.Session(&gorm.Session{NewDB: true}) }
|
||||
func quoteIdentifier(s string) string { return "`" + strings.ReplaceAll(s, "`", "``") + "`" }
|
||||
|
||||
// CreateSnapshot writes a consistent InnoDB snapshot, without external DB tools.
|
||||
// The caller supplies a dedicated connection and excludes concurrent writers
|
||||
// for the entire migration session, including any recovery.
|
||||
func CreateSnapshot(db *gorm.DB, root, label string) (string, error) {
|
||||
if err := appstate.ValidateBackupLocation(root); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if filepath.Base(label) != label {
|
||||
return "", fmt.Errorf("invalid snapshot label")
|
||||
}
|
||||
if err := os.MkdirAll(root, 0700); err != nil {
|
||||
return "", fmt.Errorf("create snapshot directory: %w", err)
|
||||
}
|
||||
if err := appstate.ValidateBackupLocation(root); err != nil {
|
||||
return "", err
|
||||
}
|
||||
file, err := os.CreateTemp(root, "qfs-mariadb-v1-"+label+"-"+time.Now().UTC().Format("20060102T150405Z")+"-*.zip")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create snapshot: %w", err)
|
||||
}
|
||||
path := file.Name()
|
||||
archive := zip.NewWriter(file)
|
||||
err = freshDB(db).Transaction(func(tx *gorm.DB) error {
|
||||
manifest, err := snapshotInventory(tx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i := range manifest.Tables {
|
||||
table := &manifest.Tables[i]
|
||||
table.Entry = fmt.Sprintf("tables/%06d.jsonl", i)
|
||||
out, err := archive.Create(table.Entry)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create table entry %s: %w", table.Name, err)
|
||||
}
|
||||
if err := snapshotRows(tx, table, out); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
out, err := archive.Create("manifest.json")
|
||||
if err != nil {
|
||||
return fmt.Errorf("create snapshot manifest: %w", err)
|
||||
}
|
||||
if err := json.NewEncoder(out).Encode(manifest); err != nil {
|
||||
return fmt.Errorf("write snapshot manifest: %w", err)
|
||||
}
|
||||
return nil
|
||||
}, &sql.TxOptions{Isolation: sql.LevelRepeatableRead, ReadOnly: true})
|
||||
zipErr := archive.Close()
|
||||
syncErr := file.Sync()
|
||||
closeErr := file.Close()
|
||||
for _, candidate := range []error{err, zipErr, syncErr, closeErr} {
|
||||
if candidate != nil {
|
||||
return "", fmt.Errorf("write snapshot %s: %w", path, candidate)
|
||||
}
|
||||
}
|
||||
if _, err := readSnapshot(path, nil); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
func snapshotInventory(db *gorm.DB) (*snapshotManifest, error) {
|
||||
m := &snapshotManifest{Format: 1}
|
||||
if err := freshDB(db).Raw("SELECT DATABASE(), @@collation_database").Row().Scan(&m.Schema, &m.Collation); err != nil {
|
||||
return nil, fmt.Errorf("read snapshot schema: %w", err)
|
||||
}
|
||||
var unsupported int64
|
||||
query := `SELECT
|
||||
(SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA=? AND (TABLE_TYPE<>'BASE TABLE' OR ENGINE<>'InnoDB')) +
|
||||
(SELECT COUNT(*) FROM information_schema.ROUTINES WHERE ROUTINE_SCHEMA=?) +
|
||||
(SELECT COUNT(*) FROM information_schema.EVENTS WHERE EVENT_SCHEMA=?) +
|
||||
(SELECT COUNT(*) FROM information_schema.KEY_COLUMN_USAGE WHERE
|
||||
(TABLE_SCHEMA=? AND REFERENCED_TABLE_SCHEMA<>?) OR (REFERENCED_TABLE_SCHEMA=? AND TABLE_SCHEMA<>?))`
|
||||
if err := freshDB(db).Raw(query, m.Schema, m.Schema, m.Schema, m.Schema, m.Schema, m.Schema, m.Schema).Scan(&unsupported).Error; err != nil {
|
||||
return nil, fmt.Errorf("check snapshot support: %w", err)
|
||||
}
|
||||
if unsupported != 0 {
|
||||
return nil, fmt.Errorf("snapshot refused: views, non-InnoDB tables, routines, events or cross-schema foreign keys require supported recovery before migration")
|
||||
}
|
||||
var names []string
|
||||
if err := freshDB(db).Raw("SELECT TABLE_NAME FROM information_schema.TABLES WHERE TABLE_SCHEMA=? ORDER BY TABLE_NAME", m.Schema).Scan(&names).Error; err != nil {
|
||||
return nil, fmt.Errorf("list snapshot tables: %w", err)
|
||||
}
|
||||
for _, name := range names {
|
||||
table := snapshotTable{Name: name}
|
||||
var ignored string
|
||||
if err := freshDB(db).Raw("SHOW CREATE TABLE "+quoteIdentifier(name)).Row().Scan(&ignored, &table.Create); err != nil {
|
||||
return nil, fmt.Errorf("read schema for %s: %w", name, err)
|
||||
}
|
||||
if err := freshDB(db).Raw(`SELECT COLUMN_NAME FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=? AND TABLE_NAME=? AND EXTRA NOT LIKE '%VIRTUAL GENERATED%' AND EXTRA NOT LIKE '%STORED GENERATED%' ORDER BY ORDINAL_POSITION`, m.Schema, name).Scan(&table.Columns).Error; err != nil {
|
||||
return nil, fmt.Errorf("read columns for %s: %w", name, err)
|
||||
}
|
||||
if len(table.Columns) == 0 {
|
||||
return nil, fmt.Errorf("snapshot table %s has no writable columns", name)
|
||||
}
|
||||
m.Tables = append(m.Tables, table)
|
||||
}
|
||||
if err := readSnapshotTriggers(db, m); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func snapshotRows(db *gorm.DB, table *snapshotTable, out io.Writer) error {
|
||||
columns := make([]string, len(table.Columns))
|
||||
for i, col := range table.Columns {
|
||||
columns[i] = "CAST(" + quoteIdentifier(col) + " AS BINARY)"
|
||||
}
|
||||
rows, err := freshDB(db).Raw("SELECT " + strings.Join(columns, ",") + " FROM " + quoteIdentifier(table.Name)).Rows()
|
||||
if err != nil {
|
||||
return fmt.Errorf("read snapshot table %s: %w", table.Name, err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var digest [32]byte
|
||||
encoder := json.NewEncoder(out)
|
||||
table.Rows = 0
|
||||
for rows.Next() {
|
||||
values := make([]sql.RawBytes, len(columns))
|
||||
args := make([]any, len(columns))
|
||||
for i := range args {
|
||||
args[i] = &values[i]
|
||||
}
|
||||
if err := rows.Scan(args...); err != nil {
|
||||
return fmt.Errorf("read row in %s: %w", table.Name, err)
|
||||
}
|
||||
if err := encoder.Encode(values); err != nil {
|
||||
return fmt.Errorf("write row in %s: %w", table.Name, err)
|
||||
}
|
||||
addRowDigest(&digest, values)
|
||||
table.Rows++
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return fmt.Errorf("iterate %s: %w", table.Name, err)
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return fmt.Errorf("close rows in %s: %w", table.Name, err)
|
||||
}
|
||||
table.Digest = hex.EncodeToString(digest[:])
|
||||
return nil
|
||||
}
|
||||
|
||||
func addRowDigest(sum *[32]byte, row []sql.RawBytes) {
|
||||
encoded, _ := json.Marshal(row) // Raw bytes and nil always marshal successfully.
|
||||
hash := sha256.Sum256(encoded)
|
||||
carry := uint16(0)
|
||||
for i := len(sum) - 1; i >= 0; i-- {
|
||||
value := uint16(sum[i]) + uint16(hash[i]) + carry
|
||||
sum[i], carry = byte(value), value>>8
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package mariadb
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/appstate"
|
||||
)
|
||||
|
||||
func readSnapshot(path string, visit func(snapshotTable, []sql.RawBytes) error) (*snapshotManifest, error) {
|
||||
if err := appstate.ValidateBackupLocation(path); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
archive, err := zip.OpenReader(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open server snapshot: %w", err)
|
||||
}
|
||||
defer archive.Close()
|
||||
files := map[string]*zip.File{}
|
||||
for _, f := range archive.File {
|
||||
if files[f.Name] != nil {
|
||||
return nil, fmt.Errorf("duplicate snapshot entry")
|
||||
}
|
||||
files[f.Name] = f
|
||||
}
|
||||
manifestFile := files["manifest.json"]
|
||||
if manifestFile == nil || manifestFile.UncompressedSize64 > 16<<20 {
|
||||
return nil, fmt.Errorf("missing or oversized snapshot manifest")
|
||||
}
|
||||
in, err := manifestFile.Open()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open server manifest: %w", err)
|
||||
}
|
||||
data, readErr := io.ReadAll(in)
|
||||
closeErr := in.Close()
|
||||
if readErr != nil || closeErr != nil {
|
||||
return nil, fmt.Errorf("read server manifest: %w", errors.Join(readErr, closeErr))
|
||||
}
|
||||
var m snapshotManifest
|
||||
if err := json.Unmarshal(data, &m); err != nil {
|
||||
return nil, fmt.Errorf("parse server manifest: %w", err)
|
||||
}
|
||||
if m.Format != 1 || m.Schema == "" || len(files) != len(m.Tables)+1 {
|
||||
return nil, fmt.Errorf("invalid server snapshot manifest")
|
||||
}
|
||||
names, entries := map[string]bool{}, map[string]bool{}
|
||||
for _, table := range m.Tables {
|
||||
if table.Name == "" || names[table.Name] || entries[table.Entry] || files[table.Entry] == nil || len(table.Columns) == 0 || table.Rows < 0 || !strings.HasPrefix(table.Create, "CREATE TABLE "+quoteIdentifier(table.Name)+" (") {
|
||||
return nil, fmt.Errorf("invalid snapshot table metadata")
|
||||
}
|
||||
names[table.Name], entries[table.Entry] = true, true
|
||||
if err := readSnapshotTable(files[table.Entry], table, visit); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
func readSnapshotTable(file *zip.File, table snapshotTable, visit func(snapshotTable, []sql.RawBytes) error) error {
|
||||
in, err := file.Open()
|
||||
if err != nil {
|
||||
return fmt.Errorf("open snapshot table %s: %w", table.Name, err)
|
||||
}
|
||||
defer in.Close()
|
||||
decoder := json.NewDecoder(in)
|
||||
var rows int64
|
||||
var digest [32]byte
|
||||
for {
|
||||
var values []sql.RawBytes
|
||||
err := decoder.Decode(&values)
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("read snapshot row in %s: %w", table.Name, err)
|
||||
}
|
||||
if len(values) != len(table.Columns) {
|
||||
return fmt.Errorf("snapshot column count mismatch in %s", table.Name)
|
||||
}
|
||||
addRowDigest(&digest, values)
|
||||
rows++
|
||||
if visit != nil {
|
||||
if err := visit(table, values); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := in.Close(); err != nil {
|
||||
return fmt.Errorf("close snapshot table %s: %w", table.Name, err)
|
||||
}
|
||||
if rows != table.Rows || hex.EncodeToString(digest[:]) != table.Digest {
|
||||
return fmt.Errorf("snapshot checksum mismatch in %s", table.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package mariadb
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// RestoreSnapshot restores and checks a snapshot on a dedicated connection.
|
||||
// The same maintenance exclusion as migration is required throughout recovery.
|
||||
func RestoreSnapshot(db *gorm.DB, path string) error {
|
||||
m, err := readSnapshot(path, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var schema string
|
||||
if err := freshDB(db).Raw("SELECT DATABASE()").Scan(&schema).Error; err != nil {
|
||||
return fmt.Errorf("read restore destination: %w", err)
|
||||
}
|
||||
if schema != m.Schema {
|
||||
return fmt.Errorf("snapshot schema does not match restore destination")
|
||||
}
|
||||
if err := restoreSnapshotTables(db, path, m); err != nil {
|
||||
return err
|
||||
}
|
||||
return compareSnapshot(db, m)
|
||||
}
|
||||
|
||||
func restoreSnapshotTables(db *gorm.DB, path string, m *snapshotManifest) (err error) {
|
||||
current, err := snapshotInventory(db)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var foreignKeys int
|
||||
var sqlMode string
|
||||
if err := freshDB(db).Raw("SELECT @@foreign_key_checks, @@sql_mode").Row().Scan(&foreignKeys, &sqlMode); err != nil {
|
||||
return fmt.Errorf("read restore session policy: %w", err)
|
||||
}
|
||||
if err := freshDB(db).Exec("SET FOREIGN_KEY_CHECKS=0").Error; err != nil {
|
||||
return fmt.Errorf("disable FK checks for restore: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
if restoreErr := freshDB(db).Exec("SET sql_mode=?", sqlMode).Error; restoreErr != nil {
|
||||
err = errors.Join(err, fmt.Errorf("restore SQL mode: %w", restoreErr))
|
||||
}
|
||||
if restoreErr := freshDB(db).Exec("SET FOREIGN_KEY_CHECKS=?", foreignKeys).Error; restoreErr != nil {
|
||||
err = errors.Join(err, fmt.Errorf("restore FK policy: %w", restoreErr))
|
||||
}
|
||||
}()
|
||||
// Preserve explicit zero auto-increment IDs and legacy zero dates. Exact
|
||||
// schema/data comparison detects any coercion before the source is changed.
|
||||
if err := freshDB(db).Exec("SET sql_mode='NO_AUTO_VALUE_ON_ZERO'").Error; err != nil {
|
||||
return fmt.Errorf("set snapshot insert mode: %w", err)
|
||||
}
|
||||
for i := len(current.Tables) - 1; i >= 0; i-- {
|
||||
if err := freshDB(db).Exec("DROP TABLE " + quoteIdentifier(current.Tables[i].Name)).Error; err != nil {
|
||||
return fmt.Errorf("drop restore table %s: %w", current.Tables[i].Name, err)
|
||||
}
|
||||
}
|
||||
for _, table := range m.Tables {
|
||||
if err := freshDB(db).Exec(table.Create).Error; err != nil {
|
||||
return fmt.Errorf("restore schema for %s: %w", table.Name, err)
|
||||
}
|
||||
}
|
||||
var prefix, rowSQL, tableName string
|
||||
var args []any
|
||||
var groups []string
|
||||
var bytes int
|
||||
flush := func() error {
|
||||
if len(groups) == 0 {
|
||||
return nil
|
||||
}
|
||||
// Bind through database/sql: GORM expands a byte slice after '('.
|
||||
// Direct execution also keeps restored values out of SQL logs.
|
||||
if _, err := db.Statement.ConnPool.ExecContext(db.Statement.Context, prefix+strings.Join(groups, ","), args...); err != nil {
|
||||
return fmt.Errorf("restore rows in %s: %w", tableName, err)
|
||||
}
|
||||
args, groups, bytes = nil, nil, 0
|
||||
return nil
|
||||
}
|
||||
_, err = readSnapshot(path, func(table snapshotTable, values []sql.RawBytes) error {
|
||||
if tableName != table.Name {
|
||||
if err := flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
tableName = table.Name
|
||||
columns, placeholders := make([]string, len(values)), make([]string, len(values))
|
||||
for i := range values {
|
||||
columns[i], placeholders[i] = quoteIdentifier(table.Columns[i]), "?"
|
||||
}
|
||||
prefix = "INSERT INTO " + quoteIdentifier(tableName) + " (" + strings.Join(columns, ",") + ") VALUES "
|
||||
rowSQL = "(" + strings.Join(placeholders, ",") + ")"
|
||||
}
|
||||
for _, value := range values {
|
||||
if value == nil {
|
||||
args = append(args, nil)
|
||||
} else {
|
||||
args = append(args, []byte(value))
|
||||
bytes += len(value)
|
||||
}
|
||||
}
|
||||
groups = append(groups, rowSQL)
|
||||
if len(groups) >= 250 || len(args)+len(values) > 60000 || bytes >= 1<<20 {
|
||||
return flush()
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
err = flush()
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return restoreSnapshotTriggers(db, m)
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package mariadb
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSnapshotRowsAndValidation(t *testing.T) {
|
||||
row := []sql.RawBytes{[]byte{0, 1, 255}, {}, nil}
|
||||
var sum [32]byte
|
||||
addRowDigest(&sum, row)
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
format int
|
||||
rows int64
|
||||
digest string
|
||||
width int
|
||||
valid bool
|
||||
}{
|
||||
{"valid", 1, 1, hex.EncodeToString(sum[:]), 3, true},
|
||||
{"bad format", 2, 1, hex.EncodeToString(sum[:]), 3, false},
|
||||
{"bad count", 1, 2, hex.EncodeToString(sum[:]), 3, false},
|
||||
{"bad digest", 1, 1, "invalid", 3, false},
|
||||
{"bad width", 1, 1, hex.EncodeToString(sum[:]), 2, false},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "backup.zip")
|
||||
file, err := os.Create(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
archive := zip.NewWriter(file)
|
||||
entry, err := archive.Create("table.jsonl")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := json.NewEncoder(entry).Encode(row); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entry, err = archive.Create("manifest.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
table := snapshotTable{Name: "example", Create: "CREATE TABLE `example` (id INT)", Entry: "table.jsonl", Columns: []string{"a", "b", "c"}[:tt.width], Rows: tt.rows, Digest: tt.digest}
|
||||
if err := json.NewEncoder(entry).Encode(snapshotManifest{Format: tt.format, Schema: "example", Tables: []snapshotTable{table}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := archive.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := file.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err = readSnapshot(path, nil)
|
||||
if (err == nil) != tt.valid {
|
||||
t.Fatalf("valid=%v err=%v", tt.valid, err)
|
||||
}
|
||||
if !tt.valid {
|
||||
if err := RestoreSnapshot(nil, path); err == nil {
|
||||
t.Fatal("invalid archive reached restore without preflight refusal")
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSnapshotDigestPreservesValues(t *testing.T) {
|
||||
a, b := []sql.RawBytes{[]byte("a")}, []sql.RawBytes{[]byte("b")}
|
||||
var forward, reverse, null, empty [32]byte
|
||||
addRowDigest(&forward, a)
|
||||
addRowDigest(&forward, b)
|
||||
addRowDigest(&reverse, b)
|
||||
addRowDigest(&reverse, a)
|
||||
if forward != reverse {
|
||||
t.Fatal("digest depends on row order")
|
||||
}
|
||||
addRowDigest(&null, []sql.RawBytes{nil})
|
||||
addRowDigest(&empty, []sql.RawBytes{{}})
|
||||
if null == empty {
|
||||
t.Fatal("digest conflates NULL and empty bytes")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
package mariadb
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Recovery covers the selected schema, so external trigger effects are refused.
|
||||
func validateSnapshotTriggerReferences(trigger snapshotTrigger, schema string) error {
|
||||
body := trigger.Body
|
||||
quoted := `'(?:''|\\.|[^'\\])*'`
|
||||
if strings.Contains(trigger.SQLMode, "NO_BACKSLASH_ESCAPES") {
|
||||
quoted = `'(?:''|[^'])*'`
|
||||
}
|
||||
body = regexp.MustCompile(quoted).ReplaceAllString(body, " ")
|
||||
if !strings.Contains(trigger.SQLMode, "ANSI_QUOTES") {
|
||||
double := `"(?:""|\\.|[^"\\])*"`
|
||||
if strings.Contains(trigger.SQLMode, "NO_BACKSLASH_ESCAPES") {
|
||||
double = `"(?:""|[^"])*"`
|
||||
}
|
||||
body = regexp.MustCompile(double).ReplaceAllString(body, " ")
|
||||
}
|
||||
if strings.Contains(body, "/*!") || strings.Contains(body, "/*M!") {
|
||||
return fmt.Errorf("trigger %s has unsupported executable comments", trigger.Name)
|
||||
}
|
||||
body = regexp.MustCompile(`(?s)/\*.*?\*/|--[\t ][^\n]*|#[^\n]*`).ReplaceAllString(body, " ")
|
||||
if regexp.MustCompile(`(?i)\bCALL\s+`).MatchString(body) {
|
||||
return fmt.Errorf("trigger %s calls a routine outside supported recovery", trigger.Name)
|
||||
}
|
||||
identifier := "(?:`(?:``|[^`])+`|\"(?:\"\"|[^\"])+\"|[\\pL_$][\\pL\\pN_$]*)"
|
||||
for _, pattern := range []string{
|
||||
"(?i)\\b(?:FROM|JOIN|UPDATE|INTO)\\s+(?:(?:LOW_PRIORITY|HIGH_PRIORITY|DELAYED|IGNORE)\\s+)*(" + identifier + ")\\s*\\.",
|
||||
"(?i)(" + identifier + ")\\s*\\.\\s*" + identifier + "\\s*\\(",
|
||||
} {
|
||||
for _, match := range regexp.MustCompile(pattern).FindAllStringSubmatch(body, -1) {
|
||||
if !strings.EqualFold(strings.Trim(match[1], "`\""), schema) {
|
||||
return fmt.Errorf("trigger %s references another schema outside supported recovery", trigger.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package mariadb
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestSnapshotTriggerReferences(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name, body, mode string
|
||||
fail bool
|
||||
}{
|
||||
{"local", "INSERT INTO audit VALUES (NEW.id, OLD.value)", "", false},
|
||||
{"local qualified", "UPDATE `example`.`audit` SET id=NEW.id", "", false},
|
||||
{"alias", "SELECT 1 FROM audit a WHERE a.id=OLD.id", "", false},
|
||||
{"literal", "INSERT INTO audit VALUES ('example.com', '$.field', 'UPDATE other.audit')", "", false},
|
||||
{"comment", "/* UPDATE other.audit */ INSERT INTO audit VALUES (NEW.id)", "", false},
|
||||
{"external write", "UPDATE other.audit SET id=NEW.id", "", true},
|
||||
{"external insert", "INSERT INTO `other`.`audit` VALUES (NEW.id)", "", true},
|
||||
{"ANSI external", `UPDATE "other"."audit" SET id=NEW.id`, "ANSI_QUOTES", true},
|
||||
{"external function", "SET NEW.value=other.example(NEW.value)", "", true},
|
||||
{"routine", "CALL example.change_rows()", "", true},
|
||||
{"executable", "/*!50000 UPDATE other.audit SET id=1 */", "", true},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := validateSnapshotTriggerReferences(snapshotTrigger{Name: "example", Body: tt.body, SQLMode: tt.mode}, "example")
|
||||
if (err != nil) != tt.fail {
|
||||
t.Fatalf("error=%v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package mariadb
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
type snapshotTrigger struct {
|
||||
Name string `gorm:"column:TRIGGER_NAME"`
|
||||
Table string `gorm:"column:EVENT_OBJECT_TABLE"`
|
||||
Event string `gorm:"column:EVENT_MANIPULATION"`
|
||||
Timing string `gorm:"column:ACTION_TIMING"`
|
||||
Order int `gorm:"column:ACTION_ORDER"`
|
||||
Body string `gorm:"column:ACTION_STATEMENT"`
|
||||
SQLMode string `gorm:"column:SQL_MODE"`
|
||||
Definer string `gorm:"column:DEFINER"`
|
||||
Charset string `gorm:"column:CHARACTER_SET_CLIENT"`
|
||||
Collation string `gorm:"column:COLLATION_CONNECTION"`
|
||||
DatabaseCollation string `gorm:"column:DATABASE_COLLATION"`
|
||||
}
|
||||
|
||||
func readSnapshotTriggers(db *gorm.DB, m *snapshotManifest) error {
|
||||
query := `SELECT TRIGGER_NAME, EVENT_OBJECT_TABLE, EVENT_MANIPULATION, ACTION_TIMING,
|
||||
ACTION_ORDER, ACTION_STATEMENT, SQL_MODE, DEFINER, CHARACTER_SET_CLIENT,
|
||||
COLLATION_CONNECTION, DATABASE_COLLATION FROM information_schema.TRIGGERS
|
||||
WHERE TRIGGER_SCHEMA=? ORDER BY EVENT_OBJECT_TABLE, ACTION_TIMING, EVENT_MANIPULATION, ACTION_ORDER`
|
||||
if err := freshDB(db).Raw(query, m.Schema).Scan(&m.Triggers).Error; err != nil {
|
||||
return fmt.Errorf("read snapshot triggers: %w", err)
|
||||
}
|
||||
for _, trigger := range m.Triggers {
|
||||
if err := validateSnapshotTriggerReferences(trigger, m.Schema); err != nil {
|
||||
return err
|
||||
}
|
||||
if trigger.DatabaseCollation != m.Collation {
|
||||
return fmt.Errorf("trigger %s has a legacy database collation; restore support is required before migration", trigger.Name)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func restoreSnapshotTriggers(db *gorm.DB, m *snapshotManifest) (err error) {
|
||||
if len(m.Triggers) == 0 {
|
||||
return nil
|
||||
}
|
||||
var mode, charset, collation, schema string
|
||||
if err := freshDB(db).Raw("SELECT @@sql_mode, @@character_set_client, @@collation_connection, DATABASE()").Row().Scan(&mode, &charset, &collation, &schema); err != nil {
|
||||
return fmt.Errorf("read trigger restore session: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
for _, reset := range []struct{ sql, value string }{{"SET sql_mode=?", mode}, {"SET character_set_client=?", charset}, {"SET collation_connection=?", collation}} {
|
||||
if e := freshDB(db).Exec(reset.sql, reset.value).Error; e != nil {
|
||||
err = errors.Join(err, fmt.Errorf("reset trigger restore session: %w", e))
|
||||
}
|
||||
}
|
||||
}()
|
||||
for _, trigger := range m.Triggers {
|
||||
if trigger.Timing != "BEFORE" && trigger.Timing != "AFTER" {
|
||||
return fmt.Errorf("invalid trigger timing")
|
||||
}
|
||||
if trigger.Event != "INSERT" && trigger.Event != "UPDATE" && trigger.Event != "DELETE" {
|
||||
return fmt.Errorf("invalid trigger event")
|
||||
}
|
||||
at := strings.LastIndex(trigger.Definer, "@")
|
||||
if at < 1 || at == len(trigger.Definer)-1 {
|
||||
return fmt.Errorf("invalid trigger definer")
|
||||
}
|
||||
for _, setting := range []struct{ sql, value string }{{"SET sql_mode=?", trigger.SQLMode}, {"SET character_set_client=?", trigger.Charset}, {"SET collation_connection=?", trigger.Collation}} {
|
||||
if err := freshDB(db).Exec(setting.sql, setting.value).Error; err != nil {
|
||||
return fmt.Errorf("set trigger restore session: %w", err)
|
||||
}
|
||||
}
|
||||
// Explicitly bind name and target table to the destination. Trigger bodies
|
||||
// retain their original SQL; verification performs no DML after creation.
|
||||
statement := "CREATE DEFINER=" + quoteIdentifier(trigger.Definer[:at]) + "@" + quoteIdentifier(trigger.Definer[at+1:]) + " TRIGGER " + quoteIdentifier(schema) + "." + quoteIdentifier(trigger.Name) + " " + trigger.Timing + " " + trigger.Event + " ON " + quoteIdentifier(schema) + "." + quoteIdentifier(trigger.Table) + " FOR EACH ROW " + trigger.Body
|
||||
if _, err := db.Statement.ConnPool.ExecContext(db.Statement.Context, statement); err != nil {
|
||||
return fmt.Errorf("restore trigger %s: %w", trigger.Name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package mariadb
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"reflect"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// CheckSnapshot compares restored schema and all writable row values.
|
||||
func CheckSnapshot(db *gorm.DB, path string) error {
|
||||
m, err := readSnapshot(path, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return compareSnapshot(db, m)
|
||||
}
|
||||
|
||||
// VerifySnapshot proves restore readiness in an isolated temporary database.
|
||||
func VerifySnapshot(db *gorm.DB, path string) (err error) {
|
||||
m, err := readSnapshot(path, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var random [12]byte
|
||||
if _, err := rand.Read(random[:]); err != nil {
|
||||
return fmt.Errorf("name verification schema: %w", err)
|
||||
}
|
||||
scratch := "qfs_verify_" + hex.EncodeToString(random[:])
|
||||
var original string
|
||||
if err := freshDB(db).Raw("SELECT DATABASE()").Scan(&original).Error; err != nil {
|
||||
return fmt.Errorf("read original schema: %w", err)
|
||||
}
|
||||
if err := freshDB(db).Exec("CREATE DATABASE " + quoteIdentifier(scratch) + " COLLATE " + quoteIdentifier(m.Collation)).Error; err != nil {
|
||||
return fmt.Errorf("create restore verification schema: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
if e := freshDB(db).Exec("USE " + quoteIdentifier(original)).Error; e != nil {
|
||||
err = errors.Join(err, fmt.Errorf("return to original schema: %w", e))
|
||||
}
|
||||
if e := freshDB(db).Exec("DROP DATABASE " + quoteIdentifier(scratch)).Error; e != nil {
|
||||
err = errors.Join(err, fmt.Errorf("remove verification schema %s: %w", scratch, e))
|
||||
}
|
||||
}()
|
||||
if err := freshDB(db).Exec("USE " + quoteIdentifier(scratch)).Error; err != nil {
|
||||
return fmt.Errorf("select verification schema: %w", err)
|
||||
}
|
||||
if err := restoreSnapshotTables(db, path, m); err != nil {
|
||||
return err
|
||||
}
|
||||
return compareSnapshot(db, m)
|
||||
}
|
||||
|
||||
func compareSnapshot(db *gorm.DB, m *snapshotManifest) error {
|
||||
current, err := snapshotInventory(db)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !reflect.DeepEqual(current.Triggers, m.Triggers) {
|
||||
return fmt.Errorf("restored trigger metadata mismatch")
|
||||
}
|
||||
if len(current.Tables) != len(m.Tables) {
|
||||
return fmt.Errorf("restored table count mismatch")
|
||||
}
|
||||
for i := range m.Tables {
|
||||
want, got := m.Tables[i], current.Tables[i]
|
||||
if err := snapshotRows(db, &got, io.Discard); err != nil {
|
||||
return err
|
||||
}
|
||||
if want.Name != got.Name || want.Create != got.Create || want.Rows != got.Rows || want.Digest != got.Digest {
|
||||
return fmt.Errorf("restored snapshot mismatch in %s (name=%t schema=%t rows=%t digest=%t)", want.Name, want.Name == got.Name, want.Create == got.Create, want.Rows == got.Rows, want.Digest == got.Digest)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
//go:build integration
|
||||
|
||||
package mariadb
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/rsa"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"database/sql"
|
||||
"encoding/pem"
|
||||
"math/big"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
mysql "github.com/go-sql-driver/mysql"
|
||||
)
|
||||
|
||||
func TestMariaDBTLSIntegration(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.Chmod(dir, 0755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
key, err := rsa.GenerateKey(rand.Reader, 2048)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cert := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "example"},
|
||||
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour),
|
||||
IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
|
||||
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}}
|
||||
der, err := x509.CreateCertificate(rand.Reader, cert, cert, &key.PublicKey, key)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ca := string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}))
|
||||
for name, data := range map[string][]byte{"server.crt": []byte(ca), "server.key": pem.EncodeToMemory(&pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(key)})} {
|
||||
// Docker's mysql user must read these disposable test files.
|
||||
if err := os.WriteFile(filepath.Join(dir, name), data, 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
otherCA, _ := testCertificate(t)
|
||||
for _, tlsEnabled := range []bool{true, false} {
|
||||
t.Run(map[bool]string{true: "TLS", false: "plaintext"}[tlsEnabled], func(t *testing.T) {
|
||||
args := []string{"run", "--detach", "--rm", "--pull=never", "-p", "127.0.0.1::3306",
|
||||
"-e", "MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1", "-e", "MARIADB_ROOT_HOST=%", "-v", dir + ":/qfs-tls:ro", "mariadb:11.8"}
|
||||
if tlsEnabled {
|
||||
args = append(args, "--ssl-cert=/qfs-tls/server.crt", "--ssl-key=/qfs-tls/server.key")
|
||||
} else {
|
||||
args = append(args, "--skip-ssl")
|
||||
}
|
||||
out, err := exec.Command("docker", args...).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("start disposable MariaDB: %v: %s", err, out)
|
||||
}
|
||||
id := strings.TrimSpace(string(out))
|
||||
t.Cleanup(func() {
|
||||
if out, err := exec.Command("docker", "stop", "-t", "2", id).CombinedOutput(); err != nil {
|
||||
t.Errorf("stop test MariaDB: %v: %s", err, out)
|
||||
}
|
||||
})
|
||||
out, err = exec.Command("docker", "port", id, "3306/tcp").CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("test MariaDB port: %v: %s", err, out)
|
||||
}
|
||||
addr := strings.TrimSpace(string(out))
|
||||
connect := func(mode, roots, serverName string) (string, error) {
|
||||
cfg := mysql.NewConfig()
|
||||
cfg.User, cfg.Net, cfg.Addr, cfg.Timeout = "root", "tcp", addr, time.Second
|
||||
if err := ConfigureTLS(cfg, mode, roots); err != nil {
|
||||
return "", err
|
||||
}
|
||||
cfg, err = mysql.ParseDSN(cfg.FormatDSN())
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if serverName != "" {
|
||||
cfg.TLS.ServerName = serverName
|
||||
}
|
||||
connector, err := mysql.NewConnector(cfg)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
db := sql.OpenDB(connector)
|
||||
defer db.Close()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
defer cancel()
|
||||
var name, cipher string
|
||||
err = db.QueryRowContext(ctx, "SHOW SESSION STATUS LIKE 'Ssl_cipher'").Scan(&name, &cipher)
|
||||
return cipher, err
|
||||
}
|
||||
deadline := time.Now().Add(45 * time.Second)
|
||||
for {
|
||||
if _, err := connect("preferred", "", ""); err == nil {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("test MariaDB did not become ready")
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
for _, tt := range []struct {
|
||||
mode, roots, name string
|
||||
success bool
|
||||
}{
|
||||
{"preferred", "", "", true}, {"required", "", "", tlsEnabled},
|
||||
{"verify-ca", ca, "", tlsEnabled}, {"verify-full", ca, "", tlsEnabled},
|
||||
{"verify-ca", otherCA, "", false}, {"verify-full", otherCA, "", false},
|
||||
{"verify-full", ca, "wrong.example", false},
|
||||
} {
|
||||
cipher, err := connect(tt.mode, tt.roots, tt.name)
|
||||
if (err == nil) != tt.success {
|
||||
t.Errorf("mode=%s name=%s success=%v error=%v", tt.mode, tt.name, tt.success, err)
|
||||
}
|
||||
if err == nil && (cipher != "") != tlsEnabled {
|
||||
t.Errorf("mode=%s cipher=%q TLS=%v", tt.mode, cipher, tlsEnabled)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
//go:build integration
|
||||
|
||||
package models
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
qfassets "git.mchus.pro/mchus/quoteforge"
|
||||
"git.mchus.pro/mchus/quoteforge/internal/mariadb"
|
||||
"gorm.io/driver/mysql"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
// Input is read-only. All SQL runs in a new, disposable local container.
|
||||
func TestProductionDumpMigrationMariaDB(t *testing.T) {
|
||||
dump := os.Getenv("QFS_TEST_PRODUCTION_DUMP")
|
||||
if dump == "" {
|
||||
t.Skip("QFS_TEST_PRODUCTION_DUMP is not set")
|
||||
}
|
||||
input, err := os.Open(dump)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer input.Close()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
|
||||
defer cancel()
|
||||
image := os.Getenv("QFS_TEST_MARIADB_IMAGE")
|
||||
if image == "" {
|
||||
image = "mariadb:11.8.3"
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, "docker", "run", "--detach", "--rm", "--pull=never", "-p", "127.0.0.1::3306",
|
||||
"-e", "MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1", "-e", "MARIADB_ROOT_HOST=%", "-e", "MARIADB_DATABASE=RFQ_LOG", image,
|
||||
"--skip-ssl", "--character-set-server=utf8mb4", "--collation-server=utf8mb4_uca1400_ai_ci", "--max-allowed-packet=256M").CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("start dump rehearsal MariaDB: %v", err)
|
||||
}
|
||||
id := strings.TrimSpace(string(out))
|
||||
t.Cleanup(func() {
|
||||
if _, err := exec.Command("docker", "stop", "-t", "2", id).CombinedOutput(); err != nil {
|
||||
t.Errorf("stop dump rehearsal MariaDB: %v", err)
|
||||
}
|
||||
})
|
||||
out, err = exec.CommandContext(ctx, "docker", "port", id, "3306/tcp").CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dsn := "root@tcp(" + strings.TrimSpace(string(out)) + ")/RFQ_LOG?parseTime=true&timeout=1s&tls=false"
|
||||
var db *gorm.DB
|
||||
deadline := time.Now().Add(45 * time.Second)
|
||||
for {
|
||||
db, err = gorm.Open(mysql.Open(dsn), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("dump rehearsal database did not become ready")
|
||||
}
|
||||
time.Sleep(150 * time.Millisecond)
|
||||
}
|
||||
pool, err := db.DB()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer pool.Close()
|
||||
pool.SetMaxOpenConns(1)
|
||||
pool.SetMaxIdleConns(1)
|
||||
importer := exec.CommandContext(ctx, "docker", "exec", "-i", id, "mariadb", "--skip-ssl", "--binary-mode", "--default-character-set=utf8mb4", "RFQ_LOG")
|
||||
importer.Stdin = input
|
||||
if output, err := importer.CombinedOutput(); err != nil {
|
||||
// Import errors can include entire data rows; report only error code/line.
|
||||
location := regexp.MustCompile(`(?m)^ERROR[^\n]*`).Find(output)
|
||||
location = regexp.MustCompile(`'[^']*'`).ReplaceAll(location, []byte("'<redacted>'"))
|
||||
if len(location) > 250 {
|
||||
location = location[:250]
|
||||
}
|
||||
t.Fatalf("load dump into disposable database: %v (%s)", err, location)
|
||||
}
|
||||
var version, collation string
|
||||
if err := db.Raw("SELECT VERSION(), @@collation_database").Row().Scan(&version, &collation); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasPrefix(version, "11.8.3-") || collation != "utf8mb4_uca1400_ai_ci" {
|
||||
t.Fatalf("unexpected rehearsal server: version=%s collation=%s", version, collation)
|
||||
}
|
||||
var tables, triggers int64
|
||||
if err := db.Raw("SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA=DATABASE()").Scan(&tables).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.Raw("SELECT COUNT(*) FROM information_schema.TRIGGERS WHERE TRIGGER_SCHEMA=DATABASE()").Scan(&triggers).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Logf("production dump imported: MariaDB=%s collation=%s tables=%d triggers=%d", version, collation, tables, triggers)
|
||||
if err := db.Exec("SET time_zone='+00:00'").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
root := t.TempDir()
|
||||
var original string
|
||||
if err := db.Connection(func(conn *gorm.DB) error {
|
||||
var err error
|
||||
original, err = mariadb.CreateSnapshot(conn, root, "production-dump")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return mariadb.VerifySnapshot(conn, original)
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Log("full dump snapshot restore verified")
|
||||
const filename = "035_add_server_updated_at.sql"
|
||||
if err := db.Where("filename=?", filename).Delete(&SQLSchemaMigration{}).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Reconstruct the real pre-035 schema only in this disposable clone.
|
||||
for _, table := range []string{"qt_configurations", "qt_projects"} {
|
||||
if err := db.Exec("ALTER TABLE " + table + " DROP INDEX IF EXISTS idx_" + table + "_server_updated_at, DROP COLUMN IF EXISTS server_updated_at").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
var before string
|
||||
if err := db.Connection(func(conn *gorm.DB) error {
|
||||
var err error
|
||||
before, err = mariadb.CreateSnapshot(conn, root, "before-rehearsal")
|
||||
return err
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
content, err := qfassets.MigrationFiles.ReadFile("migrations/" + filename)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
files := fstest.MapFS{filename: &fstest.MapFile{Data: content}, "036_recovery_probe.sql": &fstest.MapFile{Data: []byte("-- Tables affected: qt_configurations\n-- verify: deliberate rehearsal failure | SELECT 1\nALTER TABLE qt_configurations ADD COLUMN qfs_recovery_probe INT;")}}
|
||||
if _, err := RunSQLMigrations(db, files, nil, root); err == nil || !strings.Contains(err.Error(), "all session changes restored") {
|
||||
t.Fatalf("dump rollback rehearsal: %v", err)
|
||||
}
|
||||
if err := mariadb.CheckSnapshot(db, before); err != nil {
|
||||
t.Fatalf("dump changed after rollback: %v", err)
|
||||
}
|
||||
t.Log("real migration plus later failed DDL/verify restored the complete dump")
|
||||
delete(files, "036_recovery_probe.sql")
|
||||
if applied, err := RunSQLMigrations(db, files, nil, root); err != nil || len(applied) != 1 {
|
||||
t.Fatalf("real migration rehearsal: applied=%v error=%v", applied, err)
|
||||
}
|
||||
states, err := SQLMigrationStatus(db, files)
|
||||
if err != nil || len(states) != 1 || states[0].AppliedAt == nil {
|
||||
t.Fatalf("rehearsal history invalid: %v", err)
|
||||
}
|
||||
if err := RestoreSQLMigrationSession(db, original); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := mariadb.CheckSnapshot(db, original); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Logf("production dump rehearsal complete: %s, exact original restored", filename)
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gorm.io/driver/mysql"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// migrationJournalName uses the configured endpoint, which survives server restarts.
|
||||
func migrationJournalName(db *gorm.DB, schema string) (string, error) {
|
||||
dialect, ok := db.Dialector.(*mysql.Dialector)
|
||||
if !ok || dialect.DSNConfig == nil {
|
||||
return "", fmt.Errorf("migration requires a MariaDB DSN")
|
||||
}
|
||||
identity := dialect.DSNConfig.Net + "/" + dialect.DSNConfig.Addr + "/" + schema
|
||||
return fmt.Sprintf("qfs-migration-v1-%x.pending", sha256.Sum256([]byte(identity))), nil
|
||||
}
|
||||
|
||||
func requireExclusiveMigrationSession(db *gorm.DB) error {
|
||||
var grants []string
|
||||
if err := db.Session(&gorm.Session{NewDB: true}).Raw("SHOW GRANTS FOR CURRENT_USER").Scan(&grants).Error; err != nil {
|
||||
return fmt.Errorf("read maintenance privileges: %w", err)
|
||||
}
|
||||
allowed := false
|
||||
for _, grant := range grants {
|
||||
if strings.Contains(grant, " ON *.* ") && strings.Contains(grant, "ALL PRIVILEGES") {
|
||||
allowed = true
|
||||
}
|
||||
}
|
||||
if !allowed {
|
||||
return fmt.Errorf("migration recovery requires a DBA with directly granted global ALL PRIVILEGES to inspect all objects and clients")
|
||||
}
|
||||
var clients int64
|
||||
if err := db.Session(&gorm.Session{NewDB: true}).Raw("SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE ID<>CONNECTION_ID() AND DB=DATABASE()").Scan(&clients).Error; err != nil {
|
||||
return fmt.Errorf("check maintenance clients: %w", err)
|
||||
}
|
||||
if clients != 0 {
|
||||
return fmt.Errorf("migration requires maintenance window: %d other connections use this schema", clients)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"regexp"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
func planSQLMigrations(db *gorm.DB, migrations fs.FS) ([]plannedSQLMigration, error) {
|
||||
states, err := SQLMigrationStatus(db, migrations)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var plans []plannedSQLMigration
|
||||
unsupported := regexp.MustCompile(`(?i)\b(?:CREATE|DROP|ALTER)\s+(?:DATABASE|SCHEMA|VIEW|TRIGGER|PROCEDURE|FUNCTION|EVENT)\b|\b(?:USE|DELIMITER|GRANT|REVOKE)\s|\bSET\s+(?:GLOBAL|SQL_MODE)\b|\bENGINE\s*=\s*(?:MyISAM|Aria|MEMORY|BLACKHOLE)\b`)
|
||||
for _, state := range states {
|
||||
if state.AppliedAt != nil {
|
||||
continue
|
||||
}
|
||||
content, err := fs.ReadFile(migrations, state.Filename)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read migration %s: %w", state.Filename, err)
|
||||
}
|
||||
statements, err := splitSQLStatements(string(content))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse migration %s: %w", state.Filename, err)
|
||||
}
|
||||
checks, err := parseMigrationVerifications(string(content))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("migration %s verify headers: %w", state.Filename, err)
|
||||
}
|
||||
if len(checks) == 0 {
|
||||
return nil, fmt.Errorf("migration %s needs verify headers", state.Filename)
|
||||
}
|
||||
for _, stmt := range statements {
|
||||
if unsupported.MatchString(stmt) {
|
||||
return nil, fmt.Errorf("migration %s contains SQL outside supported table recovery", state.Filename)
|
||||
}
|
||||
}
|
||||
plans = append(plans, plannedSQLMigration{state.Filename, statements, checks})
|
||||
}
|
||||
return plans, nil
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/mariadb"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// RestoreSQLMigrationSession is the shipped recovery path after interruption.
|
||||
func RestoreSQLMigrationSession(db *gorm.DB, snapshot string) error {
|
||||
return db.Connection(func(conn *gorm.DB) (err error) {
|
||||
fresh := func() *gorm.DB { return conn.Session(&gorm.Session{NewDB: true}) }
|
||||
var schema, zone string
|
||||
var acquired int
|
||||
if err := fresh().Raw("SELECT DATABASE(), @@time_zone").Row().Scan(&schema, &zone); err != nil {
|
||||
return fmt.Errorf("read recovery session: %w", err)
|
||||
}
|
||||
lock := fmt.Sprintf("qfs-migrate-%x", sha256.Sum256([]byte(schema)))
|
||||
if err := fresh().Raw("SELECT GET_LOCK(?,0)", lock).Scan(&acquired).Error; err != nil || acquired != 1 {
|
||||
return fmt.Errorf("migration lock unavailable for recovery")
|
||||
}
|
||||
defer func() {
|
||||
if e := fresh().Exec("SET time_zone=?", zone).Error; e != nil {
|
||||
err = errors.Join(err, fmt.Errorf("reset recovery time zone: %w", e))
|
||||
}
|
||||
if e := fresh().Raw("SELECT RELEASE_LOCK(?)", lock).Scan(&acquired).Error; e != nil {
|
||||
err = errors.Join(err, fmt.Errorf("release recovery lock: %w", e))
|
||||
}
|
||||
}()
|
||||
if err := requireExclusiveMigrationSession(fresh()); err != nil {
|
||||
return err
|
||||
}
|
||||
journalName, err := migrationJournalName(conn, schema)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
journal := filepath.Join(filepath.Dir(snapshot), journalName)
|
||||
if content, e := os.ReadFile(journal); e == nil {
|
||||
if strings.TrimSpace(string(content)) != snapshot {
|
||||
return fmt.Errorf("pending recovery requires its original snapshot: %s", strings.TrimSpace(string(content)))
|
||||
}
|
||||
} else if !os.IsNotExist(e) {
|
||||
return fmt.Errorf("read recovery marker: %w", e)
|
||||
}
|
||||
if err := fresh().Exec("SET time_zone='+00:00'").Error; err != nil {
|
||||
return fmt.Errorf("set restore time zone: %w", err)
|
||||
}
|
||||
if err := mariadb.VerifySnapshot(fresh(), snapshot); err != nil {
|
||||
return fmt.Errorf("verify recovery snapshot: %w", err)
|
||||
}
|
||||
if err := mariadb.RestoreSnapshot(fresh(), snapshot); err != nil {
|
||||
return fmt.Errorf("restore migration session: %w", err)
|
||||
}
|
||||
if err := os.Remove(journal); err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("clear recovery marker: %w", err)
|
||||
}
|
||||
slog.Info("server migration snapshot restored", "schema", schema, "backup", snapshot, "state", "verified")
|
||||
return nil
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/appstate"
|
||||
"git.mchus.pro/mchus/quoteforge/internal/mariadb"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
type plannedSQLMigration struct {
|
||||
filename string
|
||||
statements []string
|
||||
checks []migrationVerification
|
||||
}
|
||||
|
||||
// RunSQLMigrations requires a DBA maintenance window: stop all other writers.
|
||||
// Backups are mandatory; each file gets per-table snapshots before any DDL.
|
||||
func RunSQLMigrations(db *gorm.DB, migrations fs.FS, onApply func(string), backupRoot string) ([]string, error) {
|
||||
var applied []string
|
||||
err := db.Connection(func(conn *gorm.DB) (err error) {
|
||||
fresh := func() *gorm.DB { return conn.Session(&gorm.Session{NewDB: true}) }
|
||||
var schema, mode, zone, engine string
|
||||
var foreignKeys int
|
||||
if err := fresh().Raw("SELECT DATABASE(), @@sql_mode, @@time_zone, @@foreign_key_checks, @@default_storage_engine").Row().Scan(&schema, &mode, &zone, &foreignKeys, &engine); err != nil {
|
||||
return fmt.Errorf("read migration session: %w", err)
|
||||
}
|
||||
if schema == "" || strings.Contains(mode, "NO_BACKSLASH_ESCAPES") || engine != "InnoDB" {
|
||||
return fmt.Errorf("migration requires a selected InnoDB schema and backslash escape SQL mode")
|
||||
}
|
||||
lock := fmt.Sprintf("qfs-migrate-%x", sha256.Sum256([]byte(schema)))
|
||||
var acquired int
|
||||
if err := fresh().Raw("SELECT GET_LOCK(?,0)", lock).Scan(&acquired).Error; err != nil || acquired != 1 {
|
||||
return fmt.Errorf("migration lock unavailable for %s", schema)
|
||||
}
|
||||
defer func() {
|
||||
for _, reset := range []struct {
|
||||
sql string
|
||||
value any
|
||||
}{{"SET time_zone=?", zone}, {"SET sql_mode=?", mode}, {"SET foreign_key_checks=?", foreignKeys}} {
|
||||
if e := fresh().Exec(reset.sql, reset.value).Error; e != nil {
|
||||
err = errors.Join(err, fmt.Errorf("reset migration session: %w", e))
|
||||
}
|
||||
}
|
||||
if e := fresh().Raw("SELECT RELEASE_LOCK(?)", lock).Scan(&acquired).Error; e != nil {
|
||||
err = errors.Join(err, fmt.Errorf("release migration lock: %w", e))
|
||||
}
|
||||
}()
|
||||
plans, err := planSQLMigrations(fresh(), migrations)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := appstate.ValidateBackupLocation(backupRoot); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(backupRoot, 0700); err != nil {
|
||||
return fmt.Errorf("create migration backup root: %w", err)
|
||||
}
|
||||
journalName, err := migrationJournalName(conn, schema)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
journal := filepath.Join(backupRoot, journalName)
|
||||
if content, e := os.ReadFile(journal); e == nil {
|
||||
return fmt.Errorf("interrupted migration requires restore before retry; safety snapshot: %s", strings.TrimSpace(string(content)))
|
||||
} else if !os.IsNotExist(e) {
|
||||
return fmt.Errorf("read migration recovery marker: %w", e)
|
||||
}
|
||||
if len(plans) == 0 {
|
||||
return nil
|
||||
}
|
||||
if err := requireExclusiveMigrationSession(fresh()); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := fresh().Exec("SET time_zone='+00:00'").Error; err != nil {
|
||||
return fmt.Errorf("set snapshot time zone: %w", err)
|
||||
}
|
||||
full, err := mariadb.CreateSnapshot(fresh(), backupRoot, "session")
|
||||
if err != nil {
|
||||
return fmt.Errorf("pre-migration snapshot: %w", err)
|
||||
}
|
||||
if err := mariadb.VerifySnapshot(fresh(), full); err != nil {
|
||||
return fmt.Errorf("pre-migration restore verification: %w", err)
|
||||
}
|
||||
marker, err := os.OpenFile(journal, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create migration recovery marker: %w", err)
|
||||
}
|
||||
_, writeErr := fmt.Fprintln(marker, full)
|
||||
syncErr, closeErr := marker.Sync(), marker.Close()
|
||||
if e := errors.Join(writeErr, syncErr, closeErr); e != nil {
|
||||
return fmt.Errorf("persist migration recovery marker: %w", e)
|
||||
}
|
||||
slog.Info("verified server migration backup", "schema", schema, "path", full)
|
||||
var steps []string
|
||||
for _, plan := range plans {
|
||||
if err := requireExclusiveMigrationSession(fresh()); err != nil {
|
||||
return recoverSQLMigration(fresh(), journal, full, steps, plan.filename, err)
|
||||
}
|
||||
step, err := mariadb.CreateSnapshot(fresh(), backupRoot, plan.filename)
|
||||
if err == nil {
|
||||
err = mariadb.VerifySnapshot(fresh(), step)
|
||||
}
|
||||
if err != nil {
|
||||
return recoverSQLMigration(fresh(), journal, full, steps, plan.filename, err)
|
||||
}
|
||||
steps = append(steps, step)
|
||||
if onApply != nil {
|
||||
onApply(plan.filename)
|
||||
}
|
||||
if err := ensureSQLMigrationsTable(fresh()); err != nil {
|
||||
return recoverSQLMigration(fresh(), journal, full, steps, plan.filename, err)
|
||||
}
|
||||
if err := applySQLMigrationOnSession(fresh(), plan.filename, plan.statements, plan.checks); err != nil {
|
||||
return recoverSQLMigration(fresh(), journal, full, steps, plan.filename, err)
|
||||
}
|
||||
applied = append(applied, plan.filename)
|
||||
}
|
||||
if err := os.Remove(journal); err != nil {
|
||||
return fmt.Errorf("remove completed migration marker: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return applied, nil
|
||||
}
|
||||
|
||||
func recoverSQLMigration(db *gorm.DB, journal, full string, steps []string, filename string, cause error) error {
|
||||
slog.Error("server migration failed", "step", filename, "error", cause, "backup", full)
|
||||
if err := requireExclusiveMigrationSession(db); err != nil {
|
||||
return errors.Join(cause, fmt.Errorf("recovery refused with active clients; keep maintenance window and restore %s: %w", full, err))
|
||||
}
|
||||
var recovery error
|
||||
for i := len(steps) - 1; i >= 0; i-- {
|
||||
slog.Warn("restoring migration step", "step", filename, "backup", steps[i])
|
||||
if err := mariadb.RestoreSnapshot(db, steps[i]); err != nil {
|
||||
recovery = err
|
||||
break
|
||||
}
|
||||
}
|
||||
if recovery == nil {
|
||||
recovery = mariadb.CheckSnapshot(db, full)
|
||||
}
|
||||
if recovery != nil {
|
||||
slog.Warn("restoring full migration session", "backup", full, "error", recovery)
|
||||
recovery = mariadb.RestoreSnapshot(db, full)
|
||||
}
|
||||
if recovery != nil {
|
||||
return errors.Join(cause, fmt.Errorf("recovery failed; database requires restore from %s: %w", full, recovery))
|
||||
}
|
||||
if err := os.Remove(journal); err != nil {
|
||||
return errors.Join(cause, fmt.Errorf("database restored but recovery marker remains: %w", err))
|
||||
}
|
||||
slog.Warn("server migration session restored", "step", filename, "state", "before-session", "backup", full)
|
||||
return fmt.Errorf("migration %s failed; all session changes restored: %w", filename, cause)
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
//go:build integration
|
||||
|
||||
package models
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/mariadb"
|
||||
"gorm.io/driver/mysql"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
func TestSQLMigrationSafetyMariaDB(t *testing.T) {
|
||||
out, err := exec.Command("docker", "run", "--detach", "--rm", "--pull=never", "-p", "127.0.0.1::3306", "-e", "MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1", "-e", "MARIADB_ROOT_HOST=%", "mariadb:11.8", "--skip-ssl").CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("start test MariaDB: %v: %s", err, out)
|
||||
}
|
||||
id := strings.TrimSpace(string(out))
|
||||
t.Cleanup(func() {
|
||||
if out, err := exec.Command("docker", "stop", "-t", "2", id).CombinedOutput(); err != nil {
|
||||
t.Errorf("stop test MariaDB: %v: %s", err, out)
|
||||
}
|
||||
})
|
||||
out, err = exec.Command("docker", "port", id, "3306/tcp").CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("get test MariaDB port: %v: %s", err, out)
|
||||
}
|
||||
addr := strings.TrimSpace(string(out))
|
||||
dsn := "root@tcp(" + addr + ")/?parseTime=true&timeout=1s&tls=false"
|
||||
var admin *gorm.DB
|
||||
deadline := time.Now().Add(45 * time.Second)
|
||||
for {
|
||||
admin, err = gorm.Open(mysql.Open(dsn), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("test MariaDB unavailable: %v", err)
|
||||
}
|
||||
time.Sleep(150 * time.Millisecond)
|
||||
}
|
||||
pool, err := admin.DB()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { pool.Close() })
|
||||
newDB := func(t *testing.T) *gorm.DB {
|
||||
t.Helper()
|
||||
schema := fmt.Sprintf("qfs_test_%d", time.Now().UnixNano())
|
||||
if err := admin.Exec("CREATE DATABASE " + schema + " COLLATE utf8mb4_unicode_ci").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := gorm.Open(mysql.Open(strings.Replace(dsn, ")/?", ")/"+schema+"?", 1)), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p, err := db.DB()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p.SetMaxOpenConns(1)
|
||||
p.SetMaxIdleConns(1)
|
||||
t.Cleanup(func() {
|
||||
p.Close()
|
||||
if err := admin.Exec("DROP DATABASE " + schema).Error; err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
})
|
||||
for _, sql := range []string{
|
||||
"SET time_zone='+00:00'",
|
||||
"CREATE TABLE parent (id INT PRIMARY KEY AUTO_INCREMENT, value TEXT, empty_value TEXT, nullable TEXT, raw VARBINARY(32), amount DECIMAL(20,6), stamp TIMESTAMP(6), generated INT AS (id+1) VIRTUAL) ENGINE=InnoDB",
|
||||
"INSERT INTO parent (id,value,empty_value,nullable,raw,amount,stamp) VALUES (1,'quote; and ''text''','',NULL,X'0001FF',123.456789,'2026-10-26 12:34:56.123456')",
|
||||
"SET @saved_test_mode=@@sql_mode",
|
||||
"SET sql_mode='NO_AUTO_VALUE_ON_ZERO'",
|
||||
"INSERT INTO parent (id,value,stamp) VALUES (0,'legacy','0000-00-00 00:00:00')",
|
||||
"SET sql_mode=@saved_test_mode",
|
||||
"CREATE TABLE child (id INT PRIMARY KEY, parent_id INT, CONSTRAINT child_parent FOREIGN KEY (parent_id) REFERENCES parent(id)) ENGINE=InnoDB",
|
||||
"INSERT INTO child VALUES (1,1)",
|
||||
"CREATE TABLE audit (id INT PRIMARY KEY AUTO_INCREMENT, note TEXT) ENGINE=InnoDB",
|
||||
"CREATE TRIGGER insert_value BEFORE INSERT ON parent FOR EACH ROW SET NEW.value=CONCAT(NEW.value,'!')",
|
||||
"CREATE TRIGGER audit_z AFTER UPDATE ON parent FOR EACH ROW INSERT INTO audit (note) VALUES ('first')",
|
||||
"CREATE TRIGGER audit_a AFTER UPDATE ON parent FOR EACH ROW INSERT INTO audit (note) VALUES ('second')",
|
||||
} {
|
||||
if err := db.Exec(sql).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return db
|
||||
}
|
||||
migration := func(sql, checks string) *fstest.MapFile {
|
||||
return &fstest.MapFile{Data: []byte("-- Tables affected: parent, child\n" + checks + "\n" + sql)}
|
||||
}
|
||||
for _, failure := range []string{"none", "verify", "FK"} {
|
||||
t.Run(failure, func(t *testing.T) {
|
||||
db, root := newDB(t), t.TempDir()
|
||||
var original string
|
||||
err := db.Connection(func(conn *gorm.DB) error {
|
||||
var e error
|
||||
original, e = mariadb.CreateSnapshot(conn, root, "original")
|
||||
return e
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
files := fstest.MapFS{"001_a.sql": migration("SET @example='value; quoted'; PREPARE q FROM 'SELECT 1'; EXECUTE q; DEALLOCATE PREPARE q; ALTER TABLE parent ADD COLUMN added INT; UPDATE parent SET value=@example;", "-- verify: added missing | SELECT 1 FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=DATABASE() AND TABLE_NAME='parent' AND COLUMN_NAME='added' HAVING COUNT(*)=0\n-- verify: wrong value | SELECT 1 FROM parent WHERE value<>'value; quoted'")}
|
||||
if failure == "verify" {
|
||||
files["002_b.sql"] = migration("CREATE TABLE new_table (id INT); UPDATE parent SET value='changed';", "-- verify: deliberate failure | SELECT 1")
|
||||
}
|
||||
if failure == "FK" {
|
||||
files["002_b.sql"] = migration("ALTER TABLE child ADD COLUMN added INT; ALTER TABLE child ADD CONSTRAINT broken_fk FOREIGN KEY (added) REFERENCES parent(missing);", "-- verify: should never run | SELECT 1 WHERE 0")
|
||||
}
|
||||
applied, err := RunSQLMigrations(db, files, nil, root)
|
||||
if failure == "none" {
|
||||
if err != nil || len(applied) != 1 {
|
||||
t.Fatalf("apply=%v error=%v", applied, err)
|
||||
}
|
||||
var count int64
|
||||
if err := db.Model(&SQLSchemaMigration{}).Count(&count).Error; err != nil || count != 1 {
|
||||
t.Fatalf("history=%d err=%v", count, err)
|
||||
}
|
||||
} else {
|
||||
if err == nil || !strings.Contains(err.Error(), "all session changes restored") || len(applied) != 0 {
|
||||
t.Fatalf("apply=%v error=%v", applied, err)
|
||||
}
|
||||
if err := mariadb.CheckSnapshot(db, original); err != nil {
|
||||
t.Fatalf("recovery changed original state: %v", err)
|
||||
}
|
||||
}
|
||||
markers, err := filepath.Glob(filepath.Join(root, "*.pending"))
|
||||
if err != nil || len(markers) != 0 {
|
||||
t.Fatalf("markers=%v error=%v", markers, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
t.Run("interrupted session", func(t *testing.T) {
|
||||
db, root := newDB(t), t.TempDir()
|
||||
files := fstest.MapFS{"001_a.sql": migration("ALTER TABLE parent ADD COLUMN added INT", "-- verify: ok | SELECT 1 WHERE 0")}
|
||||
func() {
|
||||
defer func() {
|
||||
if recover() == nil {
|
||||
t.Error("expected interruption")
|
||||
}
|
||||
}()
|
||||
RunSQLMigrations(db, files, func(string) { panic("interruption") }, root)
|
||||
}()
|
||||
if err := db.Exec("ALTER TABLE parent ADD COLUMN added INT").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := RunSQLMigrations(db, files, nil, root); err == nil || !strings.Contains(err.Error(), "interrupted migration") {
|
||||
t.Fatalf("retry not refused: %v", err)
|
||||
}
|
||||
markers, err := filepath.Glob(filepath.Join(root, "*.pending"))
|
||||
if err != nil || len(markers) != 1 {
|
||||
t.Fatalf("markers=%v err=%v", markers, err)
|
||||
}
|
||||
data, err := os.ReadFile(markers[0])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := RestoreSQLMigrationSession(db, strings.TrimSpace(string(data))); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if db.Table("parent").Migrator().HasColumn(&struct{ Added int }{}, "added") {
|
||||
t.Fatal("interrupted DDL not restored")
|
||||
}
|
||||
if _, err := RunSQLMigrations(db, files, nil, root); err != nil {
|
||||
t.Fatalf("retry after restore: %v", err)
|
||||
}
|
||||
})
|
||||
t.Run("unsupported object", func(t *testing.T) {
|
||||
db := newDB(t)
|
||||
if err := db.Exec("CREATE VIEW example_view AS SELECT id FROM parent").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
files := fstest.MapFS{"001_a.sql": migration("ALTER TABLE parent ADD COLUMN added INT", "-- verify: ok | SELECT 1 WHERE 0")}
|
||||
if _, err := RunSQLMigrations(db, files, nil, t.TempDir()); err == nil || !strings.Contains(err.Error(), "snapshot refused") {
|
||||
t.Fatalf("unsupported schema accepted: %v", err)
|
||||
}
|
||||
if db.Table("parent").Migrator().HasColumn(&struct{ Added int }{}, "added") {
|
||||
t.Fatal("unsupported schema was changed")
|
||||
}
|
||||
})
|
||||
t.Run("other client", func(t *testing.T) {
|
||||
db := newDB(t)
|
||||
p, err := db.DB()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p.SetMaxOpenConns(2)
|
||||
other, err := p.Conn(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer other.Close()
|
||||
files := fstest.MapFS{"001_a.sql": migration("ALTER TABLE parent ADD COLUMN added INT", "-- verify: ok | SELECT 1 WHERE 0")}
|
||||
if _, err := RunSQLMigrations(db, files, nil, t.TempDir()); err == nil || !strings.Contains(err.Error(), "other connections") {
|
||||
t.Fatalf("active client accepted: %v", err)
|
||||
}
|
||||
if db.Table("parent").Migrator().HasColumn(&struct{ Added int }{}, "added") {
|
||||
t.Fatal("active schema changed")
|
||||
}
|
||||
})
|
||||
t.Run("backup failure", func(t *testing.T) {
|
||||
db := newDB(t)
|
||||
path := filepath.Join(t.TempDir(), "not-directory")
|
||||
if err := os.WriteFile(path, []byte("example"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
files := fstest.MapFS{"001_a.sql": migration("ALTER TABLE parent ADD COLUMN added INT", "-- verify: ok | SELECT 1 WHERE 0")}
|
||||
if _, err := RunSQLMigrations(db, files, nil, path); err == nil {
|
||||
t.Fatal("backup failure ignored")
|
||||
}
|
||||
if db.Table("parent").Migrator().HasColumn(&struct{ Added int }{}, "added") {
|
||||
t.Fatal("schema changed without backup")
|
||||
}
|
||||
})
|
||||
|
||||
}
|
||||
@@ -34,31 +34,43 @@ func applySQLMigrationFile(db *gorm.DB, filename, content string) error {
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse migration %s: %w", filename, err)
|
||||
}
|
||||
statements, err := splitSQLStatements(content)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse migration %s SQL: %w", filename, err)
|
||||
}
|
||||
return db.Connection(func(conn *gorm.DB) error {
|
||||
if err := executeMigrationStatements(conn, filename, splitSQLStatements(content)); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, check := range checks {
|
||||
rows, err := conn.Raw(check.query).Rows()
|
||||
if err != nil {
|
||||
return fmt.Errorf("verify migration %s (%s): %w", filename, check.description, err)
|
||||
}
|
||||
failed := rows.Next()
|
||||
rowErr := rows.Err()
|
||||
closeErr := rows.Close()
|
||||
if rowErr != nil {
|
||||
return fmt.Errorf("read migration verification %s (%s): %w", filename, check.description, rowErr)
|
||||
}
|
||||
if closeErr != nil {
|
||||
return fmt.Errorf("close migration verification %s (%s): %w", filename, check.description, closeErr)
|
||||
}
|
||||
if failed {
|
||||
return fmt.Errorf("verify migration %s failed: %s", filename, check.description)
|
||||
}
|
||||
}
|
||||
if err := conn.Create(&SQLSchemaMigration{Filename: filename}).Error; err != nil {
|
||||
return fmt.Errorf("record migration %s: %w", filename, err)
|
||||
}
|
||||
return nil
|
||||
return applySQLMigrationOnSession(conn, filename, statements, checks)
|
||||
})
|
||||
}
|
||||
|
||||
func applySQLMigrationOnSession(conn *gorm.DB, filename string, statements []string, checks []migrationVerification) error {
|
||||
if err := executeMigrationStatements(conn, filename, statements); err != nil {
|
||||
return err
|
||||
}
|
||||
// Each query gets a fresh statement on the pinned connection: chaining
|
||||
// Raw/Create on conn itself leaks the previous query's state (with two
|
||||
// verify checks the history insert targeted a table named "rows").
|
||||
fresh := func() *gorm.DB { return conn.Session(&gorm.Session{NewDB: true}) }
|
||||
for _, check := range checks {
|
||||
rows, err := fresh().Raw(check.query).Rows()
|
||||
if err != nil {
|
||||
return fmt.Errorf("verify migration %s (%s): %w", filename, check.description, err)
|
||||
}
|
||||
failed := rows.Next()
|
||||
rowErr := rows.Err()
|
||||
closeErr := rows.Close()
|
||||
if rowErr != nil {
|
||||
return fmt.Errorf("read migration verification %s (%s): %w", filename, check.description, rowErr)
|
||||
}
|
||||
if closeErr != nil {
|
||||
return fmt.Errorf("close migration verification %s (%s): %w", filename, check.description, closeErr)
|
||||
}
|
||||
if failed {
|
||||
return fmt.Errorf("verify migration %s failed: %s", filename, check.description)
|
||||
}
|
||||
}
|
||||
if err := fresh().Create(&SQLSchemaMigration{Filename: filename}).Error; err != nil {
|
||||
return fmt.Errorf("record migration %s: %w", filename, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -15,6 +15,7 @@ func TestMigrationVerification(t *testing.T) {
|
||||
{"failed", "-- verify: bad value | SELECT 1 FROM example WHERE value <> 1\nCREATE TEMP TABLE example (value INT); INSERT INTO example VALUES (2);", "bad value"},
|
||||
{"query error", "-- verify: missing table | SELECT 1 FROM missing\nCREATE TEMP TABLE example (value INT);", "missing table"},
|
||||
{"malformed", "-- verify: no query\nCREATE TABLE example (value INT);", "invalid migration verify header"},
|
||||
{"two checks pass", "-- verify: first | SELECT 1 WHERE 0\n-- verify: second | SELECT 1 WHERE 0\nCREATE TEMP TABLE example (value INT);", ""},
|
||||
{"second check fails", "-- verify: first | SELECT 1 WHERE 0\n-- verify: second | SELECT 1\nCREATE TEMP TABLE example (value INT);", "second"},
|
||||
{"execution failure", "-- verify: none | SELECT 1 WHERE 0\nINVALID SQL;", "exec migration"},
|
||||
} {
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
@@ -124,48 +123,6 @@ func NeedsSQLMigrations(db *gorm.DB, migrations fs.FS) (bool, error) {
|
||||
return count < int64(len(files)), nil
|
||||
}
|
||||
|
||||
// RunSQLMigrations applies pending SQL files from migrations once each, in filename
|
||||
// order, and records them in qt_schema_migrations. onApply (optional) is called with
|
||||
// each filename before it runs. Local SQLite-only scripts are skipped automatically.
|
||||
// Returns the filenames applied.
|
||||
func RunSQLMigrations(db *gorm.DB, migrations fs.FS, onApply func(filename string)) ([]string, error) {
|
||||
if err := ensureSQLMigrationsTable(db); err != nil {
|
||||
return nil, fmt.Errorf("migrate qt_schema_migrations table: %w", err)
|
||||
}
|
||||
|
||||
files, err := listSQLMigrationFiles(migrations)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var applied []string
|
||||
|
||||
for _, filename := range files {
|
||||
var count int64
|
||||
if err := db.Model(&SQLSchemaMigration{}).Where("filename = ?", filename).Count(&count).Error; err != nil {
|
||||
return applied, fmt.Errorf("check migration %s: %w", filename, err)
|
||||
}
|
||||
if count > 0 {
|
||||
continue
|
||||
}
|
||||
if onApply != nil {
|
||||
onApply(filename)
|
||||
}
|
||||
|
||||
content, err := fs.ReadFile(migrations, filename)
|
||||
if err != nil {
|
||||
return applied, fmt.Errorf("read migration %s: %w", filename, err)
|
||||
}
|
||||
|
||||
if err := applySQLMigrationFile(db, filename, string(content)); err != nil {
|
||||
return applied, err
|
||||
}
|
||||
applied = append(applied, filename)
|
||||
}
|
||||
|
||||
return applied, nil
|
||||
}
|
||||
|
||||
// IsMigrationPermissionError returns true if err indicates insufficient privileges
|
||||
// to create/alter/read migration metadata or target schema objects.
|
||||
func IsMigrationPermissionError(err error) bool {
|
||||
@@ -239,35 +196,6 @@ func isIgnorableMigrationError(message string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func splitSQLStatements(script string) []string {
|
||||
scanner := bufio.NewScanner(strings.NewReader(script))
|
||||
scanner.Buffer(make([]byte, 1024), 1024*1024)
|
||||
|
||||
lines := make([]string, 0, 128)
|
||||
for scanner.Scan() {
|
||||
line := strings.TrimSpace(scanner.Text())
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(line, "--") {
|
||||
continue
|
||||
}
|
||||
lines = append(lines, scanner.Text())
|
||||
}
|
||||
|
||||
combined := strings.Join(lines, "\n")
|
||||
raw := strings.Split(combined, ";")
|
||||
stmts := make([]string, 0, len(raw))
|
||||
for _, stmt := range raw {
|
||||
trimmed := strings.TrimSpace(stmt)
|
||||
if trimmed == "" {
|
||||
continue
|
||||
}
|
||||
stmts = append(stmts, trimmed)
|
||||
}
|
||||
return stmts
|
||||
}
|
||||
|
||||
func listSQLMigrationFiles(migrations fs.FS) ([]string, error) {
|
||||
entries, err := fs.ReadDir(migrations, ".")
|
||||
if err != nil {
|
||||
|
||||
@@ -46,7 +46,10 @@ func TestSQLMigrationStatusAndListing(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestSplitSQLStatementsSkipsComments(t *testing.T) {
|
||||
got := splitSQLStatements("-- header; with semicolon\nALTER TABLE a ADD COLUMN x INT;\n\nALTER TABLE b\n ADD COLUMN y INT;\n")
|
||||
got, err := splitSQLStatements("-- header; with semicolon\nALTER TABLE a ADD COLUMN x INT;\n\nALTER TABLE b\n ADD COLUMN y INT;\n")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 2 || got[1] != "ALTER TABLE b\n ADD COLUMN y INT" {
|
||||
t.Fatalf("unexpected statements %q", got)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// splitSQLStatements supports MariaDB literals, quoted identifiers and comments.
|
||||
func splitSQLStatements(script string) ([]string, error) {
|
||||
var statements []string
|
||||
var current strings.Builder
|
||||
var quote byte
|
||||
for i := 0; i < len(script); i++ {
|
||||
ch := script[i]
|
||||
if quote != 0 {
|
||||
current.WriteByte(ch)
|
||||
if ch == '\\' && quote != '`' && i+1 < len(script) {
|
||||
i++
|
||||
current.WriteByte(script[i])
|
||||
} else if ch == quote {
|
||||
if i+1 < len(script) && script[i+1] == quote {
|
||||
i++
|
||||
current.WriteByte(script[i])
|
||||
} else {
|
||||
quote = 0
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
switch {
|
||||
case ch == '\'' || ch == '"' || ch == '`':
|
||||
quote = ch
|
||||
current.WriteByte(ch)
|
||||
case ch == '#' || (ch == '-' && i+2 < len(script) && script[i+1] == '-' && script[i+2] <= ' '):
|
||||
for i < len(script) && script[i] != '\n' {
|
||||
i++
|
||||
}
|
||||
current.WriteByte('\n')
|
||||
case ch == '/' && i+1 < len(script) && script[i+1] == '*':
|
||||
if strings.HasPrefix(script[i:], "/*!") || strings.HasPrefix(script[i:], "/*M!") {
|
||||
return nil, fmt.Errorf("executable SQL comments are unsupported")
|
||||
}
|
||||
end := strings.Index(script[i+2:], "*/")
|
||||
if end < 0 {
|
||||
return nil, fmt.Errorf("unterminated SQL comment")
|
||||
}
|
||||
i += end + 3
|
||||
current.WriteByte(' ')
|
||||
case ch == ';':
|
||||
if stmt := strings.TrimSpace(current.String()); stmt != "" {
|
||||
statements = append(statements, stmt)
|
||||
}
|
||||
current.Reset()
|
||||
default:
|
||||
current.WriteByte(ch)
|
||||
}
|
||||
}
|
||||
if quote != 0 {
|
||||
return nil, fmt.Errorf("unterminated SQL quote")
|
||||
}
|
||||
if stmt := strings.TrimSpace(current.String()); stmt != "" {
|
||||
statements = append(statements, stmt)
|
||||
}
|
||||
for _, stmt := range statements {
|
||||
if strings.HasPrefix(strings.ToUpper(stmt), "DELIMITER ") {
|
||||
return nil, fmt.Errorf("DELIMITER directives are unsupported")
|
||||
}
|
||||
}
|
||||
return statements, nil
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
qfassets "git.mchus.pro/mchus/quoteforge"
|
||||
"io/fs"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSQLStatementParser(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name, sql string
|
||||
want []string
|
||||
fail bool
|
||||
}{
|
||||
{"empty", "-- comment;\n# more;", nil, false},
|
||||
{"literal separators", "SELECT 'a;b', \"c;d\", `e;f`; SELECT 2;", []string{"SELECT 'a;b', \"c;d\", `e;f`", "SELECT 2"}, false},
|
||||
{"doubled quotes", "SELECT 'a'';b', `a``;b`;", []string{"SELECT 'a'';b', `a``;b`"}, false},
|
||||
{"backslash quote", "SELECT 'a\\';b'; SELECT 2", []string{"SELECT 'a\\';b'", "SELECT 2"}, false},
|
||||
{"inline comments", "SELECT/* ; */ 1; -- end;\nSELECT 2# end;\n;", []string{"SELECT 1", "SELECT 2"}, false},
|
||||
{"not a comment", "SELECT 4--2;", []string{"SELECT 4--2"}, false},
|
||||
{"unclosed quote", "SELECT 'a;", nil, true},
|
||||
{"unclosed block", "SELECT 1; /*", nil, true},
|
||||
{"executable comment", "/*! SELECT 1 */", nil, true},
|
||||
{"MariaDB comment", "/*M! SELECT 1 */", nil, true},
|
||||
{"delimiter", "DELIMITER $$\nSELECT 1", nil, true},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := splitSQLStatements(tt.sql)
|
||||
if (err != nil) != tt.fail {
|
||||
t.Fatalf("error=%v", err)
|
||||
}
|
||||
if !tt.fail && !reflect.DeepEqual(got, tt.want) {
|
||||
t.Fatalf("got=%q want=%q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
long := "SELECT '" + strings.Repeat("x", 2<<20) + ";y';"
|
||||
if got, err := splitSQLStatements(long); err != nil || len(got) != 1 {
|
||||
t.Fatalf("large statement: count=%d err=%v", len(got), err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmbeddedServerMigrationSQL(t *testing.T) {
|
||||
migrations, err := qfassets.MigrationsFS()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
files, err := listSQLMigrationFiles(migrations)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, name := range files {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
content, err := fs.ReadFile(migrations, name)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
statements, err := splitSQLStatements(string(content))
|
||||
if err != nil || len(statements) == 0 {
|
||||
t.Fatalf("count=%d err=%v", len(statements), err)
|
||||
}
|
||||
checks, err := parseMigrationVerifications(string(content))
|
||||
if err != nil || len(checks) == 0 {
|
||||
t.Fatalf("checks=%d err=%v", len(checks), err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -141,6 +141,31 @@ func (r *PartnumberBookRepository) HasAllBookItems(bookID uint) bool {
|
||||
return count == int64(len(book.PartnumbersJSON))
|
||||
}
|
||||
|
||||
// MissingBookItems returns the book's partnumbers that have no local catalog row.
|
||||
func (r *PartnumberBookRepository) MissingBookItems(bookID uint) []string {
|
||||
book, err := r.getBook(bookID)
|
||||
if err != nil || len(book.PartnumbersJSON) == 0 {
|
||||
return nil
|
||||
}
|
||||
var present []string
|
||||
if err := r.db.Model(&localdb.LocalPartnumberBookItem{}).
|
||||
Where("partnumber IN ?", []string(book.PartnumbersJSON)).
|
||||
Pluck("partnumber", &present).Error; err != nil {
|
||||
return nil
|
||||
}
|
||||
have := make(map[string]struct{}, len(present))
|
||||
for _, pn := range present {
|
||||
have[pn] = struct{}{}
|
||||
}
|
||||
var missing []string
|
||||
for _, pn := range book.PartnumbersJSON {
|
||||
if _, ok := have[pn]; !ok {
|
||||
missing = append(missing, pn)
|
||||
}
|
||||
}
|
||||
return missing
|
||||
}
|
||||
|
||||
func (r *PartnumberBookRepository) getBook(bookID uint) (*localdb.LocalPartnumberBook, error) {
|
||||
var book localdb.LocalPartnumberBook
|
||||
if err := r.db.First(&book, bookID).Error; err != nil {
|
||||
|
||||
@@ -156,6 +156,35 @@ func (r *PricelistRepository) GetLatestActiveBySource(source string) (*models.Pr
|
||||
return &pricelist, nil
|
||||
}
|
||||
|
||||
// GetLatestActiveIDsBySources returns, per source, the ID GetLatestActiveBySource
|
||||
// would return, in one round trip. Sources with no such pricelist are absent.
|
||||
func (r *PricelistRepository) GetLatestActiveIDsBySources(sources []string) (map[string]uint, error) {
|
||||
result := make(map[string]uint, len(sources))
|
||||
if len(sources) == 0 {
|
||||
return result, nil
|
||||
}
|
||||
parts := make([]string, 0, len(sources))
|
||||
args := make([]any, 0, len(sources))
|
||||
for _, source := range sources {
|
||||
parts = append(parts, `(SELECT id, source FROM qt_pricelists
|
||||
WHERE is_active = 1 AND source = ?
|
||||
AND EXISTS (SELECT 1 FROM qt_pricelist_items WHERE qt_pricelist_items.pricelist_id = qt_pricelists.id)
|
||||
ORDER BY created_at DESC, id DESC LIMIT 1)`)
|
||||
args = append(args, source)
|
||||
}
|
||||
var rows []struct {
|
||||
ID uint
|
||||
Source string
|
||||
}
|
||||
if err := r.db.Raw(strings.Join(parts, " UNION ALL "), args...).Scan(&rows).Error; err != nil {
|
||||
return nil, fmt.Errorf("getting latest pricelists: %w", err)
|
||||
}
|
||||
for _, row := range rows {
|
||||
result[row.Source] = row.ID
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// Create creates a new pricelist
|
||||
func (r *PricelistRepository) Create(pricelist *models.Pricelist) error {
|
||||
if err := r.db.Create(pricelist).Error; err != nil {
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
package sync_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/models"
|
||||
syncsvc "git.mchus.pro/mchus/quoteforge/internal/services/sync"
|
||||
"gorm.io/driver/mysql"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
// Runs only against a disposable MariaDB that already has the qt_projects and
|
||||
// qt_configurations tables (e.g. a container loaded with SHOW CREATE TABLE from
|
||||
// production). The test truncates them. Example:
|
||||
//
|
||||
// QF_TEST_MARIADB_DSN='root:test@tcp(127.0.0.1:33306)/qf?parseTime=true' go test ./internal/services/sync -run MariaDB
|
||||
func newMariaDBForPullTest(t *testing.T) *gorm.DB {
|
||||
t.Helper()
|
||||
dsn := os.Getenv("QF_TEST_MARIADB_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("QF_TEST_MARIADB_DSN not set")
|
||||
}
|
||||
db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
|
||||
if err != nil {
|
||||
t.Fatalf("open mariadb: %v", err)
|
||||
}
|
||||
mustExec(t, db, "SET FOREIGN_KEY_CHECKS=0")
|
||||
mustExec(t, db, "TRUNCATE TABLE qt_configurations")
|
||||
mustExec(t, db, "TRUNCATE TABLE qt_projects")
|
||||
mustExec(t, db, "SET FOREIGN_KEY_CHECKS=1")
|
||||
for _, table := range []string{"qt_configurations", "qt_projects"} {
|
||||
mustExec(t, db, "ALTER TABLE "+table+" DROP INDEX IF EXISTS idx_"+table+"_server_updated_at")
|
||||
mustExec(t, db, "ALTER TABLE "+table+" DROP COLUMN IF EXISTS server_updated_at")
|
||||
}
|
||||
return db
|
||||
}
|
||||
|
||||
func mustExec(t *testing.T, db *gorm.DB, sql string, args ...any) {
|
||||
t.Helper()
|
||||
if err := db.Exec(sql, args...).Error; err != nil {
|
||||
t.Fatalf("%s: %v", sql, err)
|
||||
}
|
||||
}
|
||||
|
||||
func applyServerUpdatedAtMigration(t *testing.T, db *gorm.DB) {
|
||||
t.Helper()
|
||||
raw, err := os.ReadFile("../../../migrations/035_add_server_updated_at.sql")
|
||||
if err != nil {
|
||||
t.Fatalf("read migration: %v", err)
|
||||
}
|
||||
var body []string
|
||||
for _, line := range strings.Split(string(raw), "\n") {
|
||||
if !strings.HasPrefix(strings.TrimSpace(line), "--") {
|
||||
body = append(body, line)
|
||||
}
|
||||
}
|
||||
for _, stmt := range strings.Split(strings.Join(body, "\n"), ";") {
|
||||
if strings.TrimSpace(stmt) != "" {
|
||||
mustExec(t, db, stmt)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// seedServerConfig writes the way any client version does: through the
|
||||
// models.Configuration struct, which knows nothing about server_updated_at.
|
||||
func seedServerConfig(t *testing.T, db *gorm.DB, uuid, name string) *models.Configuration {
|
||||
t.Helper()
|
||||
cfg := &models.Configuration{
|
||||
UUID: uuid,
|
||||
OwnerUsername: "tester",
|
||||
Name: name,
|
||||
Items: models.ConfigItems{{LotName: "CPU_PULL", Quantity: 1, UnitPrice: 100}},
|
||||
ServerCount: 1,
|
||||
Line: 10,
|
||||
}
|
||||
if err := db.Create(cfg).Error; err != nil {
|
||||
t.Fatalf("seed config %s: %v", uuid, err)
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
func TestIncrementalPullMariaDB(t *testing.T) {
|
||||
serverDB := newMariaDBForPullTest(t)
|
||||
local := newLocalDBForSyncTest(t)
|
||||
if err := local.SaveSettings("127.0.0.1", 33306, "qf", "root", "test"); err != nil {
|
||||
t.Fatalf("save settings: %v", err)
|
||||
}
|
||||
|
||||
// Before migration 035: every pull is full, exactly as before.
|
||||
seedServerConfig(t, serverDB, "cfg-a", "A")
|
||||
seedServerConfig(t, serverDB, "cfg-b", "B")
|
||||
svc := syncsvc.NewServiceWithDB(serverDB, local)
|
||||
for i := 0; i < 2; i++ {
|
||||
res, err := svc.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("pre-migration import: %v", err)
|
||||
}
|
||||
if res.Imported+res.Updated != 2 {
|
||||
t.Fatalf("pre-migration pull %d: expected full pull of 2, got %+v", i, res)
|
||||
}
|
||||
}
|
||||
|
||||
applyServerUpdatedAtMigration(t, serverDB)
|
||||
applyServerUpdatedAtMigration(t, serverDB) // re-runnable
|
||||
// A client without the column in its model still inserts and updates fine.
|
||||
seedServerConfig(t, serverDB, "cfg-c", "C")
|
||||
// Age all rows past the overlap window.
|
||||
mustExec(t, serverDB, "UPDATE qt_configurations SET server_updated_at = NOW(6) - INTERVAL 1 HOUR")
|
||||
|
||||
// First pull after the migration in this process: full, sets the cursor.
|
||||
res, err := svc.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("first post-migration import: %v", err)
|
||||
}
|
||||
if res.Imported+res.Updated != 3 {
|
||||
t.Fatalf("expected full pull of 3, got %+v", res)
|
||||
}
|
||||
|
||||
// Nothing changed on the server: nothing is pulled.
|
||||
res, err = svc.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("idle incremental import: %v", err)
|
||||
}
|
||||
if res.Imported+res.Updated != 0 {
|
||||
t.Fatalf("expected empty incremental pull, got %+v", res)
|
||||
}
|
||||
|
||||
// An old-style client update (gorm Save of the whole struct) bumps the marker.
|
||||
var b models.Configuration
|
||||
if err := serverDB.Where("uuid = ?", "cfg-b").First(&b).Error; err != nil {
|
||||
t.Fatalf("load cfg-b: %v", err)
|
||||
}
|
||||
b.Name = "B renamed"
|
||||
if err := serverDB.Save(&b).Error; err != nil {
|
||||
t.Fatalf("save cfg-b: %v", err)
|
||||
}
|
||||
seedServerConfig(t, serverDB, "cfg-d", "D")
|
||||
|
||||
res, err = svc.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("incremental import: %v", err)
|
||||
}
|
||||
if res.Updated != 1 || res.Imported != 1 {
|
||||
t.Fatalf("expected 1 updated + 1 imported, got %+v", res)
|
||||
}
|
||||
localB, err := local.GetConfigurationByUUID("cfg-b")
|
||||
if err != nil || localB.Name != "B renamed" {
|
||||
t.Fatalf("expected cfg-b renamed locally, got %+v err=%v", localB, err)
|
||||
}
|
||||
|
||||
// A new process does one full pull again.
|
||||
restarted := syncsvc.NewServiceWithDB(serverDB, local)
|
||||
res, err = restarted.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("restart import: %v", err)
|
||||
}
|
||||
if res.Imported+res.Updated != 4 {
|
||||
t.Fatalf("expected full pull of 4 after restart, got %+v", res)
|
||||
}
|
||||
|
||||
// Switching to another server invalidates the cursor.
|
||||
if err := local.SaveSettings("other-host", 3306, "qf", "root", "test"); err != nil {
|
||||
t.Fatalf("save settings: %v", err)
|
||||
}
|
||||
res, err = restarted.ImportConfigurationsToLocal()
|
||||
if err != nil {
|
||||
t.Fatalf("import after server switch: %v", err)
|
||||
}
|
||||
if res.Imported+res.Updated != 4 {
|
||||
t.Fatalf("expected full pull of 4 after server switch, got %+v", res)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIncrementalProjectPullMariaDB(t *testing.T) {
|
||||
serverDB := newMariaDBForPullTest(t)
|
||||
local := newLocalDBForSyncTest(t)
|
||||
if err := local.SaveSettings("127.0.0.1", 33306, "qf", "root", "test"); err != nil {
|
||||
t.Fatalf("save settings: %v", err)
|
||||
}
|
||||
applyServerUpdatedAtMigration(t, serverDB)
|
||||
|
||||
code := "P-1"
|
||||
project := &models.Project{UUID: "prj-1", OwnerUsername: "tester", Code: code, IsActive: true}
|
||||
if err := serverDB.Create(project).Error; err != nil {
|
||||
t.Fatalf("seed project: %v", err)
|
||||
}
|
||||
mustExec(t, serverDB, "UPDATE qt_projects SET server_updated_at = NOW(6) - INTERVAL 1 HOUR")
|
||||
|
||||
svc := syncsvc.NewServiceWithDB(serverDB, local)
|
||||
if res, err := svc.ImportProjectsToLocal(); err != nil || res.Imported != 1 {
|
||||
t.Fatalf("first project pull: %+v err=%v", res, err)
|
||||
}
|
||||
if res, err := svc.ImportProjectsToLocal(); err != nil || res.Imported+res.Updated != 0 {
|
||||
t.Fatalf("idle project pull: %+v err=%v", res, err)
|
||||
}
|
||||
mustExec(t, serverDB, "UPDATE qt_projects SET name = 'renamed' WHERE uuid = 'prj-1'")
|
||||
if res, err := svc.ImportProjectsToLocal(); err != nil || res.Updated != 1 {
|
||||
t.Fatalf("incremental project pull: %+v err=%v", res, err)
|
||||
}
|
||||
}
|
||||
@@ -23,15 +23,16 @@ func (s *Service) PullPartnumberBooks() (int, error) {
|
||||
|
||||
localBookRepo := repository.NewPartnumberBookRepository(s.localDB.DB())
|
||||
|
||||
// Headers only: partnumbers_json is several MB in total and the worker runs
|
||||
// every minute, so it is fetched per book only when that book needs items.
|
||||
type serverBook struct {
|
||||
ID int `gorm:"column:id"`
|
||||
Version string `gorm:"column:version"`
|
||||
CreatedAt time.Time `gorm:"column:created_at"`
|
||||
IsActive bool `gorm:"column:is_active"`
|
||||
PartnumbersJSON string `gorm:"column:partnumbers_json"`
|
||||
ID int `gorm:"column:id"`
|
||||
Version string `gorm:"column:version"`
|
||||
CreatedAt time.Time `gorm:"column:created_at"`
|
||||
IsActive bool `gorm:"column:is_active"`
|
||||
}
|
||||
var serverBooks []serverBook
|
||||
if err := mariaDB.Raw("SELECT id, version, created_at, is_active, partnumbers_json FROM qt_partnumber_books ORDER BY created_at DESC, id DESC").Scan(&serverBooks).Error; err != nil {
|
||||
if err := mariaDB.Raw("SELECT id, version, created_at, is_active FROM qt_partnumber_books ORDER BY created_at DESC, id DESC").Scan(&serverBooks).Error; err != nil {
|
||||
return 0, fmt.Errorf("querying server partnumber books: %w", err)
|
||||
}
|
||||
slog.Info("partnumber books found on server", "count", len(serverBooks))
|
||||
@@ -40,39 +41,58 @@ func (s *Service) PullPartnumberBooks() (int, error) {
|
||||
for _, sb := range serverBooks {
|
||||
var existing localdb.LocalPartnumberBook
|
||||
err := s.localDB.DB().Where("server_id = ?", sb.ID).First(&existing).Error
|
||||
partnumbers, errPartnumbers := decodeServerPartnumbers(sb.PartnumbersJSON)
|
||||
if errPartnumbers != nil {
|
||||
slog.Error("failed to decode server partnumbers_json", "server_id", sb.ID, "error", errPartnumbers)
|
||||
continue
|
||||
}
|
||||
if err == nil {
|
||||
existing.Version = sb.Version
|
||||
existing.CreatedAt = sb.CreatedAt
|
||||
existing.IsActive = sb.IsActive
|
||||
if existing.Version != sb.Version || existing.IsActive != sb.IsActive {
|
||||
existing.Version = sb.Version
|
||||
existing.CreatedAt = sb.CreatedAt
|
||||
existing.IsActive = sb.IsActive
|
||||
if err := localBookRepo.SaveBook(&existing); err != nil {
|
||||
slog.Error("failed to update local partnumber book header", "server_id", sb.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
if localBookRepo.HasAllBookItems(existing.ID) {
|
||||
slog.Debug("partnumber book already synced, skipping", "server_id", sb.ID, "version", sb.Version)
|
||||
continue
|
||||
}
|
||||
// A server book may list partnumbers that qt_partnumber_book_items
|
||||
// lacks; re-pulling cannot fix that, so try once per process.
|
||||
if !s.markPartnumberBookRepair(sb.ID, sb.Version) {
|
||||
slog.Debug("partnumber book still incomplete after re-pull this session, skipping", "server_id", sb.ID, "version", sb.Version)
|
||||
continue
|
||||
}
|
||||
slog.Info("partnumber book header exists but catalog items are missing, re-pulling items", "server_id", sb.ID, "version", sb.Version)
|
||||
partnumbers, err := fetchServerBookPartnumbers(mariaDB, sb.ID)
|
||||
if err != nil {
|
||||
slog.Error("failed to fetch server partnumbers_json", "server_id", sb.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
existing.PartnumbersJSON = partnumbers
|
||||
if err := localBookRepo.SaveBook(&existing); err != nil {
|
||||
slog.Error("failed to update local partnumber book header", "server_id", sb.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
|
||||
localItemCount := localBookRepo.CountBookItems(existing.ID)
|
||||
if localItemCount > 0 && localBookRepo.HasAllBookItems(existing.ID) {
|
||||
slog.Debug("partnumber book already synced, skipping", "server_id", sb.ID, "version", sb.Version, "items", localItemCount)
|
||||
continue
|
||||
}
|
||||
slog.Info("partnumber book header exists but catalog items are missing, re-pulling items", "server_id", sb.ID, "version", sb.Version)
|
||||
n, err := pullBookItems(mariaDB, localBookRepo, existing.PartnumbersJSON)
|
||||
if err != nil {
|
||||
slog.Error("failed to re-pull items for existing book", "server_id", sb.ID, "error", err)
|
||||
} else {
|
||||
slog.Info("re-pulled items for existing book", "server_id", sb.ID, "version", sb.Version, "items_saved", n)
|
||||
pulled++
|
||||
if missing := localBookRepo.MissingBookItems(existing.ID); len(missing) > 0 {
|
||||
slog.Warn("partnumber book lists partnumbers absent from server qt_partnumber_book_items", "server_id", sb.ID, "version", sb.Version, "missing", missing)
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
slog.Info("pulling new partnumber book", "server_id", sb.ID, "version", sb.Version, "is_active", sb.IsActive)
|
||||
|
||||
partnumbers, err := fetchServerBookPartnumbers(mariaDB, sb.ID)
|
||||
if err != nil {
|
||||
slog.Error("failed to fetch server partnumbers_json", "server_id", sb.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
localBook := &localdb.LocalPartnumberBook{
|
||||
ServerID: sb.ID,
|
||||
Version: sb.Version,
|
||||
@@ -99,6 +119,30 @@ func (s *Service) PullPartnumberBooks() (int, error) {
|
||||
return pulled, nil
|
||||
}
|
||||
|
||||
// markPartnumberBookRepair records a re-pull attempt for a book version and
|
||||
// reports whether this is the first attempt in the current process.
|
||||
func (s *Service) markPartnumberBookRepair(serverID int, version string) bool {
|
||||
s.pnRepairMu.Lock()
|
||||
defer s.pnRepairMu.Unlock()
|
||||
if s.pnRepairAttempts == nil {
|
||||
s.pnRepairAttempts = map[int]string{}
|
||||
}
|
||||
if v, ok := s.pnRepairAttempts[serverID]; ok && v == version {
|
||||
return false
|
||||
}
|
||||
s.pnRepairAttempts[serverID] = version
|
||||
return true
|
||||
}
|
||||
|
||||
// fetchServerBookPartnumbers loads and decodes partnumbers_json of one server book.
|
||||
func fetchServerBookPartnumbers(mariaDB *gorm.DB, serverID int) (localdb.LocalStringList, error) {
|
||||
var raw string
|
||||
if err := mariaDB.Raw("SELECT partnumbers_json FROM qt_partnumber_books WHERE id = ?", serverID).Scan(&raw).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return decodeServerPartnumbers(raw)
|
||||
}
|
||||
|
||||
// pullBookItems fetches catalog items for a partnumber list from MariaDB and saves them to SQLite.
|
||||
// Returns the number of items saved.
|
||||
func pullBookItems(mariaDB *gorm.DB, repo *repository.PartnumberBookRepository, partnumbers localdb.LocalStringList) (int, error) {
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
package sync
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// serverPullOverlap re-reads rows changed shortly before the cursor, covering
|
||||
// transactions that stamped server_updated_at before the cursor but committed
|
||||
// after the previous pull read it.
|
||||
const serverPullOverlap = "INTERVAL 120 SECOND"
|
||||
|
||||
// serverPull describes how one pull of a server table should run. See
|
||||
// bible-local/decisions/2026-10-26-incremental-config-project-pull.md.
|
||||
type serverPull struct {
|
||||
table string
|
||||
scope string
|
||||
since string // cursor; empty = full pull
|
||||
startedAt string // server NOW(6) before reading; becomes the next cursor
|
||||
trackable bool // server has server_updated_at (migration 035)
|
||||
}
|
||||
|
||||
// beginServerPull decides between a full and an incremental pull of table.
|
||||
// The first pull of each table per process is full, so rows lost locally are
|
||||
// restored at least once per app start; later pulls read only rows whose
|
||||
// server_updated_at moved since the stored cursor.
|
||||
func (s *Service) beginServerPull(mariaDB *gorm.DB, table string) serverPull {
|
||||
pull := serverPull{table: table}
|
||||
if !s.serverHasChangeMarker(mariaDB, table) {
|
||||
return pull
|
||||
}
|
||||
if err := mariaDB.Raw("SELECT DATE_FORMAT(NOW(6), '%Y-%m-%d %H:%i:%s.%f')").Scan(&pull.startedAt).Error; err != nil || pull.startedAt == "" {
|
||||
slog.Warn("server pull: failed to read server time, falling back to full pull", "table", table, "error", err)
|
||||
return serverPull{table: table}
|
||||
}
|
||||
scope, err := s.localDB.ServerPullScope()
|
||||
if err != nil {
|
||||
return serverPull{table: table}
|
||||
}
|
||||
pull.scope = scope
|
||||
pull.trackable = true
|
||||
|
||||
s.pullMu.Lock()
|
||||
fullDone := s.fullPullDone[table]
|
||||
s.pullMu.Unlock()
|
||||
if fullDone {
|
||||
if since, ok := s.localDB.GetServerPullCursor(table, scope); ok {
|
||||
pull.since = since
|
||||
}
|
||||
}
|
||||
return pull
|
||||
}
|
||||
|
||||
// incrementalScope narrows query to rows changed since the cursor.
|
||||
func (p serverPull) incrementalScope(query *gorm.DB) *gorm.DB {
|
||||
return query.Where("server_updated_at >= CAST(? AS DATETIME(6)) - "+serverPullOverlap, p.since)
|
||||
}
|
||||
|
||||
// finishServerPull records a successful pull: the cursor moves to the server
|
||||
// time taken before reading, and a full pull marks the table as verified for
|
||||
// this process.
|
||||
func (s *Service) finishServerPull(pull serverPull) {
|
||||
if !pull.trackable {
|
||||
return
|
||||
}
|
||||
if err := s.localDB.SetServerPullCursor(pull.table, pull.scope, pull.startedAt); err != nil {
|
||||
slog.Warn("server pull: failed to store cursor", "table", pull.table, "error", err)
|
||||
return
|
||||
}
|
||||
if pull.since == "" {
|
||||
s.pullMu.Lock()
|
||||
if s.fullPullDone == nil {
|
||||
s.fullPullDone = map[string]bool{}
|
||||
}
|
||||
s.fullPullDone[pull.table] = true
|
||||
s.pullMu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// serverHasChangeMarker reports whether table has server_updated_at. A positive
|
||||
// answer is cached for the process; a negative one is re-checked next time so a
|
||||
// migration applied while the client runs is picked up.
|
||||
func (s *Service) serverHasChangeMarker(mariaDB *gorm.DB, table string) bool {
|
||||
s.pullMu.Lock()
|
||||
known := s.changeMarker[table]
|
||||
s.pullMu.Unlock()
|
||||
if known {
|
||||
return true
|
||||
}
|
||||
var n int64
|
||||
if err := mariaDB.Raw(`
|
||||
SELECT COUNT(*) FROM information_schema.COLUMNS
|
||||
WHERE table_schema = DATABASE() AND table_name = ? AND column_name = 'server_updated_at'
|
||||
`, table).Scan(&n).Error; err != nil || n == 0 {
|
||||
return false
|
||||
}
|
||||
s.pullMu.Lock()
|
||||
if s.changeMarker == nil {
|
||||
s.changeMarker = map[string]bool{}
|
||||
}
|
||||
s.changeMarker[table] = true
|
||||
s.pullMu.Unlock()
|
||||
return true
|
||||
}
|
||||
@@ -30,6 +30,13 @@ type Service struct {
|
||||
directDB *gorm.DB
|
||||
pricelistMu sync.Mutex // prevents concurrent pricelist syncs
|
||||
schemaOnce sync.Once // ensures ensureClientSchemaStateTable runs at most once per process
|
||||
|
||||
pnRepairMu sync.Mutex
|
||||
pnRepairAttempts map[int]string // server book id -> version already re-pulled this process
|
||||
|
||||
pullMu sync.Mutex
|
||||
changeMarker map[string]bool // server table -> has server_updated_at
|
||||
fullPullDone map[string]bool // server table -> full pull succeeded this process
|
||||
}
|
||||
|
||||
// NewService creates a new sync service
|
||||
@@ -118,13 +125,27 @@ func (s *Service) ImportConfigurationsToLocal() (*ConfigImportResult, error) {
|
||||
|
||||
configRepo := repository.NewConfigurationRepository(mariaDB)
|
||||
result := &ConfigImportResult{}
|
||||
pull := s.beginServerPull(mariaDB, "qt_configurations")
|
||||
|
||||
offset := 0
|
||||
const limit = 200
|
||||
for {
|
||||
serverConfigs, _, err := configRepo.ListAll(offset, limit)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing server configurations: %w", err)
|
||||
var serverConfigs []models.Configuration
|
||||
if pull.since != "" {
|
||||
// Incremental: the changed set is small, read it in one go.
|
||||
if offset > 0 {
|
||||
break
|
||||
}
|
||||
if err := pull.incrementalScope(mariaDB.Model(&models.Configuration{})).
|
||||
Order("created_at DESC, id DESC").
|
||||
Find(&serverConfigs).Error; err != nil {
|
||||
return nil, fmt.Errorf("listing changed server configurations: %w", err)
|
||||
}
|
||||
} else {
|
||||
serverConfigs, _, err = configRepo.ListAll(offset, limit)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing server configurations: %w", err)
|
||||
}
|
||||
}
|
||||
if len(serverConfigs) == 0 {
|
||||
break
|
||||
@@ -171,6 +192,7 @@ func (s *Service) ImportConfigurationsToLocal() (*ConfigImportResult, error) {
|
||||
offset += len(serverConfigs)
|
||||
}
|
||||
|
||||
s.finishServerPull(pull)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
@@ -184,13 +206,27 @@ func (s *Service) ImportProjectsToLocal() (*ProjectImportResult, error) {
|
||||
|
||||
projectRepo := repository.NewProjectRepository(mariaDB)
|
||||
result := &ProjectImportResult{}
|
||||
pull := s.beginServerPull(mariaDB, "qt_projects")
|
||||
|
||||
offset := 0
|
||||
const limit = 200
|
||||
for {
|
||||
serverProjects, _, err := projectRepo.List(offset, limit, true)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing server projects: %w", err)
|
||||
var serverProjects []models.Project
|
||||
if pull.since != "" {
|
||||
// Incremental: the changed set is small, read it in one go.
|
||||
if offset > 0 {
|
||||
break
|
||||
}
|
||||
if err := pull.incrementalScope(mariaDB.Model(&models.Project{})).
|
||||
Order("created_at DESC").
|
||||
Find(&serverProjects).Error; err != nil {
|
||||
return nil, fmt.Errorf("listing changed server projects: %w", err)
|
||||
}
|
||||
} else {
|
||||
serverProjects, _, err = projectRepo.List(offset, limit, true)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing server projects: %w", err)
|
||||
}
|
||||
}
|
||||
if len(serverProjects) == 0 {
|
||||
break
|
||||
@@ -247,6 +283,7 @@ func (s *Service) ImportProjectsToLocal() (*ProjectImportResult, error) {
|
||||
offset += len(serverProjects)
|
||||
}
|
||||
|
||||
s.finishServerPull(pull)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
@@ -291,28 +328,30 @@ func (s *Service) NeedSync() (bool, error) {
|
||||
mariaDB, err := s.getDB()
|
||||
if err == nil {
|
||||
pricelistRepo := repository.NewPricelistRepository(mariaDB)
|
||||
sources := []models.PricelistSource{
|
||||
models.PricelistSourceEstimate,
|
||||
models.PricelistSourceWarehouse,
|
||||
models.PricelistSourceCompetitor,
|
||||
sources := []string{
|
||||
string(models.PricelistSourceEstimate),
|
||||
string(models.PricelistSourceWarehouse),
|
||||
string(models.PricelistSourceCompetitor),
|
||||
}
|
||||
allMatch := true
|
||||
for _, source := range sources {
|
||||
latestServer, err := pricelistRepo.GetLatestActiveBySource(string(source))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
latestLocal, err := s.localDB.GetLatestLocalPricelistBySource(string(source))
|
||||
if err != nil {
|
||||
return true, nil
|
||||
}
|
||||
if latestServer.ID != latestLocal.ServerID {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
if allMatch {
|
||||
latestServer, err := pricelistRepo.GetLatestActiveIDsBySources(sources)
|
||||
if err != nil {
|
||||
slog.Warn("failed to check latest server pricelists", "error", err)
|
||||
return false, nil
|
||||
}
|
||||
for _, source := range sources {
|
||||
serverID, ok := latestServer[source]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
latestLocal, err := s.localDB.GetLatestLocalPricelistBySource(source)
|
||||
if err != nil {
|
||||
return true, nil
|
||||
}
|
||||
if serverID != latestLocal.ServerID {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
@@ -941,6 +980,12 @@ func (s *Service) PushPendingChanges() (int, error) {
|
||||
if _, err := s.EnsureReadinessForSync(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return s.pushPendingChanges()
|
||||
}
|
||||
|
||||
// pushPendingChanges pushes without its own readiness check; the caller has
|
||||
// just run EnsureReadinessForSync (and with it the client heartbeat).
|
||||
func (s *Service) pushPendingChanges() (int, error) {
|
||||
|
||||
removed, err := s.localDB.PurgeOrphanConfigurationPendingChanges()
|
||||
if err != nil {
|
||||
|
||||
@@ -3,12 +3,29 @@ package sync
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"math/rand/v2"
|
||||
"time"
|
||||
|
||||
"git.mchus.pro/mchus/quoteforge/internal/db"
|
||||
)
|
||||
|
||||
// Worker performs background synchronization at regular intervals
|
||||
// WorkerJitter spreads background sync cycles by ±30% of the interval so that
|
||||
// clients started at the same time do not hit MariaDB in lockstep.
|
||||
const WorkerJitter = 0.3
|
||||
|
||||
// jitteredInterval returns base scaled by a random factor in [1-WorkerJitter, 1+WorkerJitter].
|
||||
func jitteredInterval(base time.Duration) time.Duration {
|
||||
factor := 1 - WorkerJitter + rand.Float64()*2*WorkerJitter
|
||||
return time.Duration(float64(base) * factor)
|
||||
}
|
||||
|
||||
// nextCycleDelay returns how long to wait so the next cycle starts a jittered
|
||||
// interval after started; zero if the cycle itself took longer than that.
|
||||
func nextCycleDelay(started time.Time, base time.Duration) time.Duration {
|
||||
return max(jitteredInterval(base)-time.Since(started), 0)
|
||||
}
|
||||
|
||||
// Worker performs background synchronization at jittered intervals
|
||||
type Worker struct {
|
||||
service *Service
|
||||
connMgr *db.ConnectionManager
|
||||
@@ -37,12 +54,14 @@ func (w *Worker) isOnline() bool {
|
||||
func (w *Worker) Start(ctx context.Context) {
|
||||
w.logger.Info("starting background sync worker", "interval", w.interval)
|
||||
|
||||
ticker := time.NewTicker(w.interval)
|
||||
defer ticker.Stop()
|
||||
|
||||
// Run once immediately
|
||||
// Run once immediately. The next cycle is scheduled from the start of the
|
||||
// previous one, so heartbeats are at most (1+WorkerJitter)*interval apart.
|
||||
started := time.Now()
|
||||
w.runSync()
|
||||
|
||||
timer := time.NewTimer(nextCycleDelay(started, w.interval))
|
||||
defer timer.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
@@ -51,8 +70,10 @@ func (w *Worker) Start(ctx context.Context) {
|
||||
case <-w.stopCh:
|
||||
w.logger.Info("background sync worker stopped")
|
||||
return
|
||||
case <-ticker.C:
|
||||
case <-timer.C:
|
||||
started = time.Now()
|
||||
w.runSync()
|
||||
timer.Reset(nextCycleDelay(started, w.interval))
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -79,7 +100,7 @@ func (w *Worker) runSync() {
|
||||
"reason_code", readiness.ReasonCode,
|
||||
"reason_text", readiness.ReasonText,
|
||||
)
|
||||
} else if pushed, err := w.service.PushPendingChanges(); err != nil {
|
||||
} else if pushed, err := w.service.pushPendingChanges(); err != nil {
|
||||
w.logger.Warn("background sync: failed to push pending changes", "error", err)
|
||||
} else if pushed > 0 {
|
||||
w.logger.Info("background sync: pushed pending changes", "count", pushed)
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
package sync
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestJitteredIntervalStaysWithinBounds(t *testing.T) {
|
||||
base := time.Minute
|
||||
lo := time.Duration(float64(base) * (1 - WorkerJitter))
|
||||
hi := time.Duration(float64(base) * (1 + WorkerJitter))
|
||||
seen := map[time.Duration]struct{}{}
|
||||
for i := 0; i < 1000; i++ {
|
||||
d := jitteredInterval(base)
|
||||
if d < lo || d > hi {
|
||||
t.Fatalf("interval %v outside [%v, %v]", d, lo, hi)
|
||||
}
|
||||
seen[d] = struct{}{}
|
||||
}
|
||||
if len(seen) < 100 {
|
||||
t.Fatalf("expected spread of intervals, got %d distinct values", len(seen))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
-- Tables affected: qt_configurations, qt_projects
|
||||
-- recovery.not-started: safe to re-run; ADD COLUMN IF NOT EXISTS / ADD INDEX IF NOT EXISTS
|
||||
-- recovery.partial: ALTER TABLE qt_configurations DROP COLUMN server_updated_at; ALTER TABLE qt_projects DROP COLUMN server_updated_at;
|
||||
-- recovery.completed: no action needed
|
||||
-- verify: qt_configurations.server_updated_at column missing | SELECT 1 FROM information_schema.COLUMNS WHERE table_schema=DATABASE() AND table_name='qt_configurations' AND column_name='server_updated_at' HAVING COUNT(*)=0
|
||||
-- verify: qt_projects.server_updated_at column missing | SELECT 1 FROM information_schema.COLUMNS WHERE table_schema=DATABASE() AND table_name='qt_projects' AND column_name='server_updated_at' HAVING COUNT(*)=0
|
||||
|
||||
-- Server-maintained change marker for incremental client pulls. Never written by
|
||||
-- clients: DEFAULT on INSERT and ON UPDATE on any row change, so clients that do not
|
||||
-- know the column keep working unchanged. See
|
||||
-- bible-local/decisions/2026-10-26-incremental-config-project-pull.md.
|
||||
ALTER TABLE qt_configurations
|
||||
ADD COLUMN IF NOT EXISTS server_updated_at TIMESTAMP(6) NOT NULL
|
||||
DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
|
||||
ADD INDEX IF NOT EXISTS idx_qt_configurations_server_updated_at (server_updated_at, id);
|
||||
|
||||
ALTER TABLE qt_projects
|
||||
ADD COLUMN IF NOT EXISTS server_updated_at TIMESTAMP(6) NOT NULL
|
||||
DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
|
||||
ADD INDEX IF NOT EXISTS idx_qt_projects_server_updated_at (server_updated_at, id);
|
||||
@@ -0,0 +1,26 @@
|
||||
# QuoteForge v3.50
|
||||
|
||||
Дата релиза: 2026-10-26
|
||||
Тег: `v3.50`
|
||||
|
||||
Предыдущий релиз: `v3.49`
|
||||
|
||||
## Ключевые изменения
|
||||
|
||||
- локальный API и первоначальная настройка защищены от CSRF и DNS rebinding; сохранённый пароль не отправляется при изменении хоста, порта или пользователя без явного ввода пароля (#2);
|
||||
- устранены XSS через LOT, BOM и сообщения синхронизации; обработчики корзины и аренды получают LOT через data-атрибуты (#4);
|
||||
- перед сбросом локальной базы создаётся отдельный проверенный архив; несинхронизированные данные защищены отказом без явного принудительного сброса, переменная QFS_RESET_LOCAL_DB больше не запускает сброс (#7);
|
||||
- в настройках подключения MariaDB доступны TLS preferred, required, verify-ca и verify-full, а также CA-сертификат; preferred допускает соединение без TLS, verify-full проверяет сертификат и имя сервера (#5);
|
||||
- SQL-миграции выполняются по одной сессии на файл, запускают verify-проверки до записи в историю и не скрывают ошибки внешних ключей (#3);
|
||||
- минимальная версия Go повышена до 1.27.3, обновлены зависимости; релизная сборка блокируется при достижимых уязвимостях по govulncheck (#6).
|
||||
|
||||
## Проверки
|
||||
|
||||
Пройдены Go-тесты, go vet, проверка XSS и сборки macOS arm64 / Windows amd64.
|
||||
Govulncheck: 0 достижимых уязвимостей; остаётся одна запись на уровне модуля,
|
||||
уязвимый код которого приложение не вызывает.
|
||||
|
||||
## Запуск на macOS
|
||||
|
||||
Снимите карантинный атрибут через терминал: `xattr -d com.apple.quarantine /path/to/qfs-darwin-arm64`
|
||||
После этого бинарник запустится без предупреждения Gatekeeper.
|
||||
@@ -0,0 +1,26 @@
|
||||
# QuoteForge v3.51
|
||||
|
||||
Дата релиза: 2026-10-26
|
||||
Тег: `v3.51`
|
||||
|
||||
Предыдущий релиз: `v3.50`
|
||||
|
||||
## Ключевые изменения
|
||||
|
||||
- устранена утечка подключений к MariaDB: при синхронизации клиент каждые 30 секунд открывал новый пул, не закрывая старый, и держал до ~30 соединений; теперь не больше 2, простаивающие закрываются через минуту;
|
||||
- фоновая синхронизация раз в минуту со случайным разбросом ±30%, чтобы клиенты не обращались к серверу одновременно; статус «онлайн» в списке пользователей считается по интервалу +30%;
|
||||
- списки проектов и конфигураций подгружают с сервера только изменённые записи (миграция `035`, колонка `server_updated_at`): ~0,35 с вместо ~27 с и 3 МБ; полная загрузка — один раз при запуске приложения; старые клиенты и сервер без миграции работают как раньше;
|
||||
- PN-книги больше не скачиваются целиком на каждом цикле: забираются только заголовки, содержимое — для новых или неполных книг; неполная книга докачивается один раз за запуск с предупреждением в логе;
|
||||
- проверка новых прайслистов одним запросом вместо трёх, одна отметка о клиенте на цикл синхронизации, настройки конфигуратора не перезаписываются без изменений;
|
||||
- исправлен мигратор: файл с двумя и более `verify`-проверками применялся, но не записывался в историю миграций.
|
||||
|
||||
## Проверки
|
||||
|
||||
Пройдены Go-тесты и go vet. Инкрементальная загрузка проверена интеграционным тестом
|
||||
на MariaDB 11.8 со схемой продакшена и на продакшен-сервере после миграции `035`.
|
||||
Govulncheck: 0 достижимых уязвимостей.
|
||||
|
||||
## Запуск на macOS
|
||||
|
||||
Снимите карантинный атрибут через терминал: `xattr -d com.apple.quarantine /path/to/qfs-darwin-arm64`
|
||||
После этого бинарник запустится без предупреждения Gatekeeper.
|
||||
@@ -0,0 +1,35 @@
|
||||
# QuoteForge v3.52
|
||||
|
||||
Дата релиза: 2026-10-26
|
||||
Тег: `v3.52`
|
||||
|
||||
Предыдущий релиз: `v3.51`
|
||||
|
||||
## Ключевые изменения
|
||||
|
||||
- серверный мигратор автоматически создаёт проверяемые архивы перед изменениями и перед каждым SQL-файлом; при ошибке восстанавливает всю сессию, включая данные, триггеры и историю миграций;
|
||||
- добавлен `-migrate-restore <archive.zip>` для восстановления после прерывания процесса; незавершённая сессия блокирует повторный запуск до восстановления;
|
||||
- SQL-парсер учитывает кавычки, комментарии и точки с запятой внутри литералов; некорректный ввод отклоняется до выполнения;
|
||||
- миграционные команды используют выделенное соединение без фоновых подключений runtime; требуют окна обслуживания и существующей DBA-учётной записи, права обычного пользователя приложения не расширяются;
|
||||
- завершена приёмка issues #2–#7: настоящие setup/runtime-роутеры, XSS в браузерном DOM, CLI сброса и восстановления, TLS на MariaDB и остановка релизной сборки при ошибке сканера уязвимостей.
|
||||
|
||||
## База данных
|
||||
|
||||
Новых SQL-миграций нет. Для обновления с v3.51 изменять схему БД не требуется.
|
||||
В этом релизе изменён движок выполнения существующих миграций.
|
||||
|
||||
## Проверки
|
||||
|
||||
Пройдены `go test ./...`, `go vet ./...`, JavaScript-регрессии, headless Chrome,
|
||||
интеграционные проверки TLS и миграций, сборки macOS arm64 и Windows amd64.
|
||||
Govulncheck: 0 достижимых уязвимостей; одна запись на уровне невызываемой зависимости.
|
||||
|
||||
На предоставленном дампе от 2026-10-26 проверены 45 таблиц и 27 триггеров
|
||||
в отдельном контейнере MariaDB 11.8.3 с utf8mb4_uca1400_ai_ci: полный restore,
|
||||
миграция 035 на копии pre-035 схемы, намеренный последующий сбой, откат всей
|
||||
сессии, успешный повтор и восстановление исходного состояния. Рабочая БД
|
||||
и исходный дамп не изменялись.
|
||||
|
||||
## Запуск на macOS
|
||||
|
||||
Снимите карантинный атрибут через терминал: `xattr -d com.apple.quarantine /path/to/qfs-darwin-arm64`
|
||||
@@ -0,0 +1,98 @@
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const {spawnSync} = require('node:child_process');
|
||||
|
||||
const index = fs.readFileSync('web/templates/index.html', 'utf8');
|
||||
const base = fs.readFileSync('web/templates/base.html', 'utf8');
|
||||
function extract(source, name) {
|
||||
const match = source.match(new RegExp(`^( *)(?:async )?function ${name}\\(`, 'm'));
|
||||
assert(match, `Missing function ${name}`);
|
||||
const end = source.indexOf(`\n${match[1]}}`, match.index);
|
||||
assert(end > match.index);
|
||||
return source.slice(match.index, end + match[1].length + 2);
|
||||
}
|
||||
|
||||
const functions = [
|
||||
...['renderMultiSelectTab', 'renderMultiSelectTabWithSections', 'renderRentalTab', '_renderBOMParsedTable'].map(n => extract(index, n)),
|
||||
...['showToast', 'loadSyncInfo', 'repairPendingChanges'].map(n => extract(base, n)),
|
||||
].join('\n');
|
||||
const ids = new Set([...functions.matchAll(/getElementById\('([^']+)'\)/g)].map(m => m[1]));
|
||||
const fixtures = [...ids].map(id => id.endsWith('-body') ? `<table><tbody id="${id}"></tbody></table>` : `<div id="${id}"></div>`).join('');
|
||||
const listeners = index.split('<script>')[1].split('// Tab configuration')[0];
|
||||
const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'qfs-browser-security-'));
|
||||
const target = path.join(directory, 'test.html');
|
||||
const test = `
|
||||
let cart = [], allComponents = [], currentTab = 'cpu', configUUID = 'example';
|
||||
let rentalUpliftPercent = 0, bomRows = [], reply = {}, calls = [];
|
||||
window.__injected = 0;
|
||||
const getTabForCategory = () => 'cpu', getDisplayPrice = () => 1;
|
||||
const qualityDotHtml = () => '', formatMoney = String, formatPriceOrNA = String, formatCurrency = String;
|
||||
const ciStr = s => s.toLowerCase(), getComponentCategory = c => c.category;
|
||||
const _setBomUIError = () => {}, _ensureBomDatalist = () => {}, _getRowLotQtyPerPN = () => 1;
|
||||
const renderBOMLotAllocationsEditor = () => '', buildRentalItemsPayload = () => [];
|
||||
const removeFromCart = lot => calls.push(['remove', lot]);
|
||||
const updateMultiQuantity = (lot, value) => calls.push(['quantity', lot, value]);
|
||||
const onRentalConditionChange = (lot, value) => calls.push(['condition', lot, value]);
|
||||
const showAutocompleteEditItem = lot => calls.push(['edit', lot]);
|
||||
window.fetch = async () => ({ok: true, json: async () => reply});
|
||||
function check(ok, message) { if (!ok) throw new Error(message); }
|
||||
function inert(id, payload) {
|
||||
const el = document.getElementById(id);
|
||||
check(el.textContent.includes(payload), id + ': text did not survive');
|
||||
check(!el.querySelector('img,script,iframe'), id + ': injected element');
|
||||
}
|
||||
(async () => {
|
||||
for (const lot of [
|
||||
"LOT'\\\"><img src=x onerror=window.__injected++>",
|
||||
'LOTa');window.__injected++;//',
|
||||
'LOT\\\\&<>"',
|
||||
]) {
|
||||
cart = [{lot_name: lot, description: lot, category: 'cpu', quantity: 1}];
|
||||
for (const render of [() => renderMultiSelectTab([]), () => renderMultiSelectTabWithSections([{title: 'CPU', categories: ['cpu']}])]) {
|
||||
render();
|
||||
const root = document.getElementById('tab-content');
|
||||
check(!root.querySelector('img,script'), 'cart: injected element');
|
||||
const quantity = root.querySelector('[data-cart-action="quantity"]');
|
||||
check(quantity.dataset.lot === lot, 'cart: dataset changed');
|
||||
quantity.value = '2'; quantity.dispatchEvent(new Event('change', {bubbles: true}));
|
||||
root.querySelector('[data-cart-action="edit"]').dispatchEvent(new FocusEvent('focusin', {bubbles: true}));
|
||||
root.querySelector('[data-cart-action="remove"] svg').dispatchEvent(new MouseEvent('click', {bubbles: true}));
|
||||
check(calls.length === 3 && calls.every(c => c[1] === lot), 'cart: delegated action changed LOT');
|
||||
calls = [];
|
||||
}
|
||||
reply = {items: [{...cart[0], condition: 'new', buy_price: 1, price: 1, one_time: 1, weekly: 1}]};
|
||||
await renderRentalTab();
|
||||
inert('rental-body', lot);
|
||||
const condition = document.querySelector('[data-cart-action="condition"]');
|
||||
condition.dispatchEvent(new Event('change', {bubbles: true}));
|
||||
check(calls.length === 1 && calls[0][1] === lot, 'rental: delegated action changed LOT'); calls = [];
|
||||
bomRows = [{resolved_lot: lot, vendor_pn: lot, description: lot, quantity: 1, resolution_source: 'manual'}];
|
||||
_renderBOMParsedTable(); inert('bom-table-body', lot);
|
||||
showToast(lot, 'error'); inert('toast', lot);
|
||||
reply = {is_online: false, config_count: 0, project_count: 0,
|
||||
pending_changes: [{entity_uuid: 'example', operation: lot, entity_type: lot, last_error: lot}],
|
||||
errors: [{message: lot}]};
|
||||
await loadSyncInfo(); inert('modal-pending-list', lot); inert('modal-errors-list', lot);
|
||||
reply = {success: true, repaired: 0, remaining_errors: [lot]};
|
||||
await repairPendingChanges(); inert('repair-result', lot);
|
||||
}
|
||||
await new Promise(resolve => setTimeout(resolve, 50));
|
||||
check(window.__injected === 0, 'payload executed');
|
||||
document.getElementById('result').textContent = 'PASS';
|
||||
})().catch(error => {document.getElementById('result').textContent = 'FAIL: ' + error.stack;});
|
||||
`;
|
||||
fs.writeFileSync(target, `<!doctype html><meta charset="utf-8">${fixtures}<pre id="result">PENDING</pre><script>${fs.readFileSync('web/static/html-escape.js', 'utf8')}\n${functions}\n${listeners}\n${test}</script>`);
|
||||
const chrome = process.env.CHROME_BIN || '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome';
|
||||
try {
|
||||
const result = spawnSync(chrome, ['--headless', '--disable-gpu', '--no-first-run', '--disable-background-networking',
|
||||
'--user-data-dir=' + path.join(directory, 'profile'), '--virtual-time-budget=1000', '--dump-dom', 'file://' + target], {encoding: 'utf8', timeout: 30000});
|
||||
assert.ifError(result.error);
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
const outcome = result.stdout.match(/<pre id="result">([\s\S]*?)<\/pre>/)?.[1];
|
||||
assert.equal(outcome, 'PASS', outcome || 'Browser did not finish');
|
||||
console.log('Browser HTML security regressions passed');
|
||||
} finally {
|
||||
fs.rmSync(directory, {recursive: true, force: true});
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const {spawnSync} = require('node:child_process');
|
||||
|
||||
const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'qfs-release-gate-'));
|
||||
try {
|
||||
fs.copyFileSync('Makefile', path.join(directory, 'Makefile'));
|
||||
fs.copyFileSync('scripts/release.sh', path.join(directory, 'release.sh'));
|
||||
const tools = path.join(directory, 'tools');
|
||||
fs.mkdirSync(tools);
|
||||
fs.writeFileSync(path.join(tools, 'go'), '#!/bin/sh\nprintf "%s\\n" "$*" >> "$QFS_TEST_GO_CALLS"\nexit 23\n', {mode: 0o755});
|
||||
const callsPath = path.join(directory, 'calls');
|
||||
const env = {...process.env, PATH: tools + path.delimiter + process.env.PATH, QFS_TEST_GO_CALLS: callsPath};
|
||||
const cases = [
|
||||
...['build-release', 'build-linux', 'build-macos', 'build-windows', 'build-all'].map(target => ['make', [target]]),
|
||||
['bash', ['release.sh']],
|
||||
];
|
||||
for (const [cmd, args] of cases) {
|
||||
fs.writeFileSync(callsPath, '');
|
||||
const result = spawnSync(cmd, args, {cwd: directory, env, encoding: 'utf8', timeout: 10000});
|
||||
assert.ifError(result.error);
|
||||
assert.notEqual(result.status, 0, `${cmd} ${args}: scanner failure was ignored`);
|
||||
const calls = fs.readFileSync(callsPath, 'utf8').trim().split('\n');
|
||||
assert.deepEqual(calls, ['run golang.org/x/vuln/cmd/govulncheck@v1.9.0 ./...']);
|
||||
assert(!fs.existsSync(path.join(directory, 'bin')), 'Build artifacts created after failed scan');
|
||||
assert(!fs.existsSync(path.join(directory, 'releases')), 'Release artifacts created after failed scan');
|
||||
}
|
||||
console.log('Release security failure checks passed');
|
||||
} finally {
|
||||
fs.rmSync(directory, {recursive: true, force: true});
|
||||
}
|
||||
Reference in new issue
Block a user