bible/rules/patterns/go-database/contract.md

Contract: Database Patterns (Go / MySQL / MariaDB)

Version: 1.9

MySQL Transaction Cursor Safety (CRITICAL)

Never execute SQL on the same transaction while iterating over a query result cursor.

This is the most common source of invalid connection and unexpected EOF driver panics.

Rule

Use a two-phase approach: read all rows first, close the cursor, then execute writes.

// WRONG — executes SQL inside rows.Next() loop on the same tx
rows, _ := tx.Query("SELECT id FROM machines")
for rows.Next() {
    var id string
    rows.Scan(&id)
    tx.Exec("UPDATE machines SET processed=1 WHERE id=?", id) // DEADLOCK / driver panic
}

// CORRECT — collect IDs first, then write
rows, _ := tx.Query("SELECT id FROM machines")
var ids []string
for rows.Next() {
    var id string
    rows.Scan(&id)
    ids = append(ids, id)
}
rows.Close() // explicit close before any write

for _, id := range ids {
    tx.Exec("UPDATE machines SET processed=1 WHERE id=?", id)
}

This applies to:

• database/sql with manual transactions
• GORM db.Raw().Scan() inside a db.Transaction() callback
• Any loop that calls a repository method while a cursor is open

Soft Delete / Archive Pattern

Do not use hard deletes for user-visible records. Use an archive flag.

// Schema: is_active bool DEFAULT true
// "Delete" = set is_active = false
// Restore = set is_active = true

// All list queries must filter:
WHERE is_active = true

• Never physically delete rows that have foreign key references or history.
• Hard delete is only acceptable for orphaned/temporary data with no audit trail requirement.
• Archive operations must be reversible from the UI.

GORM Virtual Fields

Use the correct tag based on whether the field should exist in the DB schema:

// Field computed at runtime, column must NOT exist in DB (excludes from migrations AND queries)
Count int `gorm:"-"`

// Field computed at query time via JOIN/SELECT, column must NOT be in migrations
// but IS populated from query results
DisplayName string `gorm:"-:migration"`

• gorm:"-" — fully ignored: no migration, no read, no write.
• gorm:"-:migration" — skip migration only; GORM will still read/write if the column exists.
• Do not use gorm:"-" for JOIN-populated fields — the value will always be zero.

Fail-Fast DB Check on Startup

Always verify the database connection before starting the HTTP server.

sqlDB, err := db.DB()
if err != nil || sqlDB.Ping() != nil {
    log.Fatal("database unavailable, refusing to start")
}
// then: run migrations, then: start gin/http server

Never start serving traffic with an unverified DB connection. Fail loudly at boot.

N+1 Query Prevention

Use JOINs or batch IN queries. Never query inside a loop over rows from another query.

// WRONG
for _, pricelist := range pricelists {
    items, _ := repo.GetItems(pricelist.ID) // N queries
}

// CORRECT
items, _ := repo.GetItemsByPricelistIDs(ids) // 1 query with WHERE id IN (...)
// then group in Go

Automatic Backup During Migration

The migration engine is responsible for all backup steps. The operator must never be required to take a backup manually.

Backup naming, storage, archive format, retention, and restore-readiness must follow the backup-management contract.

Full DB Backup on New Migrations

When the migration engine detects that new (unapplied) migrations exist, it must take a full database backup before applying any of them.

Rules:

• The full backup must complete and be verified before the first migration step runs.
• The backup must be triggered by the application's own backup mechanism; do not assume mysql, mysqldump, pg_dump, or any other external DB client tool is present on the operator's machine.
• Before creating the backup, verify that the backup output path resolves outside the git worktree and is not tracked or staged in git.

Per-Table Backup Before Each Table Migration

Before applying a migration step that affects a specific table, take a targeted backup of that table.

Rules:

• A per-table backup must be created immediately before the migration step that modifies that table.
• If a single migration step touches multiple tables, back up each affected table before the step runs.
• Per-table backups are in addition to the full DB backup; they are not a substitute for it.

Session Rollback on Failure

If any migration step fails during a session, the engine must roll back all migrations applied in that session.

Rules:

• "Session" means all migration steps started in a single run of the migration engine.
• On failure, roll back every step applied in the current session in reverse order before surfacing the error.
• If rollback of a step is not possible (e.g., the operation is not reversible in MySQL without the per-table backup), restore from the per-table backup taken before that step.
• After rollback or restore, the database must be in the same state as before the session started.
• The engine must emit structured diagnostics that identify which step failed, which steps were rolled back, and the final database state.

Migration Policy

• For local-first desktop applications, startup and migration recovery must follow the local-first-recovery contract.
• Migrations are numbered sequentially and never modified after merge.
• Each migration must be reversible where possible (document rollback in a comment).
• Never rename a column in one migration step — add new, backfill, drop old across separate deploys.
• Auto-apply migrations on startup is acceptable for internal tools; document if used.

SQL Migration File Format

Every .sql migration file must begin with a structured header block:

-- Tables affected: supplier, lot_log
-- recovery.not-started: No action required.
-- recovery.partial:     DELETE FROM parts_log WHERE created_by = 'migration';
-- recovery.completed:   Same as partial.
-- verify: No orphaned supplier_code | SELECT supplier_code FROM parts_log pl LEFT JOIN supplier s ON s.supplier_code = pl.supplier_code WHERE s.supplier_code IS NULL LIMIT 1
-- verify: No empty supplier_code   | SELECT supplier_name FROM supplier WHERE supplier_code = '' LIMIT 1

-- Tables affected: — comma-separated list of tables the migration touches. Used by the backup engine to take a targeted pre-migration backup. Omit only if no table can be identified; the engine falls back to full DB backup.

-- recovery.*: — human-readable rollback SQL for each migration state (not-started, partial, completed). Executed manually by an operator if automatic restore fails. Must be correct, copy-pasteable SQL.

-- verify: — post-migration assertion query. Format: -- verify: <description> | <SQL>. The engine runs the query after all statements in the file succeed. If the query returns any row, the migration is considered failed and is rolled back. Write the query so it returns a row only when something is wrong:

-- verify: Orphaned FK refs | SELECT id FROM child c LEFT JOIN parent p ON p.id = c.parent_id WHERE p.id IS NULL LIMIT 1
--                            ^ returns a row = bad  ^ returns nothing = good

• Verify queries must filter out NULL/empty values that would cause false positives: add AND col IS NOT NULL AND col != ''.
• A migration is only recorded as applied after all verify checks pass.
• Verify checks are not a substitute for testing; they are a last-resort safety net on production.

Pre-Production Migration Testing in Docker

Before applying a set of new migrations to production, always validate them against a copy of the production database in a local MariaDB Docker container that matches the production version and collation.

# Start container matching production (MariaDB 11.8, utf8mb4_uca1400_ai_ci)
docker run -d --name pf_test \
  -e MYSQL_ROOT_PASSWORD=test -e MYSQL_DATABASE=RFQ_LOG \
  mariadb:11.8 --character-set-server=utf8mb4 --collation-server=utf8mb4_uca1400_ai_ci

# Load production dump
docker exec -i pf_test mariadb -uroot -ptest RFQ_LOG < prod_dump.sql

# Run migrations via pfs (uses real migration engine + verify checks, no backup)
./pfs -migrate-dsn "root:test@tcp(127.0.0.1:3306)/RFQ_LOG?parseTime=true&charset=utf8mb4&multiStatements=true" \
      -no-backup -verbose

The -migrate-dsn flag connects to the given DSN, runs all pending migrations, runs verify checks, and exits. No config file, no server, no browser.

Rules:

• Always test on a dump of the current production database, not a fixture — schema drift and real data distributions expose bugs that fixtures miss.
• The Docker container must use the same MariaDB version and --collation-server as production.
• Each migration file is executed as a single session so SET FOREIGN_KEY_CHECKS = 0 applies to all its statements. Never test by running statements from a migration file individually across separate sessions — the session variable will reset between them.
• If any migration fails in Docker, fix the SQL before touching production. Do not rely on "it will be different in production."

SQL Migration Authoring — Common Pitfalls

Semicolons inside string literals break naive splitters. The migration engine uses a quote-aware statement splitter. Do not rely on external tools that split on bare ;. When writing supplier/product names with punctuation, use commas — not semicolons — as separators in string literals. A semicolon inside 'COMPANY; LTD' will break any naive split(";") approach.

SET FOREIGN_KEY_CHECKS = 0 only applies to the current session. This is a session variable. If statements run in separate connections (e.g. via individual subprocess calls), FK checks are re-enabled for each new connection. Always run an entire migration file as one session. The pfs migration engine runs all statements in a file on the same GORM db handle, which reuses the same connection.

Verify queries must exclude NULL values. A query like SELECT c.col FROM child c LEFT JOIN parent p ON p.id = c.id WHERE p.id IS NULL will return rows with c.col = NULL if the child table has rows with a NULL FK value. Add AND c.col IS NOT NULL AND c.col != '' to avoid false failures.

Catch-all INSERT for referential integrity before adding FK constraints. When adding a FK constraint to a table that previously had no FK (legacy data may have orphaned references), add a catch-all step before the constraint:

-- Ensure every value referenced in child table exists in parent before adding FK.
INSERT IGNORE INTO parent (name)
SELECT DISTINCT c.fk_col FROM child c
LEFT JOIN parent p ON p.name = c.fk_col
WHERE p.name IS NULL AND c.fk_col IS NOT NULL AND c.fk_col != '';

This is not a hack — it repairs data that was valid before the constraint existed. Never delete orphaned child rows unless data loss is acceptable.