PriceForge/bible-local/history.md

Change History

Architectural decisions and significant refactoring are recorded here. Every architectural decision MUST be documented in this file.

---

2026-03-07: Embedded Scheduler with MariaDB Advisory Locks

Decision

Moved periodic cron-style maintenance from an external cmd/cron runtime requirement into the main pfs application. Every app instance may run the scheduler loop, but each job is serialized across the shared MariaDB using advisory locks.

What changed

• Added embedded scheduler in internal/scheduler/scheduler.go.
• Added persisted scheduler state table qt_scheduler_runs via migration 030_add_scheduler_runs.sql.
• pfs now starts the scheduler on boot when scheduler.enabled = true.
• Added GET /api/admin/pricing/scheduler-runs and scheduler status table on the Settings page (/setup).
• Consolidated DB settings into /setup; removed the obsolete connection-settings modal flow and duplicate /api/connection-settings* routes.
• Added config section scheduler with per-job intervals:
  • alerts_interval
  • update_prices_interval
  • update_popularity_interval
  • reset_weekly_counters_interval
  • reset_monthly_counters_interval
• Coordination between multiple app instances uses MariaDB GET_LOCK/RELEASE_LOCK with one lock per job.
• cmd/cron remains available as a manual one-shot utility, but is no longer required for normal operation.

Rationale

PriceForge is deployed as local applications without a dedicated always-on server process that could own cron scheduling centrally. Embedding the scheduler into each app keeps the system self-contained while DB advisory locks prevent simultaneous job execution against the same database.

Constraints

• Scheduler safety depends on MariaDB advisory locks; without DB connectivity, jobs do not run.
• Jobs are interval-based from last_finished_at stored in qt_scheduler_runs.
• Duplicate scheduler loops across multiple app instances are acceptable because only one instance acquires the per-job lock at a time.

Files

• Scheduler: internal/scheduler/scheduler.go
• Scheduler tests: internal/scheduler/scheduler_test.go
• Config: internal/config/config.go, config.example.yaml
• App startup: cmd/pfs/main.go
• Migration: migrations/030_add_scheduler_runs.sql

2026-03-07: Remove Unused BOM Storage

Decision

Removed qt_bom and qt_configurations.bom_id because the schema was never adopted by runtime code and created dead database surface area.

What changed

• Added migration 029_drop_unused_qt_bom.sql to drop:
  • foreign key fk_qt_configurations_bom
  • index idx_qt_configurations_bom_id if present
  • column qt_configurations.bom_id
  • table qt_bom
• Removed BOM model from internal/models/lot.go.
• Removed Configuration.BOMID from internal/models/configuration.go.
• Removed BOM from internal/models/models.go auto-migration registry.

Rationale

The repository had schema and model stubs for BOM persistence, but no handler, service, repository, sync path, or UI used them. Keeping unused schema increases maintenance cost and confuses future changes.

Constraints

• This removal is safe only because no runtime code reads or writes qt_bom or qt_configurations.bom_id.
• Historical BOM payloads in qt_bom are not migrated elsewhere; the table is treated as disposable unused state.

Files

• Migration: migrations/029_drop_unused_qt_bom.sql
• Models: internal/models/lot.go, internal/models/configuration.go, internal/models/models.go

2026-03-07: Purge LOT Names Polluting Seen Registry

Decision

Rows in qt_vendor_partnumber_seen where partnumber is actually equal to lot.lot_name are treated as polluted data from external systems and must not appear in Global Vendor Mappings UI.

What changed

• Added purgeSeenLotNames in internal/services/seen_cleanup.go.
• VendorMappingService.List now deletes polluted seen rows before building the Vendor Mappings list.
• GetUnmappedPartnumbers now explicitly excludes qt_vendor_partnumber_seen.partnumber values that match an existing lot.lot_name.
• Added regression test coverage in internal/services/vendor_mapping_test.go.

Rationale

Another application writes non-partnumber LOT identifiers into qt_vendor_partnumber_seen. Those rows are noise for operators and should not be shown as unmapped vendor partnumbers.

Constraints

• Only polluted seen rows are removed: if a value equal to lot.lot_name also has an explicit mapping in lot_partnumbers, it is preserved.
• The cleanup targets the seen registry only; canonical PN mappings in lot_partnumbers are not touched.

Files

• Service: internal/services/seen_cleanup.go
• Service: internal/services/vendor_mapping.go
• Handler: internal/handlers/pricing.go
• Tests: internal/services/vendor_mapping_test.go
• Docs: bible-local/vendor-mapping.md

2026-02-21: Partnumber Book Snapshots for QuoteForge

Superseded in storage shape by the 2026-03-07 decision below. This section is retained as historical context.

Decision

Implemented versioned snapshots of the lot_partnumbers → LOT mapping in qt_partnumber_books / qt_partnumber_book_items. PriceForge writes; QuoteForge reads (SELECT only).

What changed

• Migration 026: added is_primary_pn TINYINT(1) DEFAULT 1 to lot_partnumbers; created qt_partnumber_books and qt_partnumber_book_items tables (version VARCHAR(20), later corrected).
• Migration 027: corrected version VARCHAR(20) → VARCHAR(30) — PNBOOK-YYYY-MM-DD-NNN is 21 chars and overflowed the original column.
• Migration 028: added description VARCHAR(10000) NULL to qt_partnumber_book_items — required by QuoteForge sync (SELECT partnumber, lot_name, is_primary_pn, description).
• Models PartnumberBook, PartnumberBookItem (with Description *string) added to internal/models/lot.go; IsPrimaryPN bool added to LotPartnumber.
• Service internal/services/partnumber_book.go:
  • CreateSnapshot: expands bundles (QuoteForge is bundle-unaware), copies description from lot_partnumbers to every expanded row, generates version PNBOOK-YYYY-MM-DD-NNN, deactivates previous books and activates new one atomically, then runs GFS retention cleanup.
  • expandMappings: filters out rows where lot_name is empty/whitespace; filters out partnumbers marked is_ignored = true in qt_vendor_partnumber_seen. Only valid PN→LOT pairs enter the snapshot.
  • applyRetention: deletes items explicitly (DELETE … WHERE book_id IN (…)) before deleting books — does not rely on FK CASCADE which GORM does not trigger on batch deletes.
  • ListBooks: returns all snapshots ordered newest-first with item counts.
• GFS retention policy: 7 daily · 5 weekly · 12 monthly · 10 yearly; applied automatically after each snapshot; active book is never deleted.
• Task type TaskTypePartnumberBookCreate added to internal/tasks/task.go.
• Handlers ListPartnumberBooks and CreatePartnumberBook added to internal/handlers/pricing.go; PartnumberBookService injected via constructor.
• Routes GET /api/admin/pricing/partnumber-books and POST /api/admin/pricing/partnumber-books registered in cmd/pfs/main.go.
• UI: "Снимки сопоставлений (Partnumber Books)" section with snapshot table, progress bar, and "Создать снапшот сопоставлений" button added to web/templates/vendor_mappings.html.

Rationale

QuoteForge needs a stable, versioned copy of the PN→LOT mapping to resolve BOM line items without live dependency on PriceForge. Snapshots decouple the two systems.

Constraints

• Bundles MUST be expanded: QuoteForge does not know about qt_lot_bundles.
• Snapshot rows with empty lot_name or is_ignored = true partnumbers MUST be excluded.
• description in book items comes from lot_partnumbers.description; for expanded bundle rows the description of the parent partnumber mapping is used.
• is_primary_pn is copied from lot_partnumbers into each book item; drives qty logic in QuoteForge.
• New snapshot is immediately is_active=1; all previous books are deactivated in the same transaction.
• Version format: PNBOOK-YYYY-MM-DD-NNN (VARCHAR(30)), sequential within the same day.
• Item deletion during retention MUST be explicit (items first, then books) — FK CASCADE is unreliable with GORM batch deletes.
• QuoteForge has SELECT permission only on qt_partnumber_books and qt_partnumber_book_items.

Files

• Migrations: 026_add_partnumber_books.sql, 027_fix_partnumber_books_version_length.sql, 028_add_description_to_partnumber_book_items.sql
• Models: internal/models/lot.go
• Service: internal/services/partnumber_book.go
• Handler: internal/handlers/pricing.go
• Tasks: internal/tasks/task.go
• Routes: cmd/pfs/main.go
• UI: web/templates/vendor_mappings.html

2026-03-07: Partnumber Book Catalog Deduplication

Decision

Reworked partnumber book storage so qt_partnumber_book_items is a deduplicated source-of-truth catalog by partnumber, and each snapshot book stores its included PN list in qt_partnumber_books.partnumbers_json.

What changed

• Migration 029 replaces expanded snapshot rows with lots_json in qt_partnumber_book_items.
• qt_partnumber_book_items now stores one row per partnumber with fields:
  • partnumber
  • lots_json ([{lot_name, qty}, ...])
  • description
• qt_partnumber_books now stores partnumbers_json, the sorted list of PN values included in that book.
• QuoteForge read contract is now:
  • read active book from qt_partnumber_books
  • parse partnumbers_json
  • load PN payloads via SELECT partnumber, lots_json, description FROM qt_partnumber_book_items WHERE partnumber IN (...)
• PartnumberBookService.CreateSnapshot now:
  • builds one logical item per PN,
  • serializes bundle composition into lots_json,
  • upserts the global catalog,
  • writes PN membership into the book header.
• ListBooks now derives item counts from partnumbers_json.
• Added regression tests covering:
  • direct PN -> one LOT with qty 1
  • bundle PN -> multiple LOT with explicit quantities
  • deduplication of catalog rows across multiple books

Rationale

• A real vendor PN may consist of several different LOT with different quantities.
• Expanded rows in qt_partnumber_book_items lost quantity semantics and duplicated the same logical item across books.
• The new shape keeps PN composition intact and makes the items table the canonical catalog.

Constraints

• qt_partnumber_book_items must not contain duplicate partnumber rows.
• lots_json is the only source of truth for PN composition.
• qt_partnumber_books.partnumbers_json stores membership only; the resolved PN composition comes from qt_partnumber_book_items.
• qt_partnumber_book_item_links is not part of the architecture and must not exist.
• If one snapshot build encounters multiple distinct compositions for the same PN, the build must fail instead of choosing one silently.
• Historical books remain snapshots of included PN membership; item payloads are read from the current catalog.

Files

• Migration: migrations/029_change_partnumber_book_items_to_lots_json.sql
• Models: internal/models/lot.go
• Service: internal/services/partnumber_book.go
• Tests: internal/services/partnumber_book_test.go
• Docs: bible-local/vendor-mapping.md

2026-03-07: Remove is_primary_pn From Partnumber Books

Decision

Removed is_primary_pn from PriceForge partnumber book storage and from the PriceForge → QuoteForge sync contract.

What changed

• Added migration 031_drop_is_primary_pn.sql dropping is_primary_pn from:
  • lot_partnumbers
  • qt_partnumber_book_items
• Removed IsPrimaryPN from internal/models/lot.go.
• PartnumberBookService no longer copies, compares, or upserts is_primary_pn.
• VendorMappingService no longer writes is_primary_pn into qt_partnumber_book_items.
• The sync contract for QuoteForge is now: SELECT partnumber, lots_json, description FROM qt_partnumber_book_items WHERE partnumber IN (...)

Rationale

• The flag is obsolete and does not participate in current business logic.
• PN composition is now fully represented by lots_json; qty semantics belong there.
• Keeping a dead compatibility field increases drift between documented and actual architecture.

Constraints

• lots_json is the only quantity-bearing field in the PN book contract.
• Any consumer still relying on is_primary_pn must be updated in the same change wave.

Files

• Migration: migrations/031_drop_is_primary_pn.sql
• Models: internal/models/lot.go
• Services: internal/services/partnumber_book.go, internal/services/vendor_mapping.go
• Docs: bible-local/vendor-mapping.md

2026-03-07: Drop Legacy lot_partnumbers And Bundle Tables

Decision

Removed lot_partnumbers, qt_lot_bundles, and qt_lot_bundle_items from active runtime architecture. The only canonical PN mapping store is qt_partnumber_book_items, one row per partnumber, with full composition in lots_json.

What changed

• Added migration 032_drop_legacy_vendor_mapping_tables.sql dropping:
  • qt_lot_bundle_items
  • qt_lot_bundles
  • lot_partnumbers
• internal/models/models.go no longer auto-migrates legacy mapping and bundle tables.
• Active runtime paths now read canonical mappings from qt_partnumber_book_items.
• Tests were updated to seed qt_partnumber_book_items instead of lot_partnumbers and bundle tables.
• Active Bible docs now describe qt_partnumber_book_items as the only source of truth.

Rationale

• QuoteForge and partnumber books already use lots_json.
• Keeping legacy mapping tables in runtime created two conflicting contracts.
• Multi-LOT PN composition belongs in lots_json, not in auxiliary bundle tables.

Constraints

• qt_partnumber_book_items must remain deduplicated by partnumber.
• Multi-LOT PN composition exists only in lots_json.
• Vendor is metadata in qt_vendor_partnumber_seen, not part of the canonical mapping key.
• Deprecated Go structs may remain temporarily for in-memory test compatibility, but runtime must not auto-create or depend on the dropped tables.

Files

• Migration: migrations/032_drop_legacy_vendor_mapping_tables.sql
• Models: internal/models/models.go, internal/models/lot.go
• Services: internal/services/vendor_mapping.go, internal/services/stock_import.go, internal/services/partnumber_book.go
• Warehouse: internal/warehouse/snapshot.go
• Matcher: internal/lotmatch/matcher.go
• Docs: bible-local/BIBLE.md, bible-local/vendor-mapping.md, bible-local/architecture.md, bible-local/pricelist.md, bible-local/data-rules.md

2026-03-07: Vendor Mapping LOT Autocomplete Must Not Clip Long Names

Decision

LOT autocomplete in the Vendor Mapping modal must use a custom popup, not native browser datalist, because operators work with long monospaced LOT names that must remain fully readable.

What changed

• Vendor Mapping modal uses a custom positioned suggestion popup for LOT inputs.
• Popup width may exceed the input width and is constrained by viewport, not by the text field width.
• Native datalist is not used for this field anymore.

Rationale

• Native browser suggestion UIs clip long LOT names unpredictably and differ across platforms.
• Operators need to distinguish long hardware LOT names visually before selection; clipped suffixes are operationally unsafe.

Constraints

• LOT suggestion popup must allow long names to be fully visible or significantly less clipped than the input width.
• Popup must stay above the modal overlay and follow the active LOT input.

Files

• UI: web/templates/vendor_mappings.html

---

2026-02-20: Pricelist Formation Hardening (Estimate/Warehouse/Meta)

Decision

Hardened pricelist formation rules to remove ambiguity and prevent silent data loss in UI/API.

What changed

• estimate snapshot now explicitly excludes hidden components (qt_lot_metadata.is_hidden = 1).
• Category snapshot in qt_pricelist_items.lot_category now always has a deterministic fallback: PART_ is assigned even when a LOT row is missing in lot.
• PricelistItem JSON contract was normalized to a single category field (LotCategory -> json:"category"), removing duplicate JSON-tag ambiguity.
• Meta-price source expansion now always includes the base LOT, then merges meta_prices sources with deduplication (exact + wildcard overlaps).

Rationale

• Prevent hidden/ignored components from leaking into estimate pricelists.
• Keep frontend category rendering stable for all items.
• Avoid non-deterministic JSON serialization when duplicate tags are present.
• Ensure meta-article pricing includes self-history and does not overcount duplicate sources.

Constraints

• estimate pricelist invariant: current_price > 0 AND is_hidden = 0.
• category in API must map from persisted qt_pricelist_items.lot_category.
• Missing category source must default to PART_.
• Meta source list must contain each LOT at most once.

Files

• Model: internal/models/pricelist.go
• Estimate/Warehouse snapshot service: internal/services/pricelist/service.go
• Pricing handler/meta expansion: internal/handlers/pricing.go
• Pricing service/meta expansion: internal/services/pricing/service.go
• Tests:
  • internal/services/pricelist/service_estimate_test.go
  • internal/services/pricelist/service_warehouse_test.go
  • internal/handlers/pricing_meta_expand_test.go
  • internal/services/pricing/service_meta_test.go
  • internal/services/stock_import_test.go

---

2026-02-20: Seen Registry Deduplication by Partnumber

Decision

Changed qt_vendor_partnumber_seen semantics to one row per partnumber (vendor/source are no longer part of uniqueness).

Rationale

• Eliminates duplicate seen rows when the same partnumber appears both with vendor and without vendor.
• Keeps ignore behavior consistent regardless of vendor presence.
• Simplifies operational cleanup and prevents re-creation of vendor/no-vendor duplicates.

Constraints

• partnumber is now the unique key in seen registry.
• Ignore checks are resolved by partnumber only.
• Stock provenance must be preserved (source_type='stock') when stock data exists for the partnumber.

Files

• Migration: migrations/025_dedup_vendor_seen_by_partnumber.sql
• Service: internal/services/stock_import.go
• Service: internal/services/vendor_mapping.go
• Model: internal/models/lot.go

---

2026-02-18: Global Vendor Partnumber Mapping

Decision

Implemented global vendor partnumber → LOT mapping with bundle support and seen-based ignore logic.

Key rules

• lot_partnumbers is the canonical mapping contract (1:1 per key).
• Composite mappings use internal bundle tables (qt_lot_bundles, qt_lot_bundle_items).
• Ignore logic moved from stock_ignore_rules to qt_vendor_partnumber_seen.is_ignored.
• Resolver order: exact (vendor, partnumber) → fallback (vendor='', partnumber).

Rationale

• Preserves external/client contracts (lot_partnumbers, LOT-based pricelists).
• Avoids multi-row ambiguity in lot_partnumbers.
• Supports complex assembled vendor SKUs without client-side changes.
• Centralizes ignore behavior across all sources via seen-registry.

Update (2026-02-25)

• DELETE /api/admin/pricing/vendor-mappings now removes both:
• mapping rows from lot_partnumbers
• and matching entries from qt_vendor_partnumber_seen (so "seen/unmapped" rows disappear from the global UI immediately after delete).

Constraints

• Bundle LOT is internal and must stay hidden in regular LOT list by default.
• Resolver order is mandatory and fixed.
• Bundle allocation for missing estimate: fallback from previous active warehouse pricelist; if absent → 0.

Files

• Migration: migrations/023_vendor_partnumber_global_mapping.sql
• Backfill: migrations/024_backfill_vendor_seen_from_stock_and_ignore.sql
• Resolver: internal/lotmatch/matcher.go
• Service: internal/services/vendor_mapping.go
• Warehouse calc: internal/warehouse/snapshot.go
• Stock import: internal/services/stock_import.go
• API: internal/handlers/pricing.go, cmd/pfs/main.go

---

2026-02-10: LOT Page Refactoring

Decision

Moved LOT management to a dedicated /lot page. Removed LOT tab from Pricing Admin.

What changed

• Created web/templates/lot.html — two tabs: LOT (component management) and Mappings (partnumber ↔ LOT).
• Removed LOT tab from admin_pricing.html; default tab changed to estimate.
• Removed "Сопоставление partnumber → LOT" section from Warehouse tab.
• Updated navigation: LOT → /lot, Pricing Admin → /admin/pricing.
• Added Lot() handler in internal/handlers/web.go.
• Added /lot route in cmd/pfs/main.go.

Rationale

Separation of concerns: LOT/component management is distinct from pricing administration.

---

Warehouse Pricing: Weighted Average

Decision

Warehouse pricelist uses weighted_avg (quantity-weighted average) as the sole price calculation method. Commit edff712 switched from weighted_median to weighted_avg.

• price_method field always contains "weighted_avg".
• No price_period_days, price_coefficient, manual_price, meta_prices in warehouse pricelists.

---

---

2026-03-13: Go File Modularization

Decision

Split three large Go files into focused modules within the same package.

What changed

• handlers/pricing.go (2446 lines) → pricing.go (struct + helpers) + pricing_components.go + pricing_alerts.go + pricing_stock.go + pricing_vendor.go + pricing_lots.go
• services/stock_import.go (1334 lines) → stock_import.go (core) + stock_mappings.go + stock_parse.go
• services/sync/service.go (1290 lines) → service.go (struct + status) + sync_pricelists.go + sync_changes.go + sync_import.go

Rationale

Files exceeding ~1000 lines are hard to edit safely with AI tooling. Same-package split preserves all existing APIs with zero behavior change.

---

2026-03-13: JS Extraction to Static Files

Decision

All inline <script> blocks extracted from HTML templates to web/static/js/.

What changed

• admin_pricing.html 2873 → 521 lines; JS → web/static/js/admin_pricing.js
• lot.html 1531 → 304 lines; JS → web/static/js/lot.js
• vendor_mappings.html 1063 → 169 lines; JS → web/static/js/vendor_mappings.js
• competitors.html 809 → 185 lines; JS → web/static/js/competitors.js
• pricelists.html 333 → 72 lines; JS → web/static/js/pricelists.js
• pricelist_detail.html 461 → 107 lines; JS → web/static/js/pricelist_detail.js

Rationale

No Go template interpolations exist inside any <script> block — extraction is safe. Static files served via existing /static/ route.

Constraint

Never put {{.GoVar}} interpolations inside JS files. Pass server data via API calls or data-* attributes on HTML elements.

---

2026-03-13: Competitor Pricelist Implementation

Decision

Competitor pricelist source is now fully implemented (previously "Reserved").

What changed

• Tables: qt_competitors, partnumber_log_competitors
• Migrations: 033–039
• Import: Excel upload → parse → dedup → bulk insert → p/n→lot resolution → weighted_median → create pricelist
• Unmapped p/ns written to qt_vendor_partnumber_seen (source_type = "competitor:<code>")
• Quote counts (unique p/n, total historical) shown on /admin/competitors
• price_method = "weighted_median", price_period_days = 0 stored explicitly in pricelist items

Rationale

Competitors upload Excel price lists from B2B portals. Prices are DDP — no recalculation needed. Expected discount applied at pricelist build time.

---

2026-03-13: price_method / price_period_days for Warehouse and Competitor

Decision

Warehouse and competitor pricelist items now store explicit, meaningful values instead of DB defaults.

Source	price_method	price_period_days
estimate	median / average / weighted_median	configurable (default 90)
warehouse	weighted_avg	0
competitor	weighted_median	0

price_period_days = 0 means "all available data" (no time window). Previously these fields held the DB default (median, 90) which was misleading — warehouse and competitor prices are frozen snapshots, not recalculated from lot_log.

Constraint

price_period_days and price_method are only acted upon when recalculating estimate pricelists from lot_log. For warehouse and competitor these fields are informational only.

---

Architecture Conventions

All future architectural decisions must be documented here with:

• Date
• Decision summary
• Rationale
• Constraints / invariants
• Affected files