core/bible-local/architecture/runtime-flows.md

Runtime Flows And Invariants

History-First Mutation Path

All state-changing mutations for assets/components must go through the history domain layer.

Sources covered:

• user registry edits
• user asset-page component actions (Add / Edit / Remove)
• hardware JSON ingest
• manual CSV ingest

History apply transaction invariants:

1. lock current projection row (parts or machines)
2. load latest snapshot (or bootstrap from projection)
3. apply validated patch/domain command
4. semantic dedupe (after_hash == before_hash => no-op, no event/snapshot/timeline)
5. write history event + snapshot
6. update projections
7. write timeline projection rows

Direct projection writes are allowed only inside history recompute/rebuild flows.

observations remains raw ingest trace only:

• runtime current-state reads (UI/API/history projection metadata) must use history + projections (parts, machines, installations, timeline_events, machine_firmware_states)
• observations may be used for ingest forensics, audits, and backfill/migration only

Event Time Source Priority

Use component event time over ingest time whenever possible.

• eventFallbackTime(actual, ingestedAt, collectedAt):

1. actual
2. ingestedAt
3. collectedAt

• collectedFallbackTime(collectedAt, ingestedAt):

1. collectedAt
2. ingestedAt

Status Event Time Parsing Order

parseComponentStatusEventTime resolves time in this order:

1. status_changed_at
2. Latest matching status_history item for current status
3. Latest parseable status_history item
4. status_checked_at

Failure Event Rules

For critical components:

• Timeline event type: COMPONENT_FAILED
• failure_events.failure_time uses resolved failure time (not raw ingest time)
• failure_events.external_id includes the same failure timestamp
• failure_events is a projection and may be rebuilt from component history (COMPONENT_STATUS_SET)

Manual UI failure registration invariants:

• POST /failures is history-backed for component status (COMPONENT_STATUS_SET -> FAILED)
• manual failure registration writes status history event and failure_events projection in one DB transaction
• manual failures use a dedicated projection source (manual_ui) so history recompute cleanup for ingest/history sources does not delete them
• failure_date from UI is normalized to start-of-day UTC timestamp for failure_events.failure_time in v1
• failure detail open duration is computed with date precision (UTC day difference from failure date to repair date / current date), not timestamp precision

Active failure / repair-by-replacement semantics (Failures UI):

• active failure = failure event without later replacement install in the same slot on the same server
• repair by replacement is derived from installations history: later install of a different component (part_id != failed part_id) into same machine_id + slot_name
• if slot at failure time cannot be resolved from installation history (e.g. manual failure date normalized to start of day), UI may fallback to current installation slot for display-only slot rendering in Active Failures

First Seen Rules

parts.first_seen_at must be the earliest known ingest-derived component time.

Candidate sources:

1. Parseable status_history[].changed_at
2. status_changed_at
3. status_checked_at
4. eventFallbackTime(nil, ingestedAt, collectedAt)

Persistence rule:

• Keep the minimum value over time.
• If incoming is earlier than stored value, overwrite with incoming value.
• In history mode this is persisted as component metadata correction (COMPONENT_FIRST_SEEN_CORRECTED), then projected into parts.first_seen_at.

Duplicate Component Serial Rules (CSV + JSON Ingest)

If serial numbers are not unique within the same p/n (model) inside one ingest payload:

• First occurrence keeps original vendor_serial.
• Each next duplicate occurrence is assigned a service serial placeholder:
  • Format: NO_SN-XXXXXXXX (8-digit zero-padded global counter).
• If vendor_serial is empty, a service serial placeholder is assigned as well.
• Counter is global for the whole application and stored in id_sequences under entity_type = 'no_sn_placeholder'.

Component Health Computation In UI

Component health is derived only from the latest status event among:

• COMPONENT_FAILED
• COMPONENT_WARNING
• COMPONENT_UNKNOWN
• COMPONENT_OK

Non-status timeline events (INSTALLED, REMOVED, FIRMWARE_CHANGED, FIRMWARE_INSTALLED, etc.) must not change health status.

Status event mapping notes:

• COMPONENT_STATUS_SET with runtime.health_status = UNKNOWN must emit COMPONENT_UNKNOWN timeline event.
• Unknown/unrecognized status timeline events must never default to Healthy in UI state derivation.

Firmware Timeline Rules

For component firmware observations:

• First observed version -> FIRMWARE_INSTALLED (asset + component timeline pair)
• Later version change -> FIRMWARE_CHANGED (asset + component timeline pair)
• In history mode, firmware observations are persisted as:
  • component: COMPONENT_FIRMWARE_SET
  • asset device firmware: ASSET_FIRMWARE_DEVICE_SET

Storage details:

• FIRMWARE_INSTALLED stores transition string in timeline_events.firmware_version: - -> <installed_version>
• FIRMWARE_CHANGED stores installed/new firmware value

Detection details:

• Previous observation lookup: ORDER BY observed_at DESC, id DESC LIMIT 1 OFFSET 1

Install / Remove Flow (Cross-Entity)

Install/remove operations are applied as cross-entity history commands in one transaction.

Invariants:

• component and asset history events are both written
• both events share one correlation_id
• installations is updated as a projection
• asset/component timeline rows are emitted as paired projection events
• installation slot/location (installations.slot_name) is projection state synchronized from component installation snapshot field installation.slot_name

Asset Page Current Components Actions

All asset page component actions are history-backed and transactional.

• Add:
  • either creates a new component and attaches it to current asset, or attaches an existing component
  • duplicate preflight checks vendor_serial and vendor_serial + model
  • if existing component is installed on another asset and user confirms force attach, runtime executes a move (remove old asset + install new asset) in one transaction with shared correlation_id
• Edit:
  • applies bulk history patches to selected components currently attached to the asset
  • multi-select edits must reject unique fields (vendor_serial, slot)
• Remove:
  • remove is de-assert only (no detach-only mode)
  • runtime always performs:
    1. COMPONENT_REMOVED (detach from asset)
    2. COMPONENT_STATUS_SET with user-selected status mapped from working | not_working | unknown -> OK | FAILED | UNKNOWN

Log Collected Flow

• Hardware log collection is represented in history as ASSET_LOG_COLLECTED.
• UI/API timeline renders it as LOG_COLLECTED.

Delete / Rollback / Hard Restore Flows

Mutating history operations are asynchronous and DB-backed (history_recompute_jobs).

• Delete event:
  • soft-delete logical history event (is_deleted)
  • mark linked timeline rows deleted
  • enqueue recompute job
• Rollback (compensating):
  • async job creates a new rollback history event (*_ROLLBACK_APPLIED)
  • original history remains intact
• Hard restore (admin):
  • async job physically deletes future history rows after target snapshot/version
  • operation is recorded in history_admin_audit
• Batch cancel by source (admin):
  • preview lists matching history events by source_type and optional date/source-ref filters
  • if date_from / date_to are omitted, date filters do not constrain the result set
  • date-only filters are interpreted as inclusive day bounds (00:00:00 to 23:59:59)
  • execute soft-deletes matched events and enqueues generic recompute jobs for all affected assets/components
  • affected set expands from asset events to linked component entities via correlation_id

Data Repair And Cleanup (Admin)

• Repair All is an operational best-effort flow:
  • may restore missing timeline_events.slot_name from observations (repair/backfill only)
  • may enqueue recompute jobs for affected entities
  • retries transient DB connection errors and continues on per-entity failures (best effort)
• Cleanup Orphaned Projections removes registry/projection/raw rows for entities that have no active history left after cancellations.
• observations usage in repair flows is allowed only as one-time recovery input; runtime state must still come from history + projections.

Recompute Scope And Propagation

• Component recompute rebuilds component projections (parts, timeline_events, installations, failure_events).
• Asset recompute rebuilds asset projections (machines, machine_firmware_states, timeline_events) and then triggers recompute for linked components to restore cross-entity projection consistency.
• Asset delete/hard-restore propagates to correlated component history events via correlation_id.

MySQL Tx Cursor Safety (Critical)

For MySQL/MariaDB transactional code, do not execute additional SQL on the same tx while iterating an open result set (for rows.Next()).

Observed failure signature when violating this rule:

• driver logs: [mysql] ... invalid connection, unexpected EOF
• app errors: driver: bad connection
• DB warnings: Aborted connection ... Got an error reading communication packets

Required pattern:

1. Read all rows into in-memory structs.
2. Close the cursor.
3. Perform follow-up QueryRowContext / ExecContext writes and projection rebuild logic.

This rule is mandatory for recompute/rebuild flows and any projection repair routines.

Timeline API Grouping

• History timeline endpoints group events by day by default and return timeline cards:
  • single (one event)
  • dedup (same visual action + context in one day)
  • bulk (mass operation correlated by correlation_id)
• Asset timeline adds a fallback synthetic bulk for movement noise reduction:
  • if component_installed / component_removed events do not have correlation_id,
  • they may be collapsed into a synthetic bulk card by visual_action + source_type + 1h time bucket
  • this is a UI/read-model aggregation rule only (raw timeline rows remain unchanged)
• Default timezone for grouping is UTC; callers may override with tz.
• Timeline card drilldown resolves to concrete events and history event details through history API endpoints and must use the same tz as card grouping when matching card day buckets.
• Timeline cards display source labels from history source_type (ingest_json, ingest_csv, user, system).

Timeline Color Semantics

• REMOVED -> yellow
• COMPONENT_FAILED -> red
• COMPONENT_UNKNOWN -> gray
• COMPONENT_WARNING and related warning semantics follow timelineEventClass

Regression Guardrails

Do not reintroduce these regressions:

• Using ingest timestamp when payload provides better event/failure timestamp
• Letting INSTALLED mark failed components as healthy
• Missing Previous Components section on asset page
• Missing installation history on component page
• Missing firmware information on component page timeline
• Writing ingest state transitions directly to projections/timeline while bypassing history apply
• Creating duplicate history events for semantic no-op updates
• Executing nested SQL on the same transaction while rows.Next() cursor is still open (must use two-phase read-then-write)