Mikhail ChusavitinandClaude Sonnet 5
e036a9c7b8
pci=realloc was forcing a full PCI resource reallocation on every default and toram boot, added as a GPU BAR workaround for BIOS without Above 4G Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS and lspci both saw them fine. It now only applies to the no-GUI/gsp-off troubleshooting entry, alongside a bible-local ADR for the two prior rounds of SAT/blackbox durability work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
68 lines
3.4 KiB
Markdown
68 lines
3.4 KiB
Markdown
# Decision: Stream SAT job output live to disk and kick blackbox sync on job completion
|
|
|
|
**Date:** 2026-07-27
|
|
**Status:** active
|
|
|
|
## Context
|
|
|
|
Investigating a hard reboot mid-`nvbandwidth` (see
|
|
`2026-07-27-nvbandwidth-per-socket-split.md`) showed that the SAT job runner
|
|
had no way to preserve a crashing job's own output. `streamExecOutput`
|
|
(`audit/internal/platform/sat.go`) buffered a job's stdout/stderr entirely
|
|
in memory; the job's named log file (`NN-<job>.log`) was written to disk
|
|
only once, after the process exited. A crash mid-command meant that job's
|
|
file never existed on disk at all -- `verbose.log`'s `start`/`finish`
|
|
markers survived (they are appended per-line already), but the command's
|
|
own output did not.
|
|
|
|
This matters more than it would on a normal install: the live ISO boots
|
|
with `toram` (`live-boot` copies the read-only medium into RAM and mounts
|
|
an overlay on top of it -- see `runtime-flows.md`'s Boot sequence). The
|
|
export directory bee writes to lives on that RAM-backed overlay, so a
|
|
local write surviving to "disk" is not itself durable across a hard reset
|
|
-- the only real persistence boundary is `bee-blackbox.service` mirroring
|
|
the export directory out to removable media on its own adaptive schedule
|
|
(1-30s flush period). Even a perfectly-written local file is lost if
|
|
blackbox hasn't copied it off before the crash.
|
|
|
|
## Decision
|
|
|
|
Two changes, in `audit/internal/platform/sat.go` and
|
|
`audit/internal/app/blackbox.go`:
|
|
|
|
1. `streamExecOutput` takes an optional `livePath`; when set, each output
|
|
line is written straight to that file as it arrives, in addition to the
|
|
in-memory buffer used for the final return value.
|
|
`runSATCommandCtx`/`runSATCommandWithMetrics` thread this through from
|
|
`runAcceptancePackCtx`'s job loop and the storage SAT loop, pointing at
|
|
each job's own named log file. A crash mid-command now leaves whatever
|
|
had printed up to that point instead of nothing.
|
|
2. `platform.SetJobBoundaryHook(func(jobName string))` is called right
|
|
after each job's output file is written (success or failure, every
|
|
job). `app.New()` wires this to touch a marker file
|
|
(`.blackbox-kick` under `exportDir`); `blackboxWorker.run()` polls that
|
|
marker's mtime (`blackboxKickPollInterval`, default 250ms) alongside its
|
|
normal adaptive timer and syncs immediately on a kick instead of waiting
|
|
out the current flush period.
|
|
|
|
`platform` does not import `app` or know about blackbox; the job-boundary
|
|
hook is a plain `func(string)` seam, consistent with the existing
|
|
`satExecCommand`/`satStat`/`satReadFile` seams in the same file.
|
|
|
|
## Consequences
|
|
|
|
- A job that completes cleanly reaches removable media within one poll
|
|
interval instead of waiting out whatever the adaptive flush period
|
|
happens to be (up to 30s) -- closing most of the historical gap for
|
|
postmortem analysis of a later crash.
|
|
- The job that is in flight when a crash happens still only has whatever
|
|
had streamed to its live file by that point -- this does not make output
|
|
capture instantaneous, it makes it incremental instead of all-or-nothing.
|
|
- DO NOT revert `streamExecOutput`'s per-job file write to a single
|
|
end-of-command `os.WriteFile` -- that was the exact gap that lost the
|
|
`nvbandwidth` crash's own output in the original incident.
|
|
- DO NOT assume a local write under the live-ISO's export directory is
|
|
durable on its own; anything that must survive a hard reset needs
|
|
blackbox to have copied it off first (this is why the kick exists at
|
|
all, not just the live write).
|