Mikhail ChusavitinandClaude Sonnet 5
e036a9c7b8
pci=realloc was forcing a full PCI resource reallocation on every default and toram boot, added as a GPU BAR workaround for BIOS without Above 4G Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS and lspci both saw them fine. It now only applies to the no-GUI/gsp-off troubleshooting entry, alongside a bible-local ADR for the two prior rounds of SAT/blackbox durability work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3.4 KiB
Decision: Stream SAT job output live to disk and kick blackbox sync on job completion
Date: 2026-07-27 Status: active
Context
Investigating a hard reboot mid-nvbandwidth (see
2026-07-27-nvbandwidth-per-socket-split.md) showed that the SAT job runner
had no way to preserve a crashing job's own output. streamExecOutput
(audit/internal/platform/sat.go) buffered a job's stdout/stderr entirely
in memory; the job's named log file (NN-<job>.log) was written to disk
only once, after the process exited. A crash mid-command meant that job's
file never existed on disk at all -- verbose.log's start/finish
markers survived (they are appended per-line already), but the command's
own output did not.
This matters more than it would on a normal install: the live ISO boots
with toram (live-boot copies the read-only medium into RAM and mounts
an overlay on top of it -- see runtime-flows.md's Boot sequence). The
export directory bee writes to lives on that RAM-backed overlay, so a
local write surviving to "disk" is not itself durable across a hard reset
-- the only real persistence boundary is bee-blackbox.service mirroring
the export directory out to removable media on its own adaptive schedule
(1-30s flush period). Even a perfectly-written local file is lost if
blackbox hasn't copied it off before the crash.
Decision
Two changes, in audit/internal/platform/sat.go and
audit/internal/app/blackbox.go:
streamExecOutputtakes an optionallivePath; when set, each output line is written straight to that file as it arrives, in addition to the in-memory buffer used for the final return value.runSATCommandCtx/runSATCommandWithMetricsthread this through fromrunAcceptancePackCtx's job loop and the storage SAT loop, pointing at each job's own named log file. A crash mid-command now leaves whatever had printed up to that point instead of nothing.platform.SetJobBoundaryHook(func(jobName string))is called right after each job's output file is written (success or failure, every job).app.New()wires this to touch a marker file (.blackbox-kickunderexportDir);blackboxWorker.run()polls that marker's mtime (blackboxKickPollInterval, default 250ms) alongside its normal adaptive timer and syncs immediately on a kick instead of waiting out the current flush period.
platform does not import app or know about blackbox; the job-boundary
hook is a plain func(string) seam, consistent with the existing
satExecCommand/satStat/satReadFile seams in the same file.
Consequences
- A job that completes cleanly reaches removable media within one poll interval instead of waiting out whatever the adaptive flush period happens to be (up to 30s) -- closing most of the historical gap for postmortem analysis of a later crash.
- The job that is in flight when a crash happens still only has whatever had streamed to its live file by that point -- this does not make output capture instantaneous, it makes it incremental instead of all-or-nothing.
- DO NOT revert
streamExecOutput's per-job file write to a single end-of-commandos.WriteFile-- that was the exact gap that lost thenvbandwidthcrash's own output in the original incident. - DO NOT assume a local write under the live-ISO's export directory is durable on its own; anything that must survive a hard reset needs blackbox to have copied it off first (this is why the kick exists at all, not just the live write).