boot: drop pci=realloc from default/toram GRUB entries, fold into troubleshooting entry

pci=realloc was forcing a full PCI resource reallocation on every default
and toram boot, added as a GPU BAR workaround for BIOS without Above 4G
Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and
silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS
and lspci both saw them fine. It now only applies to the no-GUI/gsp-off
troubleshooting entry, alongside a bible-local ADR for the two prior
rounds of SAT/blackbox durability work.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

bible-local/README.md

@@ -43,7 +43,7 @@ Generic engineering rules live in `bible/rules/patterns/`.
 - Inter-GPU communication check
   - `all_reduce_perf`
 - GPU bandwidth check
-  - `dcgmi diag -r nvbandwidth`
+  - `dcgmi diag -r nvbandwidth` (per CPU socket, then all selected GPUs, on multi-socket systems -- see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`)
 
 ### Validate -> Stress
 
@@ -68,4 +68,4 @@ Generic engineering rules live in `bible/rules/patterns/`.
 - Inter-GPU communication check
   - `all_reduce_perf`
 - GPU bandwidth check
-  - `dcgmi diag -r nvbandwidth`
+  - `dcgmi diag -r nvbandwidth` (per CPU socket, then all selected GPUs, on multi-socket systems -- see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`)

bible-local/architecture/runtime-flows.md

@@ -167,6 +167,64 @@ Acceptance flows:
 - Runtime overrides:
   - `BEE_MEMTESTER_SIZE_MB`
   - `BEE_MEMTESTER_PASSES`
+- NVIDIA Bandwidth SAT (`RunNvidiaBandwidthPack`, `dcgmi diag -r nvbandwidth`) on a
+  multi-socket system runs per CPU socket first, then all selected GPUs together
+  (`03-dcgmi-nvbandwidth-socket0.log`, `...-socket1.log`, `...-all.log`) --
+  see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`. Single-socket
+  systems (or systems where a GPU's NUMA node can't be resolved) keep the
+  original single `NN-dcgmi-nvbandwidth.log` shape.
+
+## SAT job output durability
+
+```
+runAcceptancePackCtx job loop (per satJob)
+  1. run the job's command; streamExecOutput writes each output line to the
+     job's log file as it arrives (not only when the process exits)
+  2. write the job's final log file (same content the live stream already
+     wrote, plus any health-check suffix)
+  3. call satJobBoundaryHook(jobName) if set
+  4. append run_at_utc / *_status to summary.txt
+```
+
+**Critical invariants:**
+- DO NOT change `streamExecOutput` back to buffering output in memory and
+  writing the job's log file only once, after the command exits -- see
+  `decisions/2026-07-27-sat-live-output-and-blackbox-kick.md`. The live ISO's
+  export directory sits on a `toram` RAM-backed overlay (see Boot sequence
+  above), so a crash mid-command is otherwise unrecoverable for that job.
+- `platform` never imports `app`; `SetJobBoundaryHook` is a plain
+  `func(string)` seam (same pattern as `satExecCommand`/`satStat`), not a
+  direct call into blackbox internals.
+
+## Blackbox sync flow
+
+```
+bee-blackbox.service (separate process from bee-web/bee-audit)
+  1. discover enrolled removable-media targets every blackboxDiscoverInterval (2s)
+  2. per enrolled target, blackboxWorker.run():
+       a. syncCycle(): mount target, mirror exportDir -> removable media,
+          write README.md/manifest docs, fsync
+       b. record lastKickSeen = mtime of exportDir/.blackbox-kick
+       c. wait for: the adaptive flushPeriod timer (1-30s), OR
+          a stop signal, OR
+          a poll tick (blackboxKickPollInterval, 250ms) showing the kick
+          file is newer than lastKickSeen -- whichever comes first
+  3. on kick-triggered wake: sync immediately, do not wait out the
+     remaining flushPeriod
+```
+
+**Critical invariants:**
+- `app.New()` wires `platform.SetJobBoundaryHook` to touch
+  `exportDir/.blackbox-kick` after every SAT job finishes. If a new SAT
+  execution path bypasses `runAcceptancePackCtx`'s job loop, it will not
+  trigger this kick -- data from that path still eventually reaches
+  blackbox via the adaptive timer, just not promptly.
+- The adaptive `flushPeriod` logic (`adjustFlushPeriod`) is unchanged by the
+  kick mechanism -- the kick only short-circuits the *wait*, it does not
+  reset `flushPeriod` itself.
+- DO NOT assume a local write under the live ISO's export directory is
+  durable on its own (RAM-backed overlay) -- blackbox's mirror to removable
+  media is the only real persistence boundary across a hard reset.
 
 ## NVIDIA SAT Web UI flow
 

bible-local/decisions/2026-07-27-nvbandwidth-per-socket-split.md

@@ -0,0 +1,59 @@
+# Decision: Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass
+
+**Date:** 2026-07-27
+**Status:** active
+
+## Context
+
+On a dual-socket AMD EPYC 9655 server with 6x NVIDIA H200 NVL (PCIe form
+factor, no NVLink bridges installed, no NVSwitch), running the NVIDIA
+Bandwidth SAT (`dcgmi diag -r nvbandwidth`) across all 6 GPUs caused the
+whole host to hard-reboot within seconds of the test starting. This
+happened reliably; other GPU tests on the same box did not reproduce it:
+
+- A compute-only GPU stress test (~20 min, no GPU-to-GPU traffic) ran
+  stably.
+- `nccl-tests` all-reduce across all 6 GPUs completed successfully, at
+  PCIe-level bandwidth (~18.7 GB/s busbw -- consistent with no direct P2P,
+  not a NVLink-class transport).
+- Only `nvbandwidth` reproducibly triggered the reboot.
+
+Topology: GPUs 0-3 sit on socket 0 (NUMA node 0), GPUs 4-5 sit on socket 1
+(NUMA node 1); the cross-socket path goes through the CPU interconnect, not
+a PCIe switch. `nvbandwidth` drives real peer-to-peer DMA between every GPU
+pair, including that cross-socket pair -- a fault domain BURN and the NCCL
+transport fallback never exercise.
+
+No prior SAT run isolated which specific GPU pairing (same-socket vs.
+cross-socket) the reboot depended on: the task queue only recorded that
+`nvidia-bandwidth` never reached `done`, and `bee-sat/gpu-nvidia-bandwidth-*/verbose.log`
+showed the crash landed mid-command, with no further detail on which
+transfer was in flight.
+
+## Decision
+
+`RunNvidiaBandwidthPack` (`audit/internal/platform/sat.go`) now resolves
+each selected GPU's NUMA node (`gpuBandwidthSocketGroups`, via
+`nvidia-smi --query-gpu=index,pci.bus_id` + the PCI device's sysfs
+`numa_node`) and, when GPUs span more than one node, runs `nvbandwidth`
+three times in sequence: once per socket group, then once across all
+selected GPUs. Job files are named
+`NN-dcgmi-nvbandwidth-socket<N>.log` / `NN-dcgmi-nvbandwidth-all.log`.
+
+Single-socket systems, or systems where the NUMA node can't be resolved for
+every GPU, keep the original one-pass shape (`NN-dcgmi-nvbandwidth.log`) --
+there is nothing meaningful to split in that case, and the log-file naming
+stays backward compatible with older support bundles.
+
+## Consequences
+
+- A future crash confined to the all-GPU pass, with both single-socket
+  passes already logged clean, pinpoints the cross-socket P2P path as the
+  trigger instead of leaving it conflated with "a general GPU/PCIe fault".
+- `SATEstimatedNvidiaBandwidthSec` no longer reflects worst-case duration on
+  multi-socket systems (now up to 3x); re-measure from real multi-socket
+  task logs once available and bump the constant -- do not just multiply by
+  3 without real data, since per-pass overhead is not purely linear.
+- DO NOT collapse this back into a single all-GPU `nvbandwidth` job for
+  multi-socket systems: that was the exact shape that made the original
+  incident unattributable to a specific GPU pairing.

bible-local/decisions/2026-07-27-sat-live-output-and-blackbox-kick.md

@@ -0,0 +1,67 @@
+# Decision: Stream SAT job output live to disk and kick blackbox sync on job completion
+
+**Date:** 2026-07-27
+**Status:** active
+
+## Context
+
+Investigating a hard reboot mid-`nvbandwidth` (see
+`2026-07-27-nvbandwidth-per-socket-split.md`) showed that the SAT job runner
+had no way to preserve a crashing job's own output. `streamExecOutput`
+(`audit/internal/platform/sat.go`) buffered a job's stdout/stderr entirely
+in memory; the job's named log file (`NN-<job>.log`) was written to disk
+only once, after the process exited. A crash mid-command meant that job's
+file never existed on disk at all -- `verbose.log`'s `start`/`finish`
+markers survived (they are appended per-line already), but the command's
+own output did not.
+
+This matters more than it would on a normal install: the live ISO boots
+with `toram` (`live-boot` copies the read-only medium into RAM and mounts
+an overlay on top of it -- see `runtime-flows.md`'s Boot sequence). The
+export directory bee writes to lives on that RAM-backed overlay, so a
+local write surviving to "disk" is not itself durable across a hard reset
+-- the only real persistence boundary is `bee-blackbox.service` mirroring
+the export directory out to removable media on its own adaptive schedule
+(1-30s flush period). Even a perfectly-written local file is lost if
+blackbox hasn't copied it off before the crash.
+
+## Decision
+
+Two changes, in `audit/internal/platform/sat.go` and
+`audit/internal/app/blackbox.go`:
+
+1. `streamExecOutput` takes an optional `livePath`; when set, each output
+   line is written straight to that file as it arrives, in addition to the
+   in-memory buffer used for the final return value.
+   `runSATCommandCtx`/`runSATCommandWithMetrics` thread this through from
+   `runAcceptancePackCtx`'s job loop and the storage SAT loop, pointing at
+   each job's own named log file. A crash mid-command now leaves whatever
+   had printed up to that point instead of nothing.
+2. `platform.SetJobBoundaryHook(func(jobName string))` is called right
+   after each job's output file is written (success or failure, every
+   job). `app.New()` wires this to touch a marker file
+   (`.blackbox-kick` under `exportDir`); `blackboxWorker.run()` polls that
+   marker's mtime (`blackboxKickPollInterval`, default 250ms) alongside its
+   normal adaptive timer and syncs immediately on a kick instead of waiting
+   out the current flush period.
+
+`platform` does not import `app` or know about blackbox; the job-boundary
+hook is a plain `func(string)` seam, consistent with the existing
+`satExecCommand`/`satStat`/`satReadFile` seams in the same file.
+
+## Consequences
+
+- A job that completes cleanly reaches removable media within one poll
+  interval instead of waiting out whatever the adaptive flush period
+  happens to be (up to 30s) -- closing most of the historical gap for
+  postmortem analysis of a later crash.
+- The job that is in flight when a crash happens still only has whatever
+  had streamed to its live file by that point -- this does not make output
+  capture instantaneous, it makes it incremental instead of all-or-nothing.
+- DO NOT revert `streamExecOutput`'s per-job file write to a single
+  end-of-command `os.WriteFile` -- that was the exact gap that lost the
+  `nvbandwidth` crash's own output in the original incident.
+- DO NOT assume a local write under the live-ISO's export directory is
+  durable on its own; anything that must survive a hard reset needs
+  blackbox to have copied it off first (this is why the kick exists at
+  all, not just the live write).

bible-local/decisions/2026-07-28-pci-realloc-out-of-default-boot-entries.md

@@ -0,0 +1,79 @@
+# Decision: Move pci=realloc out of the default/toram/no-GUI GRUB entries
+
+**Date:** 2026-07-28
+**Status:** active
+
+## Context
+
+`pci=realloc` was added to the default, toram, and no-GUI/gsp-off GRUB menu
+entries by commit 8bf8dfa ("default to KMS + pci=realloc, drop nomodeset from
+main entries") as a workaround for GPU BAR assignment failures on servers
+where the BIOS lacks "Above 4G Decoding". It forces the Linux kernel to
+fully reallocate all PCI resources instead of trusting the BIOS-assigned
+addresses.
+
+On a dual-socket AMD EPYC 9655 server (MSI CG480-S6053, 6x GPU + 8x NVMe
+behind 2-port hot-plug-capable PCIe switches), this same flag caused a
+different failure: the OS only exposed 2 of 8 NVMe drives, even though
+`lspci` and the BIOS both enumerated all 6 controllers actually present in
+the affected chassis. `dmesg` showed the kernel's bridge-window reallocation
+pass exhausting available MMIO32 address space:
+
+```
+pci 0000:f2:03.0: bridge window [mem size 0x00200000 64bit pref]: can't assign; no space
+pci 0000:f7:00.0: BAR 0 [mem size 0x00004000 64bit]: can't assign; no space
+```
+
+The BIOS had already assigned working BARs to all 6 NVMe controllers at
+boot; `pci=realloc` threw those away and failed to re-assign 4 of them,
+so the `nvme` driver never bound to those 4 (no `nvme nvmeX: pci function`
+line), and they never became block devices. The dense NVMe backplane, not
+the GPUs, hit the address-space ceiling that `pci=realloc` re-triggered on
+every device in the topology, not just the GPUs it was meant to fix.
+
+`pci=realloc` is a blanket flag: it was applied globally to the default and
+toram entries (the ones almost every operator actually boots), so any
+server with a BIOS whose GPU BARs are fine, but whose PCIe topology is
+dense enough elsewhere (many NVMe bays, many switches), pays this cost
+unconditionally.
+
+## Decision
+
+`pci=realloc` is removed from the default and toram menu entries in both
+`iso/builder/build.sh` (`write_canonical_grub_cfg`) and the static
+`iso/builder/config/bootloaders/grub-efi/grub.cfg` template. Rather than
+adding a new single-purpose menu entry for this one workaround, it is
+folded into the existing no-GUI/gsp-off entry, which is already the
+troubleshooting/fail-safe-style option operators reach for when the
+default entry does not work cleanly:
+
+```
+EASY-BEE v<version> -- no GUI / troubleshooting (gsp-off, pci=realloc)
+```
+
+An operator who hits either a GSP-related GPU issue or a GPU BAR
+assignment failure (BIOS lacking Above 4G Decoding) selects this single
+entry manually, instead of the menu growing one dedicated item per
+individual failure mode. This keeps the total entry count in
+`grub.cfg` unchanged from before the incident (still 3 real boot
+entries plus wipe/memtest/firmware-settings), rather than letting it
+grow linearly with every workaround discovered.
+
+The isolinux (BIOS/legacy boot) menu in `iso/builder/config/bootloaders/isolinux/live.cfg.in`
+already had no `pci=realloc` entry at all (pre-existing asymmetry with
+grub-efi, not introduced by this change) -- not addressed here since no
+incident has been observed via legacy boot.
+
+## Consequences
+
+- Servers with dense PCIe topologies (many NVMe bays, many GPUs) no longer
+  silently lose drives to a workaround they didn't need, by default.
+- Servers that genuinely need the GPU BAR workaround (BIOS lacking Above 4G
+  Decoding) now require the operator to notice GPU BAR failures and
+  deliberately pick the workaround entry, instead of getting it for free --
+  document this tradeoff in the support runbook / vendor-facing material
+  if GPU BAR failures turn out to be common enough to warrant it.
+- DO NOT move `pci=realloc` back into the default/toram entries without
+  first checking whether the affected server's topology can tolerate a
+  full PCI reallocation pass -- this was the exact failure mode of the
+  incident that prompted this decision.

bible-local/decisions/README.md

@@ -8,3 +8,6 @@ One file per decision, named `YYYY-MM-DD-short-topic.md`.
 | 2026-04-01 | Treat memtest as explicit ISO content | active |
 | 2026-04-29 | Treat embedded submodules as read-only | active |
 | 2026-06-12 | Skip PCIe link-speed warnings for disabled devices | active |
+| 2026-07-27 | Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass | active |
+| 2026-07-27 | Stream SAT job output live to disk and kick blackbox sync on job completion | active |
+| 2026-07-28 | Move pci=realloc out of the default/toram/no-GUI GRUB entries | active |

iso/builder/build.sh

@@ -779,16 +779,16 @@ write_canonical_grub_cfg() {
 source /boot/grub/config.cfg
 
 menuentry "EASY-BEE v${version_label}" {
-    linux   ${kernel} ${append_live} nomodeset bee.nvidia.mode=normal pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup
+    linux   ${kernel} ${append_live} nomodeset bee.nvidia.mode=normal net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup
     initrd  ${initrd}
 }
 
 menuentry "EASY-BEE v${version_label} -- load to RAM (toram)" {
-    linux   ${kernel} ${append_live} toram nomodeset bee.nvidia.mode=normal pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup
+    linux   ${kernel} ${append_live} toram nomodeset bee.nvidia.mode=normal net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup
     initrd  ${initrd}
 }
 
-menuentry "EASY-BEE v${version_label} -- no GUI / no X11" {
+menuentry "EASY-BEE v${version_label} -- no GUI / troubleshooting (gsp-off, pci=realloc)" {
     linux   ${kernel} ${append_live} nomodeset bee.gui=off bee.nvidia.mode=gsp-off pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup
     initrd  ${initrd}
 }

iso/builder/config/bootloaders/grub-efi/grub.cfg

@@ -1,16 +1,16 @@
 source /boot/grub/config.cfg
 
 menuentry "EASY-BEE v@VERSION@" {
-    linux   @KERNEL_LIVE@ @APPEND_LIVE@ nomodeset bee.nvidia.mode=normal pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup
+    linux   @KERNEL_LIVE@ @APPEND_LIVE@ nomodeset bee.nvidia.mode=normal net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup
     initrd  @INITRD_LIVE@
 }
 
 menuentry "EASY-BEE v@VERSION@ -- load to RAM (toram)" {
-    linux   @KERNEL_LIVE@ @APPEND_LIVE@ toram nomodeset bee.nvidia.mode=normal pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup
+    linux   @KERNEL_LIVE@ @APPEND_LIVE@ toram nomodeset bee.nvidia.mode=normal net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup
     initrd  @INITRD_LIVE@
 }
 
-menuentry "EASY-BEE v@VERSION@ -- no GUI / no X11" {
+menuentry "EASY-BEE v@VERSION@ -- no GUI / troubleshooting (gsp-off, pci=realloc)" {
     linux   @KERNEL_LIVE@ @APPEND_LIVE@ nomodeset bee.gui=off bee.nvidia.mode=gsp-off pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup
     initrd  @INITRD_LIVE@
 }