diff --git a/bible-local/README.md b/bible-local/README.md index d1a4ee5..3bc11fd 100644 --- a/bible-local/README.md +++ b/bible-local/README.md @@ -43,7 +43,7 @@ Generic engineering rules live in `bible/rules/patterns/`. - Inter-GPU communication check - `all_reduce_perf` - GPU bandwidth check - - `dcgmi diag -r nvbandwidth` + - `dcgmi diag -r nvbandwidth` (per CPU socket, then all selected GPUs, on multi-socket systems -- see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`) ### Validate -> Stress @@ -68,4 +68,4 @@ Generic engineering rules live in `bible/rules/patterns/`. - Inter-GPU communication check - `all_reduce_perf` - GPU bandwidth check - - `dcgmi diag -r nvbandwidth` + - `dcgmi diag -r nvbandwidth` (per CPU socket, then all selected GPUs, on multi-socket systems -- see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`) diff --git a/bible-local/architecture/runtime-flows.md b/bible-local/architecture/runtime-flows.md index 00a5aec..ff82679 100644 --- a/bible-local/architecture/runtime-flows.md +++ b/bible-local/architecture/runtime-flows.md @@ -167,6 +167,64 @@ Acceptance flows: - Runtime overrides: - `BEE_MEMTESTER_SIZE_MB` - `BEE_MEMTESTER_PASSES` +- NVIDIA Bandwidth SAT (`RunNvidiaBandwidthPack`, `dcgmi diag -r nvbandwidth`) on a + multi-socket system runs per CPU socket first, then all selected GPUs together + (`03-dcgmi-nvbandwidth-socket0.log`, `...-socket1.log`, `...-all.log`) -- + see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`. Single-socket + systems (or systems where a GPU's NUMA node can't be resolved) keep the + original single `NN-dcgmi-nvbandwidth.log` shape. + +## SAT job output durability + +``` +runAcceptancePackCtx job loop (per satJob) + 1. run the job's command; streamExecOutput writes each output line to the + job's log file as it arrives (not only when the process exits) + 2. write the job's final log file (same content the live stream already + wrote, plus any health-check suffix) + 3. call satJobBoundaryHook(jobName) if set + 4. append run_at_utc / *_status to summary.txt +``` + +**Critical invariants:** +- DO NOT change `streamExecOutput` back to buffering output in memory and + writing the job's log file only once, after the command exits -- see + `decisions/2026-07-27-sat-live-output-and-blackbox-kick.md`. The live ISO's + export directory sits on a `toram` RAM-backed overlay (see Boot sequence + above), so a crash mid-command is otherwise unrecoverable for that job. +- `platform` never imports `app`; `SetJobBoundaryHook` is a plain + `func(string)` seam (same pattern as `satExecCommand`/`satStat`), not a + direct call into blackbox internals. + +## Blackbox sync flow + +``` +bee-blackbox.service (separate process from bee-web/bee-audit) + 1. discover enrolled removable-media targets every blackboxDiscoverInterval (2s) + 2. per enrolled target, blackboxWorker.run(): + a. syncCycle(): mount target, mirror exportDir -> removable media, + write README.md/manifest docs, fsync + b. record lastKickSeen = mtime of exportDir/.blackbox-kick + c. wait for: the adaptive flushPeriod timer (1-30s), OR + a stop signal, OR + a poll tick (blackboxKickPollInterval, 250ms) showing the kick + file is newer than lastKickSeen -- whichever comes first + 3. on kick-triggered wake: sync immediately, do not wait out the + remaining flushPeriod +``` + +**Critical invariants:** +- `app.New()` wires `platform.SetJobBoundaryHook` to touch + `exportDir/.blackbox-kick` after every SAT job finishes. If a new SAT + execution path bypasses `runAcceptancePackCtx`'s job loop, it will not + trigger this kick -- data from that path still eventually reaches + blackbox via the adaptive timer, just not promptly. +- The adaptive `flushPeriod` logic (`adjustFlushPeriod`) is unchanged by the + kick mechanism -- the kick only short-circuits the *wait*, it does not + reset `flushPeriod` itself. +- DO NOT assume a local write under the live ISO's export directory is + durable on its own (RAM-backed overlay) -- blackbox's mirror to removable + media is the only real persistence boundary across a hard reset. ## NVIDIA SAT Web UI flow diff --git a/bible-local/decisions/2026-07-27-nvbandwidth-per-socket-split.md b/bible-local/decisions/2026-07-27-nvbandwidth-per-socket-split.md new file mode 100644 index 0000000..855fa9a --- /dev/null +++ b/bible-local/decisions/2026-07-27-nvbandwidth-per-socket-split.md @@ -0,0 +1,59 @@ +# Decision: Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass + +**Date:** 2026-07-27 +**Status:** active + +## Context + +On a dual-socket AMD EPYC 9655 server with 6x NVIDIA H200 NVL (PCIe form +factor, no NVLink bridges installed, no NVSwitch), running the NVIDIA +Bandwidth SAT (`dcgmi diag -r nvbandwidth`) across all 6 GPUs caused the +whole host to hard-reboot within seconds of the test starting. This +happened reliably; other GPU tests on the same box did not reproduce it: + +- A compute-only GPU stress test (~20 min, no GPU-to-GPU traffic) ran + stably. +- `nccl-tests` all-reduce across all 6 GPUs completed successfully, at + PCIe-level bandwidth (~18.7 GB/s busbw -- consistent with no direct P2P, + not a NVLink-class transport). +- Only `nvbandwidth` reproducibly triggered the reboot. + +Topology: GPUs 0-3 sit on socket 0 (NUMA node 0), GPUs 4-5 sit on socket 1 +(NUMA node 1); the cross-socket path goes through the CPU interconnect, not +a PCIe switch. `nvbandwidth` drives real peer-to-peer DMA between every GPU +pair, including that cross-socket pair -- a fault domain BURN and the NCCL +transport fallback never exercise. + +No prior SAT run isolated which specific GPU pairing (same-socket vs. +cross-socket) the reboot depended on: the task queue only recorded that +`nvidia-bandwidth` never reached `done`, and `bee-sat/gpu-nvidia-bandwidth-*/verbose.log` +showed the crash landed mid-command, with no further detail on which +transfer was in flight. + +## Decision + +`RunNvidiaBandwidthPack` (`audit/internal/platform/sat.go`) now resolves +each selected GPU's NUMA node (`gpuBandwidthSocketGroups`, via +`nvidia-smi --query-gpu=index,pci.bus_id` + the PCI device's sysfs +`numa_node`) and, when GPUs span more than one node, runs `nvbandwidth` +three times in sequence: once per socket group, then once across all +selected GPUs. Job files are named +`NN-dcgmi-nvbandwidth-socket.log` / `NN-dcgmi-nvbandwidth-all.log`. + +Single-socket systems, or systems where the NUMA node can't be resolved for +every GPU, keep the original one-pass shape (`NN-dcgmi-nvbandwidth.log`) -- +there is nothing meaningful to split in that case, and the log-file naming +stays backward compatible with older support bundles. + +## Consequences + +- A future crash confined to the all-GPU pass, with both single-socket + passes already logged clean, pinpoints the cross-socket P2P path as the + trigger instead of leaving it conflated with "a general GPU/PCIe fault". +- `SATEstimatedNvidiaBandwidthSec` no longer reflects worst-case duration on + multi-socket systems (now up to 3x); re-measure from real multi-socket + task logs once available and bump the constant -- do not just multiply by + 3 without real data, since per-pass overhead is not purely linear. +- DO NOT collapse this back into a single all-GPU `nvbandwidth` job for + multi-socket systems: that was the exact shape that made the original + incident unattributable to a specific GPU pairing. diff --git a/bible-local/decisions/2026-07-27-sat-live-output-and-blackbox-kick.md b/bible-local/decisions/2026-07-27-sat-live-output-and-blackbox-kick.md new file mode 100644 index 0000000..9072d47 --- /dev/null +++ b/bible-local/decisions/2026-07-27-sat-live-output-and-blackbox-kick.md @@ -0,0 +1,67 @@ +# Decision: Stream SAT job output live to disk and kick blackbox sync on job completion + +**Date:** 2026-07-27 +**Status:** active + +## Context + +Investigating a hard reboot mid-`nvbandwidth` (see +`2026-07-27-nvbandwidth-per-socket-split.md`) showed that the SAT job runner +had no way to preserve a crashing job's own output. `streamExecOutput` +(`audit/internal/platform/sat.go`) buffered a job's stdout/stderr entirely +in memory; the job's named log file (`NN-.log`) was written to disk +only once, after the process exited. A crash mid-command meant that job's +file never existed on disk at all -- `verbose.log`'s `start`/`finish` +markers survived (they are appended per-line already), but the command's +own output did not. + +This matters more than it would on a normal install: the live ISO boots +with `toram` (`live-boot` copies the read-only medium into RAM and mounts +an overlay on top of it -- see `runtime-flows.md`'s Boot sequence). The +export directory bee writes to lives on that RAM-backed overlay, so a +local write surviving to "disk" is not itself durable across a hard reset +-- the only real persistence boundary is `bee-blackbox.service` mirroring +the export directory out to removable media on its own adaptive schedule +(1-30s flush period). Even a perfectly-written local file is lost if +blackbox hasn't copied it off before the crash. + +## Decision + +Two changes, in `audit/internal/platform/sat.go` and +`audit/internal/app/blackbox.go`: + +1. `streamExecOutput` takes an optional `livePath`; when set, each output + line is written straight to that file as it arrives, in addition to the + in-memory buffer used for the final return value. + `runSATCommandCtx`/`runSATCommandWithMetrics` thread this through from + `runAcceptancePackCtx`'s job loop and the storage SAT loop, pointing at + each job's own named log file. A crash mid-command now leaves whatever + had printed up to that point instead of nothing. +2. `platform.SetJobBoundaryHook(func(jobName string))` is called right + after each job's output file is written (success or failure, every + job). `app.New()` wires this to touch a marker file + (`.blackbox-kick` under `exportDir`); `blackboxWorker.run()` polls that + marker's mtime (`blackboxKickPollInterval`, default 250ms) alongside its + normal adaptive timer and syncs immediately on a kick instead of waiting + out the current flush period. + +`platform` does not import `app` or know about blackbox; the job-boundary +hook is a plain `func(string)` seam, consistent with the existing +`satExecCommand`/`satStat`/`satReadFile` seams in the same file. + +## Consequences + +- A job that completes cleanly reaches removable media within one poll + interval instead of waiting out whatever the adaptive flush period + happens to be (up to 30s) -- closing most of the historical gap for + postmortem analysis of a later crash. +- The job that is in flight when a crash happens still only has whatever + had streamed to its live file by that point -- this does not make output + capture instantaneous, it makes it incremental instead of all-or-nothing. +- DO NOT revert `streamExecOutput`'s per-job file write to a single + end-of-command `os.WriteFile` -- that was the exact gap that lost the + `nvbandwidth` crash's own output in the original incident. +- DO NOT assume a local write under the live-ISO's export directory is + durable on its own; anything that must survive a hard reset needs + blackbox to have copied it off first (this is why the kick exists at + all, not just the live write). diff --git a/bible-local/decisions/2026-07-28-pci-realloc-out-of-default-boot-entries.md b/bible-local/decisions/2026-07-28-pci-realloc-out-of-default-boot-entries.md new file mode 100644 index 0000000..34a76c4 --- /dev/null +++ b/bible-local/decisions/2026-07-28-pci-realloc-out-of-default-boot-entries.md @@ -0,0 +1,79 @@ +# Decision: Move pci=realloc out of the default/toram/no-GUI GRUB entries + +**Date:** 2026-07-28 +**Status:** active + +## Context + +`pci=realloc` was added to the default, toram, and no-GUI/gsp-off GRUB menu +entries by commit 8bf8dfa ("default to KMS + pci=realloc, drop nomodeset from +main entries") as a workaround for GPU BAR assignment failures on servers +where the BIOS lacks "Above 4G Decoding". It forces the Linux kernel to +fully reallocate all PCI resources instead of trusting the BIOS-assigned +addresses. + +On a dual-socket AMD EPYC 9655 server (MSI CG480-S6053, 6x GPU + 8x NVMe +behind 2-port hot-plug-capable PCIe switches), this same flag caused a +different failure: the OS only exposed 2 of 8 NVMe drives, even though +`lspci` and the BIOS both enumerated all 6 controllers actually present in +the affected chassis. `dmesg` showed the kernel's bridge-window reallocation +pass exhausting available MMIO32 address space: + +``` +pci 0000:f2:03.0: bridge window [mem size 0x00200000 64bit pref]: can't assign; no space +pci 0000:f7:00.0: BAR 0 [mem size 0x00004000 64bit]: can't assign; no space +``` + +The BIOS had already assigned working BARs to all 6 NVMe controllers at +boot; `pci=realloc` threw those away and failed to re-assign 4 of them, +so the `nvme` driver never bound to those 4 (no `nvme nvmeX: pci function` +line), and they never became block devices. The dense NVMe backplane, not +the GPUs, hit the address-space ceiling that `pci=realloc` re-triggered on +every device in the topology, not just the GPUs it was meant to fix. + +`pci=realloc` is a blanket flag: it was applied globally to the default and +toram entries (the ones almost every operator actually boots), so any +server with a BIOS whose GPU BARs are fine, but whose PCIe topology is +dense enough elsewhere (many NVMe bays, many switches), pays this cost +unconditionally. + +## Decision + +`pci=realloc` is removed from the default and toram menu entries in both +`iso/builder/build.sh` (`write_canonical_grub_cfg`) and the static +`iso/builder/config/bootloaders/grub-efi/grub.cfg` template. Rather than +adding a new single-purpose menu entry for this one workaround, it is +folded into the existing no-GUI/gsp-off entry, which is already the +troubleshooting/fail-safe-style option operators reach for when the +default entry does not work cleanly: + +``` +EASY-BEE v -- no GUI / troubleshooting (gsp-off, pci=realloc) +``` + +An operator who hits either a GSP-related GPU issue or a GPU BAR +assignment failure (BIOS lacking Above 4G Decoding) selects this single +entry manually, instead of the menu growing one dedicated item per +individual failure mode. This keeps the total entry count in +`grub.cfg` unchanged from before the incident (still 3 real boot +entries plus wipe/memtest/firmware-settings), rather than letting it +grow linearly with every workaround discovered. + +The isolinux (BIOS/legacy boot) menu in `iso/builder/config/bootloaders/isolinux/live.cfg.in` +already had no `pci=realloc` entry at all (pre-existing asymmetry with +grub-efi, not introduced by this change) -- not addressed here since no +incident has been observed via legacy boot. + +## Consequences + +- Servers with dense PCIe topologies (many NVMe bays, many GPUs) no longer + silently lose drives to a workaround they didn't need, by default. +- Servers that genuinely need the GPU BAR workaround (BIOS lacking Above 4G + Decoding) now require the operator to notice GPU BAR failures and + deliberately pick the workaround entry, instead of getting it for free -- + document this tradeoff in the support runbook / vendor-facing material + if GPU BAR failures turn out to be common enough to warrant it. +- DO NOT move `pci=realloc` back into the default/toram entries without + first checking whether the affected server's topology can tolerate a + full PCI reallocation pass -- this was the exact failure mode of the + incident that prompted this decision. diff --git a/bible-local/decisions/README.md b/bible-local/decisions/README.md index 98a0c0a..9d1b1d9 100644 --- a/bible-local/decisions/README.md +++ b/bible-local/decisions/README.md @@ -8,3 +8,6 @@ One file per decision, named `YYYY-MM-DD-short-topic.md`. | 2026-04-01 | Treat memtest as explicit ISO content | active | | 2026-04-29 | Treat embedded submodules as read-only | active | | 2026-06-12 | Skip PCIe link-speed warnings for disabled devices | active | +| 2026-07-27 | Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass | active | +| 2026-07-27 | Stream SAT job output live to disk and kick blackbox sync on job completion | active | +| 2026-07-28 | Move pci=realloc out of the default/toram/no-GUI GRUB entries | active | diff --git a/iso/builder/build.sh b/iso/builder/build.sh index d9f4d9b..eee37aa 100755 --- a/iso/builder/build.sh +++ b/iso/builder/build.sh @@ -779,16 +779,16 @@ write_canonical_grub_cfg() { source /boot/grub/config.cfg menuentry "EASY-BEE v${version_label}" { - linux ${kernel} ${append_live} nomodeset bee.nvidia.mode=normal pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup + linux ${kernel} ${append_live} nomodeset bee.nvidia.mode=normal net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup initrd ${initrd} } menuentry "EASY-BEE v${version_label} -- load to RAM (toram)" { - linux ${kernel} ${append_live} toram nomodeset bee.nvidia.mode=normal pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup + linux ${kernel} ${append_live} toram nomodeset bee.nvidia.mode=normal net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup initrd ${initrd} } -menuentry "EASY-BEE v${version_label} -- no GUI / no X11" { +menuentry "EASY-BEE v${version_label} -- no GUI / troubleshooting (gsp-off, pci=realloc)" { linux ${kernel} ${append_live} nomodeset bee.gui=off bee.nvidia.mode=gsp-off pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup initrd ${initrd} } diff --git a/iso/builder/config/bootloaders/grub-efi/grub.cfg b/iso/builder/config/bootloaders/grub-efi/grub.cfg index e3d8211..bbd8646 100644 --- a/iso/builder/config/bootloaders/grub-efi/grub.cfg +++ b/iso/builder/config/bootloaders/grub-efi/grub.cfg @@ -1,16 +1,16 @@ source /boot/grub/config.cfg menuentry "EASY-BEE v@VERSION@" { - linux @KERNEL_LIVE@ @APPEND_LIVE@ nomodeset bee.nvidia.mode=normal pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup + linux @KERNEL_LIVE@ @APPEND_LIVE@ nomodeset bee.nvidia.mode=normal net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup initrd @INITRD_LIVE@ } menuentry "EASY-BEE v@VERSION@ -- load to RAM (toram)" { - linux @KERNEL_LIVE@ @APPEND_LIVE@ toram nomodeset bee.nvidia.mode=normal pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup + linux @KERNEL_LIVE@ @APPEND_LIVE@ toram nomodeset bee.nvidia.mode=normal net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup initrd @INITRD_LIVE@ } -menuentry "EASY-BEE v@VERSION@ -- no GUI / no X11" { +menuentry "EASY-BEE v@VERSION@ -- no GUI / troubleshooting (gsp-off, pci=realloc)" { linux @KERNEL_LIVE@ @APPEND_LIVE@ nomodeset bee.gui=off bee.nvidia.mode=gsp-off pci=realloc net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1 nowatchdog nosoftlockup initrd @INITRD_LIVE@ }