Mikhail ChusavitinandClaude Sonnet 5
e036a9c7b8
pci=realloc was forcing a full PCI resource reallocation on every default and toram boot, added as a GPU BAR workaround for BIOS without Above 4G Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS and lspci both saw them fine. It now only applies to the no-GUI/gsp-off troubleshooting entry, alongside a bible-local ADR for the two prior rounds of SAT/blackbox durability work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2.9 KiB
Decision: Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass
Date: 2026-07-27 Status: active
Context
On a dual-socket AMD EPYC 9655 server with 6x NVIDIA H200 NVL (PCIe form
factor, no NVLink bridges installed, no NVSwitch), running the NVIDIA
Bandwidth SAT (dcgmi diag -r nvbandwidth) across all 6 GPUs caused the
whole host to hard-reboot within seconds of the test starting. This
happened reliably; other GPU tests on the same box did not reproduce it:
- A compute-only GPU stress test (~20 min, no GPU-to-GPU traffic) ran stably.
nccl-testsall-reduce across all 6 GPUs completed successfully, at PCIe-level bandwidth (~18.7 GB/s busbw -- consistent with no direct P2P, not a NVLink-class transport).- Only
nvbandwidthreproducibly triggered the reboot.
Topology: GPUs 0-3 sit on socket 0 (NUMA node 0), GPUs 4-5 sit on socket 1
(NUMA node 1); the cross-socket path goes through the CPU interconnect, not
a PCIe switch. nvbandwidth drives real peer-to-peer DMA between every GPU
pair, including that cross-socket pair -- a fault domain BURN and the NCCL
transport fallback never exercise.
No prior SAT run isolated which specific GPU pairing (same-socket vs.
cross-socket) the reboot depended on: the task queue only recorded that
nvidia-bandwidth never reached done, and bee-sat/gpu-nvidia-bandwidth-*/verbose.log
showed the crash landed mid-command, with no further detail on which
transfer was in flight.
Decision
RunNvidiaBandwidthPack (audit/internal/platform/sat.go) now resolves
each selected GPU's NUMA node (gpuBandwidthSocketGroups, via
nvidia-smi --query-gpu=index,pci.bus_id + the PCI device's sysfs
numa_node) and, when GPUs span more than one node, runs nvbandwidth
three times in sequence: once per socket group, then once across all
selected GPUs. Job files are named
NN-dcgmi-nvbandwidth-socket<N>.log / NN-dcgmi-nvbandwidth-all.log.
Single-socket systems, or systems where the NUMA node can't be resolved for
every GPU, keep the original one-pass shape (NN-dcgmi-nvbandwidth.log) --
there is nothing meaningful to split in that case, and the log-file naming
stays backward compatible with older support bundles.
Consequences
- A future crash confined to the all-GPU pass, with both single-socket passes already logged clean, pinpoints the cross-socket P2P path as the trigger instead of leaving it conflated with "a general GPU/PCIe fault".
SATEstimatedNvidiaBandwidthSecno longer reflects worst-case duration on multi-socket systems (now up to 3x); re-measure from real multi-socket task logs once available and bump the constant -- do not just multiply by 3 without real data, since per-pass overhead is not purely linear.- DO NOT collapse this back into a single all-GPU
nvbandwidthjob for multi-socket systems: that was the exact shape that made the original incident unattributable to a specific GPU pairing.