Mikhail ChusavitinandClaude Sonnet 5
e036a9c7b8
pci=realloc was forcing a full PCI resource reallocation on every default and toram boot, added as a GPU BAR workaround for BIOS without Above 4G Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS and lspci both saw them fine. It now only applies to the no-GUI/gsp-off troubleshooting entry, alongside a bible-local ADR for the two prior rounds of SAT/blackbox durability work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
60 lines
2.9 KiB
Markdown
60 lines
2.9 KiB
Markdown
# Decision: Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass
|
|
|
|
**Date:** 2026-07-27
|
|
**Status:** active
|
|
|
|
## Context
|
|
|
|
On a dual-socket AMD EPYC 9655 server with 6x NVIDIA H200 NVL (PCIe form
|
|
factor, no NVLink bridges installed, no NVSwitch), running the NVIDIA
|
|
Bandwidth SAT (`dcgmi diag -r nvbandwidth`) across all 6 GPUs caused the
|
|
whole host to hard-reboot within seconds of the test starting. This
|
|
happened reliably; other GPU tests on the same box did not reproduce it:
|
|
|
|
- A compute-only GPU stress test (~20 min, no GPU-to-GPU traffic) ran
|
|
stably.
|
|
- `nccl-tests` all-reduce across all 6 GPUs completed successfully, at
|
|
PCIe-level bandwidth (~18.7 GB/s busbw -- consistent with no direct P2P,
|
|
not a NVLink-class transport).
|
|
- Only `nvbandwidth` reproducibly triggered the reboot.
|
|
|
|
Topology: GPUs 0-3 sit on socket 0 (NUMA node 0), GPUs 4-5 sit on socket 1
|
|
(NUMA node 1); the cross-socket path goes through the CPU interconnect, not
|
|
a PCIe switch. `nvbandwidth` drives real peer-to-peer DMA between every GPU
|
|
pair, including that cross-socket pair -- a fault domain BURN and the NCCL
|
|
transport fallback never exercise.
|
|
|
|
No prior SAT run isolated which specific GPU pairing (same-socket vs.
|
|
cross-socket) the reboot depended on: the task queue only recorded that
|
|
`nvidia-bandwidth` never reached `done`, and `bee-sat/gpu-nvidia-bandwidth-*/verbose.log`
|
|
showed the crash landed mid-command, with no further detail on which
|
|
transfer was in flight.
|
|
|
|
## Decision
|
|
|
|
`RunNvidiaBandwidthPack` (`audit/internal/platform/sat.go`) now resolves
|
|
each selected GPU's NUMA node (`gpuBandwidthSocketGroups`, via
|
|
`nvidia-smi --query-gpu=index,pci.bus_id` + the PCI device's sysfs
|
|
`numa_node`) and, when GPUs span more than one node, runs `nvbandwidth`
|
|
three times in sequence: once per socket group, then once across all
|
|
selected GPUs. Job files are named
|
|
`NN-dcgmi-nvbandwidth-socket<N>.log` / `NN-dcgmi-nvbandwidth-all.log`.
|
|
|
|
Single-socket systems, or systems where the NUMA node can't be resolved for
|
|
every GPU, keep the original one-pass shape (`NN-dcgmi-nvbandwidth.log`) --
|
|
there is nothing meaningful to split in that case, and the log-file naming
|
|
stays backward compatible with older support bundles.
|
|
|
|
## Consequences
|
|
|
|
- A future crash confined to the all-GPU pass, with both single-socket
|
|
passes already logged clean, pinpoints the cross-socket P2P path as the
|
|
trigger instead of leaving it conflated with "a general GPU/PCIe fault".
|
|
- `SATEstimatedNvidiaBandwidthSec` no longer reflects worst-case duration on
|
|
multi-socket systems (now up to 3x); re-measure from real multi-socket
|
|
task logs once available and bump the constant -- do not just multiply by
|
|
3 without real data, since per-pass overhead is not purely linear.
|
|
- DO NOT collapse this back into a single all-GPU `nvbandwidth` job for
|
|
multi-socket systems: that was the exact shape that made the original
|
|
incident unattributable to a specific GPU pairing.
|