# Decision: Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass **Date:** 2026-07-27 **Status:** active ## Context On a dual-socket AMD EPYC 9655 server with 6x NVIDIA H200 NVL (PCIe form factor, no NVLink bridges installed, no NVSwitch), running the NVIDIA Bandwidth SAT (`dcgmi diag -r nvbandwidth`) across all 6 GPUs caused the whole host to hard-reboot within seconds of the test starting. This happened reliably; other GPU tests on the same box did not reproduce it: - A compute-only GPU stress test (~20 min, no GPU-to-GPU traffic) ran stably. - `nccl-tests` all-reduce across all 6 GPUs completed successfully, at PCIe-level bandwidth (~18.7 GB/s busbw -- consistent with no direct P2P, not a NVLink-class transport). - Only `nvbandwidth` reproducibly triggered the reboot. Topology: GPUs 0-3 sit on socket 0 (NUMA node 0), GPUs 4-5 sit on socket 1 (NUMA node 1); the cross-socket path goes through the CPU interconnect, not a PCIe switch. `nvbandwidth` drives real peer-to-peer DMA between every GPU pair, including that cross-socket pair -- a fault domain BURN and the NCCL transport fallback never exercise. No prior SAT run isolated which specific GPU pairing (same-socket vs. cross-socket) the reboot depended on: the task queue only recorded that `nvidia-bandwidth` never reached `done`, and `bee-sat/gpu-nvidia-bandwidth-*/verbose.log` showed the crash landed mid-command, with no further detail on which transfer was in flight. ## Decision `RunNvidiaBandwidthPack` (`audit/internal/platform/sat.go`) now resolves each selected GPU's NUMA node (`gpuBandwidthSocketGroups`, via `nvidia-smi --query-gpu=index,pci.bus_id` + the PCI device's sysfs `numa_node`) and, when GPUs span more than one node, runs `nvbandwidth` three times in sequence: once per socket group, then once across all selected GPUs. Job files are named `NN-dcgmi-nvbandwidth-socket.log` / `NN-dcgmi-nvbandwidth-all.log`. Single-socket systems, or systems where the NUMA node can't be resolved for every GPU, keep the original one-pass shape (`NN-dcgmi-nvbandwidth.log`) -- there is nothing meaningful to split in that case, and the log-file naming stays backward compatible with older support bundles. ## Consequences - A future crash confined to the all-GPU pass, with both single-socket passes already logged clean, pinpoints the cross-socket P2P path as the trigger instead of leaving it conflated with "a general GPU/PCIe fault". - `SATEstimatedNvidiaBandwidthSec` no longer reflects worst-case duration on multi-socket systems (now up to 3x); re-measure from real multi-socket task logs once available and bump the constant -- do not just multiply by 3 without real data, since per-pass overhead is not purely linear. - DO NOT collapse this back into a single all-GPU `nvbandwidth` job for multi-socket systems: that was the exact shape that made the original incident unattributable to a specific GPU pairing.