boot: drop pci=realloc from default/toram GRUB entries, fold into troubleshooting entry

pci=realloc was forcing a full PCI resource reallocation on every default
and toram boot, added as a GPU BAR workaround for BIOS without Above 4G
Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and
silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS
and lspci both saw them fine. It now only applies to the no-GUI/gsp-off
troubleshooting entry, alongside a bible-local ADR for the two prior
rounds of SAT/blackbox durability work.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

bible-local/decisions/2026-07-27-nvbandwidth-per-socket-split.md

@@ -0,0 +1,59 @@
+# Decision: Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass
+
+**Date:** 2026-07-27
+**Status:** active
+
+## Context
+
+On a dual-socket AMD EPYC 9655 server with 6x NVIDIA H200 NVL (PCIe form
+factor, no NVLink bridges installed, no NVSwitch), running the NVIDIA
+Bandwidth SAT (`dcgmi diag -r nvbandwidth`) across all 6 GPUs caused the
+whole host to hard-reboot within seconds of the test starting. This
+happened reliably; other GPU tests on the same box did not reproduce it:
+
+- A compute-only GPU stress test (~20 min, no GPU-to-GPU traffic) ran
+  stably.
+- `nccl-tests` all-reduce across all 6 GPUs completed successfully, at
+  PCIe-level bandwidth (~18.7 GB/s busbw -- consistent with no direct P2P,
+  not a NVLink-class transport).
+- Only `nvbandwidth` reproducibly triggered the reboot.
+
+Topology: GPUs 0-3 sit on socket 0 (NUMA node 0), GPUs 4-5 sit on socket 1
+(NUMA node 1); the cross-socket path goes through the CPU interconnect, not
+a PCIe switch. `nvbandwidth` drives real peer-to-peer DMA between every GPU
+pair, including that cross-socket pair -- a fault domain BURN and the NCCL
+transport fallback never exercise.
+
+No prior SAT run isolated which specific GPU pairing (same-socket vs.
+cross-socket) the reboot depended on: the task queue only recorded that
+`nvidia-bandwidth` never reached `done`, and `bee-sat/gpu-nvidia-bandwidth-*/verbose.log`
+showed the crash landed mid-command, with no further detail on which
+transfer was in flight.
+
+## Decision
+
+`RunNvidiaBandwidthPack` (`audit/internal/platform/sat.go`) now resolves
+each selected GPU's NUMA node (`gpuBandwidthSocketGroups`, via
+`nvidia-smi --query-gpu=index,pci.bus_id` + the PCI device's sysfs
+`numa_node`) and, when GPUs span more than one node, runs `nvbandwidth`
+three times in sequence: once per socket group, then once across all
+selected GPUs. Job files are named
+`NN-dcgmi-nvbandwidth-socket<N>.log` / `NN-dcgmi-nvbandwidth-all.log`.
+
+Single-socket systems, or systems where the NUMA node can't be resolved for
+every GPU, keep the original one-pass shape (`NN-dcgmi-nvbandwidth.log`) --
+there is nothing meaningful to split in that case, and the log-file naming
+stays backward compatible with older support bundles.
+
+## Consequences
+
+- A future crash confined to the all-GPU pass, with both single-socket
+  passes already logged clean, pinpoints the cross-socket P2P path as the
+  trigger instead of leaving it conflated with "a general GPU/PCIe fault".
+- `SATEstimatedNvidiaBandwidthSec` no longer reflects worst-case duration on
+  multi-socket systems (now up to 3x); re-measure from real multi-socket
+  task logs once available and bump the constant -- do not just multiply by
+  3 without real data, since per-pass overhead is not purely linear.
+- DO NOT collapse this back into a single all-GPU `nvbandwidth` job for
+  multi-socket systems: that was the exact shape that made the original
+  incident unattributable to a specific GPU pairing.