boot: drop pci=realloc from default/toram GRUB entries, fold into troubleshooting entry
pci=realloc was forcing a full PCI resource reallocation on every default and toram boot, added as a GPU BAR workaround for BIOS without Above 4G Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS and lspci both saw them fine. It now only applies to the no-GUI/gsp-off troubleshooting entry, alongside a bible-local ADR for the two prior rounds of SAT/blackbox durability work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
781cf5dcbf
commit
e036a9c7b8
@@ -43,7 +43,7 @@ Generic engineering rules live in `bible/rules/patterns/`.
|
||||
- Inter-GPU communication check
|
||||
- `all_reduce_perf`
|
||||
- GPU bandwidth check
|
||||
- `dcgmi diag -r nvbandwidth`
|
||||
- `dcgmi diag -r nvbandwidth` (per CPU socket, then all selected GPUs, on multi-socket systems -- see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`)
|
||||
|
||||
### Validate -> Stress
|
||||
|
||||
@@ -68,4 +68,4 @@ Generic engineering rules live in `bible/rules/patterns/`.
|
||||
- Inter-GPU communication check
|
||||
- `all_reduce_perf`
|
||||
- GPU bandwidth check
|
||||
- `dcgmi diag -r nvbandwidth`
|
||||
- `dcgmi diag -r nvbandwidth` (per CPU socket, then all selected GPUs, on multi-socket systems -- see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`)
|
||||
|
||||
@@ -167,6 +167,64 @@ Acceptance flows:
|
||||
- Runtime overrides:
|
||||
- `BEE_MEMTESTER_SIZE_MB`
|
||||
- `BEE_MEMTESTER_PASSES`
|
||||
- NVIDIA Bandwidth SAT (`RunNvidiaBandwidthPack`, `dcgmi diag -r nvbandwidth`) on a
|
||||
multi-socket system runs per CPU socket first, then all selected GPUs together
|
||||
(`03-dcgmi-nvbandwidth-socket0.log`, `...-socket1.log`, `...-all.log`) --
|
||||
see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`. Single-socket
|
||||
systems (or systems where a GPU's NUMA node can't be resolved) keep the
|
||||
original single `NN-dcgmi-nvbandwidth.log` shape.
|
||||
|
||||
## SAT job output durability
|
||||
|
||||
```
|
||||
runAcceptancePackCtx job loop (per satJob)
|
||||
1. run the job's command; streamExecOutput writes each output line to the
|
||||
job's log file as it arrives (not only when the process exits)
|
||||
2. write the job's final log file (same content the live stream already
|
||||
wrote, plus any health-check suffix)
|
||||
3. call satJobBoundaryHook(jobName) if set
|
||||
4. append run_at_utc / *_status to summary.txt
|
||||
```
|
||||
|
||||
**Critical invariants:**
|
||||
- DO NOT change `streamExecOutput` back to buffering output in memory and
|
||||
writing the job's log file only once, after the command exits -- see
|
||||
`decisions/2026-07-27-sat-live-output-and-blackbox-kick.md`. The live ISO's
|
||||
export directory sits on a `toram` RAM-backed overlay (see Boot sequence
|
||||
above), so a crash mid-command is otherwise unrecoverable for that job.
|
||||
- `platform` never imports `app`; `SetJobBoundaryHook` is a plain
|
||||
`func(string)` seam (same pattern as `satExecCommand`/`satStat`), not a
|
||||
direct call into blackbox internals.
|
||||
|
||||
## Blackbox sync flow
|
||||
|
||||
```
|
||||
bee-blackbox.service (separate process from bee-web/bee-audit)
|
||||
1. discover enrolled removable-media targets every blackboxDiscoverInterval (2s)
|
||||
2. per enrolled target, blackboxWorker.run():
|
||||
a. syncCycle(): mount target, mirror exportDir -> removable media,
|
||||
write README.md/manifest docs, fsync
|
||||
b. record lastKickSeen = mtime of exportDir/.blackbox-kick
|
||||
c. wait for: the adaptive flushPeriod timer (1-30s), OR
|
||||
a stop signal, OR
|
||||
a poll tick (blackboxKickPollInterval, 250ms) showing the kick
|
||||
file is newer than lastKickSeen -- whichever comes first
|
||||
3. on kick-triggered wake: sync immediately, do not wait out the
|
||||
remaining flushPeriod
|
||||
```
|
||||
|
||||
**Critical invariants:**
|
||||
- `app.New()` wires `platform.SetJobBoundaryHook` to touch
|
||||
`exportDir/.blackbox-kick` after every SAT job finishes. If a new SAT
|
||||
execution path bypasses `runAcceptancePackCtx`'s job loop, it will not
|
||||
trigger this kick -- data from that path still eventually reaches
|
||||
blackbox via the adaptive timer, just not promptly.
|
||||
- The adaptive `flushPeriod` logic (`adjustFlushPeriod`) is unchanged by the
|
||||
kick mechanism -- the kick only short-circuits the *wait*, it does not
|
||||
reset `flushPeriod` itself.
|
||||
- DO NOT assume a local write under the live ISO's export directory is
|
||||
durable on its own (RAM-backed overlay) -- blackbox's mirror to removable
|
||||
media is the only real persistence boundary across a hard reset.
|
||||
|
||||
## NVIDIA SAT Web UI flow
|
||||
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# Decision: Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass
|
||||
|
||||
**Date:** 2026-07-27
|
||||
**Status:** active
|
||||
|
||||
## Context
|
||||
|
||||
On a dual-socket AMD EPYC 9655 server with 6x NVIDIA H200 NVL (PCIe form
|
||||
factor, no NVLink bridges installed, no NVSwitch), running the NVIDIA
|
||||
Bandwidth SAT (`dcgmi diag -r nvbandwidth`) across all 6 GPUs caused the
|
||||
whole host to hard-reboot within seconds of the test starting. This
|
||||
happened reliably; other GPU tests on the same box did not reproduce it:
|
||||
|
||||
- A compute-only GPU stress test (~20 min, no GPU-to-GPU traffic) ran
|
||||
stably.
|
||||
- `nccl-tests` all-reduce across all 6 GPUs completed successfully, at
|
||||
PCIe-level bandwidth (~18.7 GB/s busbw -- consistent with no direct P2P,
|
||||
not a NVLink-class transport).
|
||||
- Only `nvbandwidth` reproducibly triggered the reboot.
|
||||
|
||||
Topology: GPUs 0-3 sit on socket 0 (NUMA node 0), GPUs 4-5 sit on socket 1
|
||||
(NUMA node 1); the cross-socket path goes through the CPU interconnect, not
|
||||
a PCIe switch. `nvbandwidth` drives real peer-to-peer DMA between every GPU
|
||||
pair, including that cross-socket pair -- a fault domain BURN and the NCCL
|
||||
transport fallback never exercise.
|
||||
|
||||
No prior SAT run isolated which specific GPU pairing (same-socket vs.
|
||||
cross-socket) the reboot depended on: the task queue only recorded that
|
||||
`nvidia-bandwidth` never reached `done`, and `bee-sat/gpu-nvidia-bandwidth-*/verbose.log`
|
||||
showed the crash landed mid-command, with no further detail on which
|
||||
transfer was in flight.
|
||||
|
||||
## Decision
|
||||
|
||||
`RunNvidiaBandwidthPack` (`audit/internal/platform/sat.go`) now resolves
|
||||
each selected GPU's NUMA node (`gpuBandwidthSocketGroups`, via
|
||||
`nvidia-smi --query-gpu=index,pci.bus_id` + the PCI device's sysfs
|
||||
`numa_node`) and, when GPUs span more than one node, runs `nvbandwidth`
|
||||
three times in sequence: once per socket group, then once across all
|
||||
selected GPUs. Job files are named
|
||||
`NN-dcgmi-nvbandwidth-socket<N>.log` / `NN-dcgmi-nvbandwidth-all.log`.
|
||||
|
||||
Single-socket systems, or systems where the NUMA node can't be resolved for
|
||||
every GPU, keep the original one-pass shape (`NN-dcgmi-nvbandwidth.log`) --
|
||||
there is nothing meaningful to split in that case, and the log-file naming
|
||||
stays backward compatible with older support bundles.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A future crash confined to the all-GPU pass, with both single-socket
|
||||
passes already logged clean, pinpoints the cross-socket P2P path as the
|
||||
trigger instead of leaving it conflated with "a general GPU/PCIe fault".
|
||||
- `SATEstimatedNvidiaBandwidthSec` no longer reflects worst-case duration on
|
||||
multi-socket systems (now up to 3x); re-measure from real multi-socket
|
||||
task logs once available and bump the constant -- do not just multiply by
|
||||
3 without real data, since per-pass overhead is not purely linear.
|
||||
- DO NOT collapse this back into a single all-GPU `nvbandwidth` job for
|
||||
multi-socket systems: that was the exact shape that made the original
|
||||
incident unattributable to a specific GPU pairing.
|
||||
@@ -0,0 +1,67 @@
|
||||
# Decision: Stream SAT job output live to disk and kick blackbox sync on job completion
|
||||
|
||||
**Date:** 2026-07-27
|
||||
**Status:** active
|
||||
|
||||
## Context
|
||||
|
||||
Investigating a hard reboot mid-`nvbandwidth` (see
|
||||
`2026-07-27-nvbandwidth-per-socket-split.md`) showed that the SAT job runner
|
||||
had no way to preserve a crashing job's own output. `streamExecOutput`
|
||||
(`audit/internal/platform/sat.go`) buffered a job's stdout/stderr entirely
|
||||
in memory; the job's named log file (`NN-<job>.log`) was written to disk
|
||||
only once, after the process exited. A crash mid-command meant that job's
|
||||
file never existed on disk at all -- `verbose.log`'s `start`/`finish`
|
||||
markers survived (they are appended per-line already), but the command's
|
||||
own output did not.
|
||||
|
||||
This matters more than it would on a normal install: the live ISO boots
|
||||
with `toram` (`live-boot` copies the read-only medium into RAM and mounts
|
||||
an overlay on top of it -- see `runtime-flows.md`'s Boot sequence). The
|
||||
export directory bee writes to lives on that RAM-backed overlay, so a
|
||||
local write surviving to "disk" is not itself durable across a hard reset
|
||||
-- the only real persistence boundary is `bee-blackbox.service` mirroring
|
||||
the export directory out to removable media on its own adaptive schedule
|
||||
(1-30s flush period). Even a perfectly-written local file is lost if
|
||||
blackbox hasn't copied it off before the crash.
|
||||
|
||||
## Decision
|
||||
|
||||
Two changes, in `audit/internal/platform/sat.go` and
|
||||
`audit/internal/app/blackbox.go`:
|
||||
|
||||
1. `streamExecOutput` takes an optional `livePath`; when set, each output
|
||||
line is written straight to that file as it arrives, in addition to the
|
||||
in-memory buffer used for the final return value.
|
||||
`runSATCommandCtx`/`runSATCommandWithMetrics` thread this through from
|
||||
`runAcceptancePackCtx`'s job loop and the storage SAT loop, pointing at
|
||||
each job's own named log file. A crash mid-command now leaves whatever
|
||||
had printed up to that point instead of nothing.
|
||||
2. `platform.SetJobBoundaryHook(func(jobName string))` is called right
|
||||
after each job's output file is written (success or failure, every
|
||||
job). `app.New()` wires this to touch a marker file
|
||||
(`.blackbox-kick` under `exportDir`); `blackboxWorker.run()` polls that
|
||||
marker's mtime (`blackboxKickPollInterval`, default 250ms) alongside its
|
||||
normal adaptive timer and syncs immediately on a kick instead of waiting
|
||||
out the current flush period.
|
||||
|
||||
`platform` does not import `app` or know about blackbox; the job-boundary
|
||||
hook is a plain `func(string)` seam, consistent with the existing
|
||||
`satExecCommand`/`satStat`/`satReadFile` seams in the same file.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A job that completes cleanly reaches removable media within one poll
|
||||
interval instead of waiting out whatever the adaptive flush period
|
||||
happens to be (up to 30s) -- closing most of the historical gap for
|
||||
postmortem analysis of a later crash.
|
||||
- The job that is in flight when a crash happens still only has whatever
|
||||
had streamed to its live file by that point -- this does not make output
|
||||
capture instantaneous, it makes it incremental instead of all-or-nothing.
|
||||
- DO NOT revert `streamExecOutput`'s per-job file write to a single
|
||||
end-of-command `os.WriteFile` -- that was the exact gap that lost the
|
||||
`nvbandwidth` crash's own output in the original incident.
|
||||
- DO NOT assume a local write under the live-ISO's export directory is
|
||||
durable on its own; anything that must survive a hard reset needs
|
||||
blackbox to have copied it off first (this is why the kick exists at
|
||||
all, not just the live write).
|
||||
@@ -0,0 +1,79 @@
|
||||
# Decision: Move pci=realloc out of the default/toram/no-GUI GRUB entries
|
||||
|
||||
**Date:** 2026-07-28
|
||||
**Status:** active
|
||||
|
||||
## Context
|
||||
|
||||
`pci=realloc` was added to the default, toram, and no-GUI/gsp-off GRUB menu
|
||||
entries by commit 8bf8dfa ("default to KMS + pci=realloc, drop nomodeset from
|
||||
main entries") as a workaround for GPU BAR assignment failures on servers
|
||||
where the BIOS lacks "Above 4G Decoding". It forces the Linux kernel to
|
||||
fully reallocate all PCI resources instead of trusting the BIOS-assigned
|
||||
addresses.
|
||||
|
||||
On a dual-socket AMD EPYC 9655 server (MSI CG480-S6053, 6x GPU + 8x NVMe
|
||||
behind 2-port hot-plug-capable PCIe switches), this same flag caused a
|
||||
different failure: the OS only exposed 2 of 8 NVMe drives, even though
|
||||
`lspci` and the BIOS both enumerated all 6 controllers actually present in
|
||||
the affected chassis. `dmesg` showed the kernel's bridge-window reallocation
|
||||
pass exhausting available MMIO32 address space:
|
||||
|
||||
```
|
||||
pci 0000:f2:03.0: bridge window [mem size 0x00200000 64bit pref]: can't assign; no space
|
||||
pci 0000:f7:00.0: BAR 0 [mem size 0x00004000 64bit]: can't assign; no space
|
||||
```
|
||||
|
||||
The BIOS had already assigned working BARs to all 6 NVMe controllers at
|
||||
boot; `pci=realloc` threw those away and failed to re-assign 4 of them,
|
||||
so the `nvme` driver never bound to those 4 (no `nvme nvmeX: pci function`
|
||||
line), and they never became block devices. The dense NVMe backplane, not
|
||||
the GPUs, hit the address-space ceiling that `pci=realloc` re-triggered on
|
||||
every device in the topology, not just the GPUs it was meant to fix.
|
||||
|
||||
`pci=realloc` is a blanket flag: it was applied globally to the default and
|
||||
toram entries (the ones almost every operator actually boots), so any
|
||||
server with a BIOS whose GPU BARs are fine, but whose PCIe topology is
|
||||
dense enough elsewhere (many NVMe bays, many switches), pays this cost
|
||||
unconditionally.
|
||||
|
||||
## Decision
|
||||
|
||||
`pci=realloc` is removed from the default and toram menu entries in both
|
||||
`iso/builder/build.sh` (`write_canonical_grub_cfg`) and the static
|
||||
`iso/builder/config/bootloaders/grub-efi/grub.cfg` template. Rather than
|
||||
adding a new single-purpose menu entry for this one workaround, it is
|
||||
folded into the existing no-GUI/gsp-off entry, which is already the
|
||||
troubleshooting/fail-safe-style option operators reach for when the
|
||||
default entry does not work cleanly:
|
||||
|
||||
```
|
||||
EASY-BEE v<version> -- no GUI / troubleshooting (gsp-off, pci=realloc)
|
||||
```
|
||||
|
||||
An operator who hits either a GSP-related GPU issue or a GPU BAR
|
||||
assignment failure (BIOS lacking Above 4G Decoding) selects this single
|
||||
entry manually, instead of the menu growing one dedicated item per
|
||||
individual failure mode. This keeps the total entry count in
|
||||
`grub.cfg` unchanged from before the incident (still 3 real boot
|
||||
entries plus wipe/memtest/firmware-settings), rather than letting it
|
||||
grow linearly with every workaround discovered.
|
||||
|
||||
The isolinux (BIOS/legacy boot) menu in `iso/builder/config/bootloaders/isolinux/live.cfg.in`
|
||||
already had no `pci=realloc` entry at all (pre-existing asymmetry with
|
||||
grub-efi, not introduced by this change) -- not addressed here since no
|
||||
incident has been observed via legacy boot.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Servers with dense PCIe topologies (many NVMe bays, many GPUs) no longer
|
||||
silently lose drives to a workaround they didn't need, by default.
|
||||
- Servers that genuinely need the GPU BAR workaround (BIOS lacking Above 4G
|
||||
Decoding) now require the operator to notice GPU BAR failures and
|
||||
deliberately pick the workaround entry, instead of getting it for free --
|
||||
document this tradeoff in the support runbook / vendor-facing material
|
||||
if GPU BAR failures turn out to be common enough to warrant it.
|
||||
- DO NOT move `pci=realloc` back into the default/toram entries without
|
||||
first checking whether the affected server's topology can tolerate a
|
||||
full PCI reallocation pass -- this was the exact failure mode of the
|
||||
incident that prompted this decision.
|
||||
@@ -8,3 +8,6 @@ One file per decision, named `YYYY-MM-DD-short-topic.md`.
|
||||
| 2026-04-01 | Treat memtest as explicit ISO content | active |
|
||||
| 2026-04-29 | Treat embedded submodules as read-only | active |
|
||||
| 2026-06-12 | Skip PCIe link-speed warnings for disabled devices | active |
|
||||
| 2026-07-27 | Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass | active |
|
||||
| 2026-07-27 | Stream SAT job output live to disk and kick blackbox sync on job completion | active |
|
||||
| 2026-07-28 | Move pci=realloc out of the default/toram/no-GUI GRUB entries | active |
|
||||
|
||||
Reference in New Issue
Block a user