refactor: modularize audit and harden build validation

bible-local/architecture/api-surface.md

@@ -53,6 +53,7 @@ All SAT run endpoints enqueue an async task. Response: `{"task_id": "..."}`.
 | POST   | `/api/sat/memory-stress/run`               | Memory stress                     |
 | POST   | `/api/sat/sat-stress/run`                  | Combined storage+memory stress    |
 | POST   | `/api/sat/platform-stress/run`             | Fan + thermal stress              |
+| POST   | `/api/sat/run-all`                          | Plan + enqueue the whole validate/check set server-side. Body: `{stress_mode, amd_targets[], nvidia_gpu_indices[]}` (operator intent only). Response: `{task_ids[], task_count, notes[]}`. Hardware presence/readiness and which tasks to run are decided by `handler.planSATRunAll`, not the page. |
 | GET    | `/api/sat/stream`                          | SSE: live SAT log stream          |
 | POST   | `/api/sat/abort`                           | Abort the running SAT task        |
 

bible-local/architecture/runtime-flows.md

@@ -32,6 +32,12 @@ local-fs.target
 - `bee-network.service` uses `dhclient -nw` (background) — network bring-up is best effort and non-blocking.
 - `bee-nvidia.service` loads modules via `insmod` with absolute paths — NOT `modprobe`.
   Reason: the modules are shipped in the ISO overlay under `/usr/local/lib/nvidia/`, not in the host module tree.
+- `bee-nvidia-load` refreshes `nvidia-fabricmanager.service` / `nvidia-dcgm.service`
+  with `systemctl --no-block try-restart` only. DO NOT make it a blocking
+  `systemctl {start,restart}`: `bee-nvidia.service` is `Type=oneshot` and
+  `Before=` both units, so a synchronous call deadlocks against that ordering
+  and previously reached a 60-second wrapper timeout for each unit. See
+  `decisions/2026-08-31-bee-nvidia-restart-deadlock.md`.
 - `bee-audit.service` does not wait for `network-online.target`; audit is local and must run even if DHCP is broken.
 - `bee-audit.service` logs audit failures but does not turn partial collector problems into a boot blocker.
 - `bee-web.service` binds `0.0.0.0:80` and always renders the current `/var/log/bee-audit.json` contents.
@@ -67,7 +73,7 @@ Rules:
 
 ```
 build-in-container.sh [--authorized-keys /path/to/keys]
-  1. compile `bee` binary (skip if .go files older than binary)
+  1. compile `bee` binary (always; version/git state is part of the artifact)
   2. create a temporary overlay staging dir under `dist/`
   3. inject authorized_keys into staged `root/.ssh/` (or set password fallback marker)
   4. copy `bee` binary → staged `/usr/local/bin/bee`
@@ -93,7 +99,12 @@ build-in-container.sh [--authorized-keys /path/to/keys]
   13. patch staged `motd` with build metadata
   14. copy `iso/builder/` into a temporary live-build workdir under `dist/`
   15. sync staged overlay into workdir `config/includes.chroot/`
-  16. run `lb config && lb build` inside the privileged builder container
+  16. choose the build path from persisted content/ABI/overlay state:
+       a. full: run `lb clean --all && lb config && lb build`
+       b. fast: unpack the last squashfs, sync the staged overlay, repack it,
+          then rebuild checksums, bootloader assets, ISO, and zsync
+  17. validate the final ISO boot menus, volume label, memtest, GRUB assets,
+      and variant runtime before publishing it
 ```
 
 Build host notes:
@@ -109,7 +120,26 @@ Build host notes:
 - `bee-gpu-burn` worker must be built against cached CUDA userspace headers from `build-cublas.sh`, not against random host-installed CUDA headers.
 - The live ISO must ship `libcublas`, `libcublasLt`, and `libcudart` together with `libcuda` so tensor-core stress works without internet or package installs at boot.
 - The source overlay in `iso/overlay/` is treated as immutable source. Build-time files are injected only into the staged overlay.
+- Fast-path state lives outside the rsync-managed live-build workdir and is
+  accepted only when the heavy-input content hash and resolved kernel ABI
+  match the last successful full build. A failed full build never leaves a
+  valid completion marker. The workdir's `binary/` tree is preserved because
+  it is the source artifact for squashfs reuse.
+- Bootloader menu text has two canonical sources only:
+  `config/bootloaders/grub-efi/grub.cfg` and
+  `config/bootloaders/isolinux/live.cfg.in`. `lib/bootloader.sh` renders those
+  templates after both full and fast builds; hooks do not append duplicate
+  menu entries.
+- Build orchestration stays in `build.sh`; ISO validation, bootloader
+  rendering, fast-path/memtest recovery, and logging helpers live under
+  `iso/builder/lib/`. Run `iso/builder/test-build-libs.sh` after changing
+  those helpers or the canonical boot parameters.
 - ISO filename, squashfs filename, ISO volume label, and the live system's hostname all derive from the same `easy-bee-<variant>-v<version>` scheme (`ISO_BASENAME`/`SQUASHFS_FILENAME`/`BEE_ISO_VOLUME`/`BEE_HOSTNAME` in `build.sh`) instead of the live-build default (`debian`). Keep new naming derived from `PROJECT_VERSION_EFFECTIVE`/`BUILD_VARIANT` in sync with this set rather than hardcoding a new scheme.
+- Every live boot entry carries `udev.children_max=1`,
+  `intel_iommu=on`, `iommu.passthrough=0`, and
+  `efi=disable_early_pci_dma`. Only the single failsafe entry additionally
+  carries `pci=realloc iommu.strict=1`; `iommu=pt` is forbidden. The final-ISO
+  validator enforces this for both GRUB and isolinux.
 - The live-build workdir under `dist/` is disposable; source files under `iso/builder/` stay clean.
 - Container build requires `--privileged` because `live-build` uses mounts/chroots/loop devices during ISO assembly.
 - On macOS / Docker Desktop, the builder still must run as `linux/amd64` so the shipped ISO binaries remain `amd64`.
@@ -170,11 +200,11 @@ Acceptance flows:
 - Runtime overrides:
   - `BEE_MEMTESTER_SIZE_MB`
   - `BEE_MEMTESTER_PASSES`
-- NVIDIA Bandwidth SAT (`RunNvidiaBandwidthPack`, `dcgmi diag -r nvbandwidth`) on a
-  multi-socket system runs per CPU socket first, then all selected GPUs together
+- NVIDIA Bandwidth SAT (`RunNvidiaBandwidthPack`, `dcgmi diag -r nvbandwidth`) in
+  Stress mode runs per resolved PCI NUMA node first, then all selected GPUs together
   (`03-dcgmi-nvbandwidth-socket0.log`, `...-socket1.log`, `...-all.log`) --
-  see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`. Single-socket
-  systems (or systems where a GPU's NUMA node can't be resolved) keep the
+  see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`. Single-node
+  systems (or systems where any GPU's NUMA node cannot be resolved) keep the
   original single `NN-dcgmi-nvbandwidth.log` shape.
 
 ## SAT job output durability
@@ -228,18 +258,51 @@ bee-blackbox.service (separate process from bee-web/bee-audit)
 - DO NOT assume a local write under the live ISO's export directory is
   durable on its own (RAM-backed overlay) -- blackbox's mirror to removable
   media is the only real persistence boundary across a hard reset.
+- `BuildSupportBundle` stages into a private `os.MkdirTemp` parent, not a
+  shared `os.TempDir()/bee-support-stage-<host>-<ts>` path. DO NOT go back to
+  a time-derived staging path: two builds in the same wall-clock second (two
+  operators, or an on-demand build racing the blackbox worker) then share one
+  tree and one's deferred `os.RemoveAll` truncates the other's archive.
 
 ## NVIDIA SAT Web UI flow
 
 ```
-Web UI: Acceptance Tests page → Run Test button
-  1. POST /api/sat/nvidia/run → returns job_id
-  2. GET  /api/sat/stream?job_id=... (SSE) — streams stdout/stderr lines live
-  3. After completion — archive written to /appdata/bee/export/bee-sat/
-     summary.txt contains overall_status (OK / FAILED) and per-job status values
+Web UI: Acceptance Tests page -> Run Test button
+  1. POST /api/sat/nvidia/run -> returns job_id
+  2. GET  /api/sat/stream?job_id=... (SSE): streams stdout/stderr lines live
+  3. After completion: archive written to /appdata/bee/export/bee-sat/
+     summary.txt contains overall_status (OK / FAILED / UNSUPPORTED) and per-job status
+```
+
+## Run All (validate / check) flow
+
+```
+Web UI: "Run All" button -> POST /api/sat/run-all
+  body: operator intent only { stress_mode, amd_targets[], nvidia_gpu_indices[] }
+  server (handler.planSATRunAll):
+    1. always: cpu, memory, storage, pcie-link
+    2. tpm            - only if App.TPMPresent() finds tpm_version_major=2
+    3. nvidia-config  - if DetectGPUPresence().Nvidia || NvidiaInitializing
+    4. wait for NVIDIA enumeration: repeat fresh ListNvidiaGPUs queries until
+       at least one GPU is returned, NvidiaGSPMode=="gsp-stuck", or 75s
+    5. nvidia / nvidia-interconnect / nvidia-bandwidth / nvidia-pcie-bandwidth
+       (+ targeted-stress/power/pulse when stress_mode) - only once ready,
+       -i = App.ListNvidiaGPUs() indices (intersected with the requested subset)
+    6. amd / amd-mem / amd-bandwidth - if DetectGPUPresence().AMD and selected
+  response: { task_ids[], task_count, notes[] }  (notes = what was skipped and why)
 ```
 
 **Critical invariants:**
-- `bee-gpu-burn` / `bee-john-gpu-stress` use `exec.CommandContext` — killed on job context cancel.
+- Hardware presence, readiness, and which tasks to run are decided server-side.
+  DO NOT move this back into page JS (`satSelectedGPUIndices().length` gating):
+  a browser-cached empty GPU list then silently drops every GPU test. See
+  `decisions/2026-08-31-backend-driven-sat-planning.md`.
+- `DetectGPUPresence` is the shared detection source (existing operational
+  vendor detection plus an lspci display-class fallback).
+  `/api/gpu/presence`, `/api/gpu/tools` and the planner all use it.
+- `bee-gpu-burn` / `bee-john-gpu-stress` use `exec.CommandContext`: killed on job context cancel.
 - Metric goroutine uses stopCh/doneCh pattern; main goroutine waits `<-doneCh` before reading rows (no mutex needed).
 - SVG chart is fully offline: no JS, no external CSS, pure inline SVG.
+- `RunNvidiaBandwidthPack` runs one all-GPU `nvbandwidth` pass in Validate; the
+  per-NUMA-node matrix is Stress-tier only (`fullMatrix` arg). See
+  `decisions/2026-08-31-nvbandwidth-validate-single-deep-matrix.md`.

bible-local/decisions/2026-04-01-memtest-build-strategy.md

@@ -175,8 +175,8 @@ those files may not exist yet. Instead:
 
 - Tries to copy `memtest86+x64.bin` / `memtest86+x64.efi` from `chroot/boot/` first.
 - Falls back to extracting from the cached `.deb` (via `dpkg-deb -x`) if `chroot/boot/` is empty.
-- Appends GRUB and isolinux menu entries only if the respective cfg files already exist at hook time.
-  If they do not exist, the hook warns and continues (does not fail).
+- It does not edit bootloader menus. The complete menus are rendered later
+  from the two canonical project templates.
 
 Controlled by `BEE_REQUIRE_MEMTEST=1` env var to turn warnings into hard errors when needed.
 
@@ -186,7 +186,8 @@ After `lb build` completes, `build.sh` checks whether the fully materialized `bi
 contains all required memtest artifacts. If not:
 
 - Copies/extracts memtest binaries into `binary/boot/`.
-- Patches `binary/boot/grub/grub.cfg` and `binary/isolinux/live.cfg` directly.
+- Calls `enforce_live_build_bootloader_assets`, which renders the complete
+  GRUB and isolinux configs from the canonical project templates.
 - Reruns the late binary stages (`binary_checksums`, `binary_iso`, `binary_zsync`) to rebuild
   the ISO with the patched tree.
 

bible-local/decisions/2026-07-28-pci-realloc-out-of-default-boot-entries.md

@@ -59,10 +59,11 @@ individual failure mode. This keeps the total entry count in
 entries plus wipe/memtest/firmware-settings), rather than letting it
 grow linearly with every workaround discovered.
 
-The isolinux (BIOS/legacy boot) menu in `iso/builder/config/bootloaders/isolinux/live.cfg.in`
-already had no `pci=realloc` entry at all (pre-existing asymmetry with
-grub-efi, not introduced by this change) -- not addressed here since no
-incident has been observed via legacy boot.
+As of v13.0, GRUB and isolinux both expose exactly one troubleshooting entry
+with `pci=realloc`. It remains absent from every normal, toram, no-GUI, and
+wipe entry. The same v13.0 amendment replaces `iommu=pt` in that entry with
+translated strict IOMMU mode; this changes DMA diagnostics, not the scope of
+the PCI reallocation workaround.
 
 ## Consequences
 

bible-local/decisions/2026-08-31-backend-driven-sat-planning.md

@@ -0,0 +1,67 @@
+# "Run All" SAT planning happens on the backend, not in the browser
+
+**Date:** 2026-08-31
+**Status:** active
+
+## Symptom
+
+Support bundle `210619KUGGXGS2000017` (8x H200 NVL, identical to `...008`):
+the operator hit "Run All" and only 6 tasks were queued - cpu, memory,
+storage, tpm, nvidia-config, pcie-link. No `nvidia`, `nvidia-interconnect`,
+`nvidia-bandwidth`, `nvidia-pcie-bandwidth`. The GPUs were physically present
+and `nvidia-config` (which enumerates them) passed.
+
+## Root cause
+
+Task planning lived in page JavaScript:
+
+- `runAllCheckSAT()` / `runAllSAT()` added the NVIDIA targets only
+  `if (satSelectedGPUIndices().length)`.
+- That list came from `/api/gpu/nvidia` (`nvidia-smi --query-gpu=...`), fetched
+  **once per page load and cached** in `satNvidiaGPUsPromise`.
+- If the page first queried while the driver was still enumerating GPUs (the
+  two 60-second service timeouts described in
+  [2026-08-31-bee-nvidia-restart-deadlock.md](2026-08-31-bee-nvidia-restart-deadlock.md),
+  plus per-GPU GSP firmware boot), it got an empty list and cached it for the
+  whole session. "Run All" then silently dropped every GPU test - no banner,
+  no error. `/api/gpu/presence` (a separate `os.Stat("/dev/nvidia0")` check)
+  meanwhile said "GPU present", so the UI even contradicted itself.
+
+## Decision
+
+- **New endpoint `POST /api/sat/run-all`.** The browser sends only operator
+  *intent*: `stress_mode`, `amd_targets` (checkbox selection), and
+  an optional `nvidia_gpu_indices` subset. The server decides what hardware
+  is present/ready and what to enqueue (`handler.planSATRunAll`).
+  `runAllSAT()` / `runAllCheckSAT()` are now thin `fetch` wrappers and render
+  the `notes[]` the server returns ("TPM: no device - skipped", etc.).
+- **GPU planning waits on a readiness gate, not a device probe.**
+  `planSATRunAll` calls `waitForNvidiaReady`, which repeats the fresh
+  `ListNvidiaGPUs` query until `nvidia-smi` enumerates at least one GPU, the
+  runtime snapshot reports `NvidiaGSPMode == "gsp-stuck"`, or the 75-second
+  deadline expires. A loaded kernel module is not treated as proof that
+  user-space tools can address a GPU. `CUDAReady == false` after enumeration
+  is a note, not a blocker.
+- **One presence source with a PCI fallback.** `app.DetectGPUPresence`:
+  - primary - the existing operational vendor result (`DetectGPUVendor`);
+  - fallback - `System.PhysicalGPUVendors()` (lspci VGA/3D/Display class +
+    vendor id): a GPU on the bus not reported by operational detection sets
+    `NvidiaInitializing` / `AMDInitializing`, an explicit state distinct from
+    "absent". For NVIDIA, the page can show that enumeration is still pending
+    instead of reporting no hardware.
+  `/api/gpu/tools`, `/api/gpu/presence` and the run-all planner all go through
+  it.
+- **TPM gate.** `tpm` is planned only when `app.TPMPresent()` finds a sysfs TPM
+  whose stable `tpm_version_major` attribute is exactly `2`; otherwise a note explains the skip. This
+  matches the pack-level guard already in `RunTPMValidationPack`.
+
+## Consequences
+
+- "Run All" can no longer skip hardware because a browser-cached probe was
+  early/empty. A genuinely absent or dead GPU still gets skipped - but with a
+  `notes[]` entry, and `nvidia-config` still runs to capture diagnostics.
+- The per-card `disableSATCard('...','No NVIDIA GPU detected')` hints in the
+  page still use `/api/gpu/presence` - cosmetic only now, and self-heal on
+  reload.
+- Automated / headless callers get correct planning for free by POSTing the
+  same endpoint instead of replicating the JS logic.

bible-local/decisions/2026-08-31-bee-nvidia-restart-deadlock.md

@@ -0,0 +1,66 @@
+# bee-nvidia.service: never call blocking `systemctl restart` on units ordered After= itself
+
+**Date:** 2026-08-31
+**Status:** active
+
+## Symptom
+
+Every affected NVIDIA boot reaches two 60-second wrapper timeouts in
+`bee-nvidia.service`. `bee-nvidia.log`:
+
+```
+[bee-nvidia] restarting nvidia-fabricmanager.service (timeout 60s)
+[bee-nvidia] WARN: systemctl restart nvidia-fabricmanager.service timed out after 60s
+[bee-nvidia] restarting nvidia-dcgm.service (timeout 60s)
+[bee-nvidia] WARN: systemctl restart nvidia-dcgm.service timed out after 60s
+[bee-nvidia] done
+```
+
+Seen on both `210619KUGGXGS2000008` and `...017` (8x H200 NVL). That 120 s
+window overlaps exactly with when an operator opens the web UI and clicks
+"Run All" - during it `nvidia-smi` may not yet enumerate all GPUs, which is
+how bundle `...017` ended up running the check set with **no GPU tests**
+(see [2026-08-31-backend-driven-sat-planning.md](2026-08-31-backend-driven-sat-planning.md)).
+
+## Root cause
+
+`bee-nvidia.service` is `Type=oneshot` and `Before=nvidia-fabricmanager.service
+nvidia-dcgm.service`. Its `ExecStart` (`bee-nvidia-load`) then ran, synchronously:
+
+```
+timeout 60 systemctl restart nvidia-fabricmanager.service
+timeout 60 systemctl restart nvidia-dcgm.service
+```
+
+A oneshot unit is not "active" until `ExecStart` returns. Both target units
+are ordered `After=bee-nvidia.service`, so systemd queues them behind
+bee-nvidia and will not run them while `bee-nvidia-load` is still executing.
+`bee-nvidia-load` blocks on `systemctl restart` waiting for exactly that job
+to complete -> deadlock -> broken only when `timeout 60` fires. Twice.
+
+`nvidia-smi -q` inside `bee-check-nvswitch` (the fabricmanager ExecCondition)
+is not part of this ordering cycle; the deadlock is structural.
+
+## Decision
+
+`bee-nvidia-load` no longer blocks on those units. It calls
+`systemctl --no-block try-restart` for each:
+
+- `--no-block` returns immediately; systemd runs the job after bee-nvidia
+  exits, via the existing `Before=` ordering.
+- `try-restart` only acts if the unit is already running (the "stale instance
+  from a reload / re-run" case the old code worried about). If it is inactive,
+  this command does nothing; an enabled unit already queued by the normal boot
+  transaction starts after `bee-nvidia.service` via the declared ordering.
+
+The `SYSTEMCTL_TIMEOUT` / `timeout_systemctl` wrapper and the fallback
+`systemctl start` / `systemctl status` branches are gone. `--no-block` means
+systemctl does not wait for the queued unit job to finish.
+
+## Consequences
+
+- NVIDIA boot no longer waits for those two 60-second wrapper deadlines.
+- DO NOT reintroduce a synchronous `systemctl {start,restart}` of any unit
+  that is `After=bee-nvidia.service` from inside `bee-nvidia-load`. If a unit
+  genuinely must be up before the script returns, invert the ordering
+  instead.

bible-local/decisions/2026-08-31-nvbandwidth-validate-single-deep-matrix.md

@@ -0,0 +1,47 @@
+# nvbandwidth: single all-GPU pass in Validate, per-NUMA-node matrix only in the deep tier
+
+**Date:** 2026-08-31
+**Status:** active
+**Amends:** [2026-07-27-nvbandwidth-per-socket-split.md](2026-07-27-nvbandwidth-per-socket-split.md)
+
+## Context
+
+`2026-07-27` made `RunNvidiaBandwidthPack` split `dcgmi diag -r nvbandwidth`
+using Linux PCI `numa_node` locality, followed by an all-GPU pass. A NUMA node
+is not assumed to be identical to a physical CPU socket.
+
+Two problems showed up on an 8x H200 NVL / dual-socket EPYC box
+(`210619KUGGXGS2000008`):
+
+1. `SATEstimatedNvidiaBandwidthSec` assigns 2700 seconds to one all-GPU pass.
+   Three such invocations are assigned 8100 seconds, which is outside the
+   intended **Validate** duration.
+2. The split silently never engaged anyway: `normalizeNvidiaBDF` returned
+   nvidia-smi's upper-case PCI BDF (`0000:CB:00.0`) while `/sys/bus/pci/devices`
+   entries are lower-case, so `readPCINumaNode` failed for every GPU on a bus
+   with a hex letter and `gpuBandwidthSocketGroups` fell back to one group.
+
+## Decision
+
+- `normalizeNvidiaBDF` now lower-cases (and trims) the BDF, so sysfs
+  `numa_node` / link-speed reads actually resolve.
+- `RunNvidiaBandwidthPack` takes a `fullMatrix bool`. The **only** thing it
+  changes is which GPU set each `nvbandwidth` invocation gets via `-i` - the
+  command itself is untouched (no extra flags, no testcase filtering).
+  - **Validate** (`stress_mode=false`) -> `fullMatrix=false`: one pass,
+    `-i <all selected GPUs>`. No NUMA-locality split.
+  - **Stress / deep** (`stress_mode=true`) -> `fullMatrix=true`: the
+    `2026-07-27` behaviour - one pass per resolved NUMA-node group, then one
+    all-GPU pass. If any selected GPU has no resolved NUMA node, the code does
+    not guess a group and falls back to the single all-GPU pass.
+- `task_runner` passes `t.params.StressMode` through.
+
+## Consequences
+
+- Validate schedules exactly one nvbandwidth invocation regardless of NUMA-node count.
+- The NUMA-locality isolation from `2026-07-27` is preserved, just moved to
+  the tier where a 2-3x runtime is acceptable. DO NOT re-add the split to the
+  Validate path.
+- `SATEstimatedNvidiaBandwidthSec` still reflects a single pass; the deep
+  tier's multi-pass duration is not modelled - revisit once real
+  multi-node deep-run logs exist (same open item as `2026-07-27`).

bible-local/decisions/README.md

@@ -11,3 +11,7 @@ One file per decision, named `YYYY-MM-DD-short-topic.md`.
 | 2026-07-27 | Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass | active |
 | 2026-07-27 | Stream SAT job output live to disk and kick blackbox sync on job completion | active |
 | 2026-07-28 | Move pci=realloc out of the default/toram/no-GUI GRUB entries | active |
+| 2026-08-24 | PCIe Gen1-at-idle GPU warning: load-bearing link check, not idle sysfs | active |
+| 2026-08-31 | nvbandwidth: single all-GPU pass in Validate, per-socket matrix only in deep tier | active |
+| 2026-08-31 | bee-nvidia.service: never blocking `systemctl restart` on units ordered After= itself | active |
+| 2026-08-31 | "Run All" SAT planning happens on the backend, not the browser | active |

bible-local/docs/customer-gpu-test-methodology.md

@@ -1,54 +1,67 @@
 # GPU PCIe Test Methodology
 
+Which of the checks below run is decided by the backend (`POST
+/api/sat/run-all` -> `handler.planSATRunAll`), not the web page: it enumerates
+the hardware, waits for the NVIDIA driver to finish enumerating GPUs, and
+enqueues only the applicable checks. The TPM check is planned only when sysfs
+reports a TPM 2.x device (`tpm_version_major=2`). See
+`bible-local/decisions/2026-08-31-backend-driven-sat-planning.md`.
+
 ## Validate
 
 - CPU check
-  - `lscpu`
-  - `sensors`
-  - `stress-ng`
+ - `lscpu`
+ - `sensors`
+ - `stress-ng`
 - Memory check
-  - `free`
-  - `timeout <timeout_sec> memtester`
-  - `free`
+ - `free`
+ - `timeout <timeout_sec> memtester`
+ - `free`
 - NVMe storage check
-  - `nvme id-ctrl`
-  - `nvme smart-log`
-  - `nvme device-self-test`
+ - `nvme id-ctrl`
+ - `nvme smart-log`
+ - `nvme device-self-test`
 - SATA/SAS storage check
-  - `smartctl -H -A`
-  - `smartctl -t short`
+ - `smartctl -H -A`
+ - `smartctl -t short`
 - Basic NVIDIA GPU check
-  - `nvidia-smi -pm 1`
-  - `nvidia-smi -q`
-  - `dmidecode -t baseboard`
-  - `dmidecode -t system`
-  - `dcgmi diag -r 2`
+ - `nvidia-smi -pm 1`
+ - `nvidia-smi -q`
+ - `dmidecode -t baseboard`
+ - `dmidecode -t system`
+ - `dcgmi diag -r 2`
 - Inter-GPU communication check
-  - `all_reduce_perf`
+ - `all_reduce_perf`
 - GPU bandwidth check
-  - `dcgmi diag -r nvbandwidth`
+ - `dcgmi diag -r nvbandwidth` - **one pass across all GPUs** (`-i <all>`).
+    No NUMA-locality split in Validate.
 
 ## Validate -> Stress
 
 - Extended NVIDIA GPU check
-  - `nvidia-smi -pm 1`
-  - `nvidia-smi -q`
-  - `dmidecode -t baseboard`
-  - `dmidecode -t system`
-  - `dcgmi diag -r 3`
+ - `nvidia-smi -pm 1`
+ - `nvidia-smi -q`
+ - `dmidecode -t baseboard`
+ - `dmidecode -t system`
+ - `dcgmi diag -r 3`
 - NVIDIA targeted stress
-  - `nvidia-smi -pm 1`
-  - `nvidia-smi -q`
-  - `dcgmi diag -r targeted_stress`
+ - `nvidia-smi -pm 1`
+ - `nvidia-smi -q`
+ - `dcgmi diag -r targeted_stress`
 - NVIDIA targeted power
-  - `nvidia-smi -pm 1`
-  - `nvidia-smi -q`
-  - `dcgmi diag -r targeted_power`
+ - `nvidia-smi -pm 1`
+ - `nvidia-smi -q`
+ - `dcgmi diag -r targeted_power`
 - NVIDIA pulse test
-  - `nvidia-smi -pm 1`
-  - `nvidia-smi -q`
-  - `dcgmi diag -r pulse_test`
+ - `nvidia-smi -pm 1`
+ - `nvidia-smi -q`
+ - `dcgmi diag -r pulse_test`
 - Inter-GPU communication check
-  - `all_reduce_perf`
+ - `all_reduce_perf`
 - GPU bandwidth check
-  - `dcgmi diag -r nvbandwidth`
+ - `dcgmi diag -r nvbandwidth` - **per-NUMA-node matrix**: one pass per
+    completely resolved Linux PCI NUMA group, then one all-GPU pass. If any
+    selected GPU has no resolved `numa_node`, no group is guessed and the test
+    falls back to one all-GPU pass. See
+    `bible-local/decisions/2026-07-27-nvbandwidth-per-socket-split.md` and
+    `bible-local/decisions/2026-08-31-nvbandwidth-validate-single-deep-matrix.md`.

bible-local/docs/iso-build-rules.md

@@ -17,15 +17,18 @@ This applies to:
 
 ## Bootloader sync rule
 
-The ISO has two independent bootloader configs that must be kept in sync manually:
+The ISO has two canonical bootloader templates whose live entries must remain
+semantically equivalent:
 
 | File | Used by |
 |------|---------|
 | `config/bootloaders/grub-efi/grub.cfg` | UEFI (all modern servers) |
 | `config/bootloaders/isolinux/live.cfg.in` | CSM / legacy BIOS (syslinux) |
 
-live-build does NOT derive one from the other. Any new boot entry, kernel parameter
-change, or new mode added to one file must be manually mirrored in the other.
+live-build does not derive one from the other. `lib/bootloader.sh` renders both
+templates into the final `binary/` tree after live-build has created it, and the
+ISO validator checks every live entry. Any menu or kernel-parameter change must
+still be made in both templates.
 
 **Canonical entry list** (both files must have all of these):
 
@@ -33,18 +36,20 @@ change, or new mode added to one file must be manually mirrored in the other.
 |-------|-----------|
 | normal (default) | `nomodeset bee.nvidia.mode=normal` + full param set |
 | load to RAM | `toram nomodeset bee.nvidia.mode=normal` + full param set |
-| GSP=off | `nomodeset bee.nvidia.mode=gsp-off` + full param set |
-| KMS | no `nomodeset`, `bee.nvidia.mode=normal` + full param set |
-| KMS + GSP=off | no `nomodeset`, `bee.nvidia.mode=gsp-off` + full param set |
-| fail-safe | `nomodeset bee.nvidia.mode=gsp-off noapic noapm nodma nomce nolapic nosmp` |
+| no GUI | `nomodeset bee.gui=off bee.nvidia.mode=normal` + full param set |
+| fail-safe | normal parameters plus `pci=realloc iommu.strict=1` |
+| wipe | `toram nomodeset bee.gui=off bee.wipe=all` + reduced performance params |
 
 **Full standard param set** (append after `@APPEND_LIVE@` / `nomodeset` flags):
 ```
 net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always
 numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1
-nowatchdog nosoftlockup
+nowatchdog nosoftlockup udev.children_max=1 intel_iommu=on
+iommu.passthrough=0 efi=disable_early_pci_dma
 ```
-(fail-safe is the exception — it deliberately uses minimal params.)
+The fail-safe entry alone adds `pci=realloc iommu.strict=1`. `pci=realloc` must
+not be copied into normal entries. Memtest and firmware-setup entries are not
+Linux live entries and therefore do not carry these parameters.
 
 **Historical note:** `grub-pc/` was mistakenly used instead of `grub-efi/` until v8.25.
 live-build reads `config/bootloaders/grub-efi/` for UEFI because the build is