refactor: modularize audit and harden build validation

bible-local/architecture/runtime-flows.md

@@ -32,6 +32,12 @@ local-fs.target
 - `bee-network.service` uses `dhclient -nw` (background) — network bring-up is best effort and non-blocking.
 - `bee-nvidia.service` loads modules via `insmod` with absolute paths — NOT `modprobe`.
   Reason: the modules are shipped in the ISO overlay under `/usr/local/lib/nvidia/`, not in the host module tree.
+- `bee-nvidia-load` refreshes `nvidia-fabricmanager.service` / `nvidia-dcgm.service`
+  with `systemctl --no-block try-restart` only. DO NOT make it a blocking
+  `systemctl {start,restart}`: `bee-nvidia.service` is `Type=oneshot` and
+  `Before=` both units, so a synchronous call deadlocks against that ordering
+  and previously reached a 60-second wrapper timeout for each unit. See
+  `decisions/2026-08-31-bee-nvidia-restart-deadlock.md`.
 - `bee-audit.service` does not wait for `network-online.target`; audit is local and must run even if DHCP is broken.
 - `bee-audit.service` logs audit failures but does not turn partial collector problems into a boot blocker.
 - `bee-web.service` binds `0.0.0.0:80` and always renders the current `/var/log/bee-audit.json` contents.
@@ -67,7 +73,7 @@ Rules:
 
 ```
 build-in-container.sh [--authorized-keys /path/to/keys]
-  1. compile `bee` binary (skip if .go files older than binary)
+  1. compile `bee` binary (always; version/git state is part of the artifact)
   2. create a temporary overlay staging dir under `dist/`
   3. inject authorized_keys into staged `root/.ssh/` (or set password fallback marker)
   4. copy `bee` binary → staged `/usr/local/bin/bee`
@@ -93,7 +99,12 @@ build-in-container.sh [--authorized-keys /path/to/keys]
   13. patch staged `motd` with build metadata
   14. copy `iso/builder/` into a temporary live-build workdir under `dist/`
   15. sync staged overlay into workdir `config/includes.chroot/`
-  16. run `lb config && lb build` inside the privileged builder container
+  16. choose the build path from persisted content/ABI/overlay state:
+       a. full: run `lb clean --all && lb config && lb build`
+       b. fast: unpack the last squashfs, sync the staged overlay, repack it,
+          then rebuild checksums, bootloader assets, ISO, and zsync
+  17. validate the final ISO boot menus, volume label, memtest, GRUB assets,
+      and variant runtime before publishing it
 ```
 
 Build host notes:
@@ -109,7 +120,26 @@ Build host notes:
 - `bee-gpu-burn` worker must be built against cached CUDA userspace headers from `build-cublas.sh`, not against random host-installed CUDA headers.
 - The live ISO must ship `libcublas`, `libcublasLt`, and `libcudart` together with `libcuda` so tensor-core stress works without internet or package installs at boot.
 - The source overlay in `iso/overlay/` is treated as immutable source. Build-time files are injected only into the staged overlay.
+- Fast-path state lives outside the rsync-managed live-build workdir and is
+  accepted only when the heavy-input content hash and resolved kernel ABI
+  match the last successful full build. A failed full build never leaves a
+  valid completion marker. The workdir's `binary/` tree is preserved because
+  it is the source artifact for squashfs reuse.
+- Bootloader menu text has two canonical sources only:
+  `config/bootloaders/grub-efi/grub.cfg` and
+  `config/bootloaders/isolinux/live.cfg.in`. `lib/bootloader.sh` renders those
+  templates after both full and fast builds; hooks do not append duplicate
+  menu entries.
+- Build orchestration stays in `build.sh`; ISO validation, bootloader
+  rendering, fast-path/memtest recovery, and logging helpers live under
+  `iso/builder/lib/`. Run `iso/builder/test-build-libs.sh` after changing
+  those helpers or the canonical boot parameters.
 - ISO filename, squashfs filename, ISO volume label, and the live system's hostname all derive from the same `easy-bee-<variant>-v<version>` scheme (`ISO_BASENAME`/`SQUASHFS_FILENAME`/`BEE_ISO_VOLUME`/`BEE_HOSTNAME` in `build.sh`) instead of the live-build default (`debian`). Keep new naming derived from `PROJECT_VERSION_EFFECTIVE`/`BUILD_VARIANT` in sync with this set rather than hardcoding a new scheme.
+- Every live boot entry carries `udev.children_max=1`,
+  `intel_iommu=on`, `iommu.passthrough=0`, and
+  `efi=disable_early_pci_dma`. Only the single failsafe entry additionally
+  carries `pci=realloc iommu.strict=1`; `iommu=pt` is forbidden. The final-ISO
+  validator enforces this for both GRUB and isolinux.
 - The live-build workdir under `dist/` is disposable; source files under `iso/builder/` stay clean.
 - Container build requires `--privileged` because `live-build` uses mounts/chroots/loop devices during ISO assembly.
 - On macOS / Docker Desktop, the builder still must run as `linux/amd64` so the shipped ISO binaries remain `amd64`.
@@ -170,11 +200,11 @@ Acceptance flows:
 - Runtime overrides:
   - `BEE_MEMTESTER_SIZE_MB`
   - `BEE_MEMTESTER_PASSES`
-- NVIDIA Bandwidth SAT (`RunNvidiaBandwidthPack`, `dcgmi diag -r nvbandwidth`) on a
-  multi-socket system runs per CPU socket first, then all selected GPUs together
+- NVIDIA Bandwidth SAT (`RunNvidiaBandwidthPack`, `dcgmi diag -r nvbandwidth`) in
+  Stress mode runs per resolved PCI NUMA node first, then all selected GPUs together
   (`03-dcgmi-nvbandwidth-socket0.log`, `...-socket1.log`, `...-all.log`) --
-  see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`. Single-socket
-  systems (or systems where a GPU's NUMA node can't be resolved) keep the
+  see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`. Single-node
+  systems (or systems where any GPU's NUMA node cannot be resolved) keep the
   original single `NN-dcgmi-nvbandwidth.log` shape.
 
 ## SAT job output durability
@@ -228,18 +258,51 @@ bee-blackbox.service (separate process from bee-web/bee-audit)
 - DO NOT assume a local write under the live ISO's export directory is
   durable on its own (RAM-backed overlay) -- blackbox's mirror to removable
   media is the only real persistence boundary across a hard reset.
+- `BuildSupportBundle` stages into a private `os.MkdirTemp` parent, not a
+  shared `os.TempDir()/bee-support-stage-<host>-<ts>` path. DO NOT go back to
+  a time-derived staging path: two builds in the same wall-clock second (two
+  operators, or an on-demand build racing the blackbox worker) then share one
+  tree and one's deferred `os.RemoveAll` truncates the other's archive.
 
 ## NVIDIA SAT Web UI flow
 
 ```
-Web UI: Acceptance Tests page → Run Test button
-  1. POST /api/sat/nvidia/run → returns job_id
-  2. GET  /api/sat/stream?job_id=... (SSE) — streams stdout/stderr lines live
-  3. After completion — archive written to /appdata/bee/export/bee-sat/
-     summary.txt contains overall_status (OK / FAILED) and per-job status values
+Web UI: Acceptance Tests page -> Run Test button
+  1. POST /api/sat/nvidia/run -> returns job_id
+  2. GET  /api/sat/stream?job_id=... (SSE): streams stdout/stderr lines live
+  3. After completion: archive written to /appdata/bee/export/bee-sat/
+     summary.txt contains overall_status (OK / FAILED / UNSUPPORTED) and per-job status
+```
+
+## Run All (validate / check) flow
+
+```
+Web UI: "Run All" button -> POST /api/sat/run-all
+  body: operator intent only { stress_mode, amd_targets[], nvidia_gpu_indices[] }
+  server (handler.planSATRunAll):
+    1. always: cpu, memory, storage, pcie-link
+    2. tpm            - only if App.TPMPresent() finds tpm_version_major=2
+    3. nvidia-config  - if DetectGPUPresence().Nvidia || NvidiaInitializing
+    4. wait for NVIDIA enumeration: repeat fresh ListNvidiaGPUs queries until
+       at least one GPU is returned, NvidiaGSPMode=="gsp-stuck", or 75s
+    5. nvidia / nvidia-interconnect / nvidia-bandwidth / nvidia-pcie-bandwidth
+       (+ targeted-stress/power/pulse when stress_mode) - only once ready,
+       -i = App.ListNvidiaGPUs() indices (intersected with the requested subset)
+    6. amd / amd-mem / amd-bandwidth - if DetectGPUPresence().AMD and selected
+  response: { task_ids[], task_count, notes[] }  (notes = what was skipped and why)
 ```
 
 **Critical invariants:**
-- `bee-gpu-burn` / `bee-john-gpu-stress` use `exec.CommandContext` — killed on job context cancel.
+- Hardware presence, readiness, and which tasks to run are decided server-side.
+  DO NOT move this back into page JS (`satSelectedGPUIndices().length` gating):
+  a browser-cached empty GPU list then silently drops every GPU test. See
+  `decisions/2026-08-31-backend-driven-sat-planning.md`.
+- `DetectGPUPresence` is the shared detection source (existing operational
+  vendor detection plus an lspci display-class fallback).
+  `/api/gpu/presence`, `/api/gpu/tools` and the planner all use it.
+- `bee-gpu-burn` / `bee-john-gpu-stress` use `exec.CommandContext`: killed on job context cancel.
 - Metric goroutine uses stopCh/doneCh pattern; main goroutine waits `<-doneCh` before reading rows (no mutex needed).
 - SVG chart is fully offline: no JS, no external CSS, pure inline SVG.
+- `RunNvidiaBandwidthPack` runs one all-GPU `nvbandwidth` pass in Validate; the
+  per-NUMA-node matrix is Stress-tier only (`fullMatrix` arg). See
+  `decisions/2026-08-31-nvbandwidth-validate-single-deep-matrix.md`.