refactor: modularize audit and harden build validation

bible-local/docs/customer-gpu-test-methodology.md

@@ -1,54 +1,67 @@
 # GPU PCIe Test Methodology
 
+Which of the checks below run is decided by the backend (`POST
+/api/sat/run-all` -> `handler.planSATRunAll`), not the web page: it enumerates
+the hardware, waits for the NVIDIA driver to finish enumerating GPUs, and
+enqueues only the applicable checks. The TPM check is planned only when sysfs
+reports a TPM 2.x device (`tpm_version_major=2`). See
+`bible-local/decisions/2026-08-31-backend-driven-sat-planning.md`.
+
 ## Validate
 
 - CPU check
-  - `lscpu`
-  - `sensors`
-  - `stress-ng`
+ - `lscpu`
+ - `sensors`
+ - `stress-ng`
 - Memory check
-  - `free`
-  - `timeout <timeout_sec> memtester`
-  - `free`
+ - `free`
+ - `timeout <timeout_sec> memtester`
+ - `free`
 - NVMe storage check
-  - `nvme id-ctrl`
-  - `nvme smart-log`
-  - `nvme device-self-test`
+ - `nvme id-ctrl`
+ - `nvme smart-log`
+ - `nvme device-self-test`
 - SATA/SAS storage check
-  - `smartctl -H -A`
-  - `smartctl -t short`
+ - `smartctl -H -A`
+ - `smartctl -t short`
 - Basic NVIDIA GPU check
-  - `nvidia-smi -pm 1`
-  - `nvidia-smi -q`
-  - `dmidecode -t baseboard`
-  - `dmidecode -t system`
-  - `dcgmi diag -r 2`
+ - `nvidia-smi -pm 1`
+ - `nvidia-smi -q`
+ - `dmidecode -t baseboard`
+ - `dmidecode -t system`
+ - `dcgmi diag -r 2`
 - Inter-GPU communication check
-  - `all_reduce_perf`
+ - `all_reduce_perf`
 - GPU bandwidth check
-  - `dcgmi diag -r nvbandwidth`
+ - `dcgmi diag -r nvbandwidth` - **one pass across all GPUs** (`-i <all>`).
+    No NUMA-locality split in Validate.
 
 ## Validate -> Stress
 
 - Extended NVIDIA GPU check
-  - `nvidia-smi -pm 1`
-  - `nvidia-smi -q`
-  - `dmidecode -t baseboard`
-  - `dmidecode -t system`
-  - `dcgmi diag -r 3`
+ - `nvidia-smi -pm 1`
+ - `nvidia-smi -q`
+ - `dmidecode -t baseboard`
+ - `dmidecode -t system`
+ - `dcgmi diag -r 3`
 - NVIDIA targeted stress
-  - `nvidia-smi -pm 1`
-  - `nvidia-smi -q`
-  - `dcgmi diag -r targeted_stress`
+ - `nvidia-smi -pm 1`
+ - `nvidia-smi -q`
+ - `dcgmi diag -r targeted_stress`
 - NVIDIA targeted power
-  - `nvidia-smi -pm 1`
-  - `nvidia-smi -q`
-  - `dcgmi diag -r targeted_power`
+ - `nvidia-smi -pm 1`
+ - `nvidia-smi -q`
+ - `dcgmi diag -r targeted_power`
 - NVIDIA pulse test
-  - `nvidia-smi -pm 1`
-  - `nvidia-smi -q`
-  - `dcgmi diag -r pulse_test`
+ - `nvidia-smi -pm 1`
+ - `nvidia-smi -q`
+ - `dcgmi diag -r pulse_test`
 - Inter-GPU communication check
-  - `all_reduce_perf`
+ - `all_reduce_perf`
 - GPU bandwidth check
-  - `dcgmi diag -r nvbandwidth`
+ - `dcgmi diag -r nvbandwidth` - **per-NUMA-node matrix**: one pass per
+    completely resolved Linux PCI NUMA group, then one all-GPU pass. If any
+    selected GPU has no resolved `numa_node`, no group is guessed and the test
+    falls back to one all-GPU pass. See
+    `bible-local/decisions/2026-07-27-nvbandwidth-per-socket-split.md` and
+    `bible-local/decisions/2026-08-31-nvbandwidth-validate-single-deep-matrix.md`.

bible-local/docs/iso-build-rules.md

@@ -17,15 +17,18 @@ This applies to:
 
 ## Bootloader sync rule
 
-The ISO has two independent bootloader configs that must be kept in sync manually:
+The ISO has two canonical bootloader templates whose live entries must remain
+semantically equivalent:
 
 | File | Used by |
 |------|---------|
 | `config/bootloaders/grub-efi/grub.cfg` | UEFI (all modern servers) |
 | `config/bootloaders/isolinux/live.cfg.in` | CSM / legacy BIOS (syslinux) |
 
-live-build does NOT derive one from the other. Any new boot entry, kernel parameter
-change, or new mode added to one file must be manually mirrored in the other.
+live-build does not derive one from the other. `lib/bootloader.sh` renders both
+templates into the final `binary/` tree after live-build has created it, and the
+ISO validator checks every live entry. Any menu or kernel-parameter change must
+still be made in both templates.
 
 **Canonical entry list** (both files must have all of these):
 
@@ -33,18 +36,20 @@ change, or new mode added to one file must be manually mirrored in the other.
 |-------|-----------|
 | normal (default) | `nomodeset bee.nvidia.mode=normal` + full param set |
 | load to RAM | `toram nomodeset bee.nvidia.mode=normal` + full param set |
-| GSP=off | `nomodeset bee.nvidia.mode=gsp-off` + full param set |
-| KMS | no `nomodeset`, `bee.nvidia.mode=normal` + full param set |
-| KMS + GSP=off | no `nomodeset`, `bee.nvidia.mode=gsp-off` + full param set |
-| fail-safe | `nomodeset bee.nvidia.mode=gsp-off noapic noapm nodma nomce nolapic nosmp` |
+| no GUI | `nomodeset bee.gui=off bee.nvidia.mode=normal` + full param set |
+| fail-safe | normal parameters plus `pci=realloc iommu.strict=1` |
+| wipe | `toram nomodeset bee.gui=off bee.wipe=all` + reduced performance params |
 
 **Full standard param set** (append after `@APPEND_LIVE@` / `nomodeset` flags):
 ```
 net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always
 numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1
-nowatchdog nosoftlockup
+nowatchdog nosoftlockup udev.children_max=1 intel_iommu=on
+iommu.passthrough=0 efi=disable_early_pci_dma
 ```
-(fail-safe is the exception — it deliberately uses minimal params.)
+The fail-safe entry alone adds `pci=realloc iommu.strict=1`. `pci=realloc` must
+not be copied into normal entries. Memtest and firmware-setup entries are not
+Linux live entries and therefore do not carry these parameters.
 
 **Historical note:** `grub-pc/` was mistakenly used instead of `grub-efi/` until v8.25.
 live-build reads `config/bootloaders/grub-efi/` for UEFI because the build is