refactor: modularize audit and harden build validation

2026-08-31 21:22:16 +03:00
parent bb22ccfafe
commit ac4bc0b2b7
78 changed files with 13598 additions and 13130 deletions
@@ -175,8 +175,8 @@ those files may not exist yet. Instead:

 - Tries to copy `memtest86+x64.bin` / `memtest86+x64.efi` from `chroot/boot/` first.
 - Falls back to extracting from the cached `.deb` (via `dpkg-deb -x`) if `chroot/boot/` is empty.
- Appends GRUB and isolinux menu entries only if the respective cfg files already exist at hook time.
-  If they do not exist, the hook warns and continues (does not fail).
+- It does not edit bootloader menus. The complete menus are rendered later
+  from the two canonical project templates.

 Controlled by `BEE_REQUIRE_MEMTEST=1` env var to turn warnings into hard errors when needed.

@@ -186,7 +186,8 @@ After `lb build` completes, `build.sh` checks whether the fully materialized `bi
 contains all required memtest artifacts. If not:

 - Copies/extracts memtest binaries into `binary/boot/`.
- Patches `binary/boot/grub/grub.cfg` and `binary/isolinux/live.cfg` directly.
+- Calls `enforce_live_build_bootloader_assets`, which renders the complete
+  GRUB and isolinux configs from the canonical project templates.
 - Reruns the late binary stages (`binary_checksums`, `binary_iso`, `binary_zsync`) to rebuild
  the ISO with the patched tree.

@@ -59,10 +59,11 @@ individual failure mode. This keeps the total entry count in
 entries plus wipe/memtest/firmware-settings), rather than letting it
 grow linearly with every workaround discovered.

-The isolinux (BIOS/legacy boot) menu in `iso/builder/config/bootloaders/isolinux/live.cfg.in`
-already had no `pci=realloc` entry at all (pre-existing asymmetry with
-grub-efi, not introduced by this change) -- not addressed here since no
-incident has been observed via legacy boot.
+As of v13.0, GRUB and isolinux both expose exactly one troubleshooting entry
+with `pci=realloc`. It remains absent from every normal, toram, no-GUI, and
+wipe entry. The same v13.0 amendment replaces `iommu=pt` in that entry with
+translated strict IOMMU mode; this changes DMA diagnostics, not the scope of
+the PCI reallocation workaround.

 ## Consequences

@@ -0,0 +1,67 @@
+# "Run All" SAT planning happens on the backend, not in the browser
+
+**Date:** 2026-08-31
+**Status:** active
+
+## Symptom
+
+Support bundle `210619KUGGXGS2000017` (8x H200 NVL, identical to `...008`):
+the operator hit "Run All" and only 6 tasks were queued - cpu, memory,
+storage, tpm, nvidia-config, pcie-link. No `nvidia`, `nvidia-interconnect`,
+`nvidia-bandwidth`, `nvidia-pcie-bandwidth`. The GPUs were physically present
+and `nvidia-config` (which enumerates them) passed.
+
+## Root cause
+
+Task planning lived in page JavaScript:
+
+- `runAllCheckSAT()` / `runAllSAT()` added the NVIDIA targets only
+  `if (satSelectedGPUIndices().length)`.
+- That list came from `/api/gpu/nvidia` (`nvidia-smi --query-gpu=...`), fetched
+  **once per page load and cached** in `satNvidiaGPUsPromise`.
+- If the page first queried while the driver was still enumerating GPUs (the
+  two 60-second service timeouts described in
+  [2026-08-31-bee-nvidia-restart-deadlock.md](2026-08-31-bee-nvidia-restart-deadlock.md),
+  plus per-GPU GSP firmware boot), it got an empty list and cached it for the
+  whole session. "Run All" then silently dropped every GPU test - no banner,
+  no error. `/api/gpu/presence` (a separate `os.Stat("/dev/nvidia0")` check)
+  meanwhile said "GPU present", so the UI even contradicted itself.
+
+## Decision
+
+- **New endpoint `POST /api/sat/run-all`.** The browser sends only operator
+  *intent*: `stress_mode`, `amd_targets` (checkbox selection), and
+  an optional `nvidia_gpu_indices` subset. The server decides what hardware
+  is present/ready and what to enqueue (`handler.planSATRunAll`).
+  `runAllSAT()` / `runAllCheckSAT()` are now thin `fetch` wrappers and render
+  the `notes[]` the server returns ("TPM: no device - skipped", etc.).
+- **GPU planning waits on a readiness gate, not a device probe.**
+  `planSATRunAll` calls `waitForNvidiaReady`, which repeats the fresh
+  `ListNvidiaGPUs` query until `nvidia-smi` enumerates at least one GPU, the
+  runtime snapshot reports `NvidiaGSPMode == "gsp-stuck"`, or the 75-second
+  deadline expires. A loaded kernel module is not treated as proof that
+  user-space tools can address a GPU. `CUDAReady == false` after enumeration
+  is a note, not a blocker.
+- **One presence source with a PCI fallback.** `app.DetectGPUPresence`:
+  - primary - the existing operational vendor result (`DetectGPUVendor`);
+  - fallback - `System.PhysicalGPUVendors()` (lspci VGA/3D/Display class +
+    vendor id): a GPU on the bus not reported by operational detection sets
+    `NvidiaInitializing` / `AMDInitializing`, an explicit state distinct from
+    "absent". For NVIDIA, the page can show that enumeration is still pending
+    instead of reporting no hardware.
+  `/api/gpu/tools`, `/api/gpu/presence` and the run-all planner all go through
+  it.
+- **TPM gate.** `tpm` is planned only when `app.TPMPresent()` finds a sysfs TPM
+  whose stable `tpm_version_major` attribute is exactly `2`; otherwise a note explains the skip. This
+  matches the pack-level guard already in `RunTPMValidationPack`.
+
+## Consequences
+
+- "Run All" can no longer skip hardware because a browser-cached probe was
+  early/empty. A genuinely absent or dead GPU still gets skipped - but with a
+  `notes[]` entry, and `nvidia-config` still runs to capture diagnostics.
+- The per-card `disableSATCard('...','No NVIDIA GPU detected')` hints in the
+  page still use `/api/gpu/presence` - cosmetic only now, and self-heal on
+  reload.
+- Automated / headless callers get correct planning for free by POSTing the
+  same endpoint instead of replicating the JS logic.
@@ -0,0 +1,66 @@
+# bee-nvidia.service: never call blocking `systemctl restart` on units ordered After= itself
+
+**Date:** 2026-08-31
+**Status:** active
+
+## Symptom
+
+Every affected NVIDIA boot reaches two 60-second wrapper timeouts in
+`bee-nvidia.service`. `bee-nvidia.log`:
+
+```
+[bee-nvidia] restarting nvidia-fabricmanager.service (timeout 60s)
+[bee-nvidia] WARN: systemctl restart nvidia-fabricmanager.service timed out after 60s
+[bee-nvidia] restarting nvidia-dcgm.service (timeout 60s)
+[bee-nvidia] WARN: systemctl restart nvidia-dcgm.service timed out after 60s
+[bee-nvidia] done
+```
+
+Seen on both `210619KUGGXGS2000008` and `...017` (8x H200 NVL). That 120 s
+window overlaps exactly with when an operator opens the web UI and clicks
+"Run All" - during it `nvidia-smi` may not yet enumerate all GPUs, which is
+how bundle `...017` ended up running the check set with **no GPU tests**
+(see [2026-08-31-backend-driven-sat-planning.md](2026-08-31-backend-driven-sat-planning.md)).
+
+## Root cause
+
+`bee-nvidia.service` is `Type=oneshot` and `Before=nvidia-fabricmanager.service
+nvidia-dcgm.service`. Its `ExecStart` (`bee-nvidia-load`) then ran, synchronously:
+
+```
+timeout 60 systemctl restart nvidia-fabricmanager.service
+timeout 60 systemctl restart nvidia-dcgm.service
+```
+
+A oneshot unit is not "active" until `ExecStart` returns. Both target units
+are ordered `After=bee-nvidia.service`, so systemd queues them behind
+bee-nvidia and will not run them while `bee-nvidia-load` is still executing.
+`bee-nvidia-load` blocks on `systemctl restart` waiting for exactly that job
+to complete -> deadlock -> broken only when `timeout 60` fires. Twice.
+
+`nvidia-smi -q` inside `bee-check-nvswitch` (the fabricmanager ExecCondition)
+is not part of this ordering cycle; the deadlock is structural.
+
+## Decision
+
+`bee-nvidia-load` no longer blocks on those units. It calls
+`systemctl --no-block try-restart` for each:
+
+- `--no-block` returns immediately; systemd runs the job after bee-nvidia
+  exits, via the existing `Before=` ordering.
+- `try-restart` only acts if the unit is already running (the "stale instance
+  from a reload / re-run" case the old code worried about). If it is inactive,
+  this command does nothing; an enabled unit already queued by the normal boot
+  transaction starts after `bee-nvidia.service` via the declared ordering.
+
+The `SYSTEMCTL_TIMEOUT` / `timeout_systemctl` wrapper and the fallback
+`systemctl start` / `systemctl status` branches are gone. `--no-block` means
+systemctl does not wait for the queued unit job to finish.
+
+## Consequences
+
+- NVIDIA boot no longer waits for those two 60-second wrapper deadlines.
+- DO NOT reintroduce a synchronous `systemctl {start,restart}` of any unit
+  that is `After=bee-nvidia.service` from inside `bee-nvidia-load`. If a unit
+  genuinely must be up before the script returns, invert the ordering
+  instead.
@@ -0,0 +1,47 @@
+# nvbandwidth: single all-GPU pass in Validate, per-NUMA-node matrix only in the deep tier
+
+**Date:** 2026-08-31
+**Status:** active
+**Amends:** [2026-07-27-nvbandwidth-per-socket-split.md](2026-07-27-nvbandwidth-per-socket-split.md)
+
+## Context
+
+`2026-07-27` made `RunNvidiaBandwidthPack` split `dcgmi diag -r nvbandwidth`
+using Linux PCI `numa_node` locality, followed by an all-GPU pass. A NUMA node
+is not assumed to be identical to a physical CPU socket.
+
+Two problems showed up on an 8x H200 NVL / dual-socket EPYC box
+(`210619KUGGXGS2000008`):
+
+1. `SATEstimatedNvidiaBandwidthSec` assigns 2700 seconds to one all-GPU pass.
+   Three such invocations are assigned 8100 seconds, which is outside the
+   intended **Validate** duration.
+2. The split silently never engaged anyway: `normalizeNvidiaBDF` returned
+   nvidia-smi's upper-case PCI BDF (`0000:CB:00.0`) while `/sys/bus/pci/devices`
+   entries are lower-case, so `readPCINumaNode` failed for every GPU on a bus
+   with a hex letter and `gpuBandwidthSocketGroups` fell back to one group.
+
+## Decision
+
+- `normalizeNvidiaBDF` now lower-cases (and trims) the BDF, so sysfs
+  `numa_node` / link-speed reads actually resolve.
+- `RunNvidiaBandwidthPack` takes a `fullMatrix bool`. The **only** thing it
+  changes is which GPU set each `nvbandwidth` invocation gets via `-i` - the
+  command itself is untouched (no extra flags, no testcase filtering).
+  - **Validate** (`stress_mode=false`) -> `fullMatrix=false`: one pass,
+    `-i <all selected GPUs>`. No NUMA-locality split.
+  - **Stress / deep** (`stress_mode=true`) -> `fullMatrix=true`: the
+    `2026-07-27` behaviour - one pass per resolved NUMA-node group, then one
+    all-GPU pass. If any selected GPU has no resolved NUMA node, the code does
+    not guess a group and falls back to the single all-GPU pass.
+- `task_runner` passes `t.params.StressMode` through.
+
+## Consequences
+
+- Validate schedules exactly one nvbandwidth invocation regardless of NUMA-node count.
+- The NUMA-locality isolation from `2026-07-27` is preserved, just moved to
+  the tier where a 2-3x runtime is acceptable. DO NOT re-add the split to the
+  Validate path.
+- `SATEstimatedNvidiaBandwidthSec` still reflects a single pass; the deep
+  tier's multi-pass duration is not modelled - revisit once real
+  multi-node deep-run logs exist (same open item as `2026-07-27`).
@@ -11,3 +11,7 @@ One file per decision, named `YYYY-MM-DD-short-topic.md`.
 | 2026-07-27 | Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass | active |
 | 2026-07-27 | Stream SAT job output live to disk and kick blackbox sync on job completion | active |
 | 2026-07-28 | Move pci=realloc out of the default/toram/no-GUI GRUB entries | active |
+| 2026-08-24 | PCIe Gen1-at-idle GPU warning: load-bearing link check, not idle sysfs | active |
+| 2026-08-31 | nvbandwidth: single all-GPU pass in Validate, per-socket matrix only in deep tier | active |
+| 2026-08-31 | bee-nvidia.service: never blocking `systemctl restart` on units ordered After= itself | active |
+| 2026-08-31 | "Run All" SAT planning happens on the backend, not the browser | active |