refactor: modularize audit and harden build validation

2026-08-31 21:22:16 +03:00
parent bb22ccfafe
commit ac4bc0b2b7
78 changed files with 13598 additions and 13130 deletions
@@ -0,0 +1,67 @@
+# "Run All" SAT planning happens on the backend, not in the browser
+
+**Date:** 2026-08-31
+**Status:** active
+
+## Symptom
+
+Support bundle `210619KUGGXGS2000017` (8x H200 NVL, identical to `...008`):
+the operator hit "Run All" and only 6 tasks were queued - cpu, memory,
+storage, tpm, nvidia-config, pcie-link. No `nvidia`, `nvidia-interconnect`,
+`nvidia-bandwidth`, `nvidia-pcie-bandwidth`. The GPUs were physically present
+and `nvidia-config` (which enumerates them) passed.
+
+## Root cause
+
+Task planning lived in page JavaScript:
+
+- `runAllCheckSAT()` / `runAllSAT()` added the NVIDIA targets only
+  `if (satSelectedGPUIndices().length)`.
+- That list came from `/api/gpu/nvidia` (`nvidia-smi --query-gpu=...`), fetched
+  **once per page load and cached** in `satNvidiaGPUsPromise`.
+- If the page first queried while the driver was still enumerating GPUs (the
+  two 60-second service timeouts described in
+  [2026-08-31-bee-nvidia-restart-deadlock.md](2026-08-31-bee-nvidia-restart-deadlock.md),
+  plus per-GPU GSP firmware boot), it got an empty list and cached it for the
+  whole session. "Run All" then silently dropped every GPU test - no banner,
+  no error. `/api/gpu/presence` (a separate `os.Stat("/dev/nvidia0")` check)
+  meanwhile said "GPU present", so the UI even contradicted itself.
+
+## Decision
+
+- **New endpoint `POST /api/sat/run-all`.** The browser sends only operator
+  *intent*: `stress_mode`, `amd_targets` (checkbox selection), and
+  an optional `nvidia_gpu_indices` subset. The server decides what hardware
+  is present/ready and what to enqueue (`handler.planSATRunAll`).
+  `runAllSAT()` / `runAllCheckSAT()` are now thin `fetch` wrappers and render
+  the `notes[]` the server returns ("TPM: no device - skipped", etc.).
+- **GPU planning waits on a readiness gate, not a device probe.**
+  `planSATRunAll` calls `waitForNvidiaReady`, which repeats the fresh
+  `ListNvidiaGPUs` query until `nvidia-smi` enumerates at least one GPU, the
+  runtime snapshot reports `NvidiaGSPMode == "gsp-stuck"`, or the 75-second
+  deadline expires. A loaded kernel module is not treated as proof that
+  user-space tools can address a GPU. `CUDAReady == false` after enumeration
+  is a note, not a blocker.
+- **One presence source with a PCI fallback.** `app.DetectGPUPresence`:
+  - primary - the existing operational vendor result (`DetectGPUVendor`);
+  - fallback - `System.PhysicalGPUVendors()` (lspci VGA/3D/Display class +
+    vendor id): a GPU on the bus not reported by operational detection sets
+    `NvidiaInitializing` / `AMDInitializing`, an explicit state distinct from
+    "absent". For NVIDIA, the page can show that enumeration is still pending
+    instead of reporting no hardware.
+  `/api/gpu/tools`, `/api/gpu/presence` and the run-all planner all go through
+  it.
+- **TPM gate.** `tpm` is planned only when `app.TPMPresent()` finds a sysfs TPM
+  whose stable `tpm_version_major` attribute is exactly `2`; otherwise a note explains the skip. This
+  matches the pack-level guard already in `RunTPMValidationPack`.
+
+## Consequences
+
+- "Run All" can no longer skip hardware because a browser-cached probe was
+  early/empty. A genuinely absent or dead GPU still gets skipped - but with a
+  `notes[]` entry, and `nvidia-config` still runs to capture diagnostics.
+- The per-card `disableSATCard('...','No NVIDIA GPU detected')` hints in the
+  page still use `/api/gpu/presence` - cosmetic only now, and self-heal on
+  reload.
+- Automated / headless callers get correct planning for free by POSTing the
+  same endpoint instead of replicating the JS logic.