68 lines
3.5 KiB
Markdown
68 lines
3.5 KiB
Markdown
# "Run All" SAT planning happens on the backend, not in the browser
|
|
|
|
**Date:** 2026-08-31
|
|
**Status:** active
|
|
|
|
## Symptom
|
|
|
|
Support bundle `210619KUGGXGS2000017` (8x H200 NVL, identical to `...008`):
|
|
the operator hit "Run All" and only 6 tasks were queued - cpu, memory,
|
|
storage, tpm, nvidia-config, pcie-link. No `nvidia`, `nvidia-interconnect`,
|
|
`nvidia-bandwidth`, `nvidia-pcie-bandwidth`. The GPUs were physically present
|
|
and `nvidia-config` (which enumerates them) passed.
|
|
|
|
## Root cause
|
|
|
|
Task planning lived in page JavaScript:
|
|
|
|
- `runAllCheckSAT()` / `runAllSAT()` added the NVIDIA targets only
|
|
`if (satSelectedGPUIndices().length)`.
|
|
- That list came from `/api/gpu/nvidia` (`nvidia-smi --query-gpu=...`), fetched
|
|
**once per page load and cached** in `satNvidiaGPUsPromise`.
|
|
- If the page first queried while the driver was still enumerating GPUs (the
|
|
two 60-second service timeouts described in
|
|
[2026-08-31-bee-nvidia-restart-deadlock.md](2026-08-31-bee-nvidia-restart-deadlock.md),
|
|
plus per-GPU GSP firmware boot), it got an empty list and cached it for the
|
|
whole session. "Run All" then silently dropped every GPU test - no banner,
|
|
no error. `/api/gpu/presence` (a separate `os.Stat("/dev/nvidia0")` check)
|
|
meanwhile said "GPU present", so the UI even contradicted itself.
|
|
|
|
## Decision
|
|
|
|
- **New endpoint `POST /api/sat/run-all`.** The browser sends only operator
|
|
*intent*: `stress_mode`, `amd_targets` (checkbox selection), and
|
|
an optional `nvidia_gpu_indices` subset. The server decides what hardware
|
|
is present/ready and what to enqueue (`handler.planSATRunAll`).
|
|
`runAllSAT()` / `runAllCheckSAT()` are now thin `fetch` wrappers and render
|
|
the `notes[]` the server returns ("TPM: no device - skipped", etc.).
|
|
- **GPU planning waits on a readiness gate, not a device probe.**
|
|
`planSATRunAll` calls `waitForNvidiaReady`, which repeats the fresh
|
|
`ListNvidiaGPUs` query until `nvidia-smi` enumerates at least one GPU, the
|
|
runtime snapshot reports `NvidiaGSPMode == "gsp-stuck"`, or the 75-second
|
|
deadline expires. A loaded kernel module is not treated as proof that
|
|
user-space tools can address a GPU. `CUDAReady == false` after enumeration
|
|
is a note, not a blocker.
|
|
- **One presence source with a PCI fallback.** `app.DetectGPUPresence`:
|
|
- primary - the existing operational vendor result (`DetectGPUVendor`);
|
|
- fallback - `System.PhysicalGPUVendors()` (lspci VGA/3D/Display class +
|
|
vendor id): a GPU on the bus not reported by operational detection sets
|
|
`NvidiaInitializing` / `AMDInitializing`, an explicit state distinct from
|
|
"absent". For NVIDIA, the page can show that enumeration is still pending
|
|
instead of reporting no hardware.
|
|
`/api/gpu/tools`, `/api/gpu/presence` and the run-all planner all go through
|
|
it.
|
|
- **TPM gate.** `tpm` is planned only when `app.TPMPresent()` finds a sysfs TPM
|
|
whose stable `tpm_version_major` attribute is exactly `2`; otherwise a note explains the skip. This
|
|
matches the pack-level guard already in `RunTPMValidationPack`.
|
|
|
|
## Consequences
|
|
|
|
- "Run All" can no longer skip hardware because a browser-cached probe was
|
|
early/empty. A genuinely absent or dead GPU still gets skipped - but with a
|
|
`notes[]` entry, and `nvidia-config` still runs to capture diagnostics.
|
|
- The per-card `disableSATCard('...','No NVIDIA GPU detected')` hints in the
|
|
page still use `/api/gpu/presence` - cosmetic only now, and self-heal on
|
|
reload.
|
|
- Automated / headless callers get correct planning for free by POSTing the
|
|
same endpoint instead of replicating the JS logic.
|