# "Run All" SAT planning happens on the backend, not in the browser

**Date:** 2026-08-31
**Status:** active

## Symptom

Support bundle `210619KUGGXGS2000017` (8x H200 NVL, identical to `...008`):
the operator hit "Run All" and only 6 tasks were queued - cpu, memory,
storage, tpm, nvidia-config, pcie-link. No `nvidia`, `nvidia-interconnect`,
`nvidia-bandwidth`, `nvidia-pcie-bandwidth`. The GPUs were physically present
and `nvidia-config` (which enumerates them) passed.

## Root cause

Task planning lived in page JavaScript:

- `runAllCheckSAT()` / `runAllSAT()` added the NVIDIA targets only
  `if (satSelectedGPUIndices().length)`.
- That list came from `/api/gpu/nvidia` (`nvidia-smi --query-gpu=...`), fetched
  **once per page load and cached** in `satNvidiaGPUsPromise`.
- If the page first queried while the driver was still enumerating GPUs (the
  two 60-second service timeouts described in
  [2026-08-31-bee-nvidia-restart-deadlock.md](2026-08-31-bee-nvidia-restart-deadlock.md),
  plus per-GPU GSP firmware boot), it got an empty list and cached it for the
  whole session. "Run All" then silently dropped every GPU test - no banner,
  no error. `/api/gpu/presence` (a separate `os.Stat("/dev/nvidia0")` check)
  meanwhile said "GPU present", so the UI even contradicted itself.

## Decision

- **New endpoint `POST /api/sat/run-all`.** The browser sends only operator
  *intent*: `stress_mode`, `amd_targets` (checkbox selection), and
  an optional `nvidia_gpu_indices` subset. The server decides what hardware
  is present/ready and what to enqueue (`handler.planSATRunAll`).
  `runAllSAT()` / `runAllCheckSAT()` are now thin `fetch` wrappers and render
  the `notes[]` the server returns ("TPM: no device - skipped", etc.).
- **GPU planning waits on a readiness gate, not a device probe.**
  `planSATRunAll` calls `waitForNvidiaReady`, which repeats the fresh
  `ListNvidiaGPUs` query until `nvidia-smi` enumerates at least one GPU, the
  runtime snapshot reports `NvidiaGSPMode == "gsp-stuck"`, or the 75-second
  deadline expires. A loaded kernel module is not treated as proof that
  user-space tools can address a GPU. `CUDAReady == false` after enumeration
  is a note, not a blocker.
- **One presence source with a PCI fallback.** `app.DetectGPUPresence`:
  - primary - the existing operational vendor result (`DetectGPUVendor`);
  - fallback - `System.PhysicalGPUVendors()` (lspci VGA/3D/Display class +
    vendor id): a GPU on the bus not reported by operational detection sets
    `NvidiaInitializing` / `AMDInitializing`, an explicit state distinct from
    "absent". For NVIDIA, the page can show that enumeration is still pending
    instead of reporting no hardware.
  `/api/gpu/tools`, `/api/gpu/presence` and the run-all planner all go through
  it.
- **TPM gate.** `tpm` is planned only when `app.TPMPresent()` finds a sysfs TPM
  whose stable `tpm_version_major` attribute is exactly `2`; otherwise a note explains the skip. This
  matches the pack-level guard already in `RunTPMValidationPack`.

## Consequences

- "Run All" can no longer skip hardware because a browser-cached probe was
  early/empty. A genuinely absent or dead GPU still gets skipped - but with a
  `notes[]` entry, and `nvidia-config` still runs to capture diagnostics.
- The per-card `disableSATCard('...','No NVIDIA GPU detected')` hints in the
  page still use `/api/gpu/presence` - cosmetic only now, and self-heal on
  reload.
- Automated / headless callers get correct planning for free by POSTing the
  same endpoint instead of replicating the JS logic.