bee/bible-local/decisions/2026-08-31-backend-driven-sat-planning.md

"Run All" SAT planning happens on the backend, not in the browser

Date: 2026-08-31 Status: active

Symptom

Support bundle 210619KUGGXGS2000017 (8x H200 NVL, identical to ...008): the operator hit "Run All" and only 6 tasks were queued - cpu, memory, storage, tpm, nvidia-config, pcie-link. No nvidia, nvidia-interconnect, nvidia-bandwidth, nvidia-pcie-bandwidth. The GPUs were physically present and nvidia-config (which enumerates them) passed.

Root cause

Task planning lived in page JavaScript:

• runAllCheckSAT() / runAllSAT() added the NVIDIA targets only if (satSelectedGPUIndices().length).
• That list came from /api/gpu/nvidia (nvidia-smi --query-gpu=...), fetched once per page load and cached in satNvidiaGPUsPromise.
• If the page first queried while the driver was still enumerating GPUs (the two 60-second service timeouts described in 2026-08-31-bee-nvidia-restart-deadlock.md, plus per-GPU GSP firmware boot), it got an empty list and cached it for the whole session. "Run All" then silently dropped every GPU test - no banner, no error. /api/gpu/presence (a separate os.Stat("/dev/nvidia0") check) meanwhile said "GPU present", so the UI even contradicted itself.

Decision

• New endpoint POST /api/sat/run-all. The browser sends only operator intent: stress_mode, amd_targets (checkbox selection), and an optional nvidia_gpu_indices subset. The server decides what hardware is present/ready and what to enqueue (handler.planSATRunAll). runAllSAT() / runAllCheckSAT() are now thin fetch wrappers and render the notes[] the server returns ("TPM: no device - skipped", etc.).
• GPU planning waits on a readiness gate, not a device probe. planSATRunAll calls waitForNvidiaReady, which repeats the fresh ListNvidiaGPUs query until nvidia-smi enumerates at least one GPU, the runtime snapshot reports NvidiaGSPMode == "gsp-stuck", or the 75-second deadline expires. A loaded kernel module is not treated as proof that user-space tools can address a GPU. CUDAReady == false after enumeration is a note, not a blocker.
• One presence source with a PCI fallback. app.DetectGPUPresence:
  • primary - the existing operational vendor result (DetectGPUVendor);
  • fallback - System.PhysicalGPUVendors() (lspci VGA/3D/Display class + vendor id): a GPU on the bus not reported by operational detection sets NvidiaInitializing / AMDInitializing, an explicit state distinct from "absent". For NVIDIA, the page can show that enumeration is still pending instead of reporting no hardware. /api/gpu/tools, /api/gpu/presence and the run-all planner all go through it.
• TPM gate. tpm is planned only when app.TPMPresent() finds a sysfs TPM whose stable tpm_version_major attribute is exactly 2; otherwise a note explains the skip. This matches the pack-level guard already in RunTPMValidationPack.

Consequences

• "Run All" can no longer skip hardware because a browser-cached probe was early/empty. A genuinely absent or dead GPU still gets skipped - but with a notes[] entry, and nvidia-config still runs to capture diagnostics.
• The per-card disableSATCard('...','No NVIDIA GPU detected') hints in the page still use /api/gpu/presence - cosmetic only now, and self-heal on reload.
• Automated / headless callers get correct planning for free by POSTing the same endpoint instead of replicating the JS logic.