fix(pcie): verify GPU links under real bandwidth load

This commit is contained in:
Mikhail Chusavitin
2026-09-03 10:23:39 +03:00
parent 642e68631d
commit 347bc8310a
27 changed files with 384 additions and 902 deletions
+1 -1
View File
@@ -41,7 +41,7 @@ All SAT run endpoints enqueue an async task. Response: `{"task_id": "..."}`.
| POST | `/api/sat/nvidia-targeted-power/run` | NVIDIA targeted power |
| POST | `/api/sat/nvidia-pulse/run` | NVIDIA pulse test |
| POST | `/api/sat/nvidia-interconnect/run` | NCCL all_reduce_perf |
| POST | `/api/sat/nvidia-bandwidth/run` | NVBandwidth test |
| POST | `/api/sat/nvidia-bandwidth/run` | NVBandwidth + PCIe link under load |
| POST | `/api/sat/nvidia-stress/run` | NVIDIA stress pack |
| POST | `/api/sat/memory/run` | Memory acceptance |
| POST | `/api/sat/storage/run` | Storage acceptance (smartctl) |
+5 -2
View File
@@ -285,12 +285,12 @@ Web UI: Acceptance Tests page -> Run Test button
Web UI: "Run All" button -> POST /api/sat/run-all
body: operator intent only { stress_mode, amd_targets[], nvidia_gpu_indices[] }
server (handler.planSATRunAll):
1. always: cpu, memory, storage, pcie-link
1. always: cpu, memory, storage
2. tpm - only if App.TPMPresent() finds tpm_version_major=2
3. nvidia-config - if DetectGPUPresence().Nvidia || NvidiaInitializing
4. wait for NVIDIA enumeration: repeat fresh ListNvidiaGPUs queries until
at least one GPU is returned, NvidiaGSPMode=="gsp-stuck", or 75s
5. nvidia / nvidia-interconnect / nvidia-bandwidth / nvidia-pcie-bandwidth
5. nvidia / nvidia-interconnect / nvidia-bandwidth
(+ targeted-stress/power/pulse when stress_mode) - only once ready,
-i = App.ListNvidiaGPUs() indices (intersected with the requested subset)
6. amd / amd-mem / amd-bandwidth - if DetectGPUPresence().AMD and selected
@@ -306,6 +306,9 @@ Web UI: "Run All" button -> POST /api/sat/run-all
vendor detection plus an lspci display-class fallback).
`/api/gpu/presence`, `/api/gpu/tools` and the planner all use it.
- `bee-gpu-burn` / `bee-john-gpu-stress` use `exec.CommandContext`: killed on job context cancel.
- `nvidia-bandwidth` samples every selected GPU's negotiated PCIe speed and
width immediately after its final all-GPU `dcgmi diag -r nvbandwidth`
command exits. Idle and forced-retrain readings are not health verdicts.
- Metric goroutine uses stopCh/doneCh pattern; main goroutine waits `<-doneCh` before reading rows (no mutex needed).
- SVG chart is fully offline: no JS, no external CSS, pure inline SVG.
- `RunNvidiaBandwidthPack` runs one all-GPU `nvbandwidth` pass in Validate; the