fix(pcie): verify GPU links under real bandwidth load
This commit is contained in:
@@ -41,7 +41,7 @@ All SAT run endpoints enqueue an async task. Response: `{"task_id": "..."}`.
|
||||
| POST | `/api/sat/nvidia-targeted-power/run` | NVIDIA targeted power |
|
||||
| POST | `/api/sat/nvidia-pulse/run` | NVIDIA pulse test |
|
||||
| POST | `/api/sat/nvidia-interconnect/run` | NCCL all_reduce_perf |
|
||||
| POST | `/api/sat/nvidia-bandwidth/run` | NVBandwidth test |
|
||||
| POST | `/api/sat/nvidia-bandwidth/run` | NVBandwidth + PCIe link under load |
|
||||
| POST | `/api/sat/nvidia-stress/run` | NVIDIA stress pack |
|
||||
| POST | `/api/sat/memory/run` | Memory acceptance |
|
||||
| POST | `/api/sat/storage/run` | Storage acceptance (smartctl) |
|
||||
|
||||
@@ -285,12 +285,12 @@ Web UI: Acceptance Tests page -> Run Test button
|
||||
Web UI: "Run All" button -> POST /api/sat/run-all
|
||||
body: operator intent only { stress_mode, amd_targets[], nvidia_gpu_indices[] }
|
||||
server (handler.planSATRunAll):
|
||||
1. always: cpu, memory, storage, pcie-link
|
||||
1. always: cpu, memory, storage
|
||||
2. tpm - only if App.TPMPresent() finds tpm_version_major=2
|
||||
3. nvidia-config - if DetectGPUPresence().Nvidia || NvidiaInitializing
|
||||
4. wait for NVIDIA enumeration: repeat fresh ListNvidiaGPUs queries until
|
||||
at least one GPU is returned, NvidiaGSPMode=="gsp-stuck", or 75s
|
||||
5. nvidia / nvidia-interconnect / nvidia-bandwidth / nvidia-pcie-bandwidth
|
||||
5. nvidia / nvidia-interconnect / nvidia-bandwidth
|
||||
(+ targeted-stress/power/pulse when stress_mode) - only once ready,
|
||||
-i = App.ListNvidiaGPUs() indices (intersected with the requested subset)
|
||||
6. amd / amd-mem / amd-bandwidth - if DetectGPUPresence().AMD and selected
|
||||
@@ -306,6 +306,9 @@ Web UI: "Run All" button -> POST /api/sat/run-all
|
||||
vendor detection plus an lspci display-class fallback).
|
||||
`/api/gpu/presence`, `/api/gpu/tools` and the planner all use it.
|
||||
- `bee-gpu-burn` / `bee-john-gpu-stress` use `exec.CommandContext`: killed on job context cancel.
|
||||
- `nvidia-bandwidth` samples every selected GPU's negotiated PCIe speed and
|
||||
width immediately after its final all-GPU `dcgmi diag -r nvbandwidth`
|
||||
command exits. Idle and forced-retrain readings are not health verdicts.
|
||||
- Metric goroutine uses stopCh/doneCh pattern; main goroutine waits `<-doneCh` before reading rows (no mutex needed).
|
||||
- SVG chart is fully offline: no JS, no external CSS, pure inline SVG.
|
||||
- `RunNvidiaBandwidthPack` runs one all-GPU `nvbandwidth` pass in Validate; the
|
||||
|
||||
Reference in New Issue
Block a user