fix(pcie): verify GPU links under real bandwidth load

2026-09-03 10:23:39 +03:00
parent 642e68631d
commit 347bc8310a
27 changed files with 384 additions and 902 deletions
@@ -41,7 +41,7 @@ All SAT run endpoints enqueue an async task. Response: `{"task_id": "..."}`.
 | POST   | `/api/sat/nvidia-targeted-power/run`       | NVIDIA targeted power             |
 | POST   | `/api/sat/nvidia-pulse/run`                | NVIDIA pulse test                 |
 | POST   | `/api/sat/nvidia-interconnect/run`         | NCCL all_reduce_perf              |
-| POST   | `/api/sat/nvidia-bandwidth/run`            | NVBandwidth test                  |
+| POST   | `/api/sat/nvidia-bandwidth/run`            | NVBandwidth + PCIe link under load |
 | POST   | `/api/sat/nvidia-stress/run`               | NVIDIA stress pack                |
 | POST   | `/api/sat/memory/run`                      | Memory acceptance                 |
 | POST   | `/api/sat/storage/run`                     | Storage acceptance (smartctl)     |
@@ -285,12 +285,12 @@ Web UI: Acceptance Tests page -> Run Test button
 Web UI: "Run All" button -> POST /api/sat/run-all
  body: operator intent only { stress_mode, amd_targets[], nvidia_gpu_indices[] }
  server (handler.planSATRunAll):
-    1. always: cpu, memory, storage, pcie-link
+    1. always: cpu, memory, storage
    2. tpm            - only if App.TPMPresent() finds tpm_version_major=2
    3. nvidia-config  - if DetectGPUPresence().Nvidia || NvidiaInitializing
    4. wait for NVIDIA enumeration: repeat fresh ListNvidiaGPUs queries until
       at least one GPU is returned, NvidiaGSPMode=="gsp-stuck", or 75s
-    5. nvidia / nvidia-interconnect / nvidia-bandwidth / nvidia-pcie-bandwidth
+    5. nvidia / nvidia-interconnect / nvidia-bandwidth
       (+ targeted-stress/power/pulse when stress_mode) - only once ready,
       -i = App.ListNvidiaGPUs() indices (intersected with the requested subset)
    6. amd / amd-mem / amd-bandwidth - if DetectGPUPresence().AMD and selected
@@ -306,6 +306,9 @@ Web UI: "Run All" button -> POST /api/sat/run-all
  vendor detection plus an lspci display-class fallback).
  `/api/gpu/presence`, `/api/gpu/tools` and the planner all use it.
 - `bee-gpu-burn` / `bee-john-gpu-stress` use `exec.CommandContext`: killed on job context cancel.
+- `nvidia-bandwidth` samples every selected GPU's negotiated PCIe speed and
+  width immediately after its final all-GPU `dcgmi diag -r nvbandwidth`
+  command exits. Idle and forced-retrain readings are not health verdicts.
 - Metric goroutine uses stopCh/doneCh pattern; main goroutine waits `<-doneCh` before reading rows (no mutex needed).
 - SVG chart is fully offline: no JS, no external CSS, pure inline SVG.
 - `RunNvidiaBandwidthPack` runs one all-GPU `nvbandwidth` pass in Validate; the
@@ -1,7 +1,7 @@
 # PCIe Gen1-at-idle GPU warning: history of attempts, and the fix

 **Date:** 2026-08-24
-**Status:** active — implemented (see "Resolution" at the end)
+**Status:** superseded in part by `2026-09-03-pcie-link-verdict-under-real-traffic.md`

 ## Symptom

@@ -116,6 +116,11 @@ Pick one (or combine):

 ## Resolution (2026-08-24)

+Historical note: the two SAT targets described below were the resolution at
+the time. The 2026-09-03 follow-up removes the forced-retrain target and folds
+the GPU link verdict into the existing bandwidth SAT after field evidence
+showed that retraining without traffic can remain at Gen1.
+
 Landed a variant of options 1–3 that turned out simpler than any of them
 individually once we stopped trying to make the idle reading recoverable:

@@ -0,0 +1,51 @@
+# PCIe link verdict comes from real traffic, not forced retraining
+
+**Date:** 2026-09-03
+**Status:** active
+
+## Evidence
+
+On `bee@172.16.41.97`, build `14.02-1-g642e686`, three independent runs of
+`PCIe Link Check (forced retrain)` reported all eight NVIDIA GPUs and their
+eight upstream bridges at Gen1 x16 after retraining, against Gen5 x16
+capability. In the same application history, every `dcgmi diag -r
+nvbandwidth` run brought every GPU to Gen5 x16 and passed. There were no AER
+or NVIDIA Xid events supporting a physical link fault.
+
+This disproves the premise that setting the PCIe Retrain Link bit is a
+generic substitute for device traffic. On this hardware and driver, a link
+can complete a forced retrain and remain in its idle Gen1 state. Counting both
+the endpoint and its upstream bridge also reports one physical link twice.
+
+## Decision
+
+- Remove the standalone `pcie-link` forced-retrain SAT from code, API, UI,
+  and Run All planning.
+- Remove the separate `nvidia-pcie-bandwidth` SAT. It duplicated the same
+  approximately 45-minute `nvbandwidth` workload already run by
+  `nvidia-bandwidth`.
+- Extend `nvidia-bandwidth` to record a sysfs baseline before the test and
+  sample negotiated speed and width synchronously when the final all-GPU
+  `dcgmi diag -r nvbandwidth` command exits. The sample runs before later
+  health probes and artifact processing can let the link return to idle.
+- Append `pcie_link_under_load_status`, per-GPU PCIe status, and the degraded
+  count to the existing summary. A supported GPU that is still below its own
+  maximum speed or width after real traffic fails the overall SAT. Missing
+  sysfs attributes produce `UNSUPPORTED`, not a false failure.
+- Keep non-GPU PCIe inspection read-only and diagnostic. Without a
+  device-specific traffic generator, current speed is inventory rather than
+  a health verdict.
+- When a new `nvidia-bandwidth` result is recorded, recompute the NVIDIA GPU
+  component status while ignoring the retired `sat:pcie-link` and
+  `sat:nvidia-pcie-bandwidth` sources. Their history remains auditable. Other
+  failures, such as `nvidia-config` or Xid-derived warnings, remain sticky.
+
+## Consequences
+
+Run All performs one fewer short but misleading test and one fewer duplicate
+long GPU bandwidth test. A Gen1 idle baseline no longer alarms when the same
+link reaches Gen5 x16 under traffic. A genuine under-load speed or width
+downgrade still fails the bandwidth SAT and reaches `pcie:gpu:nvidia`.
+
+Old task directories and component history remain readable. The removed API
+targets are no longer accepted for new tasks.
@@ -11,8 +11,9 @@ One file per decision, named `YYYY-MM-DD-short-topic.md`.
 | 2026-07-27 | Split the NVIDIA Bandwidth SAT into per-socket passes before the all-GPU pass | active |
 | 2026-07-27 | Stream SAT job output live to disk and kick blackbox sync on job completion | active |
 | 2026-07-28 | Move pci=realloc out of the default/toram/no-GUI GRUB entries | active |
-| 2026-08-24 | PCIe Gen1-at-idle GPU warning: load-bearing link check, not idle sysfs | active |
+| 2026-08-24 | PCIe Gen1-at-idle GPU warning: load-bearing link check, not idle sysfs | superseded in part |
 | 2026-08-31 | nvbandwidth: single all-GPU pass in Validate, per-socket matrix only in deep tier | active |
 | 2026-08-31 | bee-nvidia.service: never blocking `systemctl restart` on units ordered After= itself | active |
 | 2026-08-31 | "Run All" SAT planning happens on the backend, not the browser | active |
 | 2026-08-31 | Support bundle uses private staging and unique atomic output | active |
+| 2026-09-03 | PCIe link verdict comes only from the existing real-traffic GPU bandwidth SAT | active |