52 lines
2.5 KiB
Markdown
52 lines
2.5 KiB
Markdown
# PCIe link verdict comes from real traffic, not forced retraining
|
|
|
|
**Date:** 2026-09-03
|
|
**Status:** active
|
|
|
|
## Evidence
|
|
|
|
On `bee@172.16.41.97`, build `14.02-1-g642e686`, three independent runs of
|
|
`PCIe Link Check (forced retrain)` reported all eight NVIDIA GPUs and their
|
|
eight upstream bridges at Gen1 x16 after retraining, against Gen5 x16
|
|
capability. In the same application history, every `dcgmi diag -r
|
|
nvbandwidth` run brought every GPU to Gen5 x16 and passed. There were no AER
|
|
or NVIDIA Xid events supporting a physical link fault.
|
|
|
|
This disproves the premise that setting the PCIe Retrain Link bit is a
|
|
generic substitute for device traffic. On this hardware and driver, a link
|
|
can complete a forced retrain and remain in its idle Gen1 state. Counting both
|
|
the endpoint and its upstream bridge also reports one physical link twice.
|
|
|
|
## Decision
|
|
|
|
- Remove the standalone `pcie-link` forced-retrain SAT from code, API, UI,
|
|
and Run All planning.
|
|
- Remove the separate `nvidia-pcie-bandwidth` SAT. It duplicated the same
|
|
approximately 45-minute `nvbandwidth` workload already run by
|
|
`nvidia-bandwidth`.
|
|
- Extend `nvidia-bandwidth` to record a sysfs baseline before the test and
|
|
sample negotiated speed and width synchronously when the final all-GPU
|
|
`dcgmi diag -r nvbandwidth` command exits. The sample runs before later
|
|
health probes and artifact processing can let the link return to idle.
|
|
- Append `pcie_link_under_load_status`, per-GPU PCIe status, and the degraded
|
|
count to the existing summary. A supported GPU that is still below its own
|
|
maximum speed or width after real traffic fails the overall SAT. Missing
|
|
sysfs attributes produce `UNSUPPORTED`, not a false failure.
|
|
- Keep non-GPU PCIe inspection read-only and diagnostic. Without a
|
|
device-specific traffic generator, current speed is inventory rather than
|
|
a health verdict.
|
|
- When a new `nvidia-bandwidth` result is recorded, recompute the NVIDIA GPU
|
|
component status while ignoring the retired `sat:pcie-link` and
|
|
`sat:nvidia-pcie-bandwidth` sources. Their history remains auditable. Other
|
|
failures, such as `nvidia-config` or Xid-derived warnings, remain sticky.
|
|
|
|
## Consequences
|
|
|
|
Run All performs one fewer short but misleading test and one fewer duplicate
|
|
long GPU bandwidth test. A Gen1 idle baseline no longer alarms when the same
|
|
link reaches Gen5 x16 under traffic. A genuine under-load speed or width
|
|
downgrade still fails the bandwidth SAT and reaches `pcie:gpu:nvidia`.
|
|
|
|
Old task directories and component history remain readable. The removed API
|
|
targets are no longer accepted for new tasks.
|