bee/bible-local/decisions/2026-09-03-pcie-link-verdict-under-real-traffic.md

PCIe link verdict comes from real traffic, not forced retraining

Date: 2026-09-03 Status: active

Evidence

On bee@172.16.41.97, build 14.02-1-g642e686, three independent runs of PCIe Link Check (forced retrain) reported all eight NVIDIA GPUs and their eight upstream bridges at Gen1 x16 after retraining, against Gen5 x16 capability. In the same application history, every dcgmi diag -r nvbandwidth run brought every GPU to Gen5 x16 and passed. There were no AER or NVIDIA Xid events supporting a physical link fault.

This disproves the premise that setting the PCIe Retrain Link bit is a generic substitute for device traffic. On this hardware and driver, a link can complete a forced retrain and remain in its idle Gen1 state. Counting both the endpoint and its upstream bridge also reports one physical link twice.

Decision

• Remove the standalone pcie-link forced-retrain SAT from code, API, UI, and Run All planning.
• Remove the separate nvidia-pcie-bandwidth SAT. It duplicated the same approximately 45-minute nvbandwidth workload already run by nvidia-bandwidth.
• Extend nvidia-bandwidth to record a sysfs baseline before the test and sample negotiated speed and width synchronously when the final all-GPU dcgmi diag -r nvbandwidth command exits. The sample runs before later health probes and artifact processing can let the link return to idle.
• Append pcie_link_under_load_status, per-GPU PCIe status, and the degraded count to the existing summary. A supported GPU that is still below its own maximum speed or width after real traffic fails the overall SAT. Missing sysfs attributes produce UNSUPPORTED, not a false failure.
• Keep non-GPU PCIe inspection read-only and diagnostic. Without a device-specific traffic generator, current speed is inventory rather than a health verdict.
• When a new nvidia-bandwidth result is recorded, recompute the NVIDIA GPU component status while ignoring the retired sat:pcie-link and sat:nvidia-pcie-bandwidth sources. Their history remains auditable. Other failures, such as nvidia-config or Xid-derived warnings, remain sticky.

Consequences

Run All performs one fewer short but misleading test and one fewer duplicate long GPU bandwidth test. A Gen1 idle baseline no longer alarms when the same link reaches Gen5 x16 under traffic. A genuine under-load speed or width downgrade still fails the bandwidth SAT and reaches pcie:gpu:nvidia.

Old task directories and component history remain readable. The removed API targets are no longer accepted for new tasks.