fix(collector): stop pinning GPU PCIe status on an unverified idle reading

NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power
saving, and applyPCIeLinkSpeedWarning fired on every idle collector
pass regardless - since component-status DB records never downgrade
(Record() only ever raises severity), one boot-time idle sample
permanently pinned pcie:gpu:nvidia to Warning for the rest of the
session even after every load-bearing GPU SAT test passed clean. Two
prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag)
didn't hold up against this hardware/driver combination - see
bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the
full history.

Rather than add a downgrade path, stop writing an unverified status in
the first place: parseLspciDevice no longer calls
applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay
populated as plain descriptive fields; only a verified-under-load
caller may now turn them into a status verdict.

Two new SAT targets provide that verified signal:

- pcie-link (platform/pcie_link_check.go): forces every enabled PCIe
  device - not just GPUs - to retrain via the PCIe spec's Link Control
  "Retrain Link" bit, then compares the negotiated speed against the
  device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn
  equivalent load tool. Classifies by PCI class code + vendor ID, not
  name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into
  their own component-status keys so a degraded NIC never reads as a
  GPU fault.
- nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives
  real host<->device traffic via dcgmi diag -r nvbandwidth and
  resamples link speed immediately after, independent of nvbandwidth's
  own pass/fail.

Both wired into the task queue/webui the same way as nvidia-config
(routes, dispatch, priority, Validate page cards, Run All Check SAT).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-08-24 18:39:19 +03:00
co-authored by Claude Sonnet 5
parent 7aa276320b
commit b11018ac5e
14 changed files with 971 additions and 7 deletions
+28 -2
View File
@@ -187,8 +187,20 @@ func parseLspciDevice(fields map[string]string) schema.HardwarePCIeDevice {
markNVLinkBridge(&dev)
}
// Warn (or Critical for NVLink bridges) if PCIe link is running below max.
applyPCIeLinkSpeedWarning(&dev)
// Deliberately not calling applyPCIeLinkSpeedWarning here: this function
// runs on every idle-time collector pass, and NVIDIA GPUs legitimately
// downclock their PCIe link at idle to save power, making "current <
// max" a false positive most of the time it fires. There's no reliable
// way to downgrade a status once recorded (component-status DB history
// is a one-way transition log, by design — see component_status_db.go),
// so writing an unverified idle reading as Warning meant it stuck for
// the rest of the session even after every load-bearing SAT test passed
// clean. See bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning-unresolved.md.
// applyPCIeLinkSpeedWarning is reserved for a future caller that samples
// link speed while the device is under verified load (a dedicated SAT,
// or a forced link retrain) — only that caller should turn this data
// into a status verdict. LinkSpeed/MaxLinkSpeed above stay populated as
// plain descriptive fields regardless.
return dev
}
@@ -283,6 +295,12 @@ func readPCIStringAttribute(bdf, attribute string) (string, bool) {
// their link state has no operational impact. This covers management endpoints
// (e.g. PCIe switch fabric controllers on HGX baseboards) that the kernel never
// activates but that lspci still reports with link stats.
//
// Not called from the idle collector path (parseLspciDevice) — see the
// comment there. Call this only from a context that just put the device
// under real traffic (a load-bearing SAT, a forced link retrain), so a
// genuine Warning/Critical is never based on an ASPM/power-managed idle
// downclock that a human or SAT would immediately clear anyway.
func applyPCIeLinkSpeedWarning(dev *schema.HardwarePCIeDevice) {
if dev.LinkSpeed == nil || dev.MaxLinkSpeed == nil {
return
@@ -330,6 +348,14 @@ func pcieLinkSpeedRank(gen string) int {
}
func normalizePCILinkSpeed(raw string) string {
return NormalizePCILinkSpeed(raw)
}
// NormalizePCILinkSpeed maps a raw sysfs current_link_speed/max_link_speed
// string (e.g. "16.0 GT/s PCIe") to a "GenN" label. Exported for reuse by
// the platform package's PCIe link-retrain SAT, which needs the same
// mapping outside the idle collector path.
func NormalizePCILinkSpeed(raw string) string {
raw = strings.TrimSpace(strings.ToLower(raw))
switch {
case strings.Contains(raw, "2.5"):