fix(collector): stop pinning GPU PCIe status on an unverified idle reading
NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power saving, and applyPCIeLinkSpeedWarning fired on every idle collector pass regardless - since component-status DB records never downgrade (Record() only ever raises severity), one boot-time idle sample permanently pinned pcie:gpu:nvidia to Warning for the rest of the session even after every load-bearing GPU SAT test passed clean. Two prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag) didn't hold up against this hardware/driver combination - see bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the full history. Rather than add a downgrade path, stop writing an unverified status in the first place: parseLspciDevice no longer calls applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay populated as plain descriptive fields; only a verified-under-load caller may now turn them into a status verdict. Two new SAT targets provide that verified signal: - pcie-link (platform/pcie_link_check.go): forces every enabled PCIe device - not just GPUs - to retrain via the PCIe spec's Link Control "Retrain Link" bit, then compares the negotiated speed against the device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn equivalent load tool. Classifies by PCI class code + vendor ID, not name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into their own component-status keys so a degraded NIC never reads as a GPU fault. - nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives real host<->device traffic via dcgmi diag -r nvbandwidth and resamples link speed immediately after, independent of nvbandwidth's own pass/fail. Both wired into the task queue/webui the same way as nvidia-config (routes, dispatch, priority, Validate page cards, Run All Check SAT). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
7aa276320b
commit
b11018ac5e
@@ -187,8 +187,20 @@ func parseLspciDevice(fields map[string]string) schema.HardwarePCIeDevice {
|
||||
markNVLinkBridge(&dev)
|
||||
}
|
||||
|
||||
// Warn (or Critical for NVLink bridges) if PCIe link is running below max.
|
||||
applyPCIeLinkSpeedWarning(&dev)
|
||||
// Deliberately not calling applyPCIeLinkSpeedWarning here: this function
|
||||
// runs on every idle-time collector pass, and NVIDIA GPUs legitimately
|
||||
// downclock their PCIe link at idle to save power, making "current <
|
||||
// max" a false positive most of the time it fires. There's no reliable
|
||||
// way to downgrade a status once recorded (component-status DB history
|
||||
// is a one-way transition log, by design — see component_status_db.go),
|
||||
// so writing an unverified idle reading as Warning meant it stuck for
|
||||
// the rest of the session even after every load-bearing SAT test passed
|
||||
// clean. See bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning-unresolved.md.
|
||||
// applyPCIeLinkSpeedWarning is reserved for a future caller that samples
|
||||
// link speed while the device is under verified load (a dedicated SAT,
|
||||
// or a forced link retrain) — only that caller should turn this data
|
||||
// into a status verdict. LinkSpeed/MaxLinkSpeed above stay populated as
|
||||
// plain descriptive fields regardless.
|
||||
|
||||
return dev
|
||||
}
|
||||
@@ -283,6 +295,12 @@ func readPCIStringAttribute(bdf, attribute string) (string, bool) {
|
||||
// their link state has no operational impact. This covers management endpoints
|
||||
// (e.g. PCIe switch fabric controllers on HGX baseboards) that the kernel never
|
||||
// activates but that lspci still reports with link stats.
|
||||
//
|
||||
// Not called from the idle collector path (parseLspciDevice) — see the
|
||||
// comment there. Call this only from a context that just put the device
|
||||
// under real traffic (a load-bearing SAT, a forced link retrain), so a
|
||||
// genuine Warning/Critical is never based on an ASPM/power-managed idle
|
||||
// downclock that a human or SAT would immediately clear anyway.
|
||||
func applyPCIeLinkSpeedWarning(dev *schema.HardwarePCIeDevice) {
|
||||
if dev.LinkSpeed == nil || dev.MaxLinkSpeed == nil {
|
||||
return
|
||||
@@ -330,6 +348,14 @@ func pcieLinkSpeedRank(gen string) int {
|
||||
}
|
||||
|
||||
func normalizePCILinkSpeed(raw string) string {
|
||||
return NormalizePCILinkSpeed(raw)
|
||||
}
|
||||
|
||||
// NormalizePCILinkSpeed maps a raw sysfs current_link_speed/max_link_speed
|
||||
// string (e.g. "16.0 GT/s PCIe") to a "GenN" label. Exported for reuse by
|
||||
// the platform package's PCIe link-retrain SAT, which needs the same
|
||||
// mapping outside the idle collector path.
|
||||
func NormalizePCILinkSpeed(raw string) string {
|
||||
raw = strings.TrimSpace(strings.ToLower(raw))
|
||||
switch {
|
||||
case strings.Contains(raw, "2.5"):
|
||||
|
||||
Reference in New Issue
Block a user