Commit Graph
3 Commits
Author SHA1 Message Date
Mikhail Chusavitin 2354599889 fix(pcie): account for downstream link capability 2026-08-28 09:12:02 +03:00
Mikhail ChusavitinandClaude Sonnet 5 9e466b8a70 fix(pcie): stop misreporting empty/vanished slots as speed degradation
pcie-link's retrain check compared post-retrain speed against a port's
max capability without checking whether anything was actually seated
downstream. A switch downstream port with no card plugged in retrains
to zero lanes but still reports a reset-state speed, so it was
misreported as "retrained to Gen1, capable of Gen5" — indistinguishable
from a real degraded link.

Now a port that was already empty before the retrain is skipped (same
treatment as a disabled device: legitimately unpopulated slots are
common and shouldn't fail the check), while a device that answered
before the retrain and disappeared right after it still fails, with an
honest "no device detected downstream" message instead of a fabricated
speed reading.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-26 12:44:57 +03:00
Mikhail ChusavitinandClaude Sonnet 5 b11018ac5e fix(collector): stop pinning GPU PCIe status on an unverified idle reading
NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power
saving, and applyPCIeLinkSpeedWarning fired on every idle collector
pass regardless - since component-status DB records never downgrade
(Record() only ever raises severity), one boot-time idle sample
permanently pinned pcie:gpu:nvidia to Warning for the rest of the
session even after every load-bearing GPU SAT test passed clean. Two
prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag)
didn't hold up against this hardware/driver combination - see
bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the
full history.

Rather than add a downgrade path, stop writing an unverified status in
the first place: parseLspciDevice no longer calls
applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay
populated as plain descriptive fields; only a verified-under-load
caller may now turn them into a status verdict.

Two new SAT targets provide that verified signal:

- pcie-link (platform/pcie_link_check.go): forces every enabled PCIe
  device - not just GPUs - to retrain via the PCIe spec's Link Control
  "Retrain Link" bit, then compares the negotiated speed against the
  device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn
  equivalent load tool. Classifies by PCI class code + vendor ID, not
  name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into
  their own component-status keys so a degraded NIC never reads as a
  GPU fault.
- nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives
  real host<->device traffic via dcgmi diag -r nvbandwidth and
  resamples link speed immediately after, independent of nvbandwidth's
  own pass/fail.

Both wired into the task queue/webui the same way as nvidia-config
(routes, dispatch, priority, Validate page cards, Run All Check SAT).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-24 18:39:19 +03:00