fix(collector): stop pinning GPU PCIe status on an unverified idle reading
NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power saving, and applyPCIeLinkSpeedWarning fired on every idle collector pass regardless - since component-status DB records never downgrade (Record() only ever raises severity), one boot-time idle sample permanently pinned pcie:gpu:nvidia to Warning for the rest of the session even after every load-bearing GPU SAT test passed clean. Two prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag) didn't hold up against this hardware/driver combination - see bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the full history. Rather than add a downgrade path, stop writing an unverified status in the first place: parseLspciDevice no longer calls applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay populated as plain descriptive fields; only a verified-under-load caller may now turn them into a status verdict. Two new SAT targets provide that verified signal: - pcie-link (platform/pcie_link_check.go): forces every enabled PCIe device - not just GPUs - to retrain via the PCIe spec's Link Control "Retrain Link" bit, then compares the negotiated speed against the device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn equivalent load tool. Classifies by PCI class code + vendor ID, not name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into their own component-status keys so a degraded NIC never reads as a GPU fault. - nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives real host<->device traffic via dcgmi diag -r nvbandwidth and resamples link speed immediately after, independent of nvbandwidth's own pass/fail. Both wired into the task queue/webui the same way as nvidia-config (routes, dispatch, priority, Validate page cards, Run All Check SAT). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
7aa276320b
commit
b11018ac5e
@@ -671,6 +671,18 @@ func renderCheck(opts HandlerOptions) string {
|
||||
`<code>nvidia-smi --query-gpu=...</code>, <code>nvidia-smi topo -m</code>, <code>nvidia-smi nvlink -s/-e</code>, <code>nvidia-smi conf-compute -q</code>, <code>dmesg</code>`,
|
||||
`Seconds — read-only query only.`,
|
||||
)) +
|
||||
renderSATCard("pcie-link", "PCIe Link Check", "runSAT('pcie-link')", "", renderValidateCardBody(
|
||||
`Every enabled PCIe device in the machine (GPUs, NICs, RAID/HBA controllers, PCIe switches).`,
|
||||
`Forces every enabled PCIe device to retrain its link (PCIe spec Link Control "Retrain Link" bit) and compares the negotiated speed against the device's own maximum. An idle sysfs reading alone can't tell a real degraded slot/riser from a device that's simply power-managed down at idle (GPUs do this routinely); a forced retrain settles that without needing a device-specific load generator, so this one check covers non-GPU PCIe hardware too, not just GPUs.`,
|
||||
`<code>setpci</code> (Link Control/Link Status registers), <code>lspci</code>`,
|
||||
`Seconds per device — brief link retrain, no sustained traffic.`,
|
||||
)) +
|
||||
renderSATCard("nvidia-pcie-bandwidth", "GPU PCIe Bandwidth", "runSAT('nvidia-pcie-bandwidth')", "", renderValidateCardBody(
|
||||
inv.NVIDIA,
|
||||
`Drives real host<->device traffic across each GPU's PCIe link and resamples link speed immediately after, to confirm the link actually trains to its negotiated maximum under real load — the load-bearing counterpart to PCIe Link Check for GPUs specifically.`,
|
||||
`<code>dcgmi diag -r nvbandwidth</code>, sysfs link-speed resample`,
|
||||
`Depends on nvbandwidth's built-in test duration.`,
|
||||
)) +
|
||||
`</div>
|
||||
<div style="height:1px;background:var(--border);margin:16px 0"></div>
|
||||
<div class="card" style="margin-bottom:16px">
|
||||
@@ -732,7 +744,7 @@ func renderCheck(opts HandlerOptions) string {
|
||||
<script>
|
||||
let satES = null;
|
||||
function satLabels() {
|
||||
return {nvidia:'Check GPU (DCGM L2)', 'nvidia-interconnect':'NVIDIA Interconnect (NCCL)', 'nvidia-bandwidth':'NVIDIA Bandwidth (NVBandwidth)', memory:'Check Memory', storage:'Check Storage', cpu:'Check CPU', amd:'Check AMD GPU', 'amd-mem':'AMD GPU MEM Integrity', 'amd-bandwidth':'AMD GPU MEM Bandwidth', 'nvidia-config':'Check GPU Config & NVLink'};
|
||||
return {nvidia:'Check GPU (DCGM L2)', 'nvidia-interconnect':'NVIDIA Interconnect (NCCL)', 'nvidia-bandwidth':'NVIDIA Bandwidth (NVBandwidth)', memory:'Check Memory', storage:'Check Storage', cpu:'Check CPU', amd:'Check AMD GPU', 'amd-mem':'AMD GPU MEM Integrity', 'amd-bandwidth':'AMD GPU MEM Bandwidth', 'nvidia-config':'Check GPU Config & NVLink', 'pcie-link':'PCIe Link Check', 'nvidia-pcie-bandwidth':'GPU PCIe Bandwidth Check'};
|
||||
}
|
||||
let satNvidiaGPUsPromise = null;
|
||||
function loadSatNvidiaGPUs() {
|
||||
@@ -867,8 +879,8 @@ function runAllCheckSAT() {
|
||||
const status = document.getElementById('sat-all-status');
|
||||
status.textContent = 'Enqueuing...';
|
||||
const nvidiaIndices = satSelectedGPUIndices();
|
||||
const nvidiaAllTargets = ['nvidia', 'nvidia-interconnect', 'nvidia-bandwidth'];
|
||||
const baseTargets = ['cpu', 'memory', 'storage', 'nvidia-config'];
|
||||
const nvidiaAllTargets = ['nvidia', 'nvidia-interconnect', 'nvidia-bandwidth', 'nvidia-pcie-bandwidth'];
|
||||
const baseTargets = ['cpu', 'memory', 'storage', 'nvidia-config', 'pcie-link'];
|
||||
const amdTargets = selectedAMDValidateTargets();
|
||||
const expanded = [];
|
||||
baseTargets.forEach(t => expanded.push({target: t}));
|
||||
|
||||
Reference in New Issue
Block a user