fix(collector): stop pinning GPU PCIe status on an unverified idle reading
NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power saving, and applyPCIeLinkSpeedWarning fired on every idle collector pass regardless - since component-status DB records never downgrade (Record() only ever raises severity), one boot-time idle sample permanently pinned pcie:gpu:nvidia to Warning for the rest of the session even after every load-bearing GPU SAT test passed clean. Two prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag) didn't hold up against this hardware/driver combination - see bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the full history. Rather than add a downgrade path, stop writing an unverified status in the first place: parseLspciDevice no longer calls applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay populated as plain descriptive fields; only a verified-under-load caller may now turn them into a status verdict. Two new SAT targets provide that verified signal: - pcie-link (platform/pcie_link_check.go): forces every enabled PCIe device - not just GPUs - to retrain via the PCIe spec's Link Control "Retrain Link" bit, then compares the negotiated speed against the device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn equivalent load tool. Classifies by PCI class code + vendor ID, not name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into their own component-status keys so a degraded NIC never reads as a GPU fault. - nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives real host<->device traffic via dcgmi diag -r nvbandwidth and resamples link speed immediately after, independent of nvbandwidth's own pass/fail. Both wired into the task queue/webui the same way as nvidia-config (routes, dispatch, priority, Validate page cards, Run All Check SAT). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
7aa276320b
commit
b11018ac5e
@@ -0,0 +1,230 @@
|
||||
# PCIe Gen1-at-idle GPU warning: history of attempts, and the fix
|
||||
|
||||
**Date:** 2026-08-24
|
||||
**Status:** active — implemented (see "Resolution" at the end)
|
||||
|
||||
## Symptom
|
||||
|
||||
On NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, `reanimator.json` /
|
||||
`status/component-status.json` (`pcie:gpu:nvidia`) report `Warning`:
|
||||
"PCIe link speed degraded: running at Gen1, capable of Gen5" for every GPU in
|
||||
the system, for the entire diagnostic session — even though every functional
|
||||
test that actually exercises the GPUs (dcgmi diag targeted-stress/targeted-power/
|
||||
pulse_test, nvbandwidth, NCCL all_reduce_perf, GPU config check) passes clean,
|
||||
link width stays x16/x16 throughout, and there is no AER/Xid activity in
|
||||
dmesg. Concretely observed in the blackbox `2026-08-13 (BEE-SP v12.84)
|
||||
NF5468-M7-A0-R0-00 28CC05483`.
|
||||
|
||||
This is not a one-off — five separate commits over five months have targeted
|
||||
this exact false-positive-vs-real-fault ambiguity, and it is still not fully
|
||||
solved. This doc is the record of what was tried and why each attempt only
|
||||
closed part of the gap, so we stop re-discovering the same dead ends.
|
||||
|
||||
## Timeline
|
||||
|
||||
| Date | Commit | What it did | Gap it left |
|
||||
|---|---|---|---|
|
||||
| 2026-04-01 | `eb60100` | NVIDIA collector switched `LinkSpeed`/`MaxLinkSpeed` from raw sysfs (`current_link_speed`) to `nvidia-smi --query-gpu=pcie.link.gen.current,...`, on the theory that "the driver knows the negotiated speed regardless of current power state" and sysfs reflects only instantaneous physical state. | **False premise, confirmed wrong by this very bundle.** `export/gpu/nvidia-smi-q.txt` (the actual `nvidia-smi -q` dump, not sysfs) shows `PCIe Generation / Device Current: 1` for every GPU. `nvidia-smi`'s own query reflects the same ASPM/power-managed downshift as sysfs on driver 580.159.03 + Blackwell — switching the data source changed nothing for this failure mode. |
|
||||
| 2026-04-02 | `99cece5` | Added `lspci -vvv` and per-GPU sysfs link files to the **on-demand support bundle** (`export/gpu/`) so a human could manually cross-check `LnkCap`/`LnkSta`. | Diagnostic aid only, no automated verdict. Also — see below — this file only ships when someone clicks "Download Support Bundle" in the web UI; it never reaches a blackbox capture. |
|
||||
| 2026-04-12 | `05c1fde` | Added `applyPCIeLinkSpeedWarning`: sets `Warning` (or `Critical` for NVLink bridges) whenever `LinkSpeed < MaxLinkSpeed`, using whatever `LinkSpeed` was populated by (at that point) `eb60100`'s nvidia-smi values. This is the commit that actually introduced the warning we're chasing. | Compares "current" to "max" as an absolute rule with no allowance for idle power states — this is the root of the false-positive on any idle GPU. |
|
||||
| 2026-04-12 | `4f94ebc` | Same day: added `pcie_aspm=off`, `intel_idle.max_cstate=1`, `processor.max_cstate=1` to the boot kernel command line, as an OS-level attempt to stop links from ever downshifting in the first place. | **Confirmed present but ineffective for this failure mode.** This exact bundle's `dmesg.txt` shows `pcie_aspm=off ... intel_idle.max_cstate=1 processor.max_cstate=1` active in the kernel command line — and the GPUs still trained down to Gen1 at idle anyway. `pcie_aspm=off` disables the platform's standard PCIe ASPM (L0s/L1) link states; it does **not** touch the GPU driver's own independent runtime power management, which downclocks the PCIe link as part of the GPU's P-state machine. Two different mechanisms, only one of which this flag controls. |
|
||||
| 2026-06-12 | `2320925` | Excluded permanently-disabled PCIe devices (e.g. Switchtec fabric-management endpoints on HGX H100 baseboards) from the warning entirely, via `sysfs enable==0`. Documented in `2026-06-12-pcie-disabled-device-link-warning.md`. | Fixed a different false-positive (management-plane chips, not GPUs at idle). Doesn't touch this bug. |
|
||||
| 2026-08-04 | `a34e823` | Fixed `enrichPCIeWithNVIDIAData` unconditionally clobbering `dev.Status` after `applyPCIeLinkSpeedWarning` had already set `Warning` — a later NVIDIA-data enrichment pass was silently downgrading it back to `OK` while leaving the stale `ErrorDescription` behind. Added severity-ordered merge (`OK/Unknown < Warning < Critical`). | Made the `Warning` **stick** reliably in the hardware snapshot (`reanimator.json`) for the first time — before this, it could randomly vanish depending on collector pass ordering. Necessary fix, but it also means the false-positive from `05c1fde` now survives *more* reliably than before. |
|
||||
| 2026-08-06 | `b1f165e` | Two things: (1) `writePCIeGPUStatusesToDB` — pushes the collector's PCIe status into `component-status.json` every audit cycle, because none of the SAT jobs (`nvidia`, `nvidia-config`, `nvidia-interconnect`, `nvidia-bandwidth`) ever checked link speed, so the DB-backed status (what the web UI "Hardware Summary" chip and `status/component-status.json` in every bundle read) had been silently showing `OK` regardless of the collector's own `Warning`. (2) Added `export/gpu/pcie-nvidia-link-under-load.txt` to the **support bundle**: runs `bee-gpu-burn --seconds 8` and resamples the same sysfs link attributes mid-load, so a human/agent can tell idle power-saving from a real degraded slot/riser by comparing it against the idle `pcie-nvidia-link.txt`. | (1) closed the "SAT says OK" mismatch — this is why `status/component-status.json` in the 08-13 bundle correctly shows `Warning` and doesn't get silently overwritten by the string of `sat:nvidia*: OK` entries in its history. (2) is the one piece of tooling actually designed to answer "is this real or just idle" — **and it is absent from this bundle** (see next section). Also: `component_status_db.go`'s `Record()` merge (`newSev > curSev`) has **no downgrade path** — once `Warning` is recorded, nothing (not even a later `audit:pcie` poll reporting the link back at Gen5) can lower it back to `OK` within that DB file's lifetime. Not new in this commit, but this is the mechanism that pins the warning for the rest of the session even if the link genuinely retrains under load later. |
|
||||
| 2026-08-17 | `e3697c0` | Fixed `matchesGPUVendor`/`isGPUDevice` matching any PCIe device with "Controller" in its class string or same-vendor ID as a GPU — same-vendor NICs/NVSwitch bridges could trip the `pcie:gpu:<vendor>` alarm. | Correctness fix for *which* devices get judged, not for the idle-vs-real ambiguity itself. |
|
||||
| 2026-08-24 (today) | `9add561` | Bounded every `support_bundle.go` subprocess (including the `bee-gpu-burn`-driven under-load capture from `b1f165e`) with a timeout, because a genuinely wedged GPU could hang bundle generation forever. | Unrelated to the false-positive itself, but relevant context: confirms the under-load capture is a live, still-evolving code path, not dead code. |
|
||||
|
||||
## Why this specific bundle still shows the confusing state
|
||||
|
||||
Two independent things line up to explain exactly what you're looking at:
|
||||
|
||||
1. **The blackbox (USB auto-sync) capture path never runs `b1f165e`'s
|
||||
disambiguation tooling at all**, regardless of build freshness.
|
||||
`pcie-nvidia-link.txt` / `pcie-nvidia-link-under-load.txt` live in
|
||||
`support_bundle.go`'s `supportBundleCommands`, which only executes inside
|
||||
`BuildSupportBundle` — triggered by a human clicking "Download Support
|
||||
Bundle" in the web UI. The blackbox worker (`blackbox.go`
|
||||
`syncCycle`/`captureSnapshots`) instead calls `categorizeExportTree` on
|
||||
whatever the periodic collector already staged in the live export
|
||||
directory, plus journalctl/dmesg/status snapshots — it does not invoke
|
||||
`supportBundleCommands`. That's also why this bundle has no
|
||||
`manifest.txt`: `blackbox.go:604` notes the blackbox path intentionally
|
||||
skips it, shipping only `README.md` as the "how to read this" doc. Net
|
||||
effect: **the one artifact designed to answer "idle or real fault" for a
|
||||
GPU PCIe warning is structurally unreachable from a blackbox pull.** You
|
||||
would only get it by opening the web UI and downloading a support bundle
|
||||
from that same host while the warning is still active — not useful for a
|
||||
server that already shipped or is already offline.
|
||||
|
||||
2. **Even where the fixes did land, they don't address the actual
|
||||
mechanism.** This bundle's own `dmesg.txt` proves `pcie_aspm=off` and
|
||||
both max_cstate=1 flags from `4f94ebc` were active at boot, and the GPU
|
||||
still trained to Gen1 at idle — and `nvidia-smi-q.txt` proves `eb60100`'s
|
||||
switch to nvidia-smi's own query (instead of sysfs) reports the exact
|
||||
same Gen1 reading. Both mitigations were built on an assumption (ASPM is
|
||||
the mechanism / nvidia-smi reports negotiated capability, not current
|
||||
power state) that this hardware+driver combination (RTX PRO 6000
|
||||
Blackwell Server Edition, driver 580.159.03) disproves. The actual
|
||||
mechanism is the NVIDIA driver's own runtime power management
|
||||
downclocking the link independent of platform ASPM — nothing in the
|
||||
current codebase distinguishes "GPU driver decided to save power" from
|
||||
"riser/slot is actually degraded."
|
||||
|
||||
3. Separately, `component_status_db.go`'s `Record()` has no downgrade path
|
||||
(see `b1f165e` row above): once any source writes `Warning` for
|
||||
`pcie:gpu:nvidia`, that status cannot go back to `OK` within the same
|
||||
session even if a later poll of the *same* source reports the link back
|
||||
at full speed. Combined with #2, a single idle-time sample taken at
|
||||
14:20:05 (before any load test ran) permanently pins the whole GPU
|
||||
subsystem's status for the rest of that ~9-day diagnostic session
|
||||
(14:21 → 08-17 10:03), regardless of how many load tests pass clean in
|
||||
between.
|
||||
|
||||
## Options going forward
|
||||
|
||||
Pick one (or combine):
|
||||
|
||||
1. **Make the under-load resample part of the periodic collector /
|
||||
blackbox path**, not support-bundle-only — e.g. run it once per boot
|
||||
from `bee-audit`/`bee-nvidia` right after the SAT GPU load tests
|
||||
(`nvidia`, `nvidia-bandwidth`, `nvidia-targeted-stress` already load the
|
||||
GPUs for minutes at a time — piggyback the sysfs resample on one of
|
||||
those instead of a dedicated 8s `bee-gpu-burn` run) and write it into
|
||||
the live export tree so blackbox mirrors it automatically.
|
||||
2. **Let a real load test clear the Warning.** Have one of the GPU SAT
|
||||
jobs that's already running a sustained load (`nvidia-bandwidth`,
|
||||
`nvidia-targeted-stress`) resample link speed at the end of its run and
|
||||
call `writePCIeGPUStatusesToDB` with the fresh reading, allowing an
|
||||
explicit downgrade path for this one key when the *same* mechanism that
|
||||
raised the warning reports it resolved under controlled load — rather
|
||||
than opening up `Record()`'s merge logic in general (which is
|
||||
deliberately sticky for a reason: don't want a real intermittent PSU/ECC
|
||||
fault to be silently forgotten because one later poll came back clean).
|
||||
3. **Stop comparing against idle sysfs/nvidia-smi state at all** for
|
||||
NVIDIA GPUs specifically, and instead rely on the SAT bandwidth numbers
|
||||
themselves (nvbandwidth's measured GB/s vs. expected-for-Gen5-x16
|
||||
threshold) as the actual link-health signal — this sidesteps the
|
||||
idle-vs-load ambiguity entirely since it measures the thing you actually
|
||||
care about (does the link deliver Gen5 throughput when asked), not a
|
||||
point-in-time speed field.
|
||||
4. **Leave detection as-is, fix only the messaging**: keep flagging Gen1 at
|
||||
idle as `Warning`, but make the `error_description` explicit that this
|
||||
is unconfirmed/idle-sampled ("Warning: idle PCIe link at Gen1 (max Gen5,
|
||||
unconfirmed under load)") so a human reading `reanimator.json` cold
|
||||
isn't misled into thinking it's a proven hardware fault — closest to a
|
||||
documentation-only fix, cheapest, but keeps the ambiguity forever.
|
||||
|
||||
## Resolution (2026-08-24)
|
||||
|
||||
Landed a variant of options 1–3 that turned out simpler than any of them
|
||||
individually once we stopped trying to make the idle reading recoverable:
|
||||
|
||||
**Stop writing a status from the idle reading at all.** `parseLspciDevice`
|
||||
(`collector/pcie.go`) no longer calls `applyPCIeLinkSpeedWarning` on every
|
||||
idle collector pass — `LinkSpeed`/`MaxLinkSpeed` stay populated as plain
|
||||
descriptive fields in `reanimator.json`, but nothing sets `Status` to
|
||||
`Warning` from them anymore. This makes `Record()`'s one-way severity merge
|
||||
(no downgrade path, see the `b1f165e` row above) a non-issue by
|
||||
construction: if nothing ever writes an unverified `Warning`, there is
|
||||
nothing that later needs downgrading. `applyPCIeLinkSpeedWarning` itself is
|
||||
kept, now documented as reserved for a caller that already put the device
|
||||
under real traffic.
|
||||
|
||||
**Two new verified-load SAT targets replace the idle signal:**
|
||||
|
||||
- `pcie-link` (`platform/pcie_link_check.go`) — forces every *enabled*
|
||||
PCIe device (not just GPUs) to retrain via the PCIe spec's Link Control
|
||||
"Retrain Link" bit (`setpci … CAP_EXP+0x10.w`, poll Link Status bit 11
|
||||
until training clears), then compares the post-retrain negotiated speed
|
||||
against the device's own max. This is the generic answer to "what about
|
||||
non-GPU PCIe cards" from the prior discussion: retraining is a mechanism
|
||||
every PCIe endpoint supports, so one check now covers NICs/HBAs/switches
|
||||
that have no `bee-gpu-burn`-equivalent load tool. Classifies devices by
|
||||
PCI class code (`0x03` = Display) + vendor ID (`0x10de`/`0x1002`), not
|
||||
name substrings, per the existing `no-hardcoded-vendors` contract.
|
||||
Disabled devices (`enable==0`) are left alone, same carve-out as the
|
||||
2026-06-12 decision. Writes `gpu_nvidia_status` / `gpu_amd_status` /
|
||||
`other_status` into `summary.txt`; `ApplySATResultToDB` routes each into
|
||||
its own component key (`pcie:gpu:nvidia`, `pcie:gpu:amd`,
|
||||
`pcie:link:other`) so a degraded NIC is never reported as a GPU fault.
|
||||
- `nvidia-pcie-bandwidth` (`platform/nvidia_pcie_bandwidth.go`) — GPU-only,
|
||||
drives `dcgmi diag -r nvbandwidth` (the same tool `nvidia-bandwidth`
|
||||
already uses for P2P throughput) and resamples each involved GPU's link
|
||||
speed via sysfs immediately after, deliberately independent of
|
||||
nvbandwidth's own GB/s pass/fail — this SAT's verdict is purely "did the
|
||||
link train up to max under real traffic." Feeds `pcie:gpu:nvidia`
|
||||
alongside `pcie-link`.
|
||||
|
||||
Both are wired into the existing task queue/webui exactly like
|
||||
`nvidia-config` (routes in `server.go`, dispatch case in `task_runner.go`,
|
||||
priority in `api.go`'s `defaultTaskPriority`, cards on the Validate "Check"
|
||||
page, included in "Run All Check SAT").
|
||||
|
||||
**Open follow-up, not yet built:** neither target runs automatically today
|
||||
— a Warning only clears/confirms when an operator runs one of these two
|
||||
SATs (or "Run All Check SAT", which now includes `pcie-link`). If that
|
||||
turns out to matter in practice, wire `pcie-link` (it's fast — a retrain is
|
||||
milliseconds per device, not a sustained burn) into `bee-audit`'s periodic
|
||||
cycle.
|
||||
|
||||
## Follow-up (2026-08-24, same day): the original artifact never reached blackbox at all
|
||||
|
||||
Separately from the SAT work above, re-examined why `pcie-nvidia-link.txt` /
|
||||
`pcie-nvidia-link-under-load.txt` (the `b1f165e` diagnostic pair) were
|
||||
missing from the blackbox analyzed earlier in this doc, even on a build new
|
||||
enough to have that code. Root cause, confirmed against the actual pipeline
|
||||
(not build/version drift as first guessed): those two files only ever lived
|
||||
in `supportBundleCommands` (`app/support_bundle.go`), which exclusively
|
||||
runs inside `BuildSupportBundle` — the on-demand "Download Support Bundle"
|
||||
web UI action. The blackbox USB auto-sync worker (`blackbox.go`
|
||||
`syncCycle`) never calls that function; it only mirrors whatever
|
||||
`platform.CaptureTechnicalDump` already wrote into the live export tree at
|
||||
boot (`bee-audit.service`, `Type=oneshot`) via `categorizeExportTree`. So
|
||||
the artifact was structurally unreachable from a blackbox pull no matter
|
||||
how fresh the ISO was — this was the real explanation for the earlier
|
||||
"missing file" mystery, not the stale-build hypothesis floated above.
|
||||
|
||||
**Fix:** moved both scripts (as shared exported constants,
|
||||
`platform.PCIeNvidiaLinkScript` / `PCIeNvidiaLinkUnderLoadScript`, so
|
||||
support-bundle and techdump run byte-identical scripts instead of two
|
||||
copies that can drift) into `platform.techDumpNvidiaCommands`, so
|
||||
`CaptureTechnicalDump` captures them once at boot alongside the existing
|
||||
`nvidia-smi-*` dumps. `bee-audit` is oneshot, so the one-time ~8s
|
||||
`bee-gpu-burn` cost for the under-load sample is a boot-time cost, not a
|
||||
per-blackbox-sync-cycle one. Added both filenames to `techdumpBucketFor`
|
||||
(→ `export/gpu/`) so `categorizeExportTree` buckets them correctly.
|
||||
`supportBundleCommands` still re-runs both on-demand for a fresher sample
|
||||
than the boot-time one — that's intentional, not a duplicate to clean up.
|
||||
|
||||
**Same-pattern audit of the rest of `supportBundleCommands`:** while in
|
||||
there, checked every other `export/*` entry (the ones the README
|
||||
tags as hardware-facing, i.e. everything except `livecd/*`) for the same
|
||||
"only reachable via on-demand support bundle, never blackbox" gap. Also
|
||||
fixed **`export/gpu/kernel-aer-nvidia.txt`** (dmesg filtered for
|
||||
AER/NVRM/Xid lines) the same way — same file, same fix, now
|
||||
`platform.KernelAERNvidiaScript` shared between both paths. This one stood
|
||||
out because it's the exact file I reached for by hand (grepping raw
|
||||
`dmesg.txt` myself) when first triaging the blackbox that started this
|
||||
whole investigation — its absence wasn't hypothetical.
|
||||
|
||||
Found but **not** changed — flagging for a decision, not fixed unilaterally,
|
||||
since promoting more of these turns boot-time audit into a slower fixed
|
||||
cost for every boot, not just support-bundle downloads:
|
||||
- `export/platform/lspci-nn.txt` (`lspci -nn`) — cheap, static, would be a
|
||||
trivial add.
|
||||
- `export/gpu/lspci-video-vv.txt`, `export/gpu/lspci-nvidia-bridges-vv.txt`
|
||||
— cheap-ish, somewhat redundant with `lspci-vvv.txt` (already in
|
||||
techdump) but with NVIDIA-specific bridge-chain framing that's genuinely
|
||||
easier to read.
|
||||
- `export/gpu/systemctl-nvidia-units.txt`, `export/gpu/dcgmi-nvlink-status.txt`,
|
||||
`export/gpu/fabric-manager-paths.txt` — cheap, static.
|
||||
- `export/gpu/nvidia-smi-topo-fresh.txt` / `-nvlink-status-fresh.txt` /
|
||||
`-nvlink-errors-fresh.txt` — deliberately **not** a gap: their entire
|
||||
purpose is being a fresher resample than the boot-time
|
||||
`nvidia-smi-topo.txt` etc. that techdump already captures and blackbox
|
||||
already mirrors; only useful as an on-demand recapture.
|
||||
- `export/gpu/nvidia-bug-report.txt` (30–120s via `nvidia-bug-report.sh`)
|
||||
and the network `export/network/ethtool-*`/`mstflint-query.txt` entries
|
||||
— left alone; meaningfully heavier or more device-count-dependent than
|
||||
the others, worth a deliberate cost/benefit call rather than folding in
|
||||
by default.
|
||||
Reference in New Issue
Block a user