fix(collector): surface PCIe link-speed degradation in component-status DB, sample it under load

pcie:gpu:<vendor> in the component-status DB (what the Hardware
Summary/webui "check passed" status reads) was only ever written by SAT
diag results, and none of the nvidia/nvidia-config/nvidia-interconnect/
nvidia-bandwidth SAT jobs check PCIe link speed. So a real Gen1/Gen4
degradation the collector already flagged as Warning in the hardware
snapshot never reached the DB-backed status, and the audit kept
reporting "OK" despite GPUs training at Gen1 with BMC showing x16.

Add writePCIeGPUStatusesToDB, mirroring the existing PSU write-through,
so RunAudit pushes the collector's PCIe status into the DB alongside
SAT results.

Also add export/gpu/pcie-nvidia-link-under-load.txt to the support
bundle: NVIDIA drivers deliberately downclock PCIe at idle to save
power and re-train to full speed under load, so an idle Gen1 reading
alone can't distinguish real hardware/riser degradation from normal
power management. Resample the same sysfs link attributes while
bee-gpu-burn is actively loading the GPUs so both bundles ship
together.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-08-06 13:51:14 +03:00
co-authored by Claude Sonnet 5
parent 3ec7ca08da
commit b1f165edb3
4 changed files with 130 additions and 1 deletions
+32
View File
@@ -308,6 +308,38 @@ for d in /sys/bus/pci/devices/*/; do
printf " %-22s %s\n" "$f" "$(cat "$d/$f" 2>/dev/null)"
done
done
`}},
// A Gen1-vs-Gen4 link speed reading at idle is ambiguous: NVIDIA drivers
// deliberately downclock PCIe in low power states and re-train to full
// speed under load, so pcie-nvidia-link.txt alone can't tell a real
// hardware/riser fault from normal power management. Re-sample the same
// sysfs attributes while bee-gpu-burn is actively loading the GPUs — if
// the link comes up here, the idle Gen1 reading above was power saving,
// not a fault.
{name: "export/gpu/pcie-nvidia-link-under-load.txt", cmd: []string{"sh", "-c", `
if ! command -v bee-gpu-burn >/dev/null 2>&1; then
echo "bee-gpu-burn not found; cannot sample PCIe link speed under load"
exit 0
fi
bee-gpu-burn --seconds 8 --size-mb 64 >/tmp/bee-pcie-load-burn.log 2>&1 &
burn_pid=$!
sleep 3
for d in /sys/bus/pci/devices/*/; do
vendor=$(cat "$d/vendor" 2>/dev/null)
[ "$vendor" = "0x10de" ] || continue
class=$(cat "$d/class" 2>/dev/null)
case "$class" in
0x030000|0x030200) ;;
*) continue ;;
esac
dev=$(basename "$d")
echo "=== $dev ==="
for f in current_link_speed current_link_width max_link_speed max_link_width; do
printf " %-22s %s\n" "$f" "$(cat "$d/$f" 2>/dev/null)"
done
done
wait "$burn_pid" 2>/dev/null || true
rm -f /tmp/bee-pcie-load-burn.log
`}},
{name: "export/gpu/pcie-aer-sysfs.txt", cmd: []string{"sh", "-c", `
found=0