fix(collector): surface PCIe link-speed degradation in component-status DB, sample it under load
pcie:gpu:<vendor> in the component-status DB (what the Hardware Summary/webui "check passed" status reads) was only ever written by SAT diag results, and none of the nvidia/nvidia-config/nvidia-interconnect/ nvidia-bandwidth SAT jobs check PCIe link speed. So a real Gen1/Gen4 degradation the collector already flagged as Warning in the hardware snapshot never reached the DB-backed status, and the audit kept reporting "OK" despite GPUs training at Gen1 with BMC showing x16. Add writePCIeGPUStatusesToDB, mirroring the existing PSU write-through, so RunAudit pushes the collector's PCIe status into the DB alongside SAT results. Also add export/gpu/pcie-nvidia-link-under-load.txt to the support bundle: NVIDIA drivers deliberately downclock PCIe at idle to save power and re-train to full speed under load, so an idle Gen1 reading alone can't distinguish real hardware/riser degradation from normal power management. Resample the same sysfs link attributes while bee-gpu-burn is actively loading the GPUs so both bundles ship together. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
3ec7ca08da
commit
b1f165edb3
@@ -308,6 +308,38 @@ for d in /sys/bus/pci/devices/*/; do
|
||||
printf " %-22s %s\n" "$f" "$(cat "$d/$f" 2>/dev/null)"
|
||||
done
|
||||
done
|
||||
`}},
|
||||
// A Gen1-vs-Gen4 link speed reading at idle is ambiguous: NVIDIA drivers
|
||||
// deliberately downclock PCIe in low power states and re-train to full
|
||||
// speed under load, so pcie-nvidia-link.txt alone can't tell a real
|
||||
// hardware/riser fault from normal power management. Re-sample the same
|
||||
// sysfs attributes while bee-gpu-burn is actively loading the GPUs — if
|
||||
// the link comes up here, the idle Gen1 reading above was power saving,
|
||||
// not a fault.
|
||||
{name: "export/gpu/pcie-nvidia-link-under-load.txt", cmd: []string{"sh", "-c", `
|
||||
if ! command -v bee-gpu-burn >/dev/null 2>&1; then
|
||||
echo "bee-gpu-burn not found; cannot sample PCIe link speed under load"
|
||||
exit 0
|
||||
fi
|
||||
bee-gpu-burn --seconds 8 --size-mb 64 >/tmp/bee-pcie-load-burn.log 2>&1 &
|
||||
burn_pid=$!
|
||||
sleep 3
|
||||
for d in /sys/bus/pci/devices/*/; do
|
||||
vendor=$(cat "$d/vendor" 2>/dev/null)
|
||||
[ "$vendor" = "0x10de" ] || continue
|
||||
class=$(cat "$d/class" 2>/dev/null)
|
||||
case "$class" in
|
||||
0x030000|0x030200) ;;
|
||||
*) continue ;;
|
||||
esac
|
||||
dev=$(basename "$d")
|
||||
echo "=== $dev ==="
|
||||
for f in current_link_speed current_link_width max_link_speed max_link_width; do
|
||||
printf " %-22s %s\n" "$f" "$(cat "$d/$f" 2>/dev/null)"
|
||||
done
|
||||
done
|
||||
wait "$burn_pid" 2>/dev/null || true
|
||||
rm -f /tmp/bee-pcie-load-burn.log
|
||||
`}},
|
||||
{name: "export/gpu/pcie-aer-sysfs.txt", cmd: []string{"sh", "-c", `
|
||||
found=0
|
||||
|
||||
Reference in New Issue
Block a user