logpile/bible-local/docs/nvidia-bug-report-analysis.md

Analyzing nvidia-bug-report.log.gz — agent instructions

Short checklist for an AI agent (or a human) given a nvidia-bug-report-*.log.gz and asked "what's wrong with this box."

1. Structured inventory pass (LOGPile)

Run it through LOGPile's nvidia_bug_report parser first — it gives board/CPU/memory/ GPU/NIC/PSU inventory as structured JSON (internal/parser/vendors/nvidia_bug_report/). As of this session the decompressor no longer hard-caps at 50MB (streams with a gzip-bomb ratio guard instead — see internal/parser/archive.go), so large dumps come through whole. Known gap: PSU present flag was buggy (fixed), sensors/FRU are legitimately empty for this format (no ipmitool sdr/fru in host-side bug reports).

2. Grep passes on the raw decompressed log

In priority order:

1. Xid (PCI: — GPU error reports. Note all PCI:xxxx addresses and Xid codes, count occurrences, get first/last timestamp per GPU.
2. SXid — NVSwitch (fabric manager) errors on HGX/NVSwitch systems.
3. GPU recovery action changed — tracks a GPU through None → Drain and Reset → GPU Reset Required — a GPU stuck at "Reset Required" for a long span is a live incident, not a transient blip.
4. fell off the bus, Uncorrectable, Double Bit ECC, Row Remap — hardware memory/PCIe failures.
5. Thermal/power: thermal slowdown, power slowdown, Reason: SW\|HW.
6. nvidia-smi -q section per GPU: ECC Errors, Retired Pages, Remapped Rows, GPU Recovery Action — cross-check current (not historical) state at capture time.
7. segfault, kernel panics, Call Trace, MCE/machine-check in dmesg/journalctl excerpts.

Cross-reference: an Xid burst that stops before the nvidia-smi -q snapshot and shows GPU Recovery Action: None + clean ECC counters at the end means the box already self-recovered (e.g. via reboot) — still worth flagging, but not an active fire.

3. Classify each Xid, don't just list it

Look it up in NVIDIA's Xid catalog (link below) before writing a verdict. Same code can mean different things depending on class:

• App-level GPU exception (bad kernel launch, illegal memory access) — usually not hardware-actionable, correlate with the offending pid/process name in the log.
• NVLink/fabric fault (MSE Degraded, link training failures) — check knvlink* messages, Fabric Manager logs — often requires GPU/node reset, can recur → RMA candidate if repeated.
• ECC/memory retirement (uncorrectable DRAM, row-remap failures) — RMA candidate, check Retired Pages / Remapped Rows counts.
• XID 79 (GPU fell off the bus) — drain the node, treat as hardware failure.

4. Escalation bar

Recommend RMA/hardware escalation when: repeated fatal Xid/SXid on the same GPU across multiple days, non-zero uncorrectable ECC or remap failures, or a GPU that required manual reset more than once. Otherwise: flag as informational / correlate with workload.

What already exists online (recipes found 2026-08-26)

• Lambda Labs — check-nvidia-bug-report.sh: an actual automated scanner for this exact file, open source. Clones and runs against the extracted report; flags Xid errors, thermal/power slowdown, segfaults, CPU throttling, "fell off the bus", RmInit failures. This is the closest thing to a ready-made "recipe" — worth using directly or mining its grep patterns. https://github.com/lambdal-support/lambda-public-tools (see check-nvidia-bug-report/) Guide: Lambda Docs — Using the nvidia-bug-report.log file to troubleshoot your system
• NVIDIA — GPU Debug Guidelines: official step-by-step (collect → dcgmi diag → classify Xid → escalate/RMA criteria), the authoritative human-oriented recipe. https://docs.nvidia.com/deploy/gpu-debug-guidelines/index.html
• NVIDIA — Xid Errors reference: the Xid code catalog itself (meanings, no fixed hardware/software verdict per code — context-dependent). https://docs.nvidia.com/deploy/xid-errors/introduction.html
• AWS re:Post — Troubleshoot Xid errors in NVIDIA GPU-accelerated instances: cloud-instance-flavored version of the same workflow. https://repost.aws/knowledge-center/ec2-linux-troubleshoot-xid-errors

No public "AI agent prompt/recipe" specifically for feeding this file to an LLM was found — the closest things are the shell-script scanners above; an agent should treat their grep pattern lists as a starting checklist (§2 above already folds them in).