# Analyzing nvidia-bug-report.log.gz — agent instructions

Short checklist for an AI agent (or a human) given a `nvidia-bug-report-*.log.gz` and
asked "what's wrong with this box."

## 1. Structured inventory pass (LOGPile)

Run it through LOGPile's `nvidia_bug_report` parser first — it gives board/CPU/memory/
GPU/NIC/PSU inventory as structured JSON (`internal/parser/vendors/nvidia_bug_report/`).
As of this session the decompressor no longer hard-caps at 50MB (streams with a
gzip-bomb ratio guard instead — see `internal/parser/archive.go`), so large dumps
come through whole. Known gap: PSU `present` flag was buggy (fixed), sensors/FRU are
legitimately empty for this format (no `ipmitool sdr`/`fru` in host-side bug reports).

## 2. Grep passes on the raw decompressed log

In priority order:

1. `Xid (PCI:` — GPU error reports. Note **all** `PCI:xxxx` addresses and Xid codes,
   count occurrences, get first/last timestamp per GPU.
2. `SXid` — NVSwitch (fabric manager) errors on HGX/NVSwitch systems.
3. `GPU recovery action changed` — tracks a GPU through `None → Drain and Reset →
   GPU Reset Required` — a GPU stuck at "Reset Required" for a long span is a live
   incident, not a transient blip.
4. `fell off the bus`, `Uncorrectable`, `Double Bit ECC`, `Row Remap` — hardware
   memory/PCIe failures.
5. Thermal/power: `thermal slowdown`, `power slowdown`, `Reason: SW\|HW`.
6. `nvidia-smi -q` section per GPU: `ECC Errors`, `Retired Pages`, `Remapped Rows`,
   `GPU Recovery Action` — cross-check current (not historical) state at capture time.
7. `segfault`, kernel panics, `Call Trace`, `MCE`/machine-check in dmesg/journalctl
   excerpts.

Cross-reference: an Xid burst that stops before the `nvidia-smi -q` snapshot and shows
`GPU Recovery Action: None` + clean ECC counters at the end means the box already
self-recovered (e.g. via reboot) — still worth flagging, but not an active fire.

## 3. Classify each Xid, don't just list it

Look it up in NVIDIA's Xid catalog (link below) before writing a verdict. Same code
can mean different things depending on class:
- App-level GPU exception (bad kernel launch, illegal memory access) — usually not
  hardware-actionable, correlate with the offending `pid`/process name in the log.
- NVLink/fabric fault (`MSE Degraded`, link training failures) — check
  `knvlink*` messages, Fabric Manager logs — often requires GPU/node reset, can
  recur → RMA candidate if repeated.
- ECC/memory retirement (uncorrectable DRAM, row-remap failures) — RMA candidate,
  check `Retired Pages` / `Remapped Rows` counts.
- `XID 79` (GPU fell off the bus) — drain the node, treat as hardware failure.

## 4. Escalation bar

Recommend RMA/hardware escalation when: repeated fatal Xid/SXid on the same GPU across
multiple days, non-zero uncorrectable ECC or remap failures, or a GPU that required
manual reset more than once. Otherwise: flag as informational / correlate with workload.

## What already exists online (recipes found 2026-08-26)

- **Lambda Labs — `check-nvidia-bug-report.sh`**: an actual automated scanner for this
  exact file, open source. Clones and runs against the extracted report; flags Xid
  errors, thermal/power slowdown, segfaults, CPU throttling, "fell off the bus",
  RmInit failures. This is the closest thing to a ready-made "recipe" — worth using
  directly or mining its grep patterns.
  https://github.com/lambdal-support/lambda-public-tools (see `check-nvidia-bug-report/`)
  Guide: [Lambda Docs — Using the nvidia-bug-report.log file to troubleshoot your system](https://docs.lambda.ai/education/linux-usage/using-the-nvidia-bug-report.log-file-to-troubleshoot-your-system/)
- **NVIDIA — GPU Debug Guidelines**: official step-by-step (collect → `dcgmi diag` →
  classify Xid → escalate/RMA criteria), the authoritative human-oriented recipe.
  https://docs.nvidia.com/deploy/gpu-debug-guidelines/index.html
- **NVIDIA — Xid Errors reference**: the Xid code catalog itself (meanings, no fixed
  hardware/software verdict per code — context-dependent).
  https://docs.nvidia.com/deploy/xid-errors/introduction.html
- **AWS re:Post — Troubleshoot Xid errors in NVIDIA GPU-accelerated instances**:
  cloud-instance-flavored version of the same workflow.
  https://repost.aws/knowledge-center/ec2-linux-troubleshoot-xid-errors

No public "AI agent prompt/recipe" specifically for feeding this file to an LLM was
found — the closest things are the shell-script scanners above; an agent should treat
their grep pattern lists as a starting checklist (§2 above already folds them in).