fix(pcie): verify GPU links under real bandwidth load
This commit is contained in:
@@ -45,8 +45,7 @@ which bee component happened to produce it:
|
||||
`UNSUPPORTED`).
|
||||
- **"What's the *current* rolled-up health per component?"** →
|
||||
`status/component-status.json`. One record per component key (e.g.
|
||||
`pcie:gpu:nvidia`, `pcie:link:other`, `cpu:all`, `psu:all`,
|
||||
`sat:pcie-link`), each with a `status` and a
|
||||
`pcie:gpu:nvidia`, `cpu:all`, `psu:all`), each with a `status` and a
|
||||
`history` array. This is what feeds `reanimator.json`'s `status` fields —
|
||||
it's the source, not a duplicate. **The history is a transition log, not a
|
||||
per-check journal** — a component whose status never changed only has one
|
||||
@@ -82,16 +81,15 @@ which bee component happened to produce it:
|
||||
for NVLink). If NVLink is expected, one `<inactive>` lane next to active
|
||||
ones on an otherwise-bonded pair is the real fault signature — a degraded
|
||||
link, not an absent one.
|
||||
- **"PCIe link speeds / retrain check?"** → `tasks/bee-sat/pcie-link-*/`:
|
||||
`summary.txt` (`overall_status`, `gpu_<vendor>_status`, `other_status`,
|
||||
`devices_tested`, `other_devices_degraded`, `warnings`) and
|
||||
`pcie-link-report.txt` (one line per BDF — `OK` / `DEGRADED` /
|
||||
`FELL OFF BUS` / `skipped: <reason>`, with `before`/`after`/`max` speed
|
||||
and width). Both endpoints and the bridges above them are listed; a
|
||||
bridge's `max` is its own port capability capped to what the device
|
||||
downstream of it supports, so `before == after == max` — even with a
|
||||
`(port capability GenX, limited by downstream device)` note — means the
|
||||
link runs at the fastest rate that pair supports, not degraded.
|
||||
- **"NVIDIA PCIe link speed under real traffic?"** ->
|
||||
`tasks/bee-sat/gpu-nvidia-bandwidth-*/summary.txt`: inspect
|
||||
`pcie_link_under_load_status`, `pcie_gpuN_status`, and `pcie_degraded`.
|
||||
`nvidia-pcie-link-under-load-report.txt` records each GPU's idle baseline,
|
||||
post-`nvbandwidth` speed, maximum speed, and negotiated/max width. A Gen1
|
||||
baseline is informational; only a link still below maximum immediately
|
||||
after real traffic fails. For NICs/HBAs and other non-GPU PCIe devices,
|
||||
use the read-only inventory and AER logs; there is no generic link-speed
|
||||
health verdict without a device-specific traffic generator.
|
||||
- **"TPM state / validation?"** → `tasks/bee-sat/tpm-*/summary.txt`
|
||||
(`overall_status` plus per-step `N-<step>_status`); raw dumps in
|
||||
`export/platform/tpm-*.txt`. Read-only — it never writes the TPM.
|
||||
@@ -122,11 +120,10 @@ which bee component happened to produce it:
|
||||
`storcli2-show-all.json`'s `"Number of Controllers"` and `lspci`/
|
||||
`nvme-list.json` for actual storage hardware. Same pattern as
|
||||
`nvlink_pairs_checked=0` above.
|
||||
- `pcie-link-report.txt` bridge line `DEGRADED` with `before == after ==
|
||||
max` and `(port capability GenX, limited by downstream device)`, while
|
||||
the endpoint below it on the same link reads `OK` → link is at the
|
||||
downstream device's native maximum, not a fault. A real bridge fault has
|
||||
`after` below the (already capped) `max`, or `FELL OFF BUS`.
|
||||
- `nvidia-pcie-link-under-load-report.txt` shows `before=Gen1` and
|
||||
`after=max` at full width -> normal GPU idle power management, not a PCIe
|
||||
downgrade. Treat `after < max` or `width < max_width` as a fault only when
|
||||
the paired `nvbandwidth` command actually ran.
|
||||
- `systemctl restart ... timed out` in a `tasks/_services/bee-*.log` →
|
||||
implicates a SAT run only if its timestamp is inside that run's
|
||||
`run_at_utc`/`started_at`–`done_at` window in
|
||||
@@ -209,7 +206,7 @@ current-state answer.
|
||||
|
||||
| Path | What it is |
|
||||
|---|---|
|
||||
| `bee-sat/<target>-<timestamp>/` | One directory per acceptance-test run — `target` is what was tested (`gpu-nvidia`, `cpu`, `memory`, `storage`, `nccl-tests`, `gpu-nvidia-bandwidth`, `nvidia-config` covering GPU config/NVLink/Confidential-Computing readiness, `pcie-link` (whole-bus link-retrain check — see "Fastest answers"), `tpm` (read-only TPM validation), etc). Each contains `summary.txt` (key=value, always has `overall_status`), a full human-readable report (e.g. `storage`'s per-drive `disk-NN-<dev>-report.txt` — a narrative verdict, not raw tool output; the raw data behind it lives in `export/storage/`), numbered per-job logs, and `verbose.log` (every subprocess invocation + exit code — the place to look when a summary doesn't explain *why* something failed). |
|
||||
| `bee-sat/<target>-<timestamp>/` | One directory per acceptance-test run — `target` is what was tested (`gpu-nvidia`, `cpu`, `memory`, `storage`, `nccl-tests`, `gpu-nvidia-bandwidth` including the NVIDIA PCIe link-under-load verdict, `nvidia-config` covering GPU config/NVLink/Confidential-Computing readiness, `tpm` (read-only TPM validation), etc). Each contains `summary.txt` (key=value, always has `overall_status`), a full human-readable report (e.g. `storage`'s per-drive `disk-NN-<dev>-report.txt` — a narrative verdict, not raw tool output; the raw data behind it lives in `export/storage/`), numbered per-job logs, and `verbose.log` (every subprocess invocation + exit code — the place to look when a summary doesn't explain *why* something failed). |
|
||||
| `bee-bench/` | Benchmark run artifacts, if any have been run. |
|
||||
| `<NNN>_<slug>_<done\|failed>/` | One directory per task launched from the web UI's task queue — see "Fastest answers" above. |
|
||||
| `_state/bee-audit.json` | The full hardware-audit document as bee last generated it — same content as `export/reanimator.json`, kept here as bee's own internal record. |
|
||||
|
||||
Reference in New Issue
Block a user