diff --git a/audit/internal/app/assets/README.md b/audit/internal/app/assets/README.md index 9c1ee42..db8b429 100644 --- a/audit/internal/app/assets/README.md +++ b/audit/internal/app/assets/README.md @@ -45,7 +45,8 @@ which bee component happened to produce it: `UNSUPPORTED`). - **"What's the *current* rolled-up health per component?"** → `status/component-status.json`. One record per component key (e.g. - `pcie:gpu:nvidia`, `cpu:all`, `psu:0`), each with a `status` and a + `pcie:gpu:nvidia`, `pcie:link:other`, `cpu:all`, `psu:all`, + `sat:pcie-link`), each with a `status` and a `history` array. This is what feeds `reanimator.json`'s `status` fields — it's the source, not a duplicate. **The history is a transition log, not a per-check journal** — a component whose status never changed only has one @@ -81,6 +82,19 @@ which bee component happened to produce it: for NVLink). If NVLink is expected, one `` lane next to active ones on an otherwise-bonded pair is the real fault signature — a degraded link, not an absent one. +- **"PCIe link speeds / retrain check?"** → `tasks/bee-sat/pcie-link-*/`: + `summary.txt` (`overall_status`, `gpu__status`, `other_status`, + `devices_tested`, `other_devices_degraded`, `warnings`) and + `pcie-link-report.txt` (one line per BDF — `OK` / `DEGRADED` / + `FELL OFF BUS` / `skipped: `, with `before`/`after`/`max` speed + and width). Both endpoints and the bridges above them are listed; a + bridge's `max` is its own port capability capped to what the device + downstream of it supports, so `before == after == max` — even with a + `(port capability GenX, limited by downstream device)` note — means the + link runs at the fastest rate that pair supports, not degraded. +- **"TPM state / validation?"** → `tasks/bee-sat/tpm-*/summary.txt` + (`overall_status` plus per-step `N-_status`); raw dumps in + `export/platform/tpm-*.txt`. Read-only — it never writes the TPM. - **"What tasks were run from the web UI, in what order, with what result?"** → `tasks/_state/tasks-state.json` is the index (id, target, status, timestamps, paths). Each task also has its own directory @@ -108,6 +122,11 @@ which bee component happened to produce it: `storcli2-show-all.json`'s `"Number of Controllers"` and `lspci`/ `nvme-list.json` for actual storage hardware. Same pattern as `nvlink_pairs_checked=0` above. +- `pcie-link-report.txt` bridge line `DEGRADED` with `before == after == + max` and `(port capability GenX, limited by downstream device)`, while + the endpoint below it on the same link reads `OK` → link is at the + downstream device's native maximum, not a fault. A real bridge fault has + `after` below the (already capped) `max`, or `FELL OFF BUS`. - `systemctl restart ... timed out` in a `tasks/_services/bee-*.log` → implicates a SAT run only if its timestamp is inside that run's `run_at_utc`/`started_at`–`done_at` window in @@ -190,7 +209,7 @@ current-state answer. | Path | What it is | |---|---| -| `bee-sat/-/` | One directory per acceptance-test run — `target` is what was tested (`gpu-nvidia`, `cpu`, `memory`, `storage`, `nccl-tests`, `gpu-nvidia-bandwidth`, `nvidia-config` covering GPU config/NVLink/Confidential-Computing readiness, etc). Each contains `summary.txt` (key=value, always has `overall_status`), a full human-readable report (e.g. `storage`'s per-drive `disk-NN--report.txt` — a narrative verdict, not raw tool output; the raw data behind it lives in `export/storage/`), numbered per-job logs, and `verbose.log` (every subprocess invocation + exit code — the place to look when a summary doesn't explain *why* something failed). | +| `bee-sat/-/` | One directory per acceptance-test run — `target` is what was tested (`gpu-nvidia`, `cpu`, `memory`, `storage`, `nccl-tests`, `gpu-nvidia-bandwidth`, `nvidia-config` covering GPU config/NVLink/Confidential-Computing readiness, `pcie-link` (whole-bus link-retrain check — see "Fastest answers"), `tpm` (read-only TPM validation), etc). Each contains `summary.txt` (key=value, always has `overall_status`), a full human-readable report (e.g. `storage`'s per-drive `disk-NN--report.txt` — a narrative verdict, not raw tool output; the raw data behind it lives in `export/storage/`), numbered per-job logs, and `verbose.log` (every subprocess invocation + exit code — the place to look when a summary doesn't explain *why* something failed). | | `bee-bench/` | Benchmark run artifacts, if any have been run. | | `__/` | One directory per task launched from the web UI's task queue — see "Fastest answers" above. | | `_state/bee-audit.json` | The full hardware-audit document as bee last generated it — same content as `export/reanimator.json`, kept here as bee's own internal record. |