platform/cmd/webui: add a scriptable test-scenario engine, load scenarios from blackbox USB, GPU status detail inventory fallback

Investigating the CG480-S6053 reboot needed a way to run an ad-hoc load
(nvbandwidth across a specific GPU set) while sampling IPMI/nvidia-smi
telemetry in the background — without hardcoding a one-off test into the
SAT pack code for a single investigation.

- audit/internal/platform/scenario.go: ScenarioSpec/ScenarioJob (JSON,
  no new dependency) + System.RunScenario. "command" jobs run sequential
  or parallel (per-job "parallel" flag); "sampler" jobs run concurrently
  in the background on their own interval until every command job
  finishes or the scenario's timeout elapses. "{{gpus}}" in a command's
  cmd is substituted from that job's gpu_indices. Command jobs are wired
  through the same satJobBoundaryHook/satSyncBracketHook seams the SAT
  job runner uses, so a scenario run gets the same durability treatment
  (evidence that a risky command started/finished reaches blackbox before
  a possible crash, not just whatever streamed to the RAM-backed export
  dir).
- export.go: ReadScenarioFromRemovableMedia mounts each removable target
  looking for scenarios/<name>.json — an air-gapped engineer can author a
  scenario elsewhere, drop it under scenarios/ on the same USB stick
  already plugged in for blackbox, and run it with no network path onto
  the host.
- cmd/bee: new `bee run <file.json|name>` (bare name = looked up on
  removable media); `bee scenario run <arg>` kept as a longer alias.
- scenarios/nvbandwidth-all-gpu-power-watch.json: the scenario that
  reproduced the actual reboot (full nvbandwidth across all GPUs, which
  crashed, vs. clean per-socket passes), with IPMI sensor + GPU power/temp
  sampling for a power-delivery correlation check.

Also: webui/page_topo.go — the /topo page's component-status-detail modal
(GET /api/component-detail/{type}) showed "No status data recorded yet"
for any component type ComponentStatusDB has no history for yet (e.g. GPU
before a SAT run this boot), even though the topology card for the same
component already showed "N OK" from the audit inventory snapshot.
inventoryFallbackRecords now synthesizes records from that same inventory
snapshot when StatusDB is empty, using the same device classifiers
(isGPUDeviceClass etc.) and severity mapping (classifyTopoSeverity) the
topology card itself uses, so the two views never disagree.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-07-28 17:51:45 +03:00
co-authored by Claude Sonnet 5
parent 49979c4da4
commit 20cd317c87
12 changed files with 998 additions and 2 deletions
+41
View File
@@ -0,0 +1,41 @@
# bee test scenarios
A scenario is a plain JSON file describing an ad-hoc diagnostic run: which
commands to execute (sequentially or in parallel) and what to sample in the
background while they run — without hardcoding a new test into bee's own
code. See `audit/internal/platform/scenario.go` (`ParseScenarioJSON`,
`ScenarioSpec`) for the full field reference.
## Running one
```
bee run <path-to-file.json>
```
or, for a scenario file dropped under `scenarios/` on a mounted removable
drive (e.g. the same USB stick already plugged in for blackbox — useful on
an air-gapped host with no other way to get a file onto it):
```
bee run <name> # looks for scenarios/<name>.json on any mounted removable media
```
`bee scenario run <arg>` is the same command under a longer name.
## Files checked in here
- `nvbandwidth-all-gpu-power-watch.json` — the scenario that reproduced the
CG480-S6053 reboot: full `nvbandwidth` across all GPUs at once (the
per-GPU-socket passes alone never reproduced it), with `ipmitool sensor`
and `nvidia-smi` power/temp sampled every 2s in the background so a crash
mid-run still leaves telemetry to check for a power-delivery correlation.
**`gpu_indices` is host-specific** — update it to match the GPU indices
`nvidia-smi -L` actually reports on the box under test before running.
## Adding more
Not every scenario needs to be checked in here. For a one-off test on a
specific host (especially air-gapped), it's simpler to write the JSON file
directly onto the blackbox USB stick under `scenarios/<name>.json` and run
`bee run <name>` — no code change, no rebuild. Check a scenario in here only
when it's worth keeping around as a reusable/named test across hosts.