reanimator/bee
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

platform/app: restructure support bundle/blackbox mirror into device-type export/, status/, tasks/, livecd/

Browse Source 
Replaces the flat export/ + techdump/ + system/ + systemd/ layout (two
same-named-but-different techdump/ dirs, disk reports duplicated between a
SAT run dir and techdump/) with one shared categorizeExportTree used by both
BuildSupportBundle and the blackbox USB mirror:

- export/{cpu,memory,storage,gpu,network,platform}/ — raw vendor-tool
  output grouped by device type, plus export/reanimator.json ready to POST
  to Reanimator's /ingest/hardware endpoint.
- status/ — computed diagnosis (component-status.json, runtime-health.*,
  and metrics.db, previously missing from every bundle entirely).
- tasks/ — bee's own task-run bookkeeping (bee-sat/, bee-bench/, task
  reports, service logs) where duplication with export/ is expected.
- livecd/{gui,host}/ — live-boot/kiosk-session-only diagnostics, kept out
  of the hardware-facing tree.

Drops the redundant disk-report mirror write in RunStorageAcceptancePack
and the now-dead syncDirectoryTree/removeMissingPaths; adds a size+mtime
skip in copyPath so the blackbox mirror doesn't rewrite unchanged files
every cycle. README.md rewritten to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-07-28 14:33:26 +03:00
co-authored by Claude Sonnet 5
parent e03267a72f
commit fa618d5abb
6 changed files with 412 additions and 233 deletions
Download Patch File Download Diff File Expand all files Collapse all files
audit/internal/app/assets/README.md
+121 -63
View File
@@ -12,50 +12,79 @@ export by: collecting a full hardware inventory, running SAT (System
Acceptance Test) checks against CPU/memory/storage/GPU/RAID, capturing raw
diagnostic command output, and logging its own systemd services.
The bundle is organized around **what kind of thing each file is**, not
which bee component happened to produce it:
- **`export/`** — raw, vendor-native tool output, grouped by physical device
type. Nothing in here needs bee context to read; a NIC vendor can be
handed `export/network/` and it means the same thing it would from any
other collector. Also holds `reanimator.json`, a ready-to-ingest hardware
snapshot.
- **`status/`** — *computed* diagnosis (component health verdicts, runtime
status, metrics history), not raw tool output and not a task log.
- **`tasks/`** — bee's own bookkeeping: full per-run logs, orchestration
state, service logs. Expect duplication with `export/` here (e.g. a
storage SAT run's narrative disk report repeats data already in
`export/storage/`) — that's intentional, this is the "show your work" tier.
- **`livecd/`** — only relevant to *this particular live-boot/kiosk session*
(GUI stack, DHCP-assigned IPs, kernel ring buffer for this boot). Not
hardware facts — ignore this entirely when troubleshooting the physical
machine.
## Fastest answers to common questions
- **"What hardware is in this server?"** → `export/bee-audit.json`. The
- **"What hardware is in this server?"** → `export/reanimator.json`. The
master inventory: board, firmware, CPUs, memory, storage, PCIe devices
(GPUs/NICs/RAID controllers), power supplies, sensors, event logs. Every
component has a `status` field (`OK`/`Warning`/`Critical`/`Unknown`).
component has a `status` field (`OK`/`Warning`/`Critical`/`Unknown`). Same
document a `POST /ingest/hardware` to Reanimator expects (see
`bible-local/docs/hardware-ingest-contract.md` in the bee repo).
- **"Did the acceptance tests pass?"** → grep `overall_status=` across every
`export/bee-sat/*/summary.txt`. That key is always present and is the
`tasks/bee-sat/*/summary.txt`. That key is always present and is the
fastest single-line verdict for that run (`OK`, `FAILED`, `PARTIAL`, or
`UNSUPPORTED`).
- **"What's the *current* rolled-up health per component?"** →
`export/component-status.json`. One record per component key (e.g.
`status/component-status.json`. One record per component key (e.g.
`pcie:gpu:nvidia`, `cpu:all`, `psu:0`), each with a `status` and a
`history` array. **The history is a transition log, not a per-check
journal** — a component whose status never changed only has one entry,
even if it was checked hundreds of times. A gap in timestamps does not
mean it stopped being monitored.
- **"Is a specific service healthy, or did it crash/restart-loop?"** →
`systemd/<service>.status.txt` (current `systemctl status`) and
`systemd/<service>.journal.log` (`journalctl -u <service>` for that
service's window). `systemd/combined.journal.log` has everything,
chronological, if you need cross-service correlation.
`history` array. This is what feeds `reanimator.json`'s `status` fields —
it's the source, not a duplicate. **The history is a transition log, not a
per-check journal** — a component whose status never changed only has one
entry, even if it was checked hundreds of times. A gap in timestamps does
not mean it stopped being monitored.
- **"Is a specific bee service healthy, or did it crash/restart-loop?"** →
`tasks/_services/<service>.status.txt` (current `systemctl status`) and
`tasks/_services/<service>.journal.log` (`journalctl -u <service>` for
that service's window). `tasks/_services/combined.journal.log` has
everything, chronological, if you need cross-service correlation. NVIDIA
DCGM/fabric-manager daemon status lives in `export/gpu/` instead (see
below); the display stack's lives in `livecd/gui/`.
- **"What's the RAID/drive state?"** →
`export/techdump/storcli64-drives.json` and `storcli2-show-all.json` (LSI
`export/storage/storcli64-drives.json` and `storcli2-show-all.json` (LSI
controllers) — per-slot state like `JBOD`, `UGood`, `Onln`, `UBad`. A
drive in `JBOD`/`UBad` state cannot join a new virtual disk without first
being converted (`set good force`).
- **"GPU topology / NVLink health?"** →
`export/techdump/nvidia-smi-topo.txt` (`NV#` = bonded NVLink pair, `PIX`/
`export/gpu/nvidia-smi-topo.txt` (`NV#` = bonded NVLink pair, `PIX`/
`NODE`/`SYS` = PCIe-only, no NVLink), `nvidia-smi-nvlink-status.txt`
(per-link active/inactive), `nvidia-smi-nvlink-errors.txt` (replay/
recovery/CRC counters, should be zero), and `bee-sat/nvidia-config-*/summary.txt`'s
`nvlink_pairs_checked` (how many pairs it found to check, not how many
passed). **Not every GPU config has NVLink bridges — check whether this
SKU/order is supposed to have them before calling their absence a fault**
(all-`PIX` topology, empty nvlink-status/errors files, and
`nvlink_pairs_checked=0` together mean "none detected," which is only a
problem if the config calls for NVLink). If NVLink is expected, one
`<inactive>` lane next to active ones on an otherwise-bonded pair is the
real fault signature — a degraded link, not an absent one.
recovery/CRC counters, should be zero), and
`tasks/bee-sat/nvidia-config-*/summary.txt`'s `nvlink_pairs_checked` (how
many pairs it found to check, not how many passed). If `export/gpu/`
also has `nvidia-smi-topo-fresh.txt` / `-nvlink-status-fresh.txt` /
`-nvlink-errors-fresh.txt`, those were recaptured live at bundle-build
time — prefer them over the non-`-fresh` ones if the two disagree, since
the non-`-fresh` files are only as current as the last audit cycle.
**Not every GPU config has NVLink bridges — check whether this SKU/order
is supposed to have them before calling their absence a fault** (all-`PIX`
topology, empty nvlink-status/errors files, and `nvlink_pairs_checked=0`
together mean "none detected," which is only a problem if the config calls
for NVLink). If NVLink is expected, one `<inactive>` lane next to active
ones on an otherwise-bonded pair is the real fault signature — a degraded
link, not an absent one.
- **"What tasks were run from the web UI, in what order, with what
result?"** → `export/tasks-state.json` is the index (id, target, status,
timestamps, paths). Each task also has its own directory
`export/tasks/<NNN>_<slug>_<done|failed>/` with `task.log` (live output),
result?"** → `tasks/_state/tasks-state.json` is the index (id, target,
status, timestamps, paths). Each task also has its own directory
`tasks/<NNN>_<slug>_<done|failed>/` with `task.log` (live output),
`report.json`/`report.html` (rendered result + charts).
- **"What build/version is this, and when was it captured?"** →
`manifest.txt` (`bee_version=`, `generated_at_utc=`) at the top level. If
@@ -63,21 +92,27 @@ diagnostic command output, and logging its own systemd services.
same version, useful for cross-referencing against the bee git repo's
release tags if you're checking whether a specific fix shipped in this
build.
- **"Was there a CPU/memory/power utilization spike around time T?"** →
`status/metrics.db`, a SQLite database (`sys_metrics` table: `ts`,
`cpu_load_pct`, `mem_load_pct`, `power_w`, `power_source`, `power_mode`,
`power_reason`). Open with any SQLite client.
## Signatures
- `ipmitool-sel-time.txt` disagrees with `manifest.txt`'s `generated_at_utc`
by more than minutes → BMC RTC drifted → all `ipmitool-sel.txt` timestamps
unreliable; use record-ID order (hex counter, column 1), not printed date.
- `export/platform/ipmitool-sel-time.txt` disagrees with `manifest.txt`'s
`generated_at_utc` by more than minutes → BMC RTC drifted → all
`ipmitool-sel.txt` timestamps unreliable; use record-ID order (hex
counter, column 1), not printed date.
- `storcli64` → `"Status": "Failure", "Description": "No Controller found"`
→ no legacy MegaRAID controller present, not a drive/RAID fault. Cross-ref
`storcli2-show-all.json`'s `"Number of Controllers"` and `lspci`/
`nvme-list.json` for actual storage hardware. Same pattern as
`nvlink_pairs_checked=0` above.
- `systemctl restart ... timed out` in a `bee-*.log` → implicates a SAT run
only if its timestamp is inside that run's `run_at_utc`/`started_at`–
`done_at` window in `tasks-state.json`; outside that window (e.g. boot
bring-up) it's unrelated.
- `systemctl restart ... timed out` in a `tasks/_services/bee-*.log` →
implicates a SAT run only if its timestamp is inside that run's
`run_at_utc`/`started_at`–`done_at` window in
`tasks/_state/tasks-state.json`; outside that window (e.g. boot bring-up)
it's unrelated.
- SEL sequence `Power Supply Failure detected`/`AC lost` (all PSUs) →
`ACPI Legacy OFF` → `Chassis intrusion` → `Power Button pressed` →
`Legacy ON` → full power-cycle, not a failing PSU. Isolated failure
@@ -89,11 +124,12 @@ diagnostic command output, and logging its own systemd services.
## Diagnosing "it rebooted/crashed during test X"
1. `tasks-state.json` → task stuck `pending`/`running`, or last
`bee-sat/<target>-*/` with no `summary.txt` = check running at crash time.
1. `tasks/_state/tasks-state.json` → task stuck `pending`/`running`, or last
`tasks/bee-sat/<target>-*/` with no `summary.txt` = check running at
crash time.
2. That directory's `verbose.log` → last subprocess with no matching
`finish`/exit-code line = the trigger.
3. `system/dmesg.txt` starting at uptime 0, ending after a few hundred
3. `livecd/host/dmesg.txt` starting at uptime 0, ending after a few hundred
seconds = fresh-boot log captured after an unclean reset (confirms
unclean reboot; does not contain the crash itself).
4. Diff against a step that passed cleanly → isolates what's specific to
@@ -108,43 +144,65 @@ manifest.txt bee_version, host, generated_at_utc, export_dir,
then a flat "path<TAB>size" listing of every file
in this bundle — a quick inventory/sanity check.
README.md this file — read this first.
export/ mirror of the live /appdata/bee/export directory —
export/ raw, vendor-native hardware data, by device type —
see "export/" below.
systemd/ <service>.status.txt + <service>.journal.log per
monitored systemd unit, plus combined.journal.log.
techdump/, system/ raw command output not tied to a specific bee-*
service — see below.
status/ computed diagnosis, not raw data — see below.
tasks/ bee's own task-run bookkeeping — see below.
livecd/ this boot/kiosk session only, not hardware — see below.
```
## `export/` in detail
This is a straight mirror of the live server's export directory, so
everything below also applies when reading a raw `bee export` output
directly (not wrapped in a support-bundle archive).
Raw tool/command output, grouped by the physical device it describes.
Vendor-neutral: nothing here assumes familiarity with bee.
| Path | What it is |
|---|---|
| `bee-audit.json` | The master hardware snapshot (see above). |
| `bee-audit.log` | Log of the hardware-audit collector itself: what it queried, what it skipped and why (e.g. a field unsupported by the current driver — not necessarily an error). |
| `bee-web.log` | Web UI service log. |
| `bee-network.log`, `bee-nvidia.log`, `bee-sshsetup.log`, `bee-selfheal.log`, `bee-blackbox.log`, `bee-hpc-tuning.log` | Per-service startup/runtime logs for the correspondingly-named systemd unit. |
| `reanimator.json` | The master hardware snapshot (see "Fastest answers" above) — ready to `POST /ingest/hardware`. |
| `cpu/` | `lscpu.txt`, `dmidecode-type4.txt` (Processor Information). |
| `memory/` | `dmidecode-type17.txt` (Memory Device, one entry per DIMM). |
| `storage/` | `smartctl-<dev>.json`, `nvme-id-ctrl-<dev>.json`, `nvme-smart-log-<dev>.json`, `nvme-list.json`, `storcli64-drives.json`, `storcli2-show-all.json`, `lsblk.json`. |
| `gpu/` | `nvidia-smi-*.txt/.csv` (state/topology/NVLink; `-fresh` variants are recaptured live at bundle-build time — see "GPU topology" above), `dcgmi-nvlink-status.txt`, `nvidia-bug-report.txt`, `nvidia-dcgm.service`/`nvidia-fabricmanager.service` status+journal, `fabric-manager-paths.txt`, `fabricmanager.log`/`nvlsm.log`, `lspci-nvidia-bridges-vv.txt`, `pcie-nvidia-link.txt`, `pcie-aer-sysfs.txt`, `kernel-aer-nvidia.txt` (AER/NVRM/Xid-filtered dmesg), `lspci-video-vv.txt`, `systemctl-nvidia-units.txt`. AMD systems get `rocm-smi*.txt` here instead. |
| `network/` | `ethtool-{info,link,module}.txt` (per-NIC), `mstflint-query.txt` (Mellanox/NVIDIA NICs). |
| `platform/` | `dmidecode-type{0,1,2}.txt` (BIOS/System/Baseboard), `ipmitool-{fru,sdr,sensor,sel,sel-time}.txt` (BMC), `sensors.json`, `lspci-{nn,vmm,vvv}.txt`. |
## `status/` in detail
Computed diagnosis, not raw data — see "Fastest answers" above for
`component-status.json` and `metrics.db`.
| Path | What it is |
|---|---|
| `component-status.json` | Current rolled-up component health, written by bee's live `/dev/kmsg` watcher and by SAT run results. Source of `reanimator.json`'s `status`/`status_history` fields. |
| `runtime-health.json` / `.log` | A lighter, more frequent health snapshot than the full audit — good for "was it fine 5 minutes ago" without the cost of a full re-audit. |
| `component-status.json` | Current rolled-up component health — see "Fastest answers" above. |
| `blackbox-state.json` | Present if continuous blackbox capture (mirroring this export dir to removable media on a schedule) is or was active; tracks the sync target and last successful sync. |
| `techdump/` | Raw diagnostic command output, captured once per audit cycle. This is ground truth: `nvidia-smi-*.txt/.csv` (GPU state/topology/NVLink), `storcli64-drives.json` / `storcli2-show-all.json` (RAID), `lspci-*.txt`, `lscpu.txt`, `lsblk.json`, `dmidecode-*.txt`, `ipmitool-*.txt` (BMC sensors/SEL/FRU), `smartctl-*.json`, `nvme-list.json`, `sensors.json`. Higher-level views (the audit JSON, the web UI's topology page) are built from these. |
| `bee-sat/<target>-<timestamp>/` | One directory per acceptance-test run — `target` is what was tested (`gpu-nvidia`, `cpu`, `memory`, `storage`, `nccl-tests`, `gpu-nvidia-bandwidth`, `nvidia-config` covering GPU config/NVLink/Confidential-Computing readiness, etc). Each contains `summary.txt` (key=value, always has `overall_status`), a full human-readable report, numbered per-job logs, and `verbose.log` (every subprocess invocation + exit code — the place to look when a summary doesn't explain *why* something failed). |
| `tasks/<NNN>_<slug>_<done\|failed>/` | One directory per task launched from the web UI's task queue — see "Fastest answers" above. |
| `tasks-state.json` | Index of every task (id, target, status, timestamps, artifact paths). |
| `systemd/` (nested) | Historical per-service snapshots captured as part of an audit cycle, same shape as the top-level `systemd/` described below. |
| `metrics.db` | SQLite time-series (CPU/mem load, power draw/source/mode) — see "Fastest answers" above. |
## `systemd/` and `system/` (top level, sibling of `export/`)
## `tasks/` in detail
- `systemd/<service>.status.txt`, `systemd/<service>.journal.log`,
`systemd/combined.journal.log` — captured fresh at bundle-build time (not
mirrored from `export/`), so this is the most current service state.
- `system/` — general OS-level diagnostics not specific to bee:
`dmesg.txt`, X server / display-manager logs. Mostly relevant to
physical/console access issues, not hardware health.
bee's own bookkeeping. Duplication with `export/` is expected here — this
tier keeps the full story of a specific run, `export/` keeps the clean
current-state answer.
| Path | What it is |
|---|---|
| `bee-sat/<target>-<timestamp>/` | One directory per acceptance-test run — `target` is what was tested (`gpu-nvidia`, `cpu`, `memory`, `storage`, `nccl-tests`, `gpu-nvidia-bandwidth`, `nvidia-config` covering GPU config/NVLink/Confidential-Computing readiness, etc). Each contains `summary.txt` (key=value, always has `overall_status`), a full human-readable report (e.g. `storage`'s per-drive `disk-NN-<dev>-report.txt` — a narrative verdict, not raw tool output; the raw data behind it lives in `export/storage/`), numbered per-job logs, and `verbose.log` (every subprocess invocation + exit code — the place to look when a summary doesn't explain *why* something failed). |
| `bee-bench/` | Benchmark run artifacts, if any have been run. |
| `<NNN>_<slug>_<done\|failed>/` | One directory per task launched from the web UI's task queue — see "Fastest answers" above. |
| `_state/bee-audit.json` | The full hardware-audit document as bee last generated it — same content as `export/reanimator.json`, kept here as bee's own internal record. |
| `_state/tasks-state.json` | Index of every task (id, target, status, timestamps, artifact paths). |
| `_state/blackbox-state.json` | Present if continuous blackbox capture (mirroring the export dir to removable media on a schedule) is or was active; tracks the sync target and last successful sync. |
| `_services/bee-*.log` | Per-service startup/runtime logs for each bee-owned systemd unit. |
| `_services/<service>.status.txt`, `.journal.log`, `combined.journal.log` | Captured fresh at bundle-build time, for every bee-owned service (`bee-audit`, `bee-web`, `bee-network`, `bee-nvidia`, `bee-preflight`, `bee-selfheal`, `bee-sshsetup`, `bee-blackbox`). |
## `livecd/` in detail
Only relevant to this particular live-boot/kiosk session — not the physical
hardware. Safe to ignore entirely when troubleshooting hardware.
| Path | What it is |
|---|---|
| `gui/` | `Xorg.0.log`, `lightdm/`, `xinput-list.txt`, `libinput-list-devices.txt`, `drm-cards.txt`, `proc-fb.txt`, `input-devices.txt`, `udevadm-input.txt`, `loginctl-{sessions,seats}.txt`, `ps-gui.txt`, `systemctl-gui-units.txt`, `home-bee-xsession-errors.log`, `home-bee-chromium-debug.log`, `dmesg-gui-video-input.txt` (GUI/video/input-filtered dmesg), `display-manager.service`/`lightdm.service` status+journal. |
| `host/` | `uname.txt`, `cmdline.txt`, `lsmod.txt`, `mount.txt`, `df-h.txt`, `dmesg.txt` (full kernel ring buffer for this boot), `ip-{addr,link,link-stats,route}.txt` (this session's network config, e.g. DHCP-assigned addresses — not a hardware fact about the NIC itself; NIC hardware/firmware state is in `export/network/`). |
## Timestamps