417 lines
17 KiB
Markdown
417 lines
17 KiB
Markdown
# BEE — Build Plan
|
||
|
||
Hardware audit LiveCD for offline server inventory.
|
||
Produces `HardwareIngestRequest` JSON compatible with core/reanimator.
|
||
|
||
**Principle:** OS-level collection — reads hardware directly, not through BMC.
|
||
Automatic boot audit plus operator console. Boot runs audit immediately, but local/SSH operators can rerun checks through the TUI and CLI.
|
||
Errors are logged and should not block boot on partial collector failures.
|
||
Fills the gaps where logpile/Redfish is blind: NVMe, DIMM serials, GPU serials, physical disks behind RAID, full SMART, NIC firmware.
|
||
|
||
---
|
||
|
||
## Status snapshot (2026-03-14)
|
||
|
||
### Phase 1 — Go Audit Binary
|
||
|
||
- 1.1 Project scaffold — **DONE**
|
||
- 1.2 Board collector — **DONE**
|
||
- 1.3 CPU collector — **DONE**
|
||
- 1.4 Memory collector — **DONE**
|
||
- 1.5 Storage collector — **DONE**
|
||
- 1.6 PCIe collector — **DONE** (with noise filtering for system/chipset devices)
|
||
- 1.7 PSU collector — **DONE (basic FRU path)**
|
||
- 1.8 NVIDIA GPU enrichment — **DONE**
|
||
- 1.8b Component wear / age telemetry — **DONE** (storage + NVMe + NVIDIA + NIC SFP/DOM + NIC packet stats)
|
||
- 1.8c Storage health verdicts — **DONE** (SMART/NVMe warning/failed status derivation)
|
||
- 1.9 Mellanox/NVIDIA NIC enrichment — **DONE** (mstflint + ethtool firmware fallback)
|
||
- 1.10 RAID controller enrichment — **DONE (initial multi-tool support)** (storcli + sas2/3ircu + arcconf + ssacli + VROC/mdstat)
|
||
- 1.11 PSU SDR health — **DONE** (`ipmitool sdr` merged with FRU inventory)
|
||
- 1.11 Output and export workflow — **DONE** (explicit file output + manual removable export via TUI)
|
||
- 1.12 Integration test (local) — **DONE** (`scripts/test-local.sh`)
|
||
|
||
### Phase 2 — Debian Live ISO
|
||
|
||
- Current implementation uses Debian 12 `live-build`, `systemd`, and OpenSSH.
|
||
- Network bring-up on boot — **DONE**
|
||
- Boot services (`bee-network`, `bee-nvidia`, `bee-audit`, `bee-sshsetup`) — **DONE**
|
||
- Local console UX (`bee` autologin on `tty1`, `menu` auto-start, TUI privilege escalation via `sudo -n`) — **DONE**
|
||
- VM/debug support (`qemu-guest-agent`, serial console, virtual GPU initramfs modules) — **DONE**
|
||
- Vendor utilities in overlay — **DONE**
|
||
- Build metadata + staged overlay injection — **DONE**
|
||
- Builder container cache persisted outside container writable layer — **DONE**
|
||
- ISO volume label `BEE` — **DONE**
|
||
- Auto-update flow remains deferred; current focus is deterministic offline audit ISO behavior.
|
||
- Real-hardware validation remains **PENDING**; current validation is limited to local/libvirt VM boot + service checks.
|
||
|
||
---
|
||
|
||
## Phase 1 — Go Audit Binary
|
||
|
||
Self-contained static binary. Runs on any Linux (including the Debian live ISO).
|
||
Calls system utilities, parses their output, produces `HardwareIngestRequest` JSON.
|
||
|
||
### 1.1 — Project scaffold
|
||
|
||
- `audit/go.mod` — module `bee/audit`
|
||
- `audit/cmd/bee/main.go` — main CLI entry point: subcommands, runtime selection, JSON output
|
||
- `audit/internal/schema/` — copy of `HardwareIngestRequest` types from core (no import dependency)
|
||
- `audit/internal/collector/` — empty package stubs for all collectors
|
||
- `const Version = "1.0"` in main
|
||
- Output modes: stdout (default), file path flag `--output /path/to/file.json`
|
||
- Tests: none yet (stubs only)
|
||
|
||
### 1.2 — Board collector
|
||
|
||
Source: `dmidecode -t 0` (BIOS), `-t 1` (System), `-t 2` (Baseboard)
|
||
|
||
Collects:
|
||
- `board.serial_number` — from System Information
|
||
- `board.manufacturer`, `board.product_name` — from System Information
|
||
- `board.part_number` — from Baseboard
|
||
- `board.uuid` — from System Information
|
||
- `firmware[BIOS]` — vendor, version, release date from BIOS Information
|
||
|
||
Tests: table tests with `testdata/dmidecode_*.txt` fixtures
|
||
|
||
### 1.3 — CPU collector
|
||
|
||
Source: `dmidecode -t 4`
|
||
|
||
Collects:
|
||
- socket index, model, manufacturer, status
|
||
- cores, threads, current/max frequency
|
||
- firmware: microcode version from `/sys/devices/system/cpu/cpu0/microcode/version`
|
||
- serial: not available on Intel Xeon → fallback `<board_serial>-CPU-<socket>` (matches core logic)
|
||
|
||
Tests: table tests with dmidecode fixtures
|
||
|
||
### 1.4 — Memory collector
|
||
|
||
Source: `dmidecode -t 17`
|
||
|
||
Collects:
|
||
- slot, location, present flag
|
||
- size_mb, type (DDR4/DDR5), max_speed_mhz, current_speed_mhz
|
||
- manufacturer, serial_number, part_number
|
||
- status from "Data Width" / "No Module Installed" detection
|
||
|
||
Tests: table tests with dmidecode fixtures (populated + empty slots)
|
||
|
||
### 1.5 — Storage collector
|
||
|
||
Sources:
|
||
- `lsblk -J -o NAME,TYPE,SIZE,SERIAL,MODEL,TRAN,MOUNTPOINT` — device enumeration
|
||
- `smartctl -j -i /dev/X` — serial, model, firmware, interface per device
|
||
- `nvme id-ctrl /dev/nvmeX -o json` — NVMe: serial (sn), firmware (fr), model (mn), size
|
||
|
||
Collects per device:
|
||
- type: SSD/HDD/NVMe
|
||
- model, serial_number, manufacturer, firmware
|
||
- size_gb, interface (SATA/SAS/NVMe)
|
||
- slot: from lsblk HCTL where available
|
||
|
||
Tests: table tests with `smartctl -j` JSON fixtures and `nvme id-ctrl` JSON fixtures
|
||
|
||
### 1.6 — PCIe collector
|
||
|
||
Sources:
|
||
- `lspci -vmm -D` — slot, vendor, device, class
|
||
- `lspci -vvv -D` — link width/speed (LnkSta, LnkCap)
|
||
- embedded pci.ids (same submodule as logpile: `third_party/pciids`) — model name lookup
|
||
- `/sys/bus/pci/devices/<bdf>/` — actual negotiated link state from kernel
|
||
|
||
Collects per device:
|
||
- bdf, vendor_id, device_id, device_class
|
||
- manufacturer, model (via pciids lookup if empty)
|
||
- link_width, link_speed, max_link_width, max_link_speed
|
||
- serial_number: device-specific (see per-type enrichment in 1.8, 1.9)
|
||
|
||
Dedup: by serial → bdf (mirrors logpile canonical device repository logic)
|
||
|
||
Tests: table tests with lspci fixtures
|
||
|
||
### 1.7 — PSU collector
|
||
|
||
Source: `ipmitool fru` — primary (only source for PSU data from OS)
|
||
Fallback: `dmidecode -t 39` (System Power Supply, limited availability)
|
||
|
||
Collects:
|
||
- slot, present, model, vendor, serial_number, part_number
|
||
- wattage_w from FRU or dmidecode
|
||
- firmware from ipmitool FRU `Board Extra` fields
|
||
- input_power_w, output_power_w, input_voltage from `ipmitool sdr`
|
||
|
||
Tests: table tests with ipmitool fru text fixtures
|
||
|
||
### 1.8 — NVIDIA GPU enrichment
|
||
|
||
Prerequisite: NVIDIA driver loaded (checked via `nvidia-smi -L` exit code)
|
||
|
||
Sources:
|
||
- `nvidia-smi --query-gpu=index,name,serial,vbios_version,temperature.gpu,power.draw,ecc.errors.uncorrected.aggregate.total --format=csv,noheader,nounits`
|
||
- BDF correlation: `nvidia-smi --query-gpu=index,pci.bus_id --format=csv,noheader` → match to PCIe collector records
|
||
|
||
Enriches PCIe records for NVIDIA devices:
|
||
- `serial_number` — real GPU serial from nvidia-smi
|
||
- `firmware` — VBIOS version
|
||
- `status` — derived from ECC uncorrected errors (0 = OK, >0 = WARNING)
|
||
- telemetry: temperature_c, power_w added to PCIe record attributes
|
||
|
||
Fallback (no driver): PCIe record stays as-is with serial fallback `<board_serial>-PCIE-<slot>`
|
||
|
||
Tests: table tests with nvidia-smi CSV fixtures
|
||
|
||
### 1.8b — Component wear / age telemetry
|
||
|
||
Every component that stores its own usage history must have that data collected and placed in the `attributes` / `telemetry` map of the respective record. This is a cross-cutting concern applied on top of the per-collector steps.
|
||
|
||
**Storage (SATA/SAS) — smartctl:**
|
||
- `Power_On_Hours` (attr 9) — total hours powered on
|
||
- `Power_Cycle_Count` (attr 12)
|
||
- `Reallocated_Sector_Ct` (attr 5) — wear indicator
|
||
- `Wear_Leveling_Count` (attr 177, SSD) — NAND wear
|
||
- `Total_LBAs_Written` (attr 241) — bytes written lifetime
|
||
- `SSD_Life_Left` (attr 231) — % remaining if reported
|
||
- Collected as `telemetry` map keys: `power_on_hours`, `power_cycles`, `reallocated_sectors`, `wear_leveling_pct`, `total_lba_written`, `life_remaining_pct`
|
||
|
||
**NVMe — nvme smart-log:**
|
||
- `power_on_hours` — lifetime hours
|
||
- `power_cycles`
|
||
- `unsafe_shutdowns` — abnormal power loss count
|
||
- `percentage_used` — % of rated lifetime consumed (0–100)
|
||
- `data_units_written` — 512KB units written lifetime
|
||
- `controller_busy_time` — hours controller was busy
|
||
- Collected via `nvme smart-log /dev/nvmeX -o json`
|
||
|
||
**NVIDIA GPU — nvidia-smi:**
|
||
- `ecc.errors.uncorrected.aggregate.total` — lifetime uncorrected ECC errors
|
||
- `ecc.errors.corrected.aggregate.total` — lifetime corrected ECC errors
|
||
- `clocks_throttle_reasons.hw_slowdown` — thermal/power throttle state
|
||
- Stored in PCIe device `telemetry`
|
||
|
||
**NIC SFP/QSFP transceivers — ethtool:**
|
||
- `ethtool -m <iface>` — DOM (Digital Optical Monitoring) if supported
|
||
- Extracts: TX power, RX power, temperature, voltage, bias current
|
||
- Also: `ethtool -i <iface>` → firmware version
|
||
- `ip -s link show <iface>` → tx_packets, rx_packets, tx_errors, rx_errors (uptime proxy)
|
||
- Stored in PCIe device `telemetry`: `sfp_temperature_c`, `sfp_tx_power_dbm`, `sfp_rx_power_dbm`
|
||
|
||
**PSU — ipmitool sdr:**
|
||
- Input/output power readings over time not stored by BMC (point-in-time only)
|
||
- `ipmitool fru` may include manufacture date for age estimation
|
||
- Stored: `input_power_w`, `output_power_w`, `input_voltage` (already in PSU schema)
|
||
|
||
**All wear telemetry placement rules:**
|
||
- Numeric wear indicators go into `telemetry` map (machine-readable, importable by core)
|
||
- Boolean flags (throttle_active, ecc_errors_present) go into `attributes` map
|
||
- Never flatten into named top-level fields not in the schema — use maps
|
||
|
||
Tests: table tests for each SMART parser (JSON fixtures from smartctl/nvme smart-log)
|
||
|
||
### 1.9 — Mellanox/NVIDIA NIC enrichment
|
||
|
||
Source: `mstflint -d <bdf> q` — if mstflint present and device is Mellanox (vendor_id 0x15b3)
|
||
Fallback: `ethtool -i <iface>` — firmware-version field
|
||
|
||
Enriches PCIe/NIC records:
|
||
- `firmware` — from mstflint `FW Version` or ethtool `firmware-version`
|
||
- `serial_number` — from mstflint `Board Serial Number` if available
|
||
|
||
Detection: by PCI vendor_id (0x15b3 = Mellanox/NVIDIA Networking) from PCIe collector
|
||
|
||
Tests: table tests with mstflint output fixtures
|
||
|
||
### 1.10 — RAID controller enrichment
|
||
|
||
Source: tool selected by PCI vendor_id:
|
||
|
||
| PCI vendor_id | Tool | Controller |
|
||
|---|---|---|
|
||
| 0x1000 | `storcli64 /c<n> show all J` | Broadcom MegaRAID |
|
||
| 0x1000 (SAS) | `sas2ircu <n> display` / `sas3ircu <n> display` | LSI SAS 2.x/3.x |
|
||
| 0x9005 | `arcconf getconfig <n>` | Adaptec |
|
||
| 0x103c | `ssacli ctrl slot=<n> pd all show detail` | HPE Smart Array |
|
||
|
||
Collects physical drives behind controller (not visible to OS as block devices):
|
||
- serial_number, model, manufacturer, firmware
|
||
- size_gb, interface (SAS/SATA), slot/bay
|
||
- status (Online/Failed/Rebuilding → OK/CRITICAL/WARNING)
|
||
|
||
No hardcoded vendor names in detection logic — pure PCI vendor_id map.
|
||
|
||
Tests: table tests with storcli/sas2ircu text fixtures
|
||
|
||
### 1.11 — Output and export workflow
|
||
|
||
`--output stdout` (default): pretty-printed JSON to stdout
|
||
`--output file:<path>`: write JSON to explicit path
|
||
|
||
Live ISO default service output: `/var/log/bee-audit.json`
|
||
|
||
Removable-media export is manual via `bee tui` (or the LiveCD wrapper `bee-tui`):
|
||
- operator chooses a removable filesystem explicitly
|
||
- TUI mounts it if needed
|
||
- TUI asks for confirmation before copying the JSON
|
||
- TUI unmounts temporary mountpoints after export
|
||
|
||
No auto-write to arbitrary removable media is allowed.
|
||
|
||
### 1.12 — Integration test (local)
|
||
|
||
`scripts/test-local.sh` — runs `bee audit` on developer machine (Linux), captures JSON,
|
||
validates required fields are present (board.serial_number non-empty, cpus non-empty, etc.)
|
||
|
||
Not a unit test — requires real hardware access. Documents how to run for verification.
|
||
|
||
---
|
||
|
||
## Phase 2 — Debian Live ISO
|
||
|
||
ISO image bootable via BMC virtual media or USB. Runs boot services automatically and writes the audit result to `/var/log/bee-audit.json`.
|
||
|
||
### 2.1 — Builder environment
|
||
|
||
`iso/builder/build-in-container.sh` is the only supported builder entrypoint.
|
||
It builds a Debian 12 builder image with `live-build`, toolchains, and pinned kernel headers,
|
||
then runs the ISO assembly in a privileged container because `live-build` needs
|
||
mount/chroot/loop capabilities.
|
||
|
||
`iso/builder/build.sh` orchestrates the full ISO build:
|
||
1. compile the Go `bee` binary
|
||
2. create a staged overlay under `dist/overlay-stage`
|
||
3. inject SSH auth, vendor tools, NVIDIA artifacts, and build metadata into the staged overlay
|
||
4. create a disposable `live-build` workdir under `dist/live-build-work`
|
||
5. sync the staged overlay into `config/includes.chroot/`
|
||
6. run `lb config && lb build`
|
||
7. copy the final ISO into `dist/`
|
||
|
||
### 2.2 — NVIDIA driver build
|
||
|
||
`iso/builder/build-nvidia-module.sh`:
|
||
- downloads the pinned NVIDIA `.run` installer
|
||
- verifies SHA256
|
||
- builds kernel modules against the pinned Debian kernel ABI
|
||
- caches modules, userspace tools, and libs in `dist/nvidia-<version>-<kver>/`
|
||
|
||
`iso/overlay/usr/local/bin/bee-nvidia-load`:
|
||
- loads `nvidia`, `nvidia-modeset`, `nvidia-uvm` via `insmod`
|
||
- creates `/dev/nvidia*` nodes if the driver registered successfully
|
||
- logs failures but does not block the rest of boot
|
||
|
||
### 2.3 — ISO assembly and overlay policy
|
||
|
||
`iso/overlay/` is source-only input for the build.
|
||
|
||
Build-time files are injected into the staged overlay only:
|
||
- `bee`
|
||
- `bee-smoketest`
|
||
- `authorized_keys`
|
||
- password-fallback marker
|
||
- `/etc/bee-release`
|
||
- vendor tools from `iso/vendor/`
|
||
|
||
The source tree must stay clean after a build.
|
||
|
||
### 2.4 — Boot services
|
||
|
||
`systemd` service order:
|
||
- `bee-sshsetup.service` → configures SSH auth before `ssh.service`
|
||
- `bee-network.service` → starts best-effort DHCP on all physical interfaces
|
||
- `bee-nvidia.service` → loads NVIDIA modules if present
|
||
- `bee-audit.service` → runs audit and logs failures without turning partial collector bugs into a boot blocker
|
||
|
||
### 2.4b — Runtime split
|
||
|
||
Target split:
|
||
- main Go application works on a normal Linux host and on the live ISO
|
||
- live-ISO specifics stay in integration glue under `iso/`
|
||
- the live ISO passes `--runtime livecd` to the Go binary
|
||
- local runs default to `--runtime auto`, which resolves to `local` unless a live marker is detected
|
||
|
||
Planned code shape:
|
||
- `audit/cmd/bee/` — main CLI entrypoint
|
||
- `audit/internal/runtimeenv/` — runtime detection and mode selection
|
||
- future `audit/internal/tui/` — host/live shared TUI logic
|
||
- `iso/overlay/` — boot-time livecd integration only
|
||
|
||
### 2.5 — Operator workflows
|
||
|
||
- Automatic boot audit writes JSON to `/var/log/bee-audit.json`
|
||
- `tty1` autologins into `bee` and auto-runs `menu`
|
||
- `menu` launches the LiveCD wrapper `bee-tui`, which escalates to `root` via `sudo -n`
|
||
- `bee tui` can rerun the audit manually
|
||
- `bee tui` can export the latest audit JSON to removable media
|
||
- `bee tui` can show health summary and run NVIDIA/memory/storage acceptance tests
|
||
- NVIDIA SAT now includes a lightweight in-image GPU stress step via `bee-gpu-stress`
|
||
- SAT summaries now expose `overall_status` plus per-job `OK/FAILED/UNSUPPORTED`
|
||
- Memory/GPU SAT runtime defaults can be overridden via `BEE_MEMTESTER_*` and `BEE_GPU_STRESS_*`
|
||
- removable export requires explicit target selection, mount, confirmation, copy, and cleanup
|
||
|
||
### 2.6 — Vendor utilities and optional assets
|
||
|
||
Optional binaries live in `iso/vendor/` and are included when present:
|
||
- `storcli64`
|
||
- `sas2ircu`, `sas3ircu`
|
||
- `arcconf`
|
||
- `ssacli`
|
||
- `mstflint` (via Debian package set)
|
||
|
||
Missing optional tools do not fail the build or boot.
|
||
|
||
### 2.7 — Release workflow
|
||
|
||
`iso/builder/VERSIONS` pins the current release inputs:
|
||
- audit version
|
||
- Debian version / kernel ABI
|
||
- Go version
|
||
- NVIDIA driver version
|
||
|
||
Current release model:
|
||
- shipping a new ISO means a full rebuild
|
||
- build metadata is embedded into `/etc/bee-release` and `motd`
|
||
- current ISO label is `BEE`
|
||
- binary self-update remains deferred; no automatic USB/network patching is part of the current runtime
|
||
|
||
---
|
||
|
||
## Eating order
|
||
|
||
Builder environment is set up early (after 1.3) so every subsequent collector
|
||
is developed and tested directly on real hardware in the actual Debian live ISO environment.
|
||
No "works on my Mac" drift.
|
||
|
||
```
|
||
1.0 keys repo setup → git.mchus.pro/mchus/keys, keygen.sh, developer pubkeys
|
||
1.1 scaffold + schema types → binary runs, outputs empty JSON
|
||
1.2 board collector → first real data
|
||
1.3 CPU collector → +CPUs
|
||
|
||
--- BUILDER + BEE ISO (unblock real-hardware testing) ---
|
||
|
||
2.1 builder setup → privileged container with build deps
|
||
2.2 debug ISO profile → minimal Debian ISO: `bee` binary + OpenSSH + all packages
|
||
2.3 boot on real server → SSH in, verify packages present, run audit manually
|
||
|
||
--- CONTINUE COLLECTORS (tested on real hardware from here) ---
|
||
|
||
1.4 memory collector → +DIMMs
|
||
1.5 storage collector → +disks (SATA/SAS/NVMe)
|
||
1.6 PCIe collector → +all PCIe devices
|
||
1.7 PSU collector → +power supplies
|
||
1.8 NVIDIA GPU enrichment → +GPU serial/VBIOS
|
||
1.8b wear/age telemetry → +SMART hours, NVMe % used, SFP DOM, ECC
|
||
1.9 Mellanox NIC enrichment → +NIC firmware/serial
|
||
1.10 RAID enrichment → +physical disks behind RAID
|
||
1.11 output + export workflow → file output + explicit removable export
|
||
|
||
--- PRODUCTION ISO ---
|
||
|
||
2.4 NVIDIA driver build → driver compiled into overlay
|
||
2.5 network bring-up on boot → DHCP on all interfaces
|
||
2.6 systemd boot service → audit runs on boot automatically
|
||
2.7 vendor utilities → storcli/sas2ircu/arcconf/ssacli in image
|
||
2.8 release workflow → versioning + release notes
|
||
2.9 operator export flow → explicit TUI export to removable media
|
||
```
|