# BEE — Build Plan

Hardware audit LiveCD for offline server inventory.
Produces `HardwareIngestRequest` JSON compatible with core/reanimator.

**Principle:** OS-level collection — reads hardware directly, not through BMC.
Automatic boot audit plus operator console. Boot runs audit immediately, but local/SSH operators can rerun checks through the TUI and CLI.
Errors are logged and should not block boot on partial collector failures.
Fills the gaps where logpile/Redfish is blind: NVMe, DIMM serials, GPU serials, physical disks behind RAID, full SMART, NIC firmware.

---

## Status snapshot (2026-03-14)

### Phase 1 — Go Audit Binary

- 1.1 Project scaffold — **DONE**
- 1.2 Board collector — **DONE**
- 1.3 CPU collector — **DONE**
- 1.4 Memory collector — **DONE**
- 1.5 Storage collector — **DONE**
- 1.6 PCIe collector — **DONE** (with noise filtering for system/chipset devices)
- 1.7 PSU collector — **DONE (basic FRU path)**
- 1.8 NVIDIA GPU enrichment — **DONE**
- 1.8b Component wear / age telemetry — **DONE** (storage + NVMe + NVIDIA + NIC SFP/DOM + NIC packet stats)
- 1.8c Storage health verdicts — **DONE** (SMART/NVMe warning/failed status derivation)
- 1.9 Mellanox/NVIDIA NIC enrichment — **DONE** (mstflint + ethtool firmware fallback)
- 1.10 RAID controller enrichment — **DONE (initial multi-tool support)** (storcli + sas2/3ircu + arcconf + ssacli + VROC/mdstat)
- 1.11 PSU SDR health — **DONE** (`ipmitool sdr` merged with FRU inventory)
- 1.11 Output and export workflow — **DONE** (explicit file output + manual removable export via TUI)
- 1.12 Integration test (local) — **DONE** (`scripts/test-local.sh`)

### Phase 2 — Debian Live ISO

- Current implementation uses Debian 12 `live-build`, `systemd`, and OpenSSH.
- Network bring-up on boot — **DONE**
- Boot services (`bee-network`, `bee-nvidia`, `bee-audit`, `bee-sshsetup`) — **DONE**
- Local console UX (`bee` autologin on `tty1`, `menu` auto-start, TUI privilege escalation via `sudo -n`) — **DONE**
- VM/debug support (`qemu-guest-agent`, serial console, virtual GPU initramfs modules) — **DONE**
- Vendor utilities in overlay — **DONE**
- Build metadata + staged overlay injection — **DONE**
- Builder container cache persisted outside container writable layer — **DONE**
- ISO volume label `BEE` — **DONE**
- Auto-update flow remains deferred; current focus is deterministic offline audit ISO behavior.
- Real-hardware validation remains **PENDING**; current validation is limited to local/libvirt VM boot + service checks.

---

## Phase 1 — Go Audit Binary

Self-contained static binary. Runs on any Linux (including the Debian live ISO).
Calls system utilities, parses their output, produces `HardwareIngestRequest` JSON.

### 1.1 — Project scaffold

- `audit/go.mod` — module `bee/audit`
- `audit/cmd/bee/main.go` — main CLI entry point: subcommands, runtime selection, JSON output
- `audit/internal/schema/` — copy of `HardwareIngestRequest` types from core (no import dependency)
- `audit/internal/collector/` — empty package stubs for all collectors
- `const Version = "1.0"` in main
- Output modes: stdout (default), file path flag `--output /path/to/file.json`
- Tests: none yet (stubs only)

### 1.2 — Board collector

Source: `dmidecode -t 0` (BIOS), `-t 1` (System), `-t 2` (Baseboard)

Collects:
- `board.serial_number` — from System Information
- `board.manufacturer`, `board.product_name` — from System Information
- `board.part_number` — from Baseboard
- `board.uuid` — from System Information
- `firmware[BIOS]` — vendor, version, release date from BIOS Information

Tests: table tests with `testdata/dmidecode_*.txt` fixtures

### 1.3 — CPU collector

Source: `dmidecode -t 4`

Collects:
- socket index, model, manufacturer, status
- cores, threads, current/max frequency
- firmware: microcode version from `/sys/devices/system/cpu/cpu0/microcode/version`
- serial: not available on Intel Xeon → fallback `<board_serial>-CPU-<socket>` (matches core logic)

Tests: table tests with dmidecode fixtures

### 1.4 — Memory collector

Source: `dmidecode -t 17`

Collects:
- slot, location, present flag
- size_mb, type (DDR4/DDR5), max_speed_mhz, current_speed_mhz
- manufacturer, serial_number, part_number
- status from "Data Width" / "No Module Installed" detection

Tests: table tests with dmidecode fixtures (populated + empty slots)

### 1.5 — Storage collector

Sources:
- `lsblk -J -o NAME,TYPE,SIZE,SERIAL,MODEL,TRAN,MOUNTPOINT` — device enumeration
- `smartctl -j -i /dev/X` — serial, model, firmware, interface per device
- `nvme id-ctrl /dev/nvmeX -o json` — NVMe: serial (sn), firmware (fr), model (mn), size

Collects per device:
- type: SSD/HDD/NVMe
- model, serial_number, manufacturer, firmware
- size_gb, interface (SATA/SAS/NVMe)
- slot: from lsblk HCTL where available

Tests: table tests with `smartctl -j` JSON fixtures and `nvme id-ctrl` JSON fixtures

### 1.6 — PCIe collector

Sources:
- `lspci -vmm -D` — slot, vendor, device, class
- `lspci -vvv -D` — link width/speed (LnkSta, LnkCap)
- embedded pci.ids (same submodule as logpile: `third_party/pciids`) — model name lookup
- `/sys/bus/pci/devices/<bdf>/` — actual negotiated link state from kernel

Collects per device:
- bdf, vendor_id, device_id, device_class
- manufacturer, model (via pciids lookup if empty)
- link_width, link_speed, max_link_width, max_link_speed
- serial_number: device-specific (see per-type enrichment in 1.8, 1.9)

Dedup: by serial → bdf (mirrors logpile canonical device repository logic)

Tests: table tests with lspci fixtures

### 1.7 — PSU collector

Source: `ipmitool fru` — primary (only source for PSU data from OS)
Fallback: `dmidecode -t 39` (System Power Supply, limited availability)

Collects:
- slot, present, model, vendor, serial_number, part_number
- wattage_w from FRU or dmidecode
- firmware from ipmitool FRU `Board Extra` fields
- input_power_w, output_power_w, input_voltage from `ipmitool sdr`

Tests: table tests with ipmitool fru text fixtures

### 1.8 — NVIDIA GPU enrichment

Prerequisite: NVIDIA driver loaded (checked via `nvidia-smi -L` exit code)

Sources:
- `nvidia-smi --query-gpu=index,name,serial,vbios_version,temperature.gpu,power.draw,ecc.errors.uncorrected.aggregate.total --format=csv,noheader,nounits`
- BDF correlation: `nvidia-smi --query-gpu=index,pci.bus_id --format=csv,noheader` → match to PCIe collector records

Enriches PCIe records for NVIDIA devices:
- `serial_number` — real GPU serial from nvidia-smi
- `firmware` — VBIOS version
- `status` — derived from ECC uncorrected errors (0 = OK, >0 = WARNING)
- telemetry: temperature_c, power_w added to PCIe record attributes

Fallback (no driver): PCIe record stays as-is with serial fallback `<board_serial>-PCIE-<slot>`

Tests: table tests with nvidia-smi CSV fixtures

### 1.8b — Component wear / age telemetry

Every component that stores its own usage history must have that data collected and placed in the `attributes` / `telemetry` map of the respective record. This is a cross-cutting concern applied on top of the per-collector steps.

**Storage (SATA/SAS) — smartctl:**
- `Power_On_Hours` (attr 9) — total hours powered on
- `Power_Cycle_Count` (attr 12)
- `Reallocated_Sector_Ct` (attr 5) — wear indicator
- `Wear_Leveling_Count` (attr 177, SSD) — NAND wear
- `Total_LBAs_Written` (attr 241) — bytes written lifetime
- `SSD_Life_Left` (attr 231) — % remaining if reported
- Collected as `telemetry` map keys: `power_on_hours`, `power_cycles`, `reallocated_sectors`, `wear_leveling_pct`, `total_lba_written`, `life_remaining_pct`

**NVMe — nvme smart-log:**
- `power_on_hours` — lifetime hours
- `power_cycles`
- `unsafe_shutdowns` — abnormal power loss count
- `percentage_used` — % of rated lifetime consumed (0–100)
- `data_units_written` — 512KB units written lifetime
- `controller_busy_time` — hours controller was busy
- Collected via `nvme smart-log /dev/nvmeX -o json`

**NVIDIA GPU — nvidia-smi:**
- `ecc.errors.uncorrected.aggregate.total` — lifetime uncorrected ECC errors
- `ecc.errors.corrected.aggregate.total` — lifetime corrected ECC errors
- `clocks_throttle_reasons.hw_slowdown` — thermal/power throttle state
- Stored in PCIe device `telemetry`

**NIC SFP/QSFP transceivers — ethtool:**
- `ethtool -m <iface>` — DOM (Digital Optical Monitoring) if supported
- Extracts: TX power, RX power, temperature, voltage, bias current
- Also: `ethtool -i <iface>` → firmware version
- `ip -s link show <iface>` → tx_packets, rx_packets, tx_errors, rx_errors (uptime proxy)
- Stored in PCIe device `telemetry`: `sfp_temperature_c`, `sfp_tx_power_dbm`, `sfp_rx_power_dbm`

**PSU — ipmitool sdr:**
- Input/output power readings over time not stored by BMC (point-in-time only)
- `ipmitool fru` may include manufacture date for age estimation
- Stored: `input_power_w`, `output_power_w`, `input_voltage` (already in PSU schema)

**All wear telemetry placement rules:**
- Numeric wear indicators go into `telemetry` map (machine-readable, importable by core)
- Boolean flags (throttle_active, ecc_errors_present) go into `attributes` map
- Never flatten into named top-level fields not in the schema — use maps

Tests: table tests for each SMART parser (JSON fixtures from smartctl/nvme smart-log)

### 1.9 — Mellanox/NVIDIA NIC enrichment

Source: `mstflint -d <bdf> q` — if mstflint present and device is Mellanox (vendor_id 0x15b3)
Fallback: `ethtool -i <iface>` — firmware-version field

Enriches PCIe/NIC records:
- `firmware` — from mstflint `FW Version` or ethtool `firmware-version`
- `serial_number` — from mstflint `Board Serial Number` if available

Detection: by PCI vendor_id (0x15b3 = Mellanox/NVIDIA Networking) from PCIe collector

Tests: table tests with mstflint output fixtures

### 1.10 — RAID controller enrichment

Source: tool selected by PCI vendor_id:

| PCI vendor_id | Tool | Controller |
|---|---|---|
| 0x1000 | `storcli64 /c<n> show all J` | Broadcom MegaRAID |
| 0x1000 (SAS) | `sas2ircu <n> display` / `sas3ircu <n> display` | LSI SAS 2.x/3.x |
| 0x9005 | `arcconf getconfig <n>` | Adaptec |
| 0x103c | `ssacli ctrl slot=<n> pd all show detail` | HPE Smart Array |

Collects physical drives behind controller (not visible to OS as block devices):
- serial_number, model, manufacturer, firmware
- size_gb, interface (SAS/SATA), slot/bay
- status (Online/Failed/Rebuilding → OK/CRITICAL/WARNING)

No hardcoded vendor names in detection logic — pure PCI vendor_id map.

Tests: table tests with storcli/sas2ircu text fixtures

### 1.11 — Output and export workflow

`--output stdout` (default): pretty-printed JSON to stdout
`--output file:<path>`: write JSON to explicit path

Live ISO default service output: `/var/log/bee-audit.json`

Removable-media export is manual via `bee tui` (or the LiveCD wrapper `bee-tui`):
- operator chooses a removable filesystem explicitly
- TUI mounts it if needed
- TUI asks for confirmation before copying the JSON
- TUI unmounts temporary mountpoints after export

No auto-write to arbitrary removable media is allowed.

### 1.12 — Integration test (local)

`scripts/test-local.sh` — runs `bee audit` on developer machine (Linux), captures JSON,
validates required fields are present (board.serial_number non-empty, cpus non-empty, etc.)

Not a unit test — requires real hardware access. Documents how to run for verification.

---

## Phase 2 — Debian Live ISO

ISO image bootable via BMC virtual media or USB. Runs boot services automatically and writes the audit result to `/var/log/bee-audit.json`.

### 2.1 — Builder environment

`iso/builder/build-in-container.sh` is the only supported builder entrypoint.
It builds a Debian 12 builder image with `live-build`, toolchains, and pinned kernel headers,
then runs the ISO assembly in a privileged container because `live-build` needs
mount/chroot/loop capabilities.

`iso/builder/build.sh` orchestrates the full ISO build:
1. compile the Go `bee` binary
2. create a staged overlay under `dist/overlay-stage`
3. inject SSH auth, vendor tools, NVIDIA artifacts, and build metadata into the staged overlay
4. create a disposable `live-build` workdir under `dist/live-build-work`
5. sync the staged overlay into `config/includes.chroot/`
6. run `lb config && lb build`
7. copy the final ISO into `dist/`

### 2.2 — NVIDIA driver build

`iso/builder/build-nvidia-module.sh`:
- downloads the pinned NVIDIA `.run` installer
- verifies SHA256
- builds kernel modules against the pinned Debian kernel ABI
- caches modules, userspace tools, and libs in `dist/nvidia-<version>-<kver>/`

`iso/overlay/usr/local/bin/bee-nvidia-load`:
- loads `nvidia`, `nvidia-modeset`, `nvidia-uvm` via `insmod`
- creates `/dev/nvidia*` nodes if the driver registered successfully
- logs failures but does not block the rest of boot

### 2.3 — ISO assembly and overlay policy

`iso/overlay/` is source-only input for the build.

Build-time files are injected into the staged overlay only:
- `bee`
- `bee-smoketest`
- `authorized_keys`
- password-fallback marker
- `/etc/bee-release`
- vendor tools from `iso/vendor/`

The source tree must stay clean after a build.

### 2.4 — Boot services

`systemd` service order:
- `bee-sshsetup.service` → configures SSH auth before `ssh.service`
- `bee-network.service` → starts best-effort DHCP on all physical interfaces
- `bee-nvidia.service` → loads NVIDIA modules if present
- `bee-audit.service` → runs audit and logs failures without turning partial collector bugs into a boot blocker

### 2.4b — Runtime split

Target split:
- main Go application works on a normal Linux host and on the live ISO
- live-ISO specifics stay in integration glue under `iso/`
- the live ISO passes `--runtime livecd` to the Go binary
- local runs default to `--runtime auto`, which resolves to `local` unless a live marker is detected

Planned code shape:
- `audit/cmd/bee/` — main CLI entrypoint
- `audit/internal/runtimeenv/` — runtime detection and mode selection
- future `audit/internal/tui/` — host/live shared TUI logic
- `iso/overlay/` — boot-time livecd integration only

### 2.5 — Operator workflows

- Automatic boot audit writes JSON to `/var/log/bee-audit.json`
- `tty1` autologins into `bee` and auto-runs `menu`
- `menu` launches the LiveCD wrapper `bee-tui`, which escalates to `root` via `sudo -n`
- `bee tui` can rerun the audit manually
- `bee tui` can export the latest audit JSON to removable media
- `bee tui` can show health summary and run NVIDIA/memory/storage acceptance tests
- NVIDIA SAT now includes a lightweight in-image GPU stress step via `bee-gpu-burn`
- SAT summaries now expose `overall_status` plus per-job `OK/FAILED/UNSUPPORTED`
- Memory SAT runtime defaults can be overridden via `BEE_MEMTESTER_*`
- removable export requires explicit target selection, mount, confirmation, copy, and cleanup

### 2.6 — Vendor utilities and optional assets

Optional binaries live in `iso/vendor/` and are included when present:
- `storcli64`
- `sas2ircu`, `sas3ircu`
- `arcconf`
- `ssacli`
- `mstflint` (via Debian package set)

Missing optional tools do not fail the build or boot.

### 2.7 — Release workflow

`iso/builder/VERSIONS` pins the current release inputs:
- audit version
- Debian version / kernel ABI
- Go version
- NVIDIA driver version

Current release model:
- shipping a new ISO means a full rebuild
- build metadata is embedded into `/etc/bee-release` and `motd`
- current ISO label is `BEE`
- binary self-update remains deferred; no automatic USB/network patching is part of the current runtime

---

## Eating order

Builder environment is set up early (after 1.3) so every subsequent collector
is developed and tested directly on real hardware in the actual Debian live ISO environment.
No "works on my Mac" drift.

```
1.0  keys repo setup               → git.mchus.pro/mchus/keys, keygen.sh, developer pubkeys
1.1  scaffold + schema types       → binary runs, outputs empty JSON
1.2  board collector               → first real data
1.3  CPU collector                 → +CPUs

--- BUILDER + BEE ISO (unblock real-hardware testing) ---

2.1  builder setup                 → privileged container with build deps
2.2  debug ISO profile             → minimal Debian ISO: `bee` binary + OpenSSH + all packages
2.3  boot on real server           → SSH in, verify packages present, run audit manually

--- CONTINUE COLLECTORS (tested on real hardware from here) ---

1.4  memory collector              → +DIMMs
1.5  storage collector             → +disks (SATA/SAS/NVMe)
1.6  PCIe collector                → +all PCIe devices
1.7  PSU collector                 → +power supplies
1.8  NVIDIA GPU enrichment         → +GPU serial/VBIOS
1.8b wear/age telemetry            → +SMART hours, NVMe % used, SFP DOM, ECC
1.9  Mellanox NIC enrichment       → +NIC firmware/serial
1.10 RAID enrichment               → +physical disks behind RAID
1.11 output + export workflow      → file output + explicit removable export

--- PRODUCTION ISO ---

2.4  NVIDIA driver build           → driver compiled into overlay
2.5  network bring-up on boot      → DHCP on all interfaces
2.6  systemd boot service          → audit runs on boot automatically
2.7  vendor utilities              → storcli/sas2ircu/arcconf/ssacli in image
2.8  release workflow              → versioning + release notes
2.9  operator export flow          → explicit TUI export to removable media
```
