Refactor bee CLI and LiveCD integration

This commit is contained in:
Mikhail Chusavitin
2026-03-13 16:52:16 +03:00
parent b7c888edb1
commit 6aca1682b9
47 changed files with 3137 additions and 1201 deletions
+101 -271
View File
@@ -25,31 +25,29 @@ Fills the gaps where logpile/Redfish is blind: NVMe, DIMM serials, GPU serials,
- 1.8b Component wear / age telemetry — **DONE** (storage + NVMe + NVIDIA + NIC SFP/DOM + NIC packet stats)
- 1.9 Mellanox/NVIDIA NIC enrichment — **DONE** (mstflint + ethtool firmware fallback)
- 1.10 RAID controller enrichment — **DONE (initial multi-tool support)** (storcli + sas2/3ircu + arcconf + ssacli + VROC/mdstat)
- 1.11 Output and USB write**DONE** (usb + /tmp fallback)
- 1.11 Output and export workflow**DONE** (explicit file output + manual removable export via TUI)
- 1.12 Integration test (local) — **DONE** (`scripts/test-local.sh`)
### Phase 2 — Alpine LiveCD
### Phase 2 — Debian Live ISO
- Debug ISO track is active (builder + overlay-debug + OpenRC services + TUI workflow).
- Production ISO track — **IN PROGRESS**.
- 2.3 Alpine mkimage profile — **DONE (production profile scaffold)**
- 2.4 Network bring-up on boot**DONE**
- 2.5 OpenRC boot service (bee-audit) — **DONE** (with explicit bee-nvidia ordering)
- 2.6 Vendor utilities in overlay — **DONE (fetch script + iso/vendor scaffold)**
- 2.7 Auto-update wiring (USB first, network second) — **PARTIAL** (shell flow done; strict Ed25519 verification intentionally deferred to final stage)
- 2.8 Release workflow — **PARTIAL** (production build now injects audit binary, NVIDIA modules/tools, vendor tools, and build metadata)
- Current implementation uses Debian 12 `live-build`, `systemd`, and OpenSSH.
- Network bring-up on boot — **DONE**
- Boot services (`bee-network`, `bee-nvidia`, `bee-audit`, `bee-sshsetup`) — **DONE**
- Vendor utilities in overlay**DONE**
- Build metadata + staged overlay injection — **DONE**
- Auto-update flow remains deferred; current focus is deterministic offline audit ISO behavior.
---
## Phase 1 — Go Audit Binary
Self-contained static binary. Runs on any Linux (including Alpine LiveCD).
Self-contained static binary. Runs on any Linux (including the Debian live ISO).
Calls system utilities, parses their output, produces `HardwareIngestRequest` JSON.
### 1.1 — Project scaffold
- `audit/go.mod` — module `bee/audit`
- `audit/cmd/audit/main.go` — CLI entry point: flags, orchestration, JSON output
- `audit/cmd/bee/main.go` main CLI entry point: subcommands, runtime selection, JSON output
- `audit/internal/schema/` — copy of `HardwareIngestRequest` types from core (no import dependency)
- `audit/internal/collector/` — empty package stubs for all collectors
- `const Version = "1.0"` in main
@@ -237,305 +235,137 @@ No hardcoded vendor names in detection logic — pure PCI vendor_id map.
Tests: table tests with storcli/sas2ircu text fixtures
### 1.11 — Output and USB write
### 1.11 — Output and export workflow
`--output stdout` (default): pretty-printed JSON to stdout
`--output file:<path>`: write JSON to explicit path
`--output usb`: auto-detect first removable block device, mount it, write `audit-<board_serial>-<YYYYMMDD-HHMMSS>.json`
USB detection: scan `/sys/block/*/removable`, pick first `1`, mount to `/tmp/bee-usb`
Live ISO default service output: `/var/log/bee-audit.json`
QR summary to stdout (always): board serial + model + component counts — fits in one QR code
Uses `qrencode` if present, else skips silently
Removable-media export is manual via `bee tui` (or the LiveCD wrapper `bee-tui`):
- operator chooses a removable filesystem explicitly
- TUI mounts it if needed
- TUI asks for confirmation before copying the JSON
- TUI unmounts temporary mountpoints after export
No auto-write to arbitrary removable media is allowed.
### 1.12 — Integration test (local)
`scripts/test-local.sh` — runs audit binary on developer machine (Linux), captures JSON,
`scripts/test-local.sh` — runs `bee audit` on developer machine (Linux), captures JSON,
validates required fields are present (board.serial_number non-empty, cpus non-empty, etc.)
Not a unit test — requires real hardware access. Documents how to run for verification.
---
## Phase 2 — Alpine LiveCD
## Phase 2 — Debian Live ISO
ISO image bootable via BMC virtual media. Runs audit binary automatically on boot.
ISO image bootable via BMC virtual media or USB. Runs boot services automatically and writes the audit result to `/var/log/bee-audit.json`.
### 2.1 — Builder environment
`iso/builder/Dockerfile` — Alpine 3.21 build environment with:
- `alpine-sdk`, `abuild`, `squashfs-tools`, `xorriso`
- Go toolchain (for binary compilation inside builder)
- NVIDIA driver `.run` pre-fetched during image build
`iso/builder/setup-builder.sh` prepares a Debian 12 host/VM with:
- `live-build`, `debootstrap`, bootloader tooling, kernel headers
- Go toolchain
- everything needed to compile the `bee` binary and NVIDIA modules
`iso/builder/build.sh` — orchestrates full ISO build:
1. Compile Go binary (static, `CGO_ENABLED=0`)
2. Compile NVIDIA kernel module against Alpine 3.21 LTS kernel headers
3. Run `mkimage.sh` with bee profile
4. Output: `dist/bee-<version>.iso`
`iso/builder/build-in-container.sh` offers the same builder stack in a Debian 12 container image.
The container run is privileged because `live-build` needs mount/chroot/loop capabilities.
`iso/builder/build.sh` orchestrates the full ISO build:
1. compile the Go `bee` binary
2. create a staged overlay under `dist/overlay-stage`
3. inject SSH auth, vendor tools, NVIDIA artifacts, and build metadata into the staged overlay
4. create a disposable `live-build` workdir under `dist/live-build-work`
5. sync the staged overlay into `config/includes.chroot/`
6. run `lb config && lb build`
7. copy the final ISO into `dist/`
### 2.2 — NVIDIA driver build
Alpine 3.21, LTS kernel 6.6 — fixed versions in builder.
`iso/builder/build-nvidia-module.sh`:
- downloads the pinned NVIDIA `.run` installer
- verifies SHA256
- builds kernel modules against the pinned Debian kernel ABI
- caches modules, userspace tools, and libs in `dist/nvidia-<version>-<kver>/`
`iso/builder/build-nvidia.sh`:
- Download `NVIDIA-Linux-x86_64-<ver>.run` (version pinned in `iso/builder/VERSIONS`)
- Extract kernel module sources
- Compile against `linux-lts-dev` headers
- Strip and package as `nvidia-<ver>-k6.6.ko.tar.gz` for inclusion in overlay
`iso/overlay/usr/local/bin/bee-nvidia-load`:
- loads `nvidia`, `nvidia-modeset`, `nvidia-uvm` via `insmod`
- creates `/dev/nvidia*` nodes if the driver registered successfully
- logs failures but does not block the rest of boot
`iso/overlay/usr/local/bin/load-nvidia.sh`:
- `insmod` sequence: nvidia.ko → nvidia-modeset.ko → nvidia-uvm.ko
- Verify: `nvidia-smi -L` → log result
- On failure: log warning, continue (audit runs without GPU enrichment)
### 2.3 — ISO assembly and overlay policy
### 2.3 — Alpine mkimage profile
`iso/overlay/` is source-only input for the build.
`iso/builder/mkimg.bee.sh` — Alpine mkimage profile:
- Base: `alpine-base`
- Kernel: `linux-lts`
- Packages: `dmidecode smartmontools nvme-cli pciutils ipmitool util-linux e2fsprogs qrencode`
- Overlay: `iso/overlay/` included as apkovl
Build-time files are injected into the staged overlay only:
- `bee`
- `bee-smoketest`
- `authorized_keys`
- password-fallback marker
- `/etc/bee-release`
- vendor tools from `iso/vendor/`
### 2.4 — Network bring-up on boot
The source tree must stay clean after a build.
`iso/overlay/usr/local/bin/bee-network.sh`:
- Enumerate all network interfaces: `ip link show` → filter out loopback and virtual (docker/bridge)
- For each physical interface: `ip link set <iface> up` + `udhcpc -i <iface> -t 5 -T 3 -n`
- Log each interface result (got IP / timeout / no carrier)
- Continue regardless — network is best-effort for auto-update
### 2.4 — Boot services
`iso/overlay/etc/init.d/bee-network`:
- runlevel: default, before: bee-update
- Calls bee-network.sh
- Does not block boot if DHCP fails on all interfaces
`systemd` service order:
- `bee-sshsetup.service` → configures SSH auth before `ssh.service`
- `bee-network.service` → starts best-effort DHCP on all physical interfaces
- `bee-nvidia.service` → loads NVIDIA modules if present
- `bee-audit.service` → runs audit and logs failures without turning partial collector bugs into a boot blocker
### 2.5OpenRC boot service (bee-audit)
### 2.4bRuntime split
`iso/overlay/etc/init.d/bee-audit`:
- runlevel: default, after: bee-update
- start(): load-nvidia.sh → /usr/local/bin/audit --output usb
- on completion: print QR summary to /dev/tty1 (always, even if USB write failed)
- log everything to /var/log/bee-audit.log
- exits 0 regardless of partial failures — unattended, no prompts, no waits
Target split:
- main Go application works on a normal Linux host and on the live ISO
- live-ISO specifics stay in integration glue under `iso/`
- the live ISO passes `--runtime livecd` to the Go binary
- local runs default to `--runtime auto`, which resolves to `local` unless a live marker is detected
Unattended invariants:
- No TTY prompts ever. All decisions are automatic.
- Missing USB: output goes to /tmp/bee-audit-<serial>-<date>.json, QR shown on screen.
- Missing NVIDIA driver: GPU records have status UNKNOWN, audit continues.
- Missing ipmitool/storcli/any tool: that collector is skipped, rest continue.
- Timeout on any external command: 30s hard limit via `timeout` wrapper, then skip.
- Boot never hangs waiting for user input.
Planned code shape:
- `audit/cmd/bee/` — main CLI entrypoint
- `audit/internal/runtimeenv/` — runtime detection and mode selection
- future `audit/internal/tui/` — host/live shared TUI logic
- `iso/overlay/` — boot-time livecd integration only
`iso/overlay/etc/runlevels/default/bee-audit` symlink
### 2.5 — Operator workflows
### 2.6 — Vendor utilities in overlay
- Automatic boot audit writes JSON to `/var/log/bee-audit.json`
- `bee tui` can rerun the audit manually
- `bee tui` can export the latest audit JSON to removable media
- removable export requires explicit target selection, mount, confirmation, copy, and cleanup
`iso/overlay/usr/local/bin/` includes pre-fetched proprietary tools:
- `storcli64` (Broadcom)
- `sas2ircu`, `sas3ircu` (Broadcom/LSI)
- `mstflint` (NVIDIA Networking / Mellanox)
### 2.6 — Vendor utilities and optional assets
`scripts/fetch-vendor.sh` — downloads and places these before ISO build.
Checksums verified. Tools not committed to git — fetched at build time.
Optional binaries live in `iso/vendor/` and are included when present:
- `storcli64`
- `sas2ircu`, `sas3ircu`
- `mstflint`
`iso/vendor/.gitkeep` — placeholder, directory gitignored except .gitkeep
Missing optional tools do not fail the build or boot.
### 2.7 — Auto-update of audit binary (USB + network)
### 2.7 — Release workflow
Two update paths, tried in order on every boot:
`iso/builder/VERSIONS` pins the current release inputs:
- audit version
- Debian version / kernel ABI
- Go version
- NVIDIA driver version
**Path A — USB (no network required, higher priority):**
`bee-update.sh` scans mounted removable media for an update package before checking network.
Looks for: `<usb>/bee-update/bee-audit-linux-amd64` + `<usb>/bee-update/bee-audit-linux-amd64.sha256`
Steps:
1. Find USB mount point (same detection as audit output: `/sys/block/*/removable`)
2. Check for `bee-update/bee-audit-linux-amd64` on the USB root
3. Read version from `bee-update/VERSION` file (plain text, e.g. `1.3`)
4. Compare with running binary version (`/usr/local/bin/audit --version`)
5. If USB version > running: verify SHA256 checksum, replace binary, log update
6. Re-run audit if updated
**Authenticity verification — Ed25519 multi-key trust (stdlib only, no external tools):**
Problem: SHA256 alone does not prevent a crafted attack — an attacker places their binary
and a matching SHA256 next to it. The LiveCD would accept it.
Solution: Ed25519 asymmetric signatures via Go stdlib `crypto/ed25519`.
Multiple developer public keys are supported. A binary update is accepted if its signature
verifies against ANY of the embedded trusted public keys.
This mirrors the SSH authorized_keys model: add a developer → add their public key.
Remove a developer → rebuild without their key.
**Key management — centralized across all projects:**
Public keys live in a dedicated repo at git.mchus.pro/mchus/keys (or similar):
```
keys/
developers/
mchusavitin.pub ← Ed25519 public key, base64, one line
developer2.pub
README.md ← how to generate a key pair
```
Public keys are safe to commit — they are not secret.
Private keys stay on each developer's machine, never committed anywhere.
Key generation (one-time per developer, run locally):
```sh
# scripts/keygen.sh — also lives in the keys repo
openssl genpkey -algorithm ed25519 -out ~/.bee-release.key
openssl pkey -in ~/.bee-release.key -pubout -outform DER \
| tail -c 32 | base64 > mchusavitin.pub
```
**Embedding public keys at release time (not compile time):**
Public keys are injected via `-ldflags` at build time from the keys repo.
The binary does not hardcode keys — they are provided by the release script.
```go
// audit/internal/updater/trust.go
// trustedKeysRaw is injected at build time via -ldflags
// format: base64(key1):base64(key2):...
var trustedKeysRaw string
func trustedKeys() ([]ed25519.PublicKey, error) {
if trustedKeysRaw == "" {
return nil, fmt.Errorf("binary built without trusted keys — updates disabled")
}
var keys []ed25519.PublicKey
for _, enc := range strings.Split(trustedKeysRaw, ":") {
b, err := base64.StdEncoding.DecodeString(strings.TrimSpace(enc))
if err != nil || len(b) != ed25519.PublicKeySize {
return nil, fmt.Errorf("invalid trusted key: %w", err)
}
keys = append(keys, ed25519.PublicKey(b))
}
return keys, nil
}
func verifySignature(binaryPath, sigPath string) error {
keys, err := trustedKeys()
if err != nil {
return err
}
data, _ := os.ReadFile(binaryPath)
sig, _ := os.ReadFile(sigPath) // 64 bytes raw Ed25519 signature
for _, key := range keys {
if ed25519.Verify(key, data, sig) {
return nil // any trusted key accepts → pass
}
}
return fmt.Errorf("signature verification failed: no trusted key matched")
}
```
Release build injects keys:
```sh
# scripts/build-release.sh
KEYS=$(paste -sd: keys/developers/*.pub)
go build -ldflags "-X bee/audit/internal/updater/trust.trustedKeysRaw=${KEYS}" \
-o dist/bee-audit-linux-amd64 ./cmd/audit
```
Signing (release engineer signs with their private key):
```sh
# scripts/sign-release.sh <binary>
openssl pkeyutl -sign -inkey ~/.bee-release.key \
-rawin -in "$1" -out "$1.sig"
```
Binary built without `-ldflags` injection (e.g. local dev build) has `trustedKeysRaw=""`
→ updates are disabled, logged as INFO, audit continues normally.
Update rejected silently (logged as WARNING, audit continues with current binary) if:
- `.sig` file missing
- Signature does not match any trusted key
- `trustedKeysRaw` empty (dev build)
Update package layout on USB:
```
/bee-update/
bee-audit-linux-amd64 ← new binary (also signed with embedded keys)
bee-audit-linux-amd64.sig ← Ed25519 signature (64 bytes raw)
VERSION ← plain version string e.g. "1.3"
```
Admin workflow: download `bee-audit-linux-amd64` + `bee-audit-linux-amd64.sig` from Gitea
release assets, place in `bee-update/` on USB.
**Path B — Network (requires DHCP on at least one interface):**
1. Check network: ping git.mchus.pro -c 1 -W 3 || skip
2. Fetch: `GET https://git.mchus.pro/api/v1/repos/<org>/bee/releases/latest`
3. Parse tag_name, asset URLs for `bee-audit-linux-amd64` + `bee-audit-linux-amd64.sig`
4. Compare tag with running version
5. If newer: download both files to /tmp, verify Ed25519 signature against all trusted keys
6. Replace binary on pass, log and skip on fail
7. Re-run audit if updated
**Ordering:** USB update checked first, network checked second.
If USB update applied and verified, network check is skipped.
`iso/overlay/etc/init.d/bee-update`:
- runlevel: default
- after: bee-network (network path needs interfaces up)
- before: bee-audit (audit runs with latest binary)
- Calls bee-update.sh
Triggered after bee-audit completes, only if network is available.
`iso/overlay/usr/local/bin/bee-update.sh`:
```
1. Check network: ping git.mchus.pro -c 1 -W 3 || exit 0
2. Fetch latest release metadata:
GET https://git.mchus.pro/api/v1/repos/<org>/bee/releases/latest
3. Parse: extract tag_name, asset URL for bee-audit-linux-amd64
4. Compare tag_name with /usr/local/bin/audit --version output
5. If newer: download to /tmp/bee-audit-new, verify SHA256 checksum from release assets
6. Replace /usr/local/bin/audit (tmpfs — survives until reboot)
7. Log: updated from vX.Y to vX.Z
8. Re-run audit if update happened: /usr/local/bin/audit --output usb
```
`iso/overlay/etc/init.d/bee-update`:
- runlevel: default
- after: bee-audit, network
- Calls bee-update.sh
Release naming convention: binary asset named `bee-audit-linux-amd64` per release tag.
### 2.8 — Release workflow
`iso/builder/VERSIONS` — pinned versions:
```
AUDIT_VERSION=1.0
ALPINE_VERSION=3.21
KERNEL_VERSION=6.12
NVIDIA_DRIVER_VERSION=590.48.01
```
LiveCD release = full ISO rebuild. Binary-only patch = new Gitea release with binary asset.
On boot with network: ISO auto-patches its binary without full rebuild.
ISO version embedded in `/etc/bee-release`:
```
BEE_ISO_VERSION=1.0
BEE_AUDIT_VERSION=1.0
BUILD_DATE=2026-03-05
```
Current release model:
- shipping a new ISO means a full rebuild
- build metadata is embedded into `/etc/bee-release` and `motd`
- binary self-update remains deferred; no automatic USB/network patching is part of the current runtime
---
## Eating order
Builder environment is set up early (after 1.3) so every subsequent collector
is developed and tested directly on real hardware in the actual Alpine environment.
is developed and tested directly on real hardware in the actual Debian live ISO environment.
No "works on my Mac" drift.
```
@@ -546,8 +376,8 @@ No "works on my Mac" drift.
--- BUILDER + DEBUG ISO (unblock real-hardware testing) ---
2.1 builder VM setup → Alpine VM with build deps + Go toolchain
2.2 debug ISO profile → minimal Alpine ISO: audit binary + dropbear SSH + all packages
2.1 builder setup → Debian host/VM or privileged container with build deps
2.2 debug ISO profile → minimal Debian ISO: `bee` binary + OpenSSH + all packages
2.3 boot on real server → SSH in, verify packages present, run audit manually
--- CONTINUE COLLECTORS (tested on real hardware from here) ---
@@ -560,14 +390,14 @@ No "works on my Mac" drift.
1.8b wear/age telemetry → +SMART hours, NVMe % used, SFP DOM, ECC
1.9 Mellanox NIC enrichment → +NIC firmware/serial
1.10 RAID enrichment → +physical disks behind RAID
1.11 output + USB write → production-ready output
1.11 output + export workflow → file output + explicit removable export
--- PRODUCTION ISO ---
2.4 NVIDIA driver build → driver compiled into overlay
2.5 network bring-up on boot → DHCP on all interfaces
2.6 OpenRC boot service → audit runs on boot automatically
2.6 systemd boot service → audit runs on boot automatically
2.7 vendor utilities → storcli/sas2ircu/mstflint in image
2.8 auto-update → binary self-patches from Gitea
2.9 release workflow → versioning + release notes
2.8 release workflow → versioning + release notes
2.9 operator export flow → explicit TUI export to removable media
```