bee/bible-local/architecture/runtime-flows.md

Runtime Flows — bee

Network isolation — CRITICAL

The live CD runs in an isolated network segment with no internet access. All binaries, kernel modules, and tools must be baked into the ISO at build time. No package installation, no downloads, and no package manager calls are allowed at boot. DHCP is used only for LAN (operator SSH access). Internet is NOT available.

Boot sequence (single ISO)

The live system is expected to boot with toram, so live-boot copies the full read-only medium into RAM before mounting the root filesystem. After that point, runtime must not depend on the original USB/BMC virtual media staying readable.

systemd boot order:

local-fs.target
  ├── bee-sshsetup.service   (enables SSH key auth; password fallback only if marker exists)
  │     └── ssh.service      (OpenSSH on port 22 — starts without network)
  ├── bee-network.service    (starts `dhclient -nw` on all physical interfaces, non-blocking)
  ├── bee-nvidia.service     (insmod nvidia*.ko from /usr/local/lib/nvidia/,
  │                           creates /dev/nvidia* nodes)
  ├── bee-audit.service      (runs `bee audit` → /var/log/bee-audit.json,
  │                            never blocks boot on partial collector failures)
  ├── bee-web.service        (runs `bee web` on :80 — full interactive web UI)
  └── bee-desktop.service    (startx → openbox + chromium http://localhost/)

Critical invariants:

• The live ISO boots with boot=live toram. Runtime binaries must continue working even if the original boot media disappears after early boot.
• OpenSSH MUST start without network. bee-sshsetup.service runs before ssh.service.
• bee-network.service uses dhclient -nw (background) — network bring-up is best effort and non-blocking.
• bee-nvidia.service loads modules via insmod with absolute paths — NOT modprobe. Reason: the modules are shipped in the ISO overlay under /usr/local/lib/nvidia/, not in the host module tree.
• bee-nvidia-load refreshes nvidia-fabricmanager.service / nvidia-dcgm.service with systemctl --no-block try-restart only. DO NOT make it a blocking systemctl {start,restart}: bee-nvidia.service is Type=oneshot and Before= both units, so a synchronous call deadlocks against that ordering and previously reached a 60-second wrapper timeout for each unit. See decisions/2026-08-31-bee-nvidia-restart-deadlock.md.
• bee-audit.service does not wait for network-online.target; audit is local and must run even if DHCP is broken.
• bee-audit.service logs audit failures but does not turn partial collector problems into a boot blocker.
• bee-web.service binds 0.0.0.0:80 and always renders the current /var/log/bee-audit.json contents.
• Audit JSON now includes a hardware.summary block with overall verdict and warning/failure counts.

Console and login flow

Local-console behavior:

tty1
  └── live-config autologin → bee
        └── /home/bee/.profile (prints web UI URLs)

display :0
  └── bee-desktop.service (User=bee)
        └── startx /usr/local/bin/bee-openbox-session -- :0
              ├── tint2 (taskbar)
              ├── chromium http://localhost/
              └── openbox (WM)

Rules:

• local tty1 lands in user bee, not directly in root
• bee-desktop.service starts X11 + openbox + Chromium automatically after bee-web.service
• Chromium opens http://localhost/ — the full interactive web UI
• SSH is independent from the desktop path
• serial console support is enabled for VM boot debugging
• Default boot keeps the server-safe graphics path (nomodeset + forced fbdev) for IPMI/BMC consoles
• Higher-resolution mode selection is expected only when booting through an explicit bee.display=kms menu entry, which disables the forced fbdev Xorg config before lightdm

ISO build sequence

build-in-container.sh [--authorized-keys /path/to/keys]
  1. compile `bee` binary (always; version/git state is part of the artifact)
  2. create a temporary overlay staging dir under `dist/`
  3. inject authorized_keys into staged `root/.ssh/` (or set password fallback marker)
  4. copy `bee` binary → staged `/usr/local/bin/bee`
  5. copy vendor binaries from `iso/vendor/` → staged `/usr/local/bin/`
     (`storcli64`, `sas2ircu`, `sas3ircu`, `arcconf`, `ssacli` — optional; `mstflint` comes from the Debian package set)
  6. `build-nvidia-module.sh`:
       a. install Debian kernel headers if missing
       b. download NVIDIA `.run` installer (sha256 verified, cached in `dist/`)
       c. extract installer
       d. build kernel modules against Debian headers
       e. create `libnvidia-ml.so.1` / `libcuda.so.1` symlinks in cache
       f. cache in `dist/nvidia-<version>-<kver>/`
  7. `build-cublas.sh`:
       a. download `libcublas`, `libcublasLt`, `libcudart` runtime + dev packages from the NVIDIA CUDA Debian repo
       b. verify packages against repo `Packages.gz`
       c. extract headers for `bee-gpu-burn` worker build
       d. cache userspace libs in `dist/cublas-<version>+cuda<series>/`
  8. build `bee-gpu-burn` worker against extracted cuBLASLt/cudart headers
  9. inject NVIDIA `.ko` → staged `/usr/local/lib/nvidia/`
  10. inject `nvidia-smi` → staged `/usr/local/bin/nvidia-smi`
  11. inject `libnvidia-ml` + `libcuda` + `libcublas` + `libcublasLt` + `libcudart` → staged `/usr/lib/`
  12. write staged `/etc/bee-release` (versions + git commit)
  13. patch staged `motd` with build metadata
  14. copy `iso/builder/` into a temporary live-build workdir under `dist/`
  15. sync staged overlay into workdir `config/includes.chroot/`
  16. choose the build path from persisted content/ABI/overlay state:
       a. full: run `lb clean --all && lb config && lb build`
       b. fast: unpack the last squashfs, sync the staged overlay, repack it,
          then rebuild checksums, bootloader assets, ISO, and zsync
  17. validate the final ISO boot menus, volume label, memtest, GRUB assets,
      and variant runtime before publishing it

Build host notes:

• build-in-container.sh targets linux/amd64 builder containers by default, including Docker Desktop on macOS / Apple Silicon.
• Override with BEE_BUILDER_PLATFORM=<os/arch> only if you intentionally need a different container platform.
• If the local builder image under the same tag was previously built for the wrong architecture, the script rebuilds it automatically.

Critical invariants:

• DEBIAN_KERNEL_ABI in iso/builder/VERSIONS pins the exact kernel ABI used in BOTH places:
  1. build-in-container.sh / build-nvidia-module.sh — Debian kernel headers for module build
  2. auto/config — linux-image-${DEBIAN_KERNEL_ABI} in the ISO
• NVIDIA modules go to staged usr/local/lib/nvidia/ — NOT to /lib/modules/<kver>/extra/.
• bee-gpu-burn worker must be built against cached CUDA userspace headers from build-cublas.sh, not against random host-installed CUDA headers.
• The live ISO must ship libcublas, libcublasLt, and libcudart together with libcuda so tensor-core stress works without internet or package installs at boot.
• The source overlay in iso/overlay/ is treated as immutable source. Build-time files are injected only into the staged overlay.
• Fast-path state lives outside the rsync-managed live-build workdir and is accepted only when the heavy-input content hash and resolved kernel ABI match the last successful full build. A failed full build never leaves a valid completion marker. The workdir's binary/ tree is preserved because it is the source artifact for squashfs reuse.
• Bootloader menu text has two canonical sources only: config/bootloaders/grub-efi/grub.cfg and config/bootloaders/isolinux/live.cfg.in. lib/bootloader.sh renders those templates after both full and fast builds; hooks do not append duplicate menu entries.
• Build orchestration stays in build.sh; ISO validation, bootloader rendering, fast-path/memtest recovery, and logging helpers live under iso/builder/lib/. Run iso/builder/test-build-libs.sh after changing those helpers or the canonical boot parameters.
• ISO filename, squashfs filename, ISO volume label, and the live system's hostname all derive from the same easy-bee-<variant>-v<version> scheme (ISO_BASENAME/SQUASHFS_FILENAME/BEE_ISO_VOLUME/BEE_HOSTNAME in build.sh) instead of the live-build default (debian). Keep new naming derived from PROJECT_VERSION_EFFECTIVE/BUILD_VARIANT in sync with this set rather than hardcoding a new scheme.
• Every live boot entry carries udev.children_max=1, intel_iommu=on, iommu.passthrough=0, and efi=disable_early_pci_dma. Only the single failsafe entry additionally carries pci=realloc iommu.strict=1; iommu=pt is forbidden. The final-ISO validator enforces this for both GRUB and isolinux.
• The live-build workdir under dist/ is disposable; source files under iso/builder/ stay clean.
• Container build requires --privileged because live-build uses mounts/chroots/loop devices during ISO assembly.
• On macOS / Docker Desktop, the builder still must run as linux/amd64 so the shipped ISO binaries remain amd64.
• Supported systems have at least 16 GB of installed RAM. This provides room for the full compressed live medium plus normal runtime overhead because toram copies the entire read-only ISO payload into memory before the system reaches steady state. A mid-copy failure must not be classified as low RAM without direct ENOSPC, OOM, or tmpfs-limit evidence.

Post-boot smoke test

After booting a live ISO, run to verify all critical components:

ssh root@<ip> 'sh -s' < iso/builder/smoketest.sh

Exit code 0 = all required checks pass. All FAIL lines must be zero before shipping.

Key checks: NVIDIA modules loaded, nvidia-smi sees all GPUs, lib symlinks present, systemd services running, audit completed with NVIDIA enrichment, LAN reachability.

Current validation state:

• local/libvirt VM boot path is validated for systemd, SSH, bee audit, bee-network, and Web UI startup
• real hardware validation is still required before treating the ISO as release-ready

Overlay mechanism

live-build copies files from config/includes.chroot/ into the ISO filesystem. build.sh prepares a staged overlay, then syncs it into a temporary workdir's config/includes.chroot/ before running lb build.

Collector flow

`bee audit` start
  1. board collector   (dmidecode -t 0,1,2)
  2. cpu collector     (dmidecode -t 4)
  3. memory collector  (dmidecode -t 17)
  4. storage collector (lsblk -J, smartctl -j, nvme id-ctrl, nvme smart-log)
  5. pcie collector    (lspci -vmm -D, /sys/bus/pci/devices/)
  6. psu collector     (ipmitool fru + sdr — silent if no /dev/ipmi0)
  7. nvidia enrichment (nvidia-smi — skipped if binary absent or driver not loaded)
  8. TPM inventory (sysfs presence + `tpm2_getcap properties-fixed`; no state changes)
  9. output JSON → /var/log/bee-audit.json

Every collector returns nil, nil on tool-not-found. Errors are logged, never fatal.

Acceptance flows:

• bee sat nvidia → diagnostic archive with nvidia-smi -q + nvidia-bug-report + lightweight bee-gpu-burn
• NVIDIA GPU burn-in can use either bee-gpu-burn or bee-john-gpu-stress (John the Ripper jumbo via OpenCL)
• bee sat memory → memtester archive
• bee sat storage → SMART/NVMe diagnostic archive and short self-test trigger where supported
• bee TPM Validate → read-only capabilities, PCR values, and existing self-test result; never starts TPM2_SelfTest
• SAT summary.txt now includes overall_status and per-job *_status values (OK, FAILED, UNSUPPORTED)
• bee-gpu-burn should prefer cuBLASLt GEMM load over the old integer/PTX burn path:
  • Ampere: fp16 + fp32/TF32 tensor-core load
  • Ada / Hopper: add fp8
  • Blackwell+: add fp4
  • PTX fallback is only for missing cuBLASLt/userspace or unsupported narrow datatypes
• Runtime overrides:
  • BEE_MEMTESTER_SIZE_MB
  • BEE_MEMTESTER_PASSES
• NVIDIA Bandwidth SAT (RunNvidiaBandwidthPack, dcgmi diag -r nvbandwidth) in Stress mode runs per resolved PCI NUMA node first, then all selected GPUs together (03-dcgmi-nvbandwidth-socket0.log, ...-socket1.log, ...-all.log) -- see decisions/2026-07-27-nvbandwidth-per-socket-split.md. Single-node systems (or systems where any GPU's NUMA node cannot be resolved) keep the original single NN-dcgmi-nvbandwidth.log shape.

SAT job output durability

runAcceptancePackCtx job loop (per satJob)
  1. run the job's command; streamExecOutput writes each output line to the
     job's log file as it arrives (not only when the process exits)
  2. write the job's final log file (same content the live stream already
     wrote, plus any health-check suffix)
  3. call satJobBoundaryHook(jobName) if set
  4. append run_at_utc / *_status to summary.txt

Critical invariants:

• DO NOT change streamExecOutput back to buffering output in memory and writing the job's log file only once, after the command exits -- see decisions/2026-07-27-sat-live-output-and-blackbox-kick.md. The live ISO's export directory sits on a toram RAM-backed overlay (see Boot sequence above), so a crash mid-command is otherwise unrecoverable for that job.
• platform never imports app; SetJobBoundaryHook is a plain func(string) seam (same pattern as satExecCommand/satStat), not a direct call into blackbox internals.

Blackbox sync flow

bee-blackbox.service (separate process from bee-web/bee-audit)
  1. discover enrolled removable-media targets every blackboxDiscoverInterval (2s)
  2. per enrolled target, blackboxWorker.run():
       a. syncCycle(): mount target, mirror exportDir -> removable media,
          write README.md/manifest docs, fsync
       b. record lastKickSeen = mtime of exportDir/.blackbox-kick
       c. wait for: the adaptive flushPeriod timer (1-30s), OR
          a stop signal, OR
          a poll tick (blackboxKickPollInterval, 250ms) showing the kick
          file is newer than lastKickSeen -- whichever comes first
  3. on kick-triggered wake: sync immediately, do not wait out the
     remaining flushPeriod

Critical invariants:

• app.New() wires platform.SetJobBoundaryHook to touch exportDir/.blackbox-kick after every SAT job finishes. If a new SAT execution path bypasses runAcceptancePackCtx's job loop, it will not trigger this kick -- data from that path still eventually reaches blackbox via the adaptive timer, just not promptly.
• The adaptive flushPeriod logic (adjustFlushPeriod) is unchanged by the kick mechanism -- the kick only short-circuits the wait, it does not reset flushPeriod itself.
• DO NOT assume a local write under the live ISO's export directory is durable on its own (RAM-backed overlay) -- blackbox's mirror to removable media is the only real persistence boundary across a hard reset.
• BuildSupportBundle stages into a private os.MkdirTemp parent, not a shared os.TempDir()/bee-support-stage-<host>-<ts> path. DO NOT go back to a time-derived staging path: two builds in the same wall-clock second (two operators, or an on-demand build racing the blackbox worker) then share one tree and one's deferred os.RemoveAll truncates the other's archive.
• The resulting archive is also unique (os.CreateTemp with the readable support-bundle name as its prefix) and remains a hidden .partial from retention cleanup until an atomic rename publishes the closed .tar.gz. A timestamp-only output path lets same-second writers share one inode and can corrupt both gzip streams.

NVIDIA SAT Web UI flow

Web UI: Acceptance Tests page -> Run Test button
  1. POST /api/sat/nvidia/run -> returns job_id
  2. GET  /api/sat/stream?job_id=... (SSE): streams stdout/stderr lines live
  3. After completion: archive written to /appdata/bee/export/bee-sat/
     summary.txt contains overall_status (OK / FAILED / UNSUPPORTED) and per-job status

Run All (validate / check) flow

Web UI: "Run All" button -> POST /api/sat/run-all
  body: operator intent only { stress_mode, amd_targets[], nvidia_gpu_indices[] }
  server (handler.planSATRunAll):
    1. always: cpu, memory, storage
    2. tpm            - only if App.TPMPresent() finds tpm_version_major=2
    3. nvidia-config  - if DetectGPUPresence().Nvidia || NvidiaInitializing
    4. wait for NVIDIA enumeration: repeat fresh ListNvidiaGPUs queries until
       at least one GPU is returned, NvidiaGSPMode=="gsp-stuck", or 75s
    5. nvidia / nvidia-interconnect / nvidia-bandwidth
       (+ targeted-stress/power/pulse when stress_mode) - only once ready,
       -i = App.ListNvidiaGPUs() indices (intersected with the requested subset)
    6. amd / amd-mem / amd-bandwidth - if DetectGPUPresence().AMD and selected
  response: { task_ids[], task_count, notes[] }  (notes = what was skipped and why)

Critical invariants:

• Hardware presence, readiness, and which tasks to run are decided server-side. DO NOT move this back into page JS (satSelectedGPUIndices().length gating): a browser-cached empty GPU list then silently drops every GPU test. See decisions/2026-08-31-backend-driven-sat-planning.md.
• DetectGPUPresence is the shared detection source (existing operational vendor detection plus an lspci display-class fallback). /api/gpu/presence, /api/gpu/tools and the planner all use it.
• bee-gpu-burn / bee-john-gpu-stress use exec.CommandContext: killed on job context cancel.
• nvidia-bandwidth samples every selected GPU's negotiated PCIe speed and width immediately after its final all-GPU dcgmi diag -r nvbandwidth command exits. Idle and forced-retrain readings are not health verdicts.
• Metric goroutine uses stopCh/doneCh pattern; main goroutine waits <-doneCh before reading rows (no mutex needed).
• SVG chart is fully offline: no JS, no external CSS, pure inline SVG.
• RunNvidiaBandwidthPack runs one all-GPU nvbandwidth pass in Validate; the per-NUMA-node matrix is Stress-tier only (fullMatrix arg). See decisions/2026-08-31-nvbandwidth-validate-single-deep-matrix.md.
• nvidia-config validates the active-link count independently at both ends of every NVx pair reported by nvidia-smi topo -m; aggregate counts must not be compared with a per-endpoint NVx value. It also persists matching NVIDIA degraded-mode kernel lines in 06-dmesg-nvlink-degraded.log.
• Generic SAT jobs may attach an output validator. A validator runs only after exit status 0 and returns OK, FAILED, or UNSUPPORTED; its detail is persisted in both the job log and <job>_detail in summary.txt.
• nccl-tests explicitly validates the pinned all_reduce_perf correctness summary and both #wrong columns, and cross-checks selected NVLink endpoints against the captured topology/status. Avg bus bandwidth is retained as a diagnostic value only: no model-independent bandwidth threshold is valid. NCCL P2P alone is not proof of NVLink because P2P can also use PCIe.