bee_layer_classify only ever tracked regular files and symlinks
(`find ... -type f -o -type l`), so any directory that is empty at
build time — like /var/log/nvidia-dcgm, correctly created and chowned
by the datacenter-gpu-manager postinst — was silently dropped from
every layer's rsync --files-from list and never reached the built
ISO. This is why bbc6fb1's 78d1b9b follow-up (seeding a marker file
just for that one path) kept the directory alive: it was a targeted
workaround for a general gap in the classifier, not a fix of it.
Replace that workaround with the general mechanism: classify also
walks every directory, computes the subset that is empty all the way
down (no file or symlink anywhere in its subtree — a directory that
does hold files needs no entry, rsync already recreates it as an
implied parent), and assigns each one to a layer via the same
dpkg-ownership / injected-rule precedence used for files. Add an
injected rule routing /var/log/nvidia-dcgm to 20-nvidia-platform,
alongside the DCGM binaries that actually use it, instead of letting
it fall through to base by default.
bee_layer_build folds each layer's empty-dir list into the same
rsync --files-from call; recursion into a directory that is
by-construction empty copies nothing extra. Revert the 9000/9999 hook
changes from 78d1b9b now that they're redundant, and cover the new
path with test-squashfs-layers.sh (ruled, unruled, and nested-empty
directories, asserted present in the merged rootfs after a real
mksquashfs/unsquashfs round-trip).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bbc6fb1 chowned the directory correctly but it stayed empty at build
time, and bee_layer_classify (squashfs-layers.sh) only tracks regular
files and symlinks via `find ... -type f -o -type l` — an empty
directory is silently dropped from every layer's rsync --files-from
list and never reaches the built ISO. dcgmi diag's deployment check
still failed with DCGM_FR_FILE_CREATE_PERMISSIONS after rebuilding on
top of that fix because the directory simply didn't exist at boot.
Seed /var/log/nvidia-dcgm/.keep so the directory rides along the
classifier as an implied parent (rsync -a preserves its ownership),
and stop 9999-slim's log sweep from deleting that marker before the
classifier ever sees it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nv-hostengine drops privileges to --service-account nvidia-dcgm for all
file I/O and writes diagnostics to DCGM_HOME_DIR (/var/log/nvidia-dcgm,
per the packaged unit's Environment=). The package postinst doesn't
reliably create/own this directory inside a live-build chroot, so
dcgmi diag's software deployment check fails with
DCGM_FR_FILE_CREATE_PERMISSIONS at boot even though every GPU passes
the actual hardware tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Booting via BMC virtual CD reads the ~2.8 GB filesystem squashfs
sequentially during the live-boot toram copy; a mid-read drop of the
redirected medium loses the whole copy and fails the boot (v14). Split
the rootfs into self-contained semantic layers so a retry re-reads at
most one ~500-700 MiB layer, not everything. This is a resilience /
reduced-re-read mechanism, not a fix for the virtual-media instability.
NVIDIA variants now ship 7 layers (00-base, 05-firmware, 08-desktop,
10-nvidia-driver, 20-nvidia-platform, 30-nvidia-cuda-libs,
40-nvidia-dcgm-cuda) plus an explicit live/filesystem.module that fixes
their OverlayFS order; amd/nogpu keep a single squashfs.
- lib/squashfs-layers.sh: deterministic classifier (dpkg file ownership
plus explicit rules for build.sh-injected files, never a path
substring), per-layer mksquashfs, 800 MiB hard ceiling, unsquashfs -s
plus strict extraction of every layer, merged-rootfs bootability check.
- build.sh: split the monolith after the full lb build, verify and merge,
write the module file, delete the monolith only then; abort before ISO
assembly on any failure. Runs the builder test suites up front.
- fast-path: force a full build for a multi-layer medium;
fast_path_repack_squashfs hard-refuses (it would drop layers).
- iso-validation.sh: validate_iso_squashfs_layers (module vs layer set
match, size ceiling, no lone giant squashfs) and
validate_iso_media_integrity (xorriso -check_media).
- bee-install: honour filesystem.module order, abort on any layer failure.
- 9013-toram-retry: record the real rsync exit code (it printed a false
rc=0) and correct the "resumes the tail" comment (rsync without
--partial keeps only fully-copied layers). No unsafe partial resume.
- tests: test-squashfs-layers.sh plus a multi-layer guard in
test-build-libs.sh; both run at the top of every build.
- docs: bible-local architecture and decision, iso/README, iso-build-rules.
Verified by a full nvidia build: 7 layers 622/199/256/466/37/567/562 MiB,
every validator passes, xorriso -check_media good, merged rootfs bootable.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The NVIDIA GSP firmware was staged into the overlay at a bare
lib/firmware/nvidia/<ver>/. On a merged-usr Debian root /lib is a symlink
to usr/lib, so the fast-path repack's `rsync -a` (overlay stage -> the
unsquashfs'd tree) replaced that symlink with a plain directory holding
only the firmware. Everything else reachable only through /lib
(/lib/systemd/systemd, /lib/x86_64-linux-gnu/ld-linux + libc, ...) was
orphaned. The squashfs still packed and mounted, but boot panicked:
run-init: can't execute '/sbin/init': No such file or directory
run-init: can't execute '/bin/sh': No such file or directory
Kernel panic - not syncing: Attempted to kill init
Full `lb build` was unaffected (live-build copies includes.chroot with
cp -a, which follows the /lib symlink), so only fast-path ISOs were bad.
Three fixes:
- build.sh: stage GSP firmware under usr/lib/firmware/nvidia/<ver>/, the
canonical merged-usr path. Changing the overlay path also makes
overlay_paths_were_removed() force one full build on the next run.
- fast-path.sh: add `rsync --keep-dirlinks` so a real directory in the
overlay stage can never again replace a symlink-to-directory in the
root. Verified: firmware lands in usr/lib/firmware, /lib stays a symlink.
- iso-validation.sh: new validate_iso_rootfs_layout, run for every variant
on both build paths. Fails the build if the squashfs has a plain-dir
/bin|/sbin|/lib|/lib64, or if /usr/sbin/init is present without a
resolvable /usr/lib/systemd/systemd and /lib symlink. Verified it flags
the broken v14.02-1-g642e686 ISO and passes a correct layout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H2LLuid8PFhBqBPcxXxkQU
On hosts where the boot medium is BMC/IPMI virtual media, the ~2.8 GB
squashfs read often drops mid-transfer: rsync dies with "Input/output
error (5)", every file "has vanished", and live-boot moves the partial
RAM copy over the medium, ending the boot with "No supported filesystem
images found at /live".
Add hook 9013-toram-retry, which patches live-boot's 9990-toram-todisk.sh
so the whole-medium copy runs in a retry loop: up to 8 attempts with a
geometrically growing pause (15s -> 900s cap), unmounting/waiting/remounting
the medium between tries and honouring rsync's exit code (upstream ignores
it). rsync resumes from where it stopped, so a retry needing only the tail
finishes fast.
Verified against live-boot 1:20230131+deb12u1: substitution applies,
sh -n is clean, hook is idempotent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018JA6Cj84yxUfk8N3ppw6mr
needs_full_build() decided the fast path was safe by hashing only config
inputs (VERSIONS, package lists, hooks, archives, auto/config, Dockerfile).
It never hashed the build logic itself, so a change to build.sh or
lib/fast-path.sh counted as a "light" file and the next build silently
reused a squashfs produced by the old code. The merged-usr /lib fix in the
previous commit only forced a full rebuild by accident (the overlay
manifest saw firmware move from lib/ to usr/lib/).
hash_heavy_config now covers every *.sh under iso/builder (build.sh, the
build-*.sh helpers, lib/*.sh) plus auto/. A change to build logic is now
as heavy as a package-list change. config/bootloaders stays excluded (the
fast path regenerates the outer ISO layer from it every time). Hash is
deterministic (LC_ALL=C sort).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H2LLuid8PFhBqBPcxXxkQU
mirror.mephi.ru was timing out during lb binary_rootfs. Default to
deb.debian.org (Fastly-backed, geo-balanced) and allow overriding via
--mirror on build-in-container.sh/build.sh, or BEE_MIRROR in auto/config.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The NVSwitch-detection fix (983f41a) was necessary but not sufficient on
HGX B200 (Kaytus KR9288-X3): ExecCondition now correctly runs fabricmanager,
but nvidia-fabricmanager-start.sh then aborts with "Kernel module ib_umad
has not been loaded" on this NVL5+ board, leaving the fabric stuck in
"In Progress" and cascading into failed dcgmi diag/NVBandwidth/NCCL/Validate
GPU SAT tasks (confirmed via support bundle 20260729-165331).
Neither nvidia-fabricmanager nor DCGM declare the NVLink5 support packages
as apt dependencies (checked against the cuda-repo Packages index directly):
ib_umad must be modprobed before FM starts, nvlsm (InfiniBand Subnet Manager)
is launched internally by FM's own start script once present, and DCGM
dlopens libnvidia-nscq at runtime for NVSwitch health monitoring.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
On HGX B200 (Kaytus KR9288-X3) NVSwitch is managed via HMC/ERoT and never
appears on host lspci as 10de:0680, so the old check always returned
"not present" and silently skipped fabricmanager — leaving the fabric
untrained and CUDA/NCCL/dcgmi failing with "system not yet initialized"
on otherwise-healthy hardware. bee-check-nvswitch now checks, in order,
nvidia-smi's GPU Fabric GUID, /proc/driver/nvidia-nvswitch/devices, and
finally the legacy host-PCIe scan for older HGX generations.
- page_scenario.go: onclick built via JSON.stringify() embedded raw double
quotes inside a double-quoted HTML attribute, truncating the attribute so
the click handler never compiled; pass the name through an escaped
data-scenario-name attribute instead.
- build.sh: overlay staging rsyncs (OVERLAY_DIR->stage, stage->includes.chroot)
ran without --delete, so a scenario removed from the repo (a9924b0) stayed
baked into every ISO built from the persistent stage cache since — the
"second script" in the Scenario page's list.
- blackbox: rewritten around a deterministic local zip + incremental
patch-the-changed-suffix onto removable media, instead of walking/copying
~90 files through a synchronous ntfs-3g FUSE mount every cycle. journalctl
captures are now "--since last sync" (were "--since boot", growing with
uptime) and metrics.db is excluded (was copied whole every cycle).
- scenario: nvbandwidth-acs-ab now escalates GPU count (same-socket pair,
other socket's pair, one cross-socket pair, all GPUs) under each ACS state
instead of always running all 6 GPUs at once, using a new `bee
gpu-bandwidth-groups` subcommand that discovers socket layout from
`nvidia-smi topo -m` at runtime — gpu_indices is host-specific, so this
can't be baked into the scenario file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bee.nvidia.mode=gsp-off loads the driver with NVreg_EnableGpuFirmware=0 and
skips nvidia-uvm/nvidia-modeset, so CUDA (nvbandwidth/dcgm/nccl) can't run —
and on Hopper H200 GSP is effectively mandatory anyway, so the mode is dead
weight in this test image. Remove it from every boot entry. fail-safe is now
bee.nvidia.mode=normal pci=realloc iommu=pt, so the GPUs come up with uvm and
the GPU test actually runs under iommu=pt (the point of the entry). All four
sources (grub.cfg + live.cfg.in templates, build.sh write_canonical_*) match.
The internal GSP auto-fallback in bee-nvidia-load (normal mode, GSP init
hangs >90s → retry with EnableGpuFirmware=0, then still load uvm) is left
intact — it serves converted SXM->PCIe cards on other test targets and keeps
CUDA working, unlike the removed explicit menu mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
no-GUI is now just the normal entry minus the GUI: bee.gui=off with
bee.nvidia.mode=normal (drops the previous gsp-off/pci=realloc it had picked
up). fail-safe is now simply the GPU/PCIe-workaround boot —
bee.nvidia.mode=gsp-off pci=realloc iommu=pt over the standard base — instead
of the old noapic/nodma/nomce/nolapic/nosmp minimal mode, so it can actually
run GPU tests (nvbandwidth) to verify the iommu=pt half of the vendor fix.
Applied identically across all four sources (grub.cfg + live.cfg.in templates
and build.sh write_canonical_* generators).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The GRUB and isolinux menus (and their two sources of truth — the staged
templates and build.sh's write_canonical_* generators that rewrite the built
menu) had drifted: GRUB shipped a lean set, isolinux carried extra
display-debug entries (gsp-off, kms, kms-gsp-off) and lacked a WIPE entry in
the canonical writer. Collapse all four to one set:
normal, load-to-RAM (default), no-GUI/troubleshooting (gsp-off, pci=realloc),
fail-safe, WIPE ALL DISKS, Memory Test, + UEFI Firmware Settings (EFI only)
The fail-safe entry gains iommu=pt (AMD-Vi passthrough) — the OS-side half of
the vendor fix for the MSI CG480-S6053 cross-socket nvbandwidth reboot; ACS
can't be toggled from a stock-kernel cmdline so that stays a BIOS setting.
default stays index 1 = toram (config.cfg set default=1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the old nvbandwidth-all-gpu-power-watch scenario (dropped: the
power/IPMI-sampler framing was a dead-end — the blackbox shows every other
load test, incl. NCCL all-reduce across all 6 GPUs, passes with power fully
connected; only nvbandwidth on cross-socket GPU P2P reboots the box).
nvbandwidth-acs-ab is a two-phase reproduction for the MSI CG480-S6053 case.
Phase 1 disables PCIe ACS redirect at runtime via setpci (saving each
bridge's original ACS control word to /run/bee-acs-orig) then runs
nvbandwidth across all 6 GPUs — expected to survive. Phase 2 restores the
saved ACS values and repeats — expected to reproduce the reboot. Safe phase
first so its full blackbox reaches the USB stick before the risky phase can
reset the box (sync-bracket before the final job flushes phase-1 evidence).
ACS mutation runs via `bash -c` since scenario cmd is exec'd directly with no
shell. Both shipped copies (repo scenarios/ and image overlay) identical;
verified both parse via ParseScenarioJSON (4 jobs).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ScenarioSpec gains an optional "description" field. Listing (both
ListLocalScenarioFiles and ListScenarioFilesOnRemovableMedia, via the new
scenarioDescription helper) reads it out of each file without requiring
full ParseScenarioJSON validation to succeed, so a listing never hides a
scenario over an unrelated validation issue. The webui Scenario page now
renders Name/Description/Found-on/Run instead of just Name/Found-on — a
bare filename rarely tells anyone but the author what a scenario actually
does.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
platform.LocalScenariosDir (/usr/share/bee/scenarios, populated from
iso/overlay/usr/share/bee/scenarios/ by build.sh's overlay rsync) is now
checked before removable media for both `bee run <name>` and the "6.
Scenario" page — a scenario shipped with the image works with no USB
stick required. ReadScenario/ListAvailableScenarios merge local + USB;
the removable-media-only functions from the previous commit are kept
as-is (still used directly where that's actually what's wanted) rather
than renamed out from under existing callers/tests.
iso/overlay/usr/share/bee/scenarios/nvbandwidth-all-gpu-power-watch.json
is a copy of scenarios/nvbandwidth-all-gpu-power-watch.json — the two
aren't auto-synced (documented in scenarios/README.md), so shipping a
scenario baked-in means checking it into both places.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- gpuBandwidthSocketGroups: a single GPU whose NUMA node fails to resolve
no longer collapses the whole per-socket nvbandwidth split into one
fallback pass — it now folds into the last resolved group instead,
preserving isolation for the sockets that did resolve.
- blackbox discoverMarkedTargets: skip mounting/unmounting devices that
already have a running worker on every 2s discovery tick. This was
observed hammering the same USB target continuously (mount+unmount
every ~2s for the whole session) and contending with the worker's own
sync cycle, plausibly explaining multi-minute sync cycles seen on a
real crash bundle.
- syncFilesystem now calls syscall.Sync() directly instead of spawning
/bin/sync per copied file; blackbox mounts removable targets with
-o sync so writes are durable without relying on the app-level sync as
the primary mechanism.
- New platform.SetSyncBracketHook / satJob.syncBracket: blocks (with a
bounded timeout) on blackbox actually reaching removable media right
before and right after a diagnostic's real load step (nvbandwidth,
memtester, stress-ng, dcgmi diag, nccl, smartctl/nvme self-test...),
instead of only firing a fire-and-forget kick after the job's own log
file is written. A crash mid-load now has durable evidence the load
started, not just whatever streamed to the RAM-backed export dir before
blackbox's next scheduled cycle.
Found investigating a real support bundle where blackbox's last
successful sync (19:55:25) predated both the previous job finishing and
the crashing nvbandwidth job starting (19:56:57) — none of the crash
window ever reached durable media.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds BEE_HOSTNAME, derived with the same easy-bee-<variant>-v<version>
scheme already used for ISO_BASENAME/SQUASHFS_FILENAME/BEE_ISO_VOLUME, and
wires it into auto/config via lb config --hostname. Documented the shared
naming scheme as a runtime-flows.md invariant.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pci=realloc was forcing a full PCI resource reallocation on every default
and toram boot, added as a GPU BAR workaround for BIOS without Above 4G
Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and
silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS
and lspci both saw them fine. It now only applies to the no-GUI/gsp-off
troubleshooting entry, alongside a bible-local ADR for the two prior
rounds of SAT/blackbox durability work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The bundled storcli2 (008.0005.0000.0010, Feb 2023) reports zero
controllers on a live SAS3808-iMR system even though storcli64 and
lspci both see it fine (confirmed the PCI device is correctly bound to
megaraid_sas — mpi3mr doesn't even list it as a candidate driver, so
this isn't a driver-binding issue storcli2 could route around).
Swapping in Broadcom's newer 08.017.000.0008 release on the chance it
recognizes this controller/firmware combo where the older build
doesn't; storcli64's drive-listing path (already fixed to parse the
real per-slot JSON shape) remains the primary, confirmed-working
source regardless of whether this update changes storcli2's behavior
for this card.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
needs_full_build() compared mtimes of VERSIONS/package-lists/hooks against
a marker file, which is fragile: a failed build attempt never touches the
marker, and git checkouts/rsync can reorder mtimes relative to real edit
order. This let the fast-path silently reuse a stale squashfs (baked
against old package pins) with no error — root-caused after a driver/DCGM
pin change (590/DCGM 4.6.0 -> 580/DCGM 4.5.2) produced a ~290MB larger ISO
that only shrank back down with --clean-build.
Switch to a content hash (VERSIONS + package-lists + hooks + archives +
bootloaders + auto/config + Dockerfile) instead of mtimes, and add a real
version check in validate_iso_nvidia_runtime: extract dpkg status from the
squashfs and confirm the installed datacenter-gpu-manager-4-core version
matches DCGM_VERSION. The previous check only confirmed dcgmi/nv-hostengine
/dcgmproftester exist, which stays true across DCGM versions and can't
catch this class of staleness.
dcgmi discovery -l is a preflight/metadata step ahead of the real DCGM
diag jobs; a transient failure racing nv-hostengine startup shouldn't
flip the whole pack's status, so it's now marked informational with a
couple of retries. Separately, bound the fabricmanager/nvidia-dcgm
systemctl restart/start calls in bee-nvidia-load with a timeout so a
wedged unit (e.g. fabric training stuck on a bad NVSwitch fabric)
can't hang bee-nvidia.service forever and block dcgm from ever
starting.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
storcli64 enumerates these controllers but reports zero drives; storcli2
is the tool Broadcom ships for Tri-Mode/MegaRAID8 hardware and uses a
compatible JSON schema for drive listing. Wires storcli2 into both the
collector (structured drive data) and the webui RAID Management page
(dedup so a controller isn't double-listed if storcli64 already sees it
with zero drives), plus techdump raw collection and the ISO vendor-tool
build step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add NVLink port status (nvidia-smi nvlink -s), error counters
(nvidia-smi nvlink -e), and dcgmi nvlink status to the support bundle,
and enrich HardwarePCIeDevice entries with per-link telemetry.
Replace the raw nv-hostengine pkill/restart dance in bee-nvidia-load
with systemctl restart/start of nvidia-dcgm.service, and order
bee-nvidia.service Before= nvidia-dcgm.service and
nvidia-fabricmanager.service so modules/device nodes exist before
those units start.
v11.52's ISO build failed at chroot_install-packages:
datacenter-gpu-manager-4-cuda13 : Depends: datacenter-gpu-manager-4-core
(= 1:4.5.2-1) but 1:4.6.0-1 is to be installed
cuda13/proprietary/proprietary-cuda13 all declare an exact-version
dependency on datacenter-gpu-manager-4-core, but core itself was never
listed, so apt's solver picked the newest core in the repo (4.6.0-1)
for the transaction and then reported the pinned variants as unmet
instead of resolving everything to 4.5.2-1. Pin core explicitly so
there's no ambiguity for the solver.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.51 pinned 580.126.20 (DGX OS 7.5.0's validated combo), but
build-nvidia-module.sh downloads the .run installer from
download.nvidia.com/XFree86/Linux-x86_64/<version>/, a separate catalog
from the CUDA apt repo checked previously. 580.126.20 404s there —
same failure class as the original 550.54.15 CDN-404 incident.
Checked all 580.x point releases on that CDN; only some are mirrored
(580.126.09, 580.159.03/.04, 580.173.02 are 200; 580.126.16, 580.126.20,
580.167.08 are 404). Switched to 580.159.03: confirmed downloadable,
confirmed present as .deb in the apt repo, and it's the exact driver
version running on the real DGX Spark.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
590.48.01/DCGM 4.6.0-1 was never a deliberate compatibility choice — the
driver version traces back to a CDN 404 on the originally pinned
550.54.15 (commit 559fc29), and DCGM was bumped twice since purely
because NVIDIA pulled the previously-pinned core package from the repo
(6dee8f3, b2e177a). The 590 branch only has two point releases so far,
and its nvvs plugin fails dcgmi diag with "Detected driver major
version 0 is not between the required versions 346 and 2000" on H100 —
confirmed via a staging support bundle.
Switch to 580.126.20 / DCGM 4.5.2-1, the combo NVIDIA's own DGX OS 7.5.0
release notes validate for H100. Both packages confirmed present in the
NVIDIA CUDA debian12/x86_64 apt repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- sat.go: DetectGPUVendor lspci fallback now checks GPU device classes
([0300]/[0302]/[0380]) per line instead of scanning the whole output for
vendor name; AMD EPYC servers have dozens of AMD-branded PCIe entries
(Root Complex, IOMMU, Host Bridge) that were triggering the old check
- blackbox.go: fix deadlock in finishCycle — it held w.mu while calling
persistState(), which acquires rt.mu then re-acquires w.mu inside
persistStateLocked(); now w.mu is released before persistState()
- build.sh: remove NVIDIA-specific overlay files (bee-gpu-burn,
bee-john-gpu-stress, bee-nccl-gpu-stress, bee-nvidia-recover,
bee-dcgmproftester-staggered, bee-check-nvswitch,
nvidia-fabricmanager.service.d/) for non-nvidia build variants
- bee-selfheal: gate NVIDIA recovery on BEE_GPU_VENDOR=nvidia so the
script does not attempt to restart bee-nvidia.service on NOGPU builds
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- saa_dmi.go: set cmd.Dir=/usr/local/bin on all saa exec calls so
acpica_bin/acpidump is found relative to correct working directory
- build.sh: copy all saa companion dirs (acpica_bin, ExternalData,
tool, stunnel, GO_SNMP) to /usr/local/bin/ preserving structure
- iso/vendor: add acpica_bin/acpiexec, ExternalData/, tool/gpu/nVidia/x64/,
tool/USBController/, stunnel/, GO_SNMP/ from SAA 1.5.0 release package
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extract saa 1.5.0 (Linux x86_64) into iso/vendor/saa — baked into ISO
at /usr/local/bin/saa via the existing vendor loop in build.sh
- Add saa to the vendor tool loop in iso/builder/build.sh
- Rename the web UI card from "SAA - DMI" to "Supermicro - DMI"
- Remove the redundant description hint about saa on PATH
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NVIDIA removed datacenter-gpu-manager-4-core 1:4.5.3-1 from the
repository and published 1:4.6.0-1. The cuda13 and proprietary
packages still declared an exact-version dependency on 4.5.3-1 core,
making the old pin unresolvable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bee-nvidia.service loads NVIDIA kernel modules; without After=bee-nvidia.service
fabricmanager starts before /dev/nvidiactl is ready, fails, and relies on
systemd restart to recover (~38s delay on affected systems).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nvtop pulled nvidia-tesla-470-* via Recommends into the nogpu build.
Move it from bee.list.chroot into bee-nvidia and bee-amd lists so it
only appears in GPU variants.
Also remove the stray git-bible/ directory (was not gitignored) and
move grub-bitmap-error docs into bible-local/docs/.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- storage: add jsonInt64 dual-format unmarshaler to handle lsblk output
change in util-linux 2.38 (LOG-SEC/PHY-SEC now emitted as JSON
integers, not quoted strings); fixes SATA disks invisible on Debian 12
- pcie: detect NVLink bridge mezzanine CX-7 cards (Mellanox x2, no host
net ifaces, DeviceName contains "NVLINK" in lspci -v) and mark them
with device_class="NVLinkBridge"; escalate PCIe link speed downgrade to
Critical for these cards (Gen3 on a fixed internal connector = hardware
fault, not a transient warning)
- pcie: cross-reference nvidia-smi topo to capture NVLink bond counts and
active status for all NVLink bridge cards
- packages: add infiniband-diags to ISO; provides ibstat required by
nvidia-fabricmanager-start.sh to enumerate IB devices before FM launch
(absence causes CUDA_ERROR_SYSTEM_NOT_READY)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Network: green if at least one interface has IPv4 (drop PARTIAL state).
Bee Services: treat inactive as OK — oneshot services (bee-sshsetup,
bee-preflight, bee-network, bee-audit, etc.) complete successfully and
exit to inactive; only failed is a real problem.
nvidia-fabricmanager: add ExecCondition=bee-check-nvswitch drop-in so
the service is silently skipped (inactive, not failed) on systems
without NVSwitch hardware (e.g. H200 NVL with direct NVLink, no
NVSwitch chips). bee-check-nvswitch detects NVSwitch via lspci
(vendor 10de, class 0680).
bee-nvidia.service: add ConditionPathExists=/usr/local/bin/bee-nvidia-load
so the unit is a no-op if somehow present in a non-nvidia build.
bee-boot-status: read /etc/bee-gpu-vendor and exclude bee-nvidia from
CRITICAL/ALL on non-nvidia builds, preventing boot hang if the unit
is unexpectedly present.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
EASY_BEE_NVIDIA_LEGACY_V<date> is 33 characters; ISO 9660 volid is
limited to 32. Compute the maximum token length dynamically from the
prefix length and trim ISO_VERSION_LABEL_TOKEN with cut before
assembling BEE_ISO_VOLUME. All four variants now fit within the limit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All intermediate build artifacts (binaries, live-build work dirs, overlay
stages, NVIDIA/NCCL/cuBLAS/john caches) now live under dist/cache/.
Final ISOs go to dist/release/ instead of scattered dist/easy-bee-v*/ and iso/out/.
dist/ is already gitignored, iso/out/ entry removed as redundant.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Installs a local-premount initramfs hook that intercepts bee.wipe=all before
squashfs is mounted. Shows a numbered disk selection TUI (pure POSIX sh), wipes
selected disks (nvme format / blkdiscard / dd fallback), syncs, and reboots.
Works even when squashfs fails to mount.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a "WIPE ALL DISKS" entry to both GRUB and isolinux menus (bee.wipe=all).
Includes bee-wipe-disks for manual use from a running live system.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
