mirror.mephi.ru was timing out during lb binary_rootfs. Default to
deb.debian.org (Fastly-backed, geo-balanced) and allow overriding via
--mirror on build-in-container.sh/build.sh, or BEE_MIRROR in auto/config.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The NVSwitch-detection fix (983f41a) was necessary but not sufficient on
HGX B200 (Kaytus KR9288-X3): ExecCondition now correctly runs fabricmanager,
but nvidia-fabricmanager-start.sh then aborts with "Kernel module ib_umad
has not been loaded" on this NVL5+ board, leaving the fabric stuck in
"In Progress" and cascading into failed dcgmi diag/NVBandwidth/NCCL/Validate
GPU SAT tasks (confirmed via support bundle 20260729-165331).
Neither nvidia-fabricmanager nor DCGM declare the NVLink5 support packages
as apt dependencies (checked against the cuda-repo Packages index directly):
ib_umad must be modprobed before FM starts, nvlsm (InfiniBand Subnet Manager)
is launched internally by FM's own start script once present, and DCGM
dlopens libnvidia-nscq at runtime for NVSwitch health monitoring.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- page_scenario.go: onclick built via JSON.stringify() embedded raw double
quotes inside a double-quoted HTML attribute, truncating the attribute so
the click handler never compiled; pass the name through an escaped
data-scenario-name attribute instead.
- build.sh: overlay staging rsyncs (OVERLAY_DIR->stage, stage->includes.chroot)
ran without --delete, so a scenario removed from the repo (a9924b0) stayed
baked into every ISO built from the persistent stage cache since — the
"second script" in the Scenario page's list.
- blackbox: rewritten around a deterministic local zip + incremental
patch-the-changed-suffix onto removable media, instead of walking/copying
~90 files through a synchronous ntfs-3g FUSE mount every cycle. journalctl
captures are now "--since last sync" (were "--since boot", growing with
uptime) and metrics.db is excluded (was copied whole every cycle).
- scenario: nvbandwidth-acs-ab now escalates GPU count (same-socket pair,
other socket's pair, one cross-socket pair, all GPUs) under each ACS state
instead of always running all 6 GPUs at once, using a new `bee
gpu-bandwidth-groups` subcommand that discovers socket layout from
`nvidia-smi topo -m` at runtime — gpu_indices is host-specific, so this
can't be baked into the scenario file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bee.nvidia.mode=gsp-off loads the driver with NVreg_EnableGpuFirmware=0 and
skips nvidia-uvm/nvidia-modeset, so CUDA (nvbandwidth/dcgm/nccl) can't run —
and on Hopper H200 GSP is effectively mandatory anyway, so the mode is dead
weight in this test image. Remove it from every boot entry. fail-safe is now
bee.nvidia.mode=normal pci=realloc iommu=pt, so the GPUs come up with uvm and
the GPU test actually runs under iommu=pt (the point of the entry). All four
sources (grub.cfg + live.cfg.in templates, build.sh write_canonical_*) match.
The internal GSP auto-fallback in bee-nvidia-load (normal mode, GSP init
hangs >90s → retry with EnableGpuFirmware=0, then still load uvm) is left
intact — it serves converted SXM->PCIe cards on other test targets and keeps
CUDA working, unlike the removed explicit menu mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
no-GUI is now just the normal entry minus the GUI: bee.gui=off with
bee.nvidia.mode=normal (drops the previous gsp-off/pci=realloc it had picked
up). fail-safe is now simply the GPU/PCIe-workaround boot —
bee.nvidia.mode=gsp-off pci=realloc iommu=pt over the standard base — instead
of the old noapic/nodma/nomce/nolapic/nosmp minimal mode, so it can actually
run GPU tests (nvbandwidth) to verify the iommu=pt half of the vendor fix.
Applied identically across all four sources (grub.cfg + live.cfg.in templates
and build.sh write_canonical_* generators).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The GRUB and isolinux menus (and their two sources of truth — the staged
templates and build.sh's write_canonical_* generators that rewrite the built
menu) had drifted: GRUB shipped a lean set, isolinux carried extra
display-debug entries (gsp-off, kms, kms-gsp-off) and lacked a WIPE entry in
the canonical writer. Collapse all four to one set:
normal, load-to-RAM (default), no-GUI/troubleshooting (gsp-off, pci=realloc),
fail-safe, WIPE ALL DISKS, Memory Test, + UEFI Firmware Settings (EFI only)
The fail-safe entry gains iommu=pt (AMD-Vi passthrough) — the OS-side half of
the vendor fix for the MSI CG480-S6053 cross-socket nvbandwidth reboot; ACS
can't be toggled from a stock-kernel cmdline so that stays a BIOS setting.
default stays index 1 = toram (config.cfg set default=1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- gpuBandwidthSocketGroups: a single GPU whose NUMA node fails to resolve
no longer collapses the whole per-socket nvbandwidth split into one
fallback pass — it now folds into the last resolved group instead,
preserving isolation for the sockets that did resolve.
- blackbox discoverMarkedTargets: skip mounting/unmounting devices that
already have a running worker on every 2s discovery tick. This was
observed hammering the same USB target continuously (mount+unmount
every ~2s for the whole session) and contending with the worker's own
sync cycle, plausibly explaining multi-minute sync cycles seen on a
real crash bundle.
- syncFilesystem now calls syscall.Sync() directly instead of spawning
/bin/sync per copied file; blackbox mounts removable targets with
-o sync so writes are durable without relying on the app-level sync as
the primary mechanism.
- New platform.SetSyncBracketHook / satJob.syncBracket: blocks (with a
bounded timeout) on blackbox actually reaching removable media right
before and right after a diagnostic's real load step (nvbandwidth,
memtester, stress-ng, dcgmi diag, nccl, smartctl/nvme self-test...),
instead of only firing a fire-and-forget kick after the job's own log
file is written. A crash mid-load now has durable evidence the load
started, not just whatever streamed to the RAM-backed export dir before
blackbox's next scheduled cycle.
Found investigating a real support bundle where blackbox's last
successful sync (19:55:25) predated both the previous job finishing and
the crashing nvbandwidth job starting (19:56:57) — none of the crash
window ever reached durable media.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds BEE_HOSTNAME, derived with the same easy-bee-<variant>-v<version>
scheme already used for ISO_BASENAME/SQUASHFS_FILENAME/BEE_ISO_VOLUME, and
wires it into auto/config via lb config --hostname. Documented the shared
naming scheme as a runtime-flows.md invariant.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pci=realloc was forcing a full PCI resource reallocation on every default
and toram boot, added as a GPU BAR workaround for BIOS without Above 4G
Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and
silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS
and lspci both saw them fine. It now only applies to the no-GUI/gsp-off
troubleshooting entry, alongside a bible-local ADR for the two prior
rounds of SAT/blackbox durability work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
needs_full_build() compared mtimes of VERSIONS/package-lists/hooks against
a marker file, which is fragile: a failed build attempt never touches the
marker, and git checkouts/rsync can reorder mtimes relative to real edit
order. This let the fast-path silently reuse a stale squashfs (baked
against old package pins) with no error — root-caused after a driver/DCGM
pin change (590/DCGM 4.6.0 -> 580/DCGM 4.5.2) produced a ~290MB larger ISO
that only shrank back down with --clean-build.
Switch to a content hash (VERSIONS + package-lists + hooks + archives +
bootloaders + auto/config + Dockerfile) instead of mtimes, and add a real
version check in validate_iso_nvidia_runtime: extract dpkg status from the
squashfs and confirm the installed datacenter-gpu-manager-4-core version
matches DCGM_VERSION. The previous check only confirmed dcgmi/nv-hostengine
/dcgmproftester exist, which stays true across DCGM versions and can't
catch this class of staleness.
storcli64 enumerates these controllers but reports zero drives; storcli2
is the tool Broadcom ships for Tri-Mode/MegaRAID8 hardware and uses a
compatible JSON schema for drive listing. Wires storcli2 into both the
collector (structured drive data) and the webui RAID Management page
(dedup so a controller isn't double-listed if storcli64 already sees it
with zero drives), plus techdump raw collection and the ISO vendor-tool
build step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.52's ISO build failed at chroot_install-packages:
datacenter-gpu-manager-4-cuda13 : Depends: datacenter-gpu-manager-4-core
(= 1:4.5.2-1) but 1:4.6.0-1 is to be installed
cuda13/proprietary/proprietary-cuda13 all declare an exact-version
dependency on datacenter-gpu-manager-4-core, but core itself was never
listed, so apt's solver picked the newest core in the repo (4.6.0-1)
for the transaction and then reported the pinned variants as unmet
instead of resolving everything to 4.5.2-1. Pin core explicitly so
there's no ambiguity for the solver.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.51 pinned 580.126.20 (DGX OS 7.5.0's validated combo), but
build-nvidia-module.sh downloads the .run installer from
download.nvidia.com/XFree86/Linux-x86_64/<version>/, a separate catalog
from the CUDA apt repo checked previously. 580.126.20 404s there —
same failure class as the original 550.54.15 CDN-404 incident.
Checked all 580.x point releases on that CDN; only some are mirrored
(580.126.09, 580.159.03/.04, 580.173.02 are 200; 580.126.16, 580.126.20,
580.167.08 are 404). Switched to 580.159.03: confirmed downloadable,
confirmed present as .deb in the apt repo, and it's the exact driver
version running on the real DGX Spark.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
590.48.01/DCGM 4.6.0-1 was never a deliberate compatibility choice — the
driver version traces back to a CDN 404 on the originally pinned
550.54.15 (commit 559fc29), and DCGM was bumped twice since purely
because NVIDIA pulled the previously-pinned core package from the repo
(6dee8f3, b2e177a). The 590 branch only has two point releases so far,
and its nvvs plugin fails dcgmi diag with "Detected driver major
version 0 is not between the required versions 346 and 2000" on H100 —
confirmed via a staging support bundle.
Switch to 580.126.20 / DCGM 4.5.2-1, the combo NVIDIA's own DGX OS 7.5.0
release notes validate for H100. Both packages confirmed present in the
NVIDIA CUDA debian12/x86_64 apt repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- sat.go: DetectGPUVendor lspci fallback now checks GPU device classes
([0300]/[0302]/[0380]) per line instead of scanning the whole output for
vendor name; AMD EPYC servers have dozens of AMD-branded PCIe entries
(Root Complex, IOMMU, Host Bridge) that were triggering the old check
- blackbox.go: fix deadlock in finishCycle — it held w.mu while calling
persistState(), which acquires rt.mu then re-acquires w.mu inside
persistStateLocked(); now w.mu is released before persistState()
- build.sh: remove NVIDIA-specific overlay files (bee-gpu-burn,
bee-john-gpu-stress, bee-nccl-gpu-stress, bee-nvidia-recover,
bee-dcgmproftester-staggered, bee-check-nvswitch,
nvidia-fabricmanager.service.d/) for non-nvidia build variants
- bee-selfheal: gate NVIDIA recovery on BEE_GPU_VENDOR=nvidia so the
script does not attempt to restart bee-nvidia.service on NOGPU builds
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- saa_dmi.go: set cmd.Dir=/usr/local/bin on all saa exec calls so
acpica_bin/acpidump is found relative to correct working directory
- build.sh: copy all saa companion dirs (acpica_bin, ExternalData,
tool, stunnel, GO_SNMP) to /usr/local/bin/ preserving structure
- iso/vendor: add acpica_bin/acpiexec, ExternalData/, tool/gpu/nVidia/x64/,
tool/USBController/, stunnel/, GO_SNMP/ from SAA 1.5.0 release package
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extract saa 1.5.0 (Linux x86_64) into iso/vendor/saa — baked into ISO
at /usr/local/bin/saa via the existing vendor loop in build.sh
- Add saa to the vendor tool loop in iso/builder/build.sh
- Rename the web UI card from "SAA - DMI" to "Supermicro - DMI"
- Remove the redundant description hint about saa on PATH
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NVIDIA removed datacenter-gpu-manager-4-core 1:4.5.3-1 from the
repository and published 1:4.6.0-1. The cuda13 and proprietary
packages still declared an exact-version dependency on 4.5.3-1 core,
making the old pin unresolvable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nvtop pulled nvidia-tesla-470-* via Recommends into the nogpu build.
Move it from bee.list.chroot into bee-nvidia and bee-amd lists so it
only appears in GPU variants.
Also remove the stray git-bible/ directory (was not gitignored) and
move grub-bitmap-error docs into bible-local/docs/.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- storage: add jsonInt64 dual-format unmarshaler to handle lsblk output
change in util-linux 2.38 (LOG-SEC/PHY-SEC now emitted as JSON
integers, not quoted strings); fixes SATA disks invisible on Debian 12
- pcie: detect NVLink bridge mezzanine CX-7 cards (Mellanox x2, no host
net ifaces, DeviceName contains "NVLINK" in lspci -v) and mark them
with device_class="NVLinkBridge"; escalate PCIe link speed downgrade to
Critical for these cards (Gen3 on a fixed internal connector = hardware
fault, not a transient warning)
- pcie: cross-reference nvidia-smi topo to capture NVLink bond counts and
active status for all NVLink bridge cards
- packages: add infiniband-diags to ISO; provides ibstat required by
nvidia-fabricmanager-start.sh to enumerate IB devices before FM launch
(absence causes CUDA_ERROR_SYSTEM_NOT_READY)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Network: green if at least one interface has IPv4 (drop PARTIAL state).
Bee Services: treat inactive as OK — oneshot services (bee-sshsetup,
bee-preflight, bee-network, bee-audit, etc.) complete successfully and
exit to inactive; only failed is a real problem.
nvidia-fabricmanager: add ExecCondition=bee-check-nvswitch drop-in so
the service is silently skipped (inactive, not failed) on systems
without NVSwitch hardware (e.g. H200 NVL with direct NVLink, no
NVSwitch chips). bee-check-nvswitch detects NVSwitch via lspci
(vendor 10de, class 0680).
bee-nvidia.service: add ConditionPathExists=/usr/local/bin/bee-nvidia-load
so the unit is a no-op if somehow present in a non-nvidia build.
bee-boot-status: read /etc/bee-gpu-vendor and exclude bee-nvidia from
CRITICAL/ALL on non-nvidia builds, preventing boot hang if the unit
is unexpectedly present.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
EASY_BEE_NVIDIA_LEGACY_V<date> is 33 characters; ISO 9660 volid is
limited to 32. Compute the maximum token length dynamically from the
prefix length and trim ISO_VERSION_LABEL_TOKEN with cut before
assembling BEE_ISO_VOLUME. All four variants now fit within the limit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All intermediate build artifacts (binaries, live-build work dirs, overlay
stages, NVIDIA/NCCL/cuBLAS/john caches) now live under dist/cache/.
Final ISOs go to dist/release/ instead of scattered dist/easy-bee-v*/ and iso/out/.
dist/ is already gitignored, iso/out/ entry removed as redundant.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Installs a local-premount initramfs hook that intercepts bee.wipe=all before
squashfs is mounted. Shows a numbered disk selection TUI (pure POSIX sh), wipes
selected disks (nvme format / blkdiscard / dd fallback), syncs, and reboots.
Works even when squashfs fails to mount.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a "WIPE ALL DISKS" entry to both GRUB and isolinux menus (bee.wipe=all).
Includes bee-wipe-disks for manual use from a running live system.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Squashfs versioning:
- ISO now contains filesystem-v<VERSION>.squashfs instead of the generic
filesystem.squashfs, making it immediately visible which build is
running (visible in /run/live/medium/live/ at boot time).
- Full build path: rename filesystem.squashfs → filesystem-v*.squashfs
after lb build, before lb binary_checksums/binary_iso.
- Fast path: find and unpack whatever filesystem*.squashfs exists, repack
as the new versioned name, remove the old file, update the ISO.
- needs_full_build: accept any filesystem*.squashfs so version changes
alone don't force a full rebuild.
Media selection hardening:
- Add live-media=/dev/disk/by-label/<LABEL> to the kernel boot line in
addition to the existing live-media-label=<LABEL>. live-boot will now
open exactly the labeled device rather than scanning all block devices,
preventing accidental use of squashfs files from local disks or
stale virtual media attached via IPMI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mksquashfs 4.5.1 (bookworm) writes a non-SQUASHFS_INVALID_BLK value for
xattr_id_table_start in the superblock even when -no-xattrs is passed, if
the source chroot contains POSIX ACL xattrs set by dpkg at install time.
Linux 6.1 squashfs driver then fails with "unable to read xattr id index
table" and refuses to mount the filesystem.
Strip all xattrs from the chroot via Python3 (already present) immediately
before mksquashfs runs. With an xattr-free source tree the resulting
squashfs is guaranteed to have SQUASHFS_INVALID_BLK in the xattr field.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--chroot-squashfs-compression-options does not exist in live-build
bookworm (1:20230502). The correct mechanism is the MKSQUASHFS_OPTIONS
environment variable read by binary_rootfs.
Export MKSQUASHFS_OPTIONS="-no-xattrs" before lb build so live-build's
binary_rootfs picks it up, and add -no-xattrs explicitly to every
direct mksquashfs call in build.sh (fast-path repack and the dormant
split-layers function). Remove the invalid lb config option.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--chroot-squashfs-options is not a valid lb_config option; the correct
name is --chroot-squashfs-compression-options. Without this fix lb config
aborts immediately, so the -no-xattrs flag (which prevents the
"unable to read xattr id index table" boot failure) was never applied.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Kernel squashfs driver fails with "unable to read xattr id index table"
when the squashfs contains POSIX ACL xattrs (system.posix_acl_*) written
by mksquashfs as unrecognised entries. This caused every built ISO to
drop to an initramfs shell at boot.
Add -no-xattrs to mksquashfs options so xattrs are stripped at build
time. xattrs are not needed in a live read-only rootfs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
split_live_squashfs_layers moved /usr out of filesystem.squashfs into a
separate 10-usr.squashfs, leaving a rootfs skeleton that live-boot
(1:20230131+deb12u1) cannot mount: the initramfs panics with
"Can not mount /dev/loop0 ... filesystem.squashfs".
live-boot in bookworm expects a single self-contained filesystem.squashfs.
Revert to the standard single-squashfs layout and remove the dead
multi-squashfs guard in needs_full_build().
The split_live_squashfs_layers function is kept for future reference.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
trap RETURN is a bash extension not supported by /bin/sh on Debian.
With set -e active the unsupported trap call exited the build immediately
after lb build, before bootloader sync and ISO copy steps ran.
Remove both trap RETURN lines — explicit rm -rf at the end of the
function is sufficient for cleanup on the happy path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
