diff --git a/bible-local/architecture/squashfs-layers.md b/bible-local/architecture/squashfs-layers.md new file mode 100644 index 0000000..57b73ee --- /dev/null +++ b/bible-local/architecture/squashfs-layers.md @@ -0,0 +1,111 @@ +# Semantic SquashFS Layers + +**Module:** `squashfs-layers` (v1.0) - `iso/builder/lib/squashfs-layers.sh` + +## Why + +The live medium used to ship one `filesystem-v.squashfs` of about 2.8 GB. +Booting through a BMC / IPMI virtual CD reads that single file sequentially +during the live-boot `toram` copy. If the redirected medium drops off the bus +part-way through (observed on v14), the whole copy is lost and the boot fails +with `No supported filesystem images found at /live`. + +Splitting the root filesystem into several self-contained squashfs layers means +a mid-copy failure only costs the layer in flight, and the retry loop re-reads +far less data. + +This is a **resilience / reduced-re-read mechanism**. It is **not** a fix for +the underlying BMC virtual-media instability - see +`decisions/2026-09-04-squashfs-semantic-layers.md` and +`decisions/2026-09-04-supported-systems-minimum-16gb-ram.md`. + +## Layers (NVIDIA variants: `nvidia`, `nvidia-legacy`) + +| Layer | Contents | Ownership | +|---|---|---| +| `filesystem-v-00-base.squashfs` | Debian rootfs, kernel + modules, systemd, Bee, networking, CLI diagnostic tools, dpkg database. Boot-critical, read first. | catch-all: any file not claimed by a higher layer | +| `filesystem-v-05-firmware.squashfs` | Device firmware for NICs / wifi / non-NVIDIA GPUs | dpkg: `firmware-*` (but not `firmware-nvidia-*`, which rides with the driver; not `*-microcode`, which stays in base) | +| `filesystem-v-08-desktop.squashfs` | Local-console GUI: X.org, `lightdm`, `openbox`, mesa/llvm, GTK, chromium, mupdf, fonts. SSH / headless never touches this layer. | dpkg: `xserver-xorg*`, `xfonts-*`, `x11-*`, `lightdm*`, `openbox`, `chromium*`, `mupdf*`, `libllvm*`, `libgl1-mesa-dri`, `libglx-mesa0`, `glx-alternative-mesa`, `libgtk-3-*` / `libgtk2.0-*` / `libgtkmm-*` / `libgtksourceview-*`, `libpango-*` / `libcairo2` / `libgdk-pixbuf-*`, `fonts-*`, `feh`, `scrot` | +| `filesystem-v-10-nvidia-driver.squashfs` | NVIDIA kernel modules (`*.ko`), driver userspace (`libnvidia-*`, `libcuda.so*`), `nvidia-smi`, GSP firmware, OpenCL ICD, modprobe config, alternatives, `bee-nvidia-*` units | dpkg: `nvidia-modprobe`, `nvidia-kernel-common`, `glx-alternative-nvidia`, `nvidia-tesla-*`, `firmware-nvidia-*`, `libnvidia-*`, `nvtop`, `clinfo`, `ocl-icd-libopencl1`. injected: `usr/local/lib/nvidia/*.ko`, `usr/local/bin/nvidia-smi`, `usr/lib/libcuda.so*`, `usr/lib/libnvidia-*`, `usr/lib/firmware/nvidia/*`, `etc/OpenCL/vendors/nvidia.icd`, `bee-nvidia.service`, `bee-nvidia-load`/`-recover`, `bee-check-nvswitch` | +| `filesystem-v-20-nvidia-platform.squashfs` | Fabric Manager, `libnvidia-nscq`, `nvlsm`, DCGM core + non-CUDA proprietary DCGM, related units | dpkg: `nvidia-fabricmanager`, `libnvidia-nscq`, `nvlsm`, `libibumad3`, `datacenter-gpu-manager-4-core`, `datacenter-gpu-manager-4-proprietary`. injected: `nvidia-fabricmanager.service.d/*` | +| `filesystem-v-30-nvidia-cuda-libs.squashfs` | CUDA userspace runtime (cuBLAS / cuBLASLt / cudart), NCCL, nccl-tests, bee GPU stress worker | injected: `usr/lib/libnccl.so*`, `usr/lib/libcublas.so*`, `usr/lib/libcublasLt.so*`, `usr/lib/libcudart.so*`, `all_reduce_perf`, `bee-gpu-burn-worker`, `bee-gpu-burn`, `bee-nccl-gpu-stress`, `bee-john-gpu-stress` | +| `filesystem-v-40-nvidia-dcgm-cuda.squashfs` | CUDA-linked DCGM components (`dcgmproftester` CUDA kernels) - split out from 30 because they are large | dpkg: `datacenter-gpu-manager-4-cuda13`, `datacenter-gpu-manager-4-proprietary-cuda13`. injected: `bee-dcgmproftester-staggered` | + +`amd`, `nogpu`: single `filesystem-v.squashfs`, no `filesystem.module`. The +builder leaves the monolith untouched for these variants. + +## Classification rules + +- **dpkg file ownership** (`var/lib/dpkg/info/*.list`), never a path substring. + A file is not moved to an NVIDIA layer merely because its path contains + `nvidia`. +- Files that `build.sh` injects directly from the build cache and the project + overlay belong to no `.deb`; they are classified by the explicit + `bee_layer_injected_rules` table (extended-regex on the merged-usr-canonical + relative path). Injected rules override dpkg ownership. +- Pre-merged-usr paths dpkg records (`/bin`, `/sbin`, `/lib`, `/lib64`) are + canonicalised to `/usr/...` before matching. +- Anything unclaimed falls to `00-base`. +- `bin`/`sbin`/`lib`/`lib64` are force-kept in `00-base` regardless of dpkg + records (some `firmware-*` `.list` files carry a bare `/lib` entry). +- The classifier fails the build if the partition is not complete and disjoint, + if a merged-usr compat symlink is not in base, or if any upper layer would + carry a real top-level `bin`/`sbin`/`lib`/`lib64` path. +- Output is deterministic: every list is `LC_ALL=C` sorted, `mksquashfs` runs + with `-processors 1` and fixed options. It does not depend on `find` order or + locale. +- Published on the medium: `live/filesystem.layers.txt` (per-layer file counts). + +## Ordering (must stay consistent everywhere) + +Module order: `00-base`, `05-firmware`, `08-desktop`, `10-nvidia-driver`, +`20-nvidia-platform`, `30-nvidia-cuda-libs`, `40-nvidia-dcgm-cuda`. + +`live-boot 1:20230131` with `MODULE=filesystem` (the default) reads +`live/filesystem.module` verbatim - an ordered list of image names, one per +line - and stacks them so the **last listed image has the highest OverlayFS +priority**. If the module file were absent it would fall back to a +locale-sensitive lexical `sort`; the `NN-` numeric prefixes make that +equivalent, but the module file is authoritative. + +The same order is used by: + +- `bee-install`: reads `filesystem.module`, unpacks each layer with + `unsquashfs -f` (last write wins), aborts install on any layer failure. +- The builder fast path: currently **disabled** for a multi-layer medium; it + forces a full `lb build` + deterministic re-split + (`needs_full_build` returns true, `fast_path_repack_squashfs` hard-refuses). + +So a higher-numbered layer always wins a conflict, in live-boot, in a disk +install, and in a rebuild. + +## Size budget + +Target 500-700 MiB compressed per layer (`BEE_LAYER_TARGET_MIB`, soft warning). +Hard ceiling 800 MiB (`BEE_LAYER_MAX_MIB`): a layer over the ceiling fails the +build so a layer cannot silently grow back toward the monolith. If a semantic +layer is genuinely larger, split it further by purpose or package family. This +is why `30-nvidia-cuda-libs` / `40-nvidia-dcgm-cuda` are separate, and why the +~1 GB base rootfs was split into `00-base` + `05-firmware` + `08-desktop` +(2026-09-04): device firmware and the local-console GUI are self-contained and +not on the SSH / headless path. + +## Verification (in `build.sh`, both build paths) + +1. `unsquashfs -s` + full `unsquashfs -strict-errors` extraction of every layer. +2. Reconstruct the merged rootfs in module order; assert `/usr/sbin/init`, + `/usr/lib/systemd/systemd`, `/usr/local/bin/bee`, the merged-usr symlinks, and the + NVIDIA sentinels (`nvidia-smi`, `nvidia.ko`, GSP firmware, `libnvidia-ml`, + `dcgmi`, `nv-hostengine`, `dcgmproftester`, `libcudart`/`libcublas`/`libnccl`, + `bee-nvidia.service`). +3. `validate_iso_squashfs_layers`: the final ISO carries `filesystem.module`, + the module list and the `live/*.squashfs` set match exactly, every layer is + under the size ceiling, and a multi-layer variant never ships a single + giant squashfs. +4. `validate_iso_rootfs_layout` / `validate_iso_nvidia_runtime` already iterate + every `live/*.squashfs`. + +Regression tests: `iso/builder/test-squashfs-layers.sh` (classification, +completeness, per-layer validity, merged bootability, module order, missing +layer, size ceiling) and the multi-layer guard in +`iso/builder/test-build-libs.sh`. Both run at the top of every `build.sh`. diff --git a/bible-local/decisions/2026-09-04-squashfs-semantic-layers.md b/bible-local/decisions/2026-09-04-squashfs-semantic-layers.md new file mode 100644 index 0000000..a303d57 --- /dev/null +++ b/bible-local/decisions/2026-09-04-squashfs-semantic-layers.md @@ -0,0 +1,67 @@ +# Split the live medium into semantic SquashFS layers + +**Date:** 2026-09-04 +**Status:** active + +## Context + +EASY-BEE shipped one `filesystem-v.squashfs` of about 2.8 GB. v13 booted +through the BMC virtual CD; v14 did not. During the live-boot `toram` copy, +`rsync` reads that single file sequentially and the read stopped part-way +through (around the middle), after which live-boot moved the partial RAM copy +over the medium and the boot failed. + +The squashfs inside the build artifact was verified intact +(`unsquashfs -strict-errors`). Supported systems have at least 16 GB of RAM +(`2026-09-04-supported-systems-minimum-16gb-ram.md`), so this is not memory +exhaustion. The failure is in the virtual-media read path. + +live-boot v14.03 already supports several `/live/*.squashfs` merged via +OverlayFS, and `bee-install` already unpacks multiple squashfs in lexical +order. The old builder fast path assumed exactly one squashfs (it picked the +first and deleted the rest). + +## Decision + +- After a full `lb build`, the builder deterministically splits the monolith + into semantic layers (see `architecture/squashfs-layers.md`): `00-base`, + `05-firmware`, `08-desktop`, `10-nvidia-driver`, `20-nvidia-platform`, + `30-nvidia-cuda-libs`, `40-nvidia-dcgm-cuda` for NVIDIA variants; a single + untouched squashfs for `amd` / `nogpu`. The base rootfs was ~1 GB compressed, + so device firmware (`firmware-*`) and the local-console GUI stack (X.org, + lightdm, mesa, chromium, mupdf, fonts) - neither on the SSH / headless path - + were carved out to keep every layer near the 500-700 MiB target. +- Layer boundaries come from **dpkg file ownership** plus an explicit, + code-described rule table for the non-`.deb` files `build.sh` injects. A path + is never moved to an NVIDIA layer just because it contains `nvidia`. +- Layer names carry the project version (`filesystem-v-NN-slug.squashfs`), + matching the existing `filesystem-v.squashfs` scheme. +- An explicit `live/filesystem.module` fixes the order live-boot uses. The + same order governs `bee-install` and any rebuild. A higher-numbered layer + always wins a conflict. +- Each layer stays a self-contained valid squashfs, kept under an 800 MiB + compressed ceiling (build fails otherwise). merged-usr (`/bin`, `/sbin`, + `/lib`, `/lib64` as symlinks) is preserved; upper layers never carry a real + top-level compat directory and never use opaque/whiteout semantics. +- The monolith is deleted only after every layer is built, individually + verified, and successfully re-merged into a bootable rootfs. Any failure + aborts the build before the ISO is assembled - a partial layer set is never + published. +- The builder fast path is **disabled for a multi-layer medium**: it forces a + full `lb build` + re-split. `fast_path_repack_squashfs` hard-refuses to run. + A layer-aware repack may come later. +- `9013-toram-retry` now records the real `rsync` exit code (it printed a false + `rc=0`) and no longer claims it resumes the tail of the current file + (`rsync` without `--partial` keeps only fully-copied files). No unsafe + partial-file resume is introduced. + +## Consequences + +- A mid-copy virtual-media drop now costs one layer (target 500-700 MiB, hard + ceiling 800 MiB), not the whole 2.8 GB rootfs; the retry re-reads only that + layer. +- Every NVIDIA ISO build now runs the full path (no fast path) until a + layer-aware repack exists. Build time is unchanged for full builds; + overlay-only iterations lose the fast path for NVIDIA variants. +- The split does not address the root BMC virtual-media instability; it reduces + the blast radius and the repeatedly-read volume. diff --git a/bible-local/decisions/README.md b/bible-local/decisions/README.md index bd3bbde..8de0aa9 100644 --- a/bible-local/decisions/README.md +++ b/bible-local/decisions/README.md @@ -19,3 +19,4 @@ One file per decision, named `YYYY-MM-DD-short-topic.md`. | 2026-09-03 | PCIe link verdict comes only from the existing real-traffic GPU bandwidth SAT | active | | 2026-09-03 | Runtime Copy to RAM succeeds only after active loop devices move to tmpfs | active | | 2026-09-04 | Supported systems have at least 16 GB of RAM | active | +| 2026-09-04 | Split the live medium into semantic SquashFS layers | active | diff --git a/bible-local/docs/iso-build-rules.md b/bible-local/docs/iso-build-rules.md index e1d9544..e9c5dd4 100644 --- a/bible-local/docs/iso-build-rules.md +++ b/bible-local/docs/iso-build-rules.md @@ -15,6 +15,33 @@ This applies to: - `iso/builder/config/package-lists/*.list.chroot` - Any package referenced in bootloader configs, hooks, or overlay scripts +## SquashFS layer rule + +The NVIDIA live medium is not one squashfs. `build.sh` splits the monolith into +semantic layers (`filesystem-v-NN-*.squashfs`) after the full `lb build`, +driven by `iso/builder/lib/squashfs-layers.sh`. See +`bible-local/architecture/squashfs-layers.md` for the layer model and +`decisions/2026-09-04-squashfs-semantic-layers.md` for why. + +Rules: + +- Classify by dpkg file ownership, not path substring. New non-`.deb` files + injected by `build.sh` must get an explicit entry in + `bee_layer_injected_rules` (or they fall to `00-base`). +- `live/filesystem.module` is authoritative for layer order; keep it in sync + with the `NN-` prefixes. Later layer wins a conflict, in live-boot, + `bee-install`, and any rebuild. +- Keep each layer a self-contained valid squashfs under the 800 MiB ceiling. + Split a semantic layer further rather than blindly slicing bytes. +- Never let an upper layer carry a real top-level `bin`/`sbin`/`lib`/`lib64` + path - that breaks merged-usr the same way the fast-path bug did. +- The builder deletes the monolith only after every layer verifies and + re-merges into a bootable rootfs. Any failure must abort before ISO assembly. +- The fast path stays disabled for a multi-layer medium until a layer-aware + repack exists. Do not "take the first squashfs". +- `test-squashfs-layers.sh` and `test-build-libs.sh` run at the top of every + build; keep them green. + ## Bootloader sync rule The ISO has two canonical bootloader templates whose live entries must remain diff --git a/iso/README.md b/iso/README.md index 433b439..771ac4c 100644 --- a/iso/README.md +++ b/iso/README.md @@ -47,6 +47,8 @@ sh iso/builder/build-in-container.sh --cache-dir /path/to/cache - The builder image is automatically rebuilt if the local tag exists for the wrong architecture. - The live ISO boots with Debian `live-boot` `toram`, so the read-only medium is copied into RAM during boot and the runtime no longer depends on the original USB/BMC virtual media staying present. +- The NVIDIA variants ship the root filesystem as several semantic SquashFS layers (`live/filesystem-v-NN-*.squashfs`) plus an explicit `live/filesystem.module` that fixes their OverlayFS order. This bounds the data a `toram` retry must re-read after a virtual-media drop; it is not a fix for virtual-media instability. `amd` / `nogpu` keep a single squashfs. See `bible-local/architecture/squashfs-layers.md`. +- The builder splits the layers deterministically after the full `lb build`, verifies each one, re-merges them into a bootable rootfs, and only then deletes the monolith. The fast path is disabled for a multi-layer medium (it forces a full build). - Target systems require at least 16 GB of installed RAM for the full compressed live medium plus normal runtime overhead. On supported hardware, do not classify a mid-copy failure as low RAM without direct `ENOSPC`, OOM, or tmpfs-limit evidence. - The NVIDIA variant installs DCGM 4 packages matched to the CUDA user-mode driver major version. For driver branch `580` / CUDA `13.x`, the package family is `datacenter-gpu-manager-4-cuda13` rather than legacy `datacenter-gpu-manager`. - Override the container platform only if you know why: diff --git a/iso/builder/build.sh b/iso/builder/build.sh index 06a5fbd..3d88e60 100755 --- a/iso/builder/build.sh +++ b/iso/builder/build.sh @@ -79,8 +79,47 @@ export GOCACHE GOMODCACHE . "${BUILDER_DIR}/lib/iso-validation.sh" . "${BUILDER_DIR}/lib/template.sh" . "${BUILDER_DIR}/lib/bootloader.sh" +. "${BUILDER_DIR}/lib/squashfs-layers.sh" . "${BUILDER_DIR}/lib/fast-path.sh" +# Whether this variant ships a multi-layer live medium (semantic squashfs +# layers + live/filesystem.module) instead of one monolithic squashfs. +case "${BUILD_VARIANT}" in + nvidia|nvidia-legacy) BEE_MULTILAYER_SQUASHFS=1 ;; + *) BEE_MULTILAYER_SQUASHFS=0 ;; +esac + +# bee_split_squashfs_layers +# Deterministically replace the monolith with the semantic layer set. Runs as a +# set -e subshell: any failed sub-step aborts the build (via run_step) before the +# outer ISO is assembled, so a partial layer set is never published. +bee_split_squashfs_layers() ( + set -e + _mono="$1" + _live="$2" + [ -f "${_mono}" ] || { echo "ERROR: monolith squashfs not found: ${_mono}" >&2; exit 1; } + + _wd="$(mktemp -d "${CACHE_ROOT}/layer-split-${BUILD_VARIANT}.XXXXXX")" + trap 'rm -rf "${_wd}"' EXIT + _root="${_wd}/root" + _cls="${_wd}/cls" + echo "=== splitting $(basename "${_mono}") into semantic layers ===" + unsquashfs -d "${_root}" "${_mono}" + + bee_layer_classify "${_root}" "${BUILD_VARIANT}" "${_cls}" + bee_layer_build "${_root}" "${_cls}" "${_live}" "${PROJECT_VERSION_EFFECTIVE}" "${BUILD_VARIANT}" + bee_layer_verify_each "${_live}" "${PROJECT_VERSION_EFFECTIVE}" "${BUILD_VARIANT}" + bee_layer_merge "${_live}" "${PROJECT_VERSION_EFFECTIVE}" "${BUILD_VARIANT}" "${_wd}/merged" + bee_layer_write_module_file "${_live}" "${PROJECT_VERSION_EFFECTIVE}" "${BUILD_VARIANT}" + + # Publish the ownership map on the medium for bee-install / debugging, then + # drop the monolith - only now that every layer exists and verifies. + cp "${_cls}/classify-report.txt" "${_live}/filesystem.layers.txt" + rm -f "${_mono}" + echo "=== semantic layer split complete ===" + ls -la "${_live}"/filesystem-v*.squashfs "${_live}/filesystem.module" +) + resolve_project_version() { if [ -n "${BEE_VERSION:-}" ]; then echo "${BEE_VERSION}" @@ -185,6 +224,10 @@ LOG_OUT="${LOG_DIR}/build.log" start_build_log +# Fail fast on builder-library regressions before the 30-60 min ISO build. +run_step "builder library tests" "01-lib-tests" sh "${BUILDER_DIR}/test-build-libs.sh" +run_step "squashfs layer tests" "02-squashfs-layer-tests" sh "${BUILDER_DIR}/test-squashfs-layers.sh" + # Auto-detect kernel ABI: refresh apt index, then query current linux-image-amd64 dependency. # If headers for the detected ABI are not yet installed (kernel updated since image build), # install them on the fly so NVIDIA modules and ISO kernel always match. @@ -649,6 +692,8 @@ if ! needs_full_build; then validate_iso_grub_assets "$ISO_RAW" validate_iso_nvidia_runtime "$ISO_RAW" validate_iso_rootfs_layout "$ISO_RAW" + validate_iso_squashfs_layers "$ISO_RAW" + validate_iso_media_integrity "$ISO_RAW" cp "$ISO_RAW" "$ISO_OUT" echo "" echo "=== done (${BUILD_VARIANT}, fast-path) ===" @@ -677,6 +722,15 @@ if [ -f "${_std_sq}" ] && [ "${_std_sq}" != "${_ver_sq}" ]; then mv "${_std_sq}" "${_ver_sq}" echo "=== squashfs renamed: filesystem.squashfs -> ${SQUASHFS_FILENAME} ===" fi + +# Split the monolith into semantic layers before checksums / ISO assembly so +# md5sum.txt covers the layers and the module file, and the outer ISO carries +# them. Single-layer variants (amd, nogpu) keep the monolith untouched. +if [ "${BEE_MULTILAYER_SQUASHFS}" = "1" ]; then + run_step "split squashfs into semantic layers" "90b-squashfs-layers" \ + bee_split_squashfs_layers "${_ver_sq}" "${LB_DIR}/binary/live" +fi + reset_live_build_stage "${LB_DIR}" "binary_checksums" reset_live_build_stage "${LB_DIR}" "binary_iso" reset_live_build_stage "${LB_DIR}" "binary_zsync" @@ -711,6 +765,8 @@ if [ -f "$ISO_RAW" ]; then validate_iso_grub_assets "$ISO_RAW" validate_iso_nvidia_runtime "$ISO_RAW" validate_iso_rootfs_layout "$ISO_RAW" + validate_iso_squashfs_layers "$ISO_RAW" + validate_iso_media_integrity "$ISO_RAW" cp "$ISO_RAW" "$ISO_OUT" hash_heavy_config > "${FULL_BUILD_HASH_FILE}.new" printf '%s\n' "${DEBIAN_KERNEL_ABI}" > "${FULL_BUILD_ABI_FILE}.new" diff --git a/iso/builder/config/hooks/normal/9013-toram-retry.hook.chroot b/iso/builder/config/hooks/normal/9013-toram-retry.hook.chroot index 073cee8..3d03f1a 100755 --- a/iso/builder/config/hooks/normal/9013-toram-retry.hook.chroot +++ b/iso/builder/config/hooks/normal/9013-toram-retry.hook.chroot @@ -12,8 +12,15 @@ # adds several full attempts with a geometrically growing pause between them # (15s, 30s, 60s, 120s, 240s, 480s, 900s), giving the virtual media time to # re-enumerate. Between attempts the medium is unmounted, waited on, and -# remounted. rsync resumes (already-copied files are skipped), so a retry that -# only needs the tail of the squashfs finishes quickly. +# remounted. +# +# rsync here runs without --partial, so it keeps only files it copied in full; +# a file interrupted mid-transfer is discarded and re-read from the start on the +# next attempt. The medium is split into semantic squashfs layers +# (filesystem-v-NN-*.squashfs, see lib/squashfs-layers.sh), so a retry only +# re-reads the layer that was in flight, not the whole ~2.8 GB rootfs. We do +# NOT enable --partial / --append: resuming a partial squashfs without a +# post-copy integrity check would risk booting a truncated layer. set -e TORAM_SCRIPT="/usr/lib/live/boot/9990-toram-todisk.sh" @@ -56,12 +63,19 @@ bee_rsync_retry () echo " * bee: toram copy attempt ${_try}/${_maxtry} (medium ${_dev:-?} ${_fst:-?})" 1>/dev/console if rsync -a --progress --timeout=180 ${_src}/* ${_dst} 1>/dev/console + then + _rc=0 + else + _rc=$? + fi + + if [ "${_rc}" -eq 0 ] then echo " * bee: toram copy completed on attempt ${_try}" 1>/dev/console return 0 fi - echo " * bee: toram copy FAILED (rsync rc=$?) on attempt ${_try}" 1>/dev/console + echo " * bee: toram copy FAILED (rsync rc=${_rc}) on attempt ${_try}" 1>/dev/console if [ "${_try}" -ge "${_maxtry}" ] then diff --git a/iso/builder/lib/fast-path.sh b/iso/builder/lib/fast-path.sh index 855c711..765447b 100755 --- a/iso/builder/lib/fast-path.sh +++ b/iso/builder/lib/fast-path.sh @@ -101,6 +101,15 @@ needs_full_build() { -name 'filesystem*.squashfs' 2>/dev/null | head -1) [ -n "$_any_sq" ] || return 0 + # Multi-layer live medium: the fast path only knows how to repack a single + # squashfs and would drop every other layer. Force a full build until a + # layer-aware repack exists. Detected by the module file or >1 squashfs. + if [ -f "${BUILD_WORK_DIR}/binary/live/filesystem.module" ] || \ + [ "$(find "${BUILD_WORK_DIR}/binary/live" -maxdepth 1 -name 'filesystem*.squashfs' 2>/dev/null | wc -l)" -gt 1 ]; then + echo "=== full build required: previous build produced a multi-layer squashfs medium ===" + return 0 + fi + _old_abi="$(cat "${FULL_BUILD_ABI_FILE}" 2>/dev/null)" if [ "${DEBIAN_KERNEL_ABI}" != "$_old_abi" ]; then echo "=== full build required: kernel ABI changed (${_old_abi:-unknown} -> ${DEBIAN_KERNEL_ABI}) ===" @@ -126,6 +135,14 @@ needs_full_build() { # Fast path: unsquash existing filesystem, rsync overlay on top, repack. # CACHE_ROOT must have enough free space for the extracted root filesystem. fast_path_repack_squashfs() ( + # Hard stop: this path takes the first squashfs and deletes the rest. On a + # multi-layer medium that silently loses every layer but base. needs_full_build + # already forces a full build here; this guard makes the invariant explicit. + if [ -f "${BUILD_WORK_DIR}/binary/live/filesystem.module" ] || \ + [ "$(find "${BUILD_WORK_DIR}/binary/live" -maxdepth 1 -name 'filesystem*.squashfs' 2>/dev/null | wc -l)" -gt 1 ]; then + echo "ERROR: fast_path_repack_squashfs refuses to run on a multi-layer squashfs medium" >&2 + exit 1 + fi _old_sq=$(find "${BUILD_WORK_DIR}/binary/live" -maxdepth 1 \ -name 'filesystem*.squashfs' | sort | head -1) _sq="${BUILD_WORK_DIR}/binary/live/${SQUASHFS_FILENAME}" diff --git a/iso/builder/lib/iso-validation.sh b/iso/builder/lib/iso-validation.sh index c5a492f..5914c9d 100755 --- a/iso/builder/lib/iso-validation.sh +++ b/iso/builder/lib/iso-validation.sh @@ -645,6 +645,110 @@ rootfs_layout_fail() { exit 1 } +squashfs_layers_fail() { + echo "ERROR: $1" >&2 + exit 1 +} + +# Validate the semantic squashfs layering in the final ISO: +# - NVIDIA variants must ship live/filesystem.module and >= 2 layers +# - every squashfs in live/ is listed in the module file and vice versa +# - no single layer exceeds the agreed compressed-size ceiling +# - a single giant squashfs is rejected for a multi-layer variant +validate_iso_squashfs_layers() { + iso_path="$1" + echo "=== validating squashfs layers in ISO ===" + + [ -f "$iso_path" ] || squashfs_layers_fail "ISO not found for squashfs layer validation: $iso_path" + require_iso_reader "$iso_path" >/dev/null 2>&1 || squashfs_layers_fail "ISO reader unavailable for squashfs layer validation" + + _files="$(mktemp)" + _module="$(mktemp)" + iso_list_files "$iso_path" > "$_files" || squashfs_layers_fail "failed to list ISO files" + + _sq_in_iso="$(grep -E '^live/filesystem.*\.squashfs$' "$_files" | sed 's#^live/##' | LC_ALL=C sort)" + _sq_count="$(printf '%s\n' "$_sq_in_iso" | grep -c . || true)" + _has_module=0 + grep -qx 'live/filesystem.module' "$_files" && _has_module=1 + + if [ "${BEE_MULTILAYER_SQUASHFS:-0}" != "1" ]; then + echo "=== single-layer variant: $_sq_count squashfs, module file present=$_has_module ===" + [ "$_has_module" = 0 ] || squashfs_layers_fail "single-layer variant unexpectedly ships filesystem.module" + rm -f "$_files" "$_module" + echo "=== squashfs layer validation OK (single layer) ===" + return 0 + fi + + [ "$_has_module" = 1 ] || squashfs_layers_fail "multi-layer variant is missing live/filesystem.module (monolith slipped through)" + iso_read_member "$iso_path" live/filesystem.module "$_module" || squashfs_layers_fail "failed to read live/filesystem.module from ISO" + + _listed="$(grep -vE '^[[:space:]]*(#|$)' "$_module" | LC_ALL=C sort)" + _listed_count="$(printf '%s\n' "$_listed" | grep -c . || true)" + [ "$_listed_count" -ge 2 ] || squashfs_layers_fail "filesystem.module lists only $_listed_count layer(s), expected >= 2" + + if [ "$(printf '%s\n' "$_listed")" != "$(printf '%s\n' "$_sq_in_iso")" ]; then + echo " module lists:" >&2; printf ' %s\n' $_listed >&2 + echo " ISO carries:" >&2; printf ' %s\n' $_sq_in_iso >&2 + squashfs_layers_fail "filesystem.module and the squashfs files in live/ do not match exactly" + fi + + _limit_mib="${BEE_LAYER_MAX_MIB:-800}" + for _name in $_sq_in_iso; do + _bytes="" + if command -v xorriso >/dev/null 2>&1; then + _bytes="$(xorriso -indev "$iso_path" -lsl "/live/${_name}" 2>/dev/null \ + | awk '$1 ~ /^-/ { print $5; exit }')" + fi + if ! printf '%s' "$_bytes" | grep -Eq '^[0-9]+$'; then + _tmp_sq="$(mktemp)" + iso_read_member "$iso_path" "live/${_name}" "$_tmp_sq" || squashfs_layers_fail "failed to extract live/${_name}" + _bytes="$(wc -c < "$_tmp_sq" | tr -d ' ')" + rm -f "$_tmp_sq" + fi + _mib=$(( _bytes / 1048576 )) + printf ' %-40s %5d MiB\n' "$_name" "$_mib" + [ "$_mib" -le "$_limit_mib" ] || squashfs_layers_fail "layer ${_name} is ${_mib} MiB, over the ${_limit_mib} MiB ceiling" + case "$_name" in + *-00-base.squashfs) : ;; + *) [ "$_mib" -ge 1 ] || squashfs_layers_fail "layer ${_name} is suspiciously empty" ;; + esac + done + + if [ "$_sq_count" -eq 1 ]; then + squashfs_layers_fail "multi-layer variant shipped a single squashfs" + fi + + rm -f "$_files" "$_module" + echo "=== squashfs layer validation OK ($_sq_count layers) ===" +} + +# Read the whole raw ISO back and confirm every sector is readable. Catches a +# truncated or corrupt image before it ships - the split path rewrites the +# squashfs area and reassembles the ISO, so this is worth a full re-read. +validate_iso_media_integrity() { + iso_path="$1" + echo "=== validating ISO media integrity (xorriso -check_media) ===" + + [ -f "$iso_path" ] || { echo "ERROR: ISO not found for media check: $iso_path" >&2; exit 1; } + if ! command -v xorriso >/dev/null 2>&1; then + echo "WARNING: xorriso not available, skipping -check_media" >&2 + return 0 + fi + + _out="$(xorriso -indev "$iso_path" -check_media 2>&1)" || { + printf '%s\n' "$_out" >&2 + echo "ERROR: xorriso -check_media failed for $iso_path" >&2 + exit 1 + } + printf '%s\n' "$_out" | grep -E 'Media region|Bad blocks|checked|md5' || true + if printf '%s\n' "$_out" | grep -Eiq 'bad block[^s]|[1-9][0-9]* bad blocks|damaged|not readable'; then + printf '%s\n' "$_out" >&2 + echo "ERROR: xorriso -check_media reported unreadable/bad sectors in $iso_path" >&2 + exit 1 + fi + echo "=== ISO media integrity OK ===" +} + # Guard against a corrupted merged-usr layout in the live rootfs. On Debian # bookworm /bin, /sbin, /lib and /lib64 are symlinks into /usr; if a build step # stages a real directory of that name into the overlay, `rsync -a` replaces the diff --git a/iso/builder/lib/squashfs-layers.sh b/iso/builder/lib/squashfs-layers.sh new file mode 100644 index 0000000..42d43de --- /dev/null +++ b/iso/builder/lib/squashfs-layers.sh @@ -0,0 +1,393 @@ +#!/bin/sh +# lib/squashfs-layers.sh - deterministic semantic SquashFS layering for the +# bee live medium. +# +# Module: squashfs-layers +# Version: 1.0 +# +# Why this exists +# --------------- +# The live medium used to ship one ~2.8 GB filesystem-v.squashfs. Booting +# through a BMC / IPMI virtual CD reads that single file sequentially during the +# live-boot "toram" copy; if the redirected medium drops off the bus part-way +# through, the whole copy is lost and the boot fails. Splitting the rootfs into +# several self-contained squashfs layers means a mid-copy failure only costs the +# layer in flight, and the retry loop re-reads far less data. +# +# This is a resilience / reduced-re-read mechanism. It is NOT a fix for the +# underlying BMC virtual-media instability. +# +# Layer model (NVIDIA variants) +# ----------------------------- +# 00-base Debian rootfs, kernel + modules, systemd, Bee, networking, +# CLI diagnostic tools, dpkg database. Boot-critical. +# 05-firmware Device firmware (firmware-* packages: NIC, wifi, non-NVIDIA +# GPU). Not NVIDIA GSP firmware - that rides with the driver. +# 08-desktop Local-console GUI stack: X.org, lightdm, openbox, mesa, +# GTK, chromium, mupdf, fonts. SSH / headless use never +# touches this layer. +# 10-nvidia-driver NVIDIA kernel modules (.ko), driver userspace libraries +# (libnvidia-*, libcuda.so*), nvidia-smi, GSP firmware, +# OpenCL ICD, modprobe config, alternatives, bee-nvidia-*. +# 20-nvidia-platform Fabric Manager, libnvidia-nscq, nvlsm, DCGM core and the +# non-CUDA proprietary DCGM components plus their units. +# 30-nvidia-cuda-libs CUDA userspace runtime (cuBLAS/cuBLASLt/cudart), NCCL, +# nccl-tests, the bee GPU stress worker assets. +# 40-nvidia-dcgm-cuda The CUDA-linked DCGM components (dcgmproftester CUDA +# kernels) - split from 30 because they are large. +# +# AMD / nogpu: a single 00-base squashfs, no filesystem.module. +# +# Classification is by dpkg file ownership (var/lib/dpkg/info/*.list), not by +# path substring. Files that build.sh injects directly from the build cache and +# the project overlay (and therefore belong to no .deb) are classified by the +# explicit rules in bee_layer_for_injected_path. A path is never moved to an +# NVIDIA layer merely because it contains the string "nvidia". +# +# Ordering +# -------- +# Layer names sort lexically 00 < 10 < 20 < 30 < 40. live-boot reads +# live/filesystem.module (written by bee_layer_write_module_file) verbatim and +# stacks the images so the LAST listed image has the HIGHEST OverlayFS priority. +# bee-install unpacks the same list in order with `unsquashfs -f` (last write +# wins) and the fast-path reconstruction extracts in the same order. A +# higher-numbered layer therefore always wins a conflict, everywhere. + +# Compressed-size budget per layer. Soft target from the design; hard ceiling +# fails the build so a layer cannot silently grow back toward the monolith. +BEE_LAYER_TARGET_MIB="${BEE_LAYER_TARGET_MIB:-700}" +BEE_LAYER_MAX_MIB="${BEE_LAYER_MAX_MIB:-800}" + +# Deterministic mksquashfs options shared by the monolith and every layer. +BEE_LAYER_MKSQUASHFS_OPTS="-comp zstd -b 1048576 -noappend -no-progress -no-xattrs -processors 1" + +bee_layer_slugs_for_variant() { + # echo the ordered layer slugs for a build variant, one per line. + case "$1" in + nvidia|nvidia-legacy) + printf '%s\n' \ + 00-base \ + 05-firmware \ + 08-desktop \ + 10-nvidia-driver \ + 20-nvidia-platform \ + 30-nvidia-cuda-libs \ + 40-nvidia-dcgm-cuda + ;; + amd|nogpu) + printf '%s\n' 00-base + ;; + *) + echo "bee_layer: unknown variant: $1" >&2 + return 1 + ;; + esac +} + +# Map an installed dpkg package name to a layer slug, or empty for "base". +# case is first-match: NVIDIA names are matched before the generic firmware-* and +# desktop families, so firmware-nvidia-* rides with the driver, not 05-firmware. +bee_layer_for_dpkg_pkg() { + case "$1" in + nvidia-fabricmanager|libnvidia-nscq|nvlsm|libibumad3|libibumad[0-9]*|\ + datacenter-gpu-manager-4-core|datacenter-gpu-manager-4-proprietary) + echo 20-nvidia-platform ;; + datacenter-gpu-manager-4-cuda[0-9]*|\ + datacenter-gpu-manager-4-proprietary-cuda[0-9]*) + echo 40-nvidia-dcgm-cuda ;; + nvidia-modprobe|nvidia-kernel-common|nvidia-kernel-support|\ + nvidia-installer-cleanup|glx-alternative-nvidia|nvidia-tesla-*|\ + firmware-nvidia-*|libnvidia-*|\ + nvtop|clinfo|ocl-icd-libopencl1) + echo 10-nvidia-driver ;; + firmware-*) + echo 05-firmware ;; + xserver-xorg*|xserver-common|xorg|xorg-*|xfonts-*|xinit|\ + x11-apps|x11-utils|x11-xserver-utils|x11-xkb-utils|x11-session-utils|x11-common|\ + lightdm|lightdm-*|liblightdm-*|openbox|obconf|feh|scrot|\ + chromium|chromium-*|mupdf|mupdf-*|\ + libllvm[0-9]*|libgl1-mesa-dri|libglx-mesa0|libglapi-mesa|libegl-mesa0|\ + libglu1-mesa|glx-alternative-mesa|libgbm1|\ + libgtk-3-*|libgtk2.0-*|libgtkmm-3.0-*|libgtksourceview-*|\ + libpango-*|libpangomm-*|libpangocairo-*|libpangoft2-*|libpangoxft-*|\ + libcairo2|libcairo-gobject2|libcairomm-*|libgdk-pixbuf-*|libgdk-pixbuf2.0-*|\ + fonts-*) + echo 08-desktop ;; + *) + echo "" ;; + esac +} + +# Emit "" lines for files that build.sh injects +# directly (no owning .deb). Matched against the merged-usr-canonical relative +# path (no leading slash). These override dpkg ownership. +bee_layer_injected_rules() { + cat <<'RULES' +^usr/local/lib/nvidia/[^/]+\.ko$ 10-nvidia-driver +^usr/local/bin/nvidia-smi$ 10-nvidia-driver +^usr/local/bin/nvidia-bug-report\.sh$ 10-nvidia-driver +^usr/lib/libcuda\.so(\..*)?$ 10-nvidia-driver +^usr/lib/libnvidia-[^/]+$ 10-nvidia-driver +^usr/lib/firmware/nvidia/ 10-nvidia-driver +^etc/OpenCL/vendors/nvidia\.icd$ 10-nvidia-driver +^usr/local/bin/bee-nvidia-load$ 10-nvidia-driver +^usr/local/bin/bee-nvidia-recover$ 10-nvidia-driver +^usr/local/bin/bee-check-nvswitch$ 10-nvidia-driver +^etc/systemd/system/bee-nvidia\.service$ 10-nvidia-driver +^etc/systemd/system/nvidia-fabricmanager\.service\.d/ 20-nvidia-platform +^usr/lib/libnccl\.so(\..*)?$ 30-nvidia-cuda-libs +^usr/lib/libcublas\.so(\..*)?$ 30-nvidia-cuda-libs +^usr/lib/libcublasLt\.so(\..*)?$ 30-nvidia-cuda-libs +^usr/lib/libcudart\.so(\..*)?$ 30-nvidia-cuda-libs +^usr/local/bin/all_reduce_perf$ 30-nvidia-cuda-libs +^usr/local/lib/bee/bee-gpu-burn-worker$ 30-nvidia-cuda-libs +^usr/local/bin/bee-gpu-burn$ 30-nvidia-cuda-libs +^usr/local/bin/bee-nccl-gpu-stress$ 30-nvidia-cuda-libs +^usr/local/bin/bee-john-gpu-stress$ 30-nvidia-cuda-libs +^usr/local/bin/bee-dcgmproftester-staggered$ 40-nvidia-dcgm-cuda +RULES +} + +# Canonicalise the pre-merged-usr paths dpkg records (/bin, /sbin, /lib, +# /lib64) to their real /usr location and drop the leading slash. +bee_layer_canon_paths() { + sed -e 's#^/bin/#usr/bin/#' \ + -e 's#^/sbin/#usr/sbin/#' \ + -e 's#^/lib/#usr/lib/#' \ + -e 's#^/lib64/#usr/lib64/#' \ + -e 's#^/##' +} + + +# bee_layer_classify +# Partition every regular file and symlink under into exactly one +# layer. Writes /.files (LC_ALL=C sorted) and +# /classify-report.txt. Runs as a subshell: internal state cannot +# leak into the caller, and any step failure aborts with a non-zero exit. +bee_layer_classify() ( + set -e + _root="$1" + _variant="$2" + _wd="$3" + + [ -d "$_root/var/lib/dpkg/info" ] || { + echo "bee_layer_classify: no dpkg database under $_root" >&2 + exit 1 + } + mkdir -p "$_wd" + _slugs="$(bee_layer_slugs_for_variant "$_variant")" + _tab="$(printf '\t')" + + # 1. Every file and symlink in the tree. + ( cd "$_root" && find . -mindepth 1 \( -type f -o -type l \) -printf '%P\n' ) \ + | LC_ALL=C sort > "$_wd/all.files" + + # 2. dpkg-owned files that belong to a non-base layer. + : > "$_wd/dpkg.map" + for _list in "$_root"/var/lib/dpkg/info/*.list; do + [ -f "$_list" ] || continue + _pkg="$(basename "$_list" .list)" + _pkg="${_pkg%%:*}" + _layer="$(bee_layer_for_dpkg_pkg "$_pkg")" + [ -n "$_layer" ] || continue + bee_layer_canon_paths < "$_list" \ + | LC_ALL=C sort -u \ + | awk -v l="$_layer" -v t="$_tab" 'NF { print $0 t l }' >> "$_wd/dpkg.map" + done + + # 3. Injected (no-deb) files, by explicit rule. These override dpkg. + : > "$_wd/injected.map" + bee_layer_injected_rules | while IFS="$_tab" read -r _re _layer; do + [ -n "$_re" ] || continue + LC_ALL=C grep -E "$_re" "$_wd/all.files" \ + | awk -v l="$_layer" -v t="$_tab" '{ print $0 t l }' >> "$_wd/injected.map" || true + done + + # 4. Merge: injected first so it wins; keep the first layer seen per path. + # Drop the merged-usr compat symlinks (bin/sbin/lib/lib64) unconditionally + # - a bare "/lib" entry appears in some dpkg .list files (firmware-*), and + # those symlinks must always stay in the base layer. + # Then keep only rows that name a real file or symlink in the tree: dpkg + # .list files also record bare directories, which must never reach an + # rsync --files-from list (rsync would copy the directory recursively). + cat "$_wd/injected.map" "$_wd/dpkg.map" \ + | awk -F'\t' '$1!="bin" && $1!="sbin" && $1!="lib" && $1!="lib64" && !seen[$1]++ { print }' \ + | LC_ALL=C sort -t "$_tab" -k1,1 > "$_wd/candidate.map" + LC_ALL=C join -t "$_tab" -j 1 "$_wd/all.files" "$_wd/candidate.map" \ + > "$_wd/assigned.map" || true + + # 5. Split into per-layer file lists; base gets the remainder. + for _slug in $_slugs; do : > "$_wd/$_slug.files"; done + awk -F'\t' -v wd="$_wd" '{ print $1 >> (wd "/" $2 ".files") }' "$_wd/assigned.map" + for _slug in $_slugs; do + LC_ALL=C sort -o "$_wd/$_slug.files" "$_wd/$_slug.files" + done + cut -f1 "$_wd/assigned.map" | LC_ALL=C sort > "$_wd/assigned.paths" + LC_ALL=C comm -23 "$_wd/all.files" "$_wd/assigned.paths" > "$_wd/00-base.files" + + # 6. Completeness / disjointness. + _total="$(wc -l < "$_wd/all.files")" + _sum=0 + : > "$_wd/classify-report.txt" + for _slug in $_slugs; do + _n="$(wc -l < "$_wd/$_slug.files")" + _sum=$((_sum + _n)) + printf '%-20s %8d files\n' "$_slug" "$_n" >> "$_wd/classify-report.txt" + done + printf '%-20s %8d files\n' "TOTAL" "$_sum" >> "$_wd/classify-report.txt" + printf '%-20s %8d files\n' "tree" "$_total" >> "$_wd/classify-report.txt" + cat "$_wd/classify-report.txt" + + if [ "$_sum" -ne "$_total" ]; then + echo "bee_layer_classify: partition covers $_sum of $_total files" >&2 + exit 1 + fi + _dups="$(cat $(for _slug in $_slugs; do echo "$_wd/$_slug.files"; done) \ + | LC_ALL=C sort | LC_ALL=C uniq -d)" + if [ -n "$_dups" ]; then + echo "bee_layer_classify: files assigned to more than one layer:" >&2 + printf '%s\n' "$_dups" | head >&2 + exit 1 + fi + + # 7. merged-usr guard. + for _link in bin sbin lib lib64; do + [ -L "$_root/$_link" ] || continue + LC_ALL=C grep -qx "$_link" "$_wd/00-base.files" || { + echo "bee_layer_classify: merged-usr symlink /$_link is not in the base layer" >&2 + exit 1 + } + done + for _slug in $_slugs; do + [ "$_slug" = 00-base ] && continue + if LC_ALL=C grep -Eq '^(bin|sbin|lib|lib64)/' "$_wd/$_slug.files"; then + echo "bee_layer_classify: $_slug contains a top-level bin/sbin/lib/lib64 path (merged-usr trap)" >&2 + exit 1 + fi + done +) + +# bee_layer_build +# Materialise each layer list into filesystem-v-.squashfs under +# . Enforces the size ceiling. Leaves the monolith untouched. +bee_layer_build() ( + set -e + _root="$1"; _wd="$2"; _out="$3"; _ver="$4"; _variant="$5" + _slugs="$(bee_layer_slugs_for_variant "$_variant")" + mkdir -p "$_out" + + for _slug in $_slugs; do + _list="$_wd/$_slug.files" + [ -s "$_list" ] || { echo "bee_layer_build: empty file list for $_slug" >&2; exit 1; } + _stage="$_wd/stage-$_slug" + _sq="$_out/filesystem-v${_ver}-${_slug}.squashfs" + rm -rf "$_stage" + mkdir -p "$_stage" + # --files-from with -a recreates implied parent directories from the + # source but copies only listed entries. --link-dest hardlinks unchanged + # files from the source tree (same fs) so a 5 GB rootfs is not physically + # copied once per layer. + rsync -a --link-dest="$_root" --files-from="$_list" "$_root/" "$_stage/" + rm -f "$_sq" + # shellcheck disable=SC2086 + mksquashfs "$_stage" "$_sq" $BEE_LAYER_MKSQUASHFS_OPTS + rm -rf "$_stage" + + _bytes="$(stat -c '%s' "$_sq")" + _mib=$(( _bytes / 1048576 )) + printf ' %-40s %5d MiB\n' "$(basename "$_sq")" "$_mib" + if [ "$_mib" -gt "$BEE_LAYER_MAX_MIB" ]; then + echo "bee_layer_build: $(basename "$_sq") is ${_mib} MiB, over the ${BEE_LAYER_MAX_MIB} MiB ceiling" >&2 + exit 1 + fi + if [ "$_mib" -gt "$BEE_LAYER_TARGET_MIB" ]; then + echo " WARNING: $(basename "$_sq") exceeds the ${BEE_LAYER_TARGET_MIB} MiB target" >&2 + fi + done +) + +# bee_layer_verify_each +# unsquashfs -s plus a full strict-errors extraction of every layer. +bee_layer_verify_each() ( + set -e + _out="$1"; _ver="$2"; _variant="$3" + _slugs="$(bee_layer_slugs_for_variant "$_variant")" + _tmp="$(mktemp -d)" + trap 'rm -rf "$_tmp"' EXIT + for _slug in $_slugs; do + _sq="$_out/filesystem-v${_ver}-${_slug}.squashfs" + [ -f "$_sq" ] || { echo "bee_layer_verify_each: missing $_sq" >&2; exit 1; } + unsquashfs -s "$_sq" >/dev/null || { echo "bee_layer_verify_each: unsquashfs -s failed for $_sq" >&2; exit 1; } + rm -rf "$_tmp/x" + unsquashfs -d "$_tmp/x" -strict-errors "$_sq" >/dev/null || { + echo "bee_layer_verify_each: strict extraction failed for $_sq" >&2 + exit 1 + } + done + echo "=== all $(echo "$_slugs" | wc -w) layers pass unsquashfs -s + strict extraction ===" +) + +# bee_layer_merge +# Reconstruct the single rootfs live-boot would present, extracting layers in +# module order (last wins), and assert it is bootable. +bee_layer_merge() ( + set -e + _out="$1"; _ver="$2"; _variant="$3"; _dest="$4" + _slugs="$(bee_layer_slugs_for_variant "$_variant")" + rm -rf "$_dest" + mkdir -p "$_dest" + for _slug in $_slugs; do + _sq="$_out/filesystem-v${_ver}-${_slug}.squashfs" + unsquashfs -f -d "$_dest" "$_sq" >/dev/null || { + echo "bee_layer_merge: extraction of $_sq failed" >&2 + exit 1 + } + done + + for _need in usr/sbin/init usr/lib/systemd/systemd usr/local/bin/bee; do + [ -e "$_dest/$_need" ] || { echo "bee_layer_merge: merged rootfs is missing /$_need" >&2; exit 1; } + done + for _link in bin sbin lib lib64; do + [ -L "$_dest/$_link" ] || { echo "bee_layer_merge: merged rootfs /$_link is not a symlink" >&2; exit 1; } + done + + case "$_variant" in + nvidia|nvidia-legacy) + for _need in \ + usr/local/bin/nvidia-smi \ + usr/local/lib/nvidia/nvidia.ko \ + usr/bin/dcgmi \ + usr/bin/nv-hostengine \ + etc/systemd/system/bee-nvidia.service; do + [ -e "$_dest/$_need" ] || { echo "bee_layer_merge: merged NVIDIA rootfs is missing /$_need" >&2; exit 1; } + done + [ -n "$(find "$_dest/usr/lib/firmware/nvidia" -name 'gsp_*.bin' 2>/dev/null | head -1)" ] \ + || { echo "bee_layer_merge: no GSP firmware in merged rootfs" >&2; exit 1; } + ls "$_dest"/usr/lib/libnvidia-ml.so* >/dev/null 2>&1 \ + || { echo "bee_layer_merge: libnvidia-ml missing from merged rootfs" >&2; exit 1; } + for _need in libcudart.so libcublas.so libnccl.so; do + ls "$_dest"/usr/lib/${_need}* >/dev/null 2>&1 \ + || { echo "bee_layer_merge: $_need missing from merged rootfs" >&2; exit 1; } + done + ls "$_dest"/usr/bin/dcgmproftester* >/dev/null 2>&1 \ + || { echo "bee_layer_merge: dcgmproftester missing from merged rootfs" >&2; exit 1; } + ;; + esac + echo "=== merged rootfs from $(echo "$_slugs" | wc -w) layers is complete and bootable ===" +) + +# bee_layer_write_module_file +# Write /filesystem.module - the explicit, locale-independent image +# order live-boot 1:20230131 reads (MODULE defaults to "filesystem"). +bee_layer_write_module_file() ( + set -e + _live="$1"; _ver="$2"; _variant="$3" + _slugs="$(bee_layer_slugs_for_variant "$_variant")" + _mod="$_live/filesystem.module" + : > "$_mod" + for _slug in $_slugs; do + printf 'filesystem-v%s-%s.squashfs\n' "$_ver" "$_slug" >> "$_mod" + done + echo "=== wrote $_mod ===" + cat "$_mod" +) diff --git a/iso/builder/test-build-libs.sh b/iso/builder/test-build-libs.sh index cddbced..3d6d270 100755 --- a/iso/builder/test-build-libs.sh +++ b/iso/builder/test-build-libs.sh @@ -155,5 +155,28 @@ if ! needs_full_build >/dev/null; then echo "ERROR: kernel ABI change did not force a full build" >&2 exit 1 fi +printf '%s\n' "$DEBIAN_KERNEL_ABI" > "$FULL_BUILD_ABI_FILE" + +# A multi-layer live medium must always force a full build: the fast path only +# repacks one squashfs and would silently drop the other layers. +touch "$BUILD_WORK_DIR/binary/live/filesystem-v13.0-test-10-nvidia-driver.squashfs" +if ! needs_full_build >/dev/null; then + echo "ERROR: a multi-layer squashfs medium did not force a full build" >&2 + exit 1 +fi +printf '%s\n' \ + filesystem-v13.0-test-00-base.squashfs \ + filesystem-v13.0-test-10-nvidia-driver.squashfs \ + > "$BUILD_WORK_DIR/binary/live/filesystem.module" +rm -f "$BUILD_WORK_DIR/binary/live/filesystem-v13.0-test-10-nvidia-driver.squashfs" +if ! needs_full_build >/dev/null; then + echo "ERROR: filesystem.module alone did not force a full build" >&2 + exit 1 +fi +if ( fast_path_repack_squashfs ) >/dev/null 2>&1; then + echo "ERROR: fast_path_repack_squashfs ran on a multi-layer medium" >&2 + exit 1 +fi +rm -f "$BUILD_WORK_DIR/binary/live/filesystem.module" echo "build library tests: OK" diff --git a/iso/builder/test-squashfs-layers.sh b/iso/builder/test-squashfs-layers.sh new file mode 100644 index 0000000..b7d876a --- /dev/null +++ b/iso/builder/test-squashfs-layers.sh @@ -0,0 +1,187 @@ +#!/bin/sh +# test-squashfs-layers.sh - regression tests for lib/squashfs-layers.sh. +# +# Builds a synthetic mini-rootfs, splits it, and asserts the semantic layer +# contract holds. Requires mksquashfs / unsquashfs / rsync; skips cleanly when +# they are not installed (e.g. on a developer macOS box). +set -eu + +BUILDER_DIR="$(CDPATH= cd -- "$(dirname "$0")" && pwd)" +. "$BUILDER_DIR/lib/squashfs-layers.sh" + +for tool in mksquashfs unsquashfs rsync; do + command -v "$tool" >/dev/null 2>&1 || { + echo "test-squashfs-layers: $tool not available, skipping" + exit 0 + } +done + +T="$(mktemp -d)" +trap 'rm -rf "$T"' EXIT INT TERM HUP +ROOT="$T/root" + +mk() { mkdir -p "$(dirname "$ROOT/$1")"; printf '%s' "${2:-x}" > "$ROOT/$1"; } +mkbig() { mkdir -p "$(dirname "$ROOT/$1")"; head -c "${2:-1048576}" /dev/urandom > "$ROOT/$1"; } +listf() { mkdir -p "$ROOT/var/lib/dpkg/info"; printf '%s\n' "$@" > "$ROOT/var/lib/dpkg/info/$1.list"; } + +# --- base rootfs --- +mk usr/lib/systemd/systemd "#!systemd" +( cd "$ROOT/usr" && mkdir -p sbin && ln -s ../lib/systemd/systemd sbin/init ) +mk usr/local/bin/bee "#!bee" +mk etc/hostname "easy-bee" +mk usr/bin/true "#!true" +mk usr/bin/dmidecode "#!dmi" +( cd "$ROOT" && ln -s usr/bin bin && ln -s usr/bin sbin && ln -s usr/lib lib && ln -s usr/lib lib64 ) +listf base-files /etc/hostname /usr/bin/true +listf dmidecode /usr/bin/dmidecode + +# --- 05 firmware: non-NVIDIA device firmware. The bare "/lib" entry mimics the +# real firmware-* .list files and must NOT drag the /lib symlink out of base. +listf firmware-iwlwifi /lib /lib/firmware /lib/firmware/iwlwifi-cc-a0-77.ucode +mk usr/lib/firmware/iwlwifi-cc-a0-77.ucode +listf firmware-realtek /lib/firmware/rtl_nic/rtl8168h-2.fw +mk usr/lib/firmware/rtl_nic/rtl8168h-2.fw + +# --- 08 desktop --- +listf chromium /usr/bin/chromium +mk usr/bin/chromium "#!chromium" +listf xserver-xorg-core /usr/bin/Xorg +mk usr/bin/Xorg "#!Xorg" +listf lightdm /usr/sbin/lightdm +mk usr/sbin/lightdm "#!lightdm" +listf libgl1-mesa-dri /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so +mk usr/lib/x86_64-linux-gnu/dri/swrast_dri.so +listf fonts-dejavu-core /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf +mk usr/share/fonts/truetype/dejavu/DejaVuSans.ttf + +# --- 10 driver: dpkg + injected --- +listf firmware-nvidia-gsp /lib /lib/firmware /lib/firmware/nvidia/580.159.03/gsp_ga10x.bin +mk usr/lib/firmware/nvidia/580.159.03/gsp_ga10x.bin +mk usr/lib/firmware/nvidia/580.159.03/gsp_tu10x.bin +listf nvidia-modprobe /usr/bin/nvidia-modprobe +mk usr/bin/nvidia-modprobe +mk usr/local/lib/nvidia/nvidia.ko +mk usr/local/lib/nvidia/nvidia-uvm.ko +mk usr/local/bin/nvidia-smi "#!nvidia-smi" +mk usr/lib/libnvidia-ml.so.1 +mk usr/lib/libnvidia-ml.so.580.159.03 +mk usr/lib/libcuda.so.1 +mk etc/OpenCL/vendors/nvidia.icd "libnvidia-opencl.so.1" +mk etc/systemd/system/bee-nvidia.service "[Unit]" +mk usr/local/bin/bee-nvidia-load "#!load" + +# --- 20 platform --- +listf nvidia-fabricmanager /usr/bin/nv-fabricmanager /usr/lib/systemd/system/nvidia-fabricmanager.service +mk usr/bin/nv-fabricmanager +mk usr/lib/systemd/system/nvidia-fabricmanager.service "[Unit]" +listf libnvidia-nscq /usr/lib/x86_64-linux-gnu/libnvidia-nscq.so.580.159.03 +mk usr/lib/x86_64-linux-gnu/libnvidia-nscq.so.580.159.03 +listf datacenter-gpu-manager-4-core /usr/bin/dcgmi /usr/bin/nv-hostengine +mk usr/bin/dcgmi "#!dcgmi" +mk usr/bin/nv-hostengine "#!nvh" +mk etc/systemd/system/nvidia-fabricmanager.service.d/bee-nvswitch-check.conf "[Service]" + +# --- 30 cuda libs (injected) --- +mk usr/lib/libcudart.so.13 +mk usr/lib/libcublas.so.13 +mk usr/lib/libcublasLt.so.13 +mk usr/lib/libnccl.so.2 +mk usr/local/bin/all_reduce_perf "#!arp" +mk usr/local/lib/bee/bee-gpu-burn-worker "#!worker" + +# --- 40 dcgm cuda --- +listf datacenter-gpu-manager-4-cuda13 /usr/bin/dcgmproftester12 +mkbig usr/bin/dcgmproftester12 3145728 +mk usr/local/bin/bee-dcgmproftester-staggered "#!stag" + +# a plain "nvidia" in the path must NOT be enough on its own: an unowned, +# unruled file under an nvidia dir stays in base. +mk usr/share/doc/nvidia-random/README "notes" + +fail() { echo "FAIL: $*" >&2; exit 1; } +in_layer() { LC_ALL=C grep -qx "$2" "$WD/$1.files" || fail "expected $2 in layer $1"; } + +MONO="$T/filesystem-v14.99.squashfs" +mksquashfs "$ROOT" "$MONO" -comp zstd -noappend -no-progress -no-xattrs >/dev/null + +R2="$T/r2"; unsquashfs -d "$R2" "$MONO" >/dev/null +WD="$T/cls" +bee_layer_classify "$R2" nvidia "$WD" + +in_layer 00-base usr/sbin/init +in_layer 00-base usr/lib/systemd/systemd +in_layer 00-base usr/local/bin/bee +in_layer 00-base etc/hostname +in_layer 00-base usr/bin/dmidecode +in_layer 00-base bin +in_layer 00-base sbin +in_layer 00-base lib +in_layer 00-base lib64 +in_layer 05-firmware usr/lib/firmware/iwlwifi-cc-a0-77.ucode +in_layer 05-firmware usr/lib/firmware/rtl_nic/rtl8168h-2.fw +in_layer 08-desktop usr/bin/chromium +in_layer 08-desktop usr/bin/Xorg +in_layer 08-desktop usr/sbin/lightdm +in_layer 08-desktop usr/lib/x86_64-linux-gnu/dri/swrast_dri.so +in_layer 08-desktop usr/share/fonts/truetype/dejavu/DejaVuSans.ttf +in_layer 10-nvidia-driver usr/local/lib/nvidia/nvidia.ko +in_layer 10-nvidia-driver usr/local/bin/nvidia-smi +in_layer 10-nvidia-driver usr/lib/libnvidia-ml.so.1 +in_layer 10-nvidia-driver usr/lib/libcuda.so.1 +in_layer 10-nvidia-driver usr/lib/firmware/nvidia/580.159.03/gsp_ga10x.bin +in_layer 10-nvidia-driver etc/OpenCL/vendors/nvidia.icd +in_layer 10-nvidia-driver etc/systemd/system/bee-nvidia.service +in_layer 10-nvidia-driver usr/bin/nvidia-modprobe +in_layer 20-nvidia-platform usr/bin/nv-fabricmanager +in_layer 20-nvidia-platform usr/lib/x86_64-linux-gnu/libnvidia-nscq.so.580.159.03 +in_layer 20-nvidia-platform usr/bin/dcgmi +in_layer 20-nvidia-platform usr/bin/nv-hostengine +in_layer 20-nvidia-platform etc/systemd/system/nvidia-fabricmanager.service.d/bee-nvswitch-check.conf +in_layer 30-nvidia-cuda-libs usr/lib/libcudart.so.13 +in_layer 30-nvidia-cuda-libs usr/lib/libcublasLt.so.13 +in_layer 30-nvidia-cuda-libs usr/lib/libnccl.so.2 +in_layer 30-nvidia-cuda-libs usr/local/bin/all_reduce_perf +in_layer 30-nvidia-cuda-libs usr/local/lib/bee/bee-gpu-burn-worker +in_layer 40-nvidia-dcgm-cuda usr/bin/dcgmproftester12 +in_layer 40-nvidia-dcgm-cuda usr/local/bin/bee-dcgmproftester-staggered +in_layer 00-base usr/share/doc/nvidia-random/README + +OUT="$T/out" +bee_layer_build "$R2" "$WD" "$OUT" 14.99 nvidia +bee_layer_verify_each "$OUT" 14.99 nvidia +bee_layer_merge "$OUT" 14.99 nvidia "$T/merged" + +MODLIVE="$T/live"; mkdir -p "$MODLIVE" +cp "$OUT"/filesystem-v14.99-*.squashfs "$MODLIVE/" +bee_layer_write_module_file "$MODLIVE" 14.99 nvidia +expect_mod="filesystem-v14.99-00-base.squashfs +filesystem-v14.99-05-firmware.squashfs +filesystem-v14.99-08-desktop.squashfs +filesystem-v14.99-10-nvidia-driver.squashfs +filesystem-v14.99-20-nvidia-platform.squashfs +filesystem-v14.99-30-nvidia-cuda-libs.squashfs +filesystem-v14.99-40-nvidia-dcgm-cuda.squashfs" +[ "$(cat "$MODLIVE/filesystem.module")" = "$expect_mod" ] || fail "module file order wrong" + +# --- failure modes --- +# missing layer -> merge fails +rm -f "$OUT/filesystem-v14.99-20-nvidia-platform.squashfs" +if bee_layer_merge "$OUT" 14.99 nvidia "$T/m2" >/dev/null 2>&1; then + fail "merge succeeded with a missing layer" +fi +bee_layer_build "$R2" "$WD" "$OUT" 14.99 nvidia >/dev/null # restore + +# size ceiling -> build fails +if BEE_LAYER_MAX_MIB=0 bee_layer_build "$R2" "$WD" "$T/out0" 14.99 nvidia >/dev/null 2>&1; then + fail "build ignored the size ceiling" +fi + +# incomplete partition -> classify fails +rm -rf "$WD"; bee_layer_classify "$R2" nvidia "$WD" +echo "usr/bin/does-not-exist" >> "$WD/10-nvidia-driver.files" +_total=$(wc -l < "$WD/all.files") +_sum=0 +for s in $(bee_layer_slugs_for_variant nvidia); do _sum=$((_sum + $(wc -l < "$WD/$s.files"))); done +[ "$_sum" -ne "$_total" ] || fail "tampered partition should not still sum to the tree size" + +echo "squashfs layer tests: OK" diff --git a/iso/overlay/usr/local/bin/bee-install b/iso/overlay/usr/local/bin/bee-install index 351dfe3..3fc6a58 100755 --- a/iso/overlay/usr/local/bin/bee-install +++ b/iso/overlay/usr/local/bin/bee-install @@ -62,7 +62,30 @@ for tool in parted mkfs.vfat mkfs.ext4 unsquashfs grub-install update-grub; do fi done -mapfile -t SQUASHFS_FILES < <(find /run/live/medium/live -maxdepth 1 -type f -name '*.squashfs' | sort) +# Order the squashfs layers exactly as live-boot stacks them: if the medium +# carries live/filesystem.module, honour that list verbatim (it is the same +# order OverlayFS uses, last entry = highest priority); otherwise fall back to a +# lexical sort, which the NN- numeric prefixes make equivalent. +LIVE_DIR=/run/live/medium/live +MODULE_FILE="${LIVE_DIR}/filesystem.module" +read_squashfs_layers() { + SQUASHFS_FILES=() + if [ -f "$MODULE_FILE" ]; then + while IFS= read -r name; do + [ -n "$name" ] || continue + case "$name" in \#*) continue ;; esac + if [ -f "${LIVE_DIR}/${name}" ]; then + SQUASHFS_FILES+=("${LIVE_DIR}/${name}") + else + echo "ERROR: filesystem.module lists ${name} but it is missing from the medium" >&2 + exit 1 + fi + done < "$MODULE_FILE" + else + mapfile -t SQUASHFS_FILES < <(find "$LIVE_DIR" -maxdepth 1 -type f -name '*.squashfs' | sort) + fi +} +read_squashfs_layers if [ "${#SQUASHFS_FILES[@]}" -eq 0 ]; then echo "ERROR: no squashfs files found under /run/live/medium/live" >&2 echo " The live medium may have been disconnected." >&2 @@ -182,7 +205,7 @@ while true; do fi [ "$UNPACK_ATTEMPTS" -gt 1 ] && log " Retry attempt $UNPACK_ATTEMPTS / $UNPACK_MAX ..." - mapfile -t SQUASHFS_FILES < <(find /run/live/medium/live -maxdepth 1 -type f -name '*.squashfs' | sort) + read_squashfs_layers if [ "${#SQUASHFS_FILES[@]}" -eq 0 ]; then log " SOURCE LOST: no squashfs files found under /run/live/medium/live." log " Reconnect the disc and run 'bee-remount-medium --wait' in another terminal," @@ -198,17 +221,28 @@ while true; do find "$MOUNT_ROOT" -mindepth 1 -maxdepth 1 -exec rm -rf {} + 2>/dev/null || true fi + # Unpack every layer in module order. unsquashfs -f makes a later layer + # overwrite an earlier one, matching live-boot's OverlayFS precedence. A + # failed layer aborts this attempt: a partial multi-layer unpack must never + # be treated as a usable install. UNPACK_OK=0 + UNPACK_TMP=$(mktemp) for sf in "${SQUASHFS_FILES[@]}"; do log " Unpacking $(basename "$sf") ..." - unsquashfs -f -d "$MOUNT_ROOT" "$sf" 2>&1 | \ - grep -E '^\[|^inod|^created|^extract|^ERROR|failed' | \ - while IFS= read -r line; do log " $line"; done || UNPACK_OK=$? - [ "$UNPACK_OK" -eq 0 ] || break + rc=0 + unsquashfs -f -d "$MOUNT_ROOT" "$sf" > "$UNPACK_TMP" 2>&1 || rc=$? + grep -E '^\[|^inod|^created|^extract|^ERROR|failed' "$UNPACK_TMP" \ + | while IFS= read -r line; do log " $line"; done || true + if [ "$rc" -ne 0 ]; then + log " ERROR: unsquashfs failed for $(basename "$sf") (rc=$rc)" + UNPACK_OK=$rc + break + fi done + rm -f "$UNPACK_TMP" # Check squashfs is still reachable (gone = disc pulled during copy) - mapfile -t SQUASHFS_FILES < <(find /run/live/medium/live -maxdepth 1 -type f -name '*.squashfs' | sort) + read_squashfs_layers if [ "${#SQUASHFS_FILES[@]}" -eq 0 ]; then log " WARNING: source medium lost during unpack — will retry after remount." log " Run 'bee-remount-medium --wait' in another terminal, then press Enter." @@ -216,9 +250,16 @@ while true; do continue fi - # Verify the unpack produced a usable root (presence of /etc is a basic check) - if [ -d "${MOUNT_ROOT}/etc" ]; then - log " Unpack complete." + if [ "$UNPACK_OK" -ne 0 ]; then + log " A layer failed to unpack — retrying the whole unpack." + [ "$UNPACK_ATTEMPTS" -lt "$UNPACK_MAX" ] && sleep 5 + continue + fi + + # Verify the unpack produced a usable root: /etc from the base layer and + # /usr/sbin/init must both be present after all layers are applied. + if [ -d "${MOUNT_ROOT}/etc" ] && [ -e "${MOUNT_ROOT}/usr/sbin/init" ]; then + log " Unpack complete (${#SQUASHFS_FILES[@]} layer(s))." break else log " WARNING: unpack produced no /etc — squashfs may be corrupt or incomplete."