Files
bee/bible-local/docs/iso-build-rules.md
T
Mikhail ChusavitinandClaude Sonnet 5 b8c45d54c1 feat(iso): split the live medium into semantic SquashFS layers
Booting via BMC virtual CD reads the ~2.8 GB filesystem squashfs
sequentially during the live-boot toram copy; a mid-read drop of the
redirected medium loses the whole copy and fails the boot (v14). Split
the rootfs into self-contained semantic layers so a retry re-reads at
most one ~500-700 MiB layer, not everything. This is a resilience /
reduced-re-read mechanism, not a fix for the virtual-media instability.

NVIDIA variants now ship 7 layers (00-base, 05-firmware, 08-desktop,
10-nvidia-driver, 20-nvidia-platform, 30-nvidia-cuda-libs,
40-nvidia-dcgm-cuda) plus an explicit live/filesystem.module that fixes
their OverlayFS order; amd/nogpu keep a single squashfs.

- lib/squashfs-layers.sh: deterministic classifier (dpkg file ownership
  plus explicit rules for build.sh-injected files, never a path
  substring), per-layer mksquashfs, 800 MiB hard ceiling, unsquashfs -s
  plus strict extraction of every layer, merged-rootfs bootability check.
- build.sh: split the monolith after the full lb build, verify and merge,
  write the module file, delete the monolith only then; abort before ISO
  assembly on any failure. Runs the builder test suites up front.
- fast-path: force a full build for a multi-layer medium;
  fast_path_repack_squashfs hard-refuses (it would drop layers).
- iso-validation.sh: validate_iso_squashfs_layers (module vs layer set
  match, size ceiling, no lone giant squashfs) and
  validate_iso_media_integrity (xorriso -check_media).
- bee-install: honour filesystem.module order, abort on any layer failure.
- 9013-toram-retry: record the real rsync exit code (it printed a false
  rc=0) and correct the "resumes the tail" comment (rsync without
  --partial keeps only fully-copied layers). No unsafe partial resume.
- tests: test-squashfs-layers.sh plus a multi-layer guard in
  test-build-libs.sh; both run at the top of every build.
- docs: bible-local architecture and decision, iso/README, iso-build-rules.

Verified by a full nvidia build: 7 layers 622/199/256/466/37/567/562 MiB,
every validator passes, xorriso -check_media good, merged rootfs bootable.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-04 15:38:14 +03:00

6.0 KiB
Raw Blame History

ISO Build Rules

Verify package names before use

ISO builds take 3060 minutes. A wrong package name wastes an entire build cycle.

Rule: before adding any Debian package name to the ISO config, verify it exists and check its file list.

Use one of:

  • https://packages.debian.org/bookworm/<package-name> — existence + description
  • https://packages.debian.org/bookworm/amd64/<package-name>/filelist — exact files installed
  • apt-cache show <package> inside a Debian bookworm container

This applies to:

  • iso/builder/config/package-lists/*.list.chroot
  • Any package referenced in bootloader configs, hooks, or overlay scripts

SquashFS layer rule

The NVIDIA live medium is not one squashfs. build.sh splits the monolith into semantic layers (filesystem-v<ver>-NN-*.squashfs) after the full lb build, driven by iso/builder/lib/squashfs-layers.sh. See bible-local/architecture/squashfs-layers.md for the layer model and decisions/2026-09-04-squashfs-semantic-layers.md for why.

Rules:

  • Classify by dpkg file ownership, not path substring. New non-.deb files injected by build.sh must get an explicit entry in bee_layer_injected_rules (or they fall to 00-base).
  • live/filesystem.module is authoritative for layer order; keep it in sync with the NN- prefixes. Later layer wins a conflict, in live-boot, bee-install, and any rebuild.
  • Keep each layer a self-contained valid squashfs under the 800 MiB ceiling. Split a semantic layer further rather than blindly slicing bytes.
  • Never let an upper layer carry a real top-level bin/sbin/lib/lib64 path - that breaks merged-usr the same way the fast-path bug did.
  • The builder deletes the monolith only after every layer verifies and re-merges into a bootable rootfs. Any failure must abort before ISO assembly.
  • The fast path stays disabled for a multi-layer medium until a layer-aware repack exists. Do not "take the first squashfs".
  • test-squashfs-layers.sh and test-build-libs.sh run at the top of every build; keep them green.

Bootloader sync rule

The ISO has two canonical bootloader templates whose live entries must remain semantically equivalent:

File Used by
config/bootloaders/grub-efi/grub.cfg UEFI (all modern servers)
config/bootloaders/isolinux/live.cfg.in CSM / legacy BIOS (syslinux)

live-build does not derive one from the other. lib/bootloader.sh renders both templates into the final binary/ tree after live-build has created it, and the ISO validator checks every live entry. Any menu or kernel-parameter change must still be made in both templates.

Canonical entry list (both files must have all of these):

Label Key params
normal (default) nomodeset bee.nvidia.mode=normal + full param set
load to RAM toram nomodeset bee.nvidia.mode=normal + full param set
no GUI nomodeset bee.gui=off bee.nvidia.mode=normal + full param set
fail-safe normal parameters plus pci=realloc iommu.strict=1
wipe toram nomodeset bee.gui=off bee.wipe=all + reduced performance params

Full standard param set (append after @APPEND_LIVE@ / nomodeset flags):

net.ifnames=0 biosdevname=0 mitigations=off transparent_hugepage=always
numa_balancing=disable pcie_aspm=off intel_idle.max_cstate=1 processor.max_cstate=1
nowatchdog nosoftlockup udev.children_max=1 intel_iommu=on
iommu.passthrough=0 efi=disable_early_pci_dma

The fail-safe entry alone adds pci=realloc iommu.strict=1. pci=realloc must not be copied into normal entries. Memtest and firmware-setup entries are not Linux live entries and therefore do not carry these parameters.

Historical note: grub-pc/ was mistakenly used instead of grub-efi/ until v8.25. live-build reads config/bootloaders/grub-efi/ for UEFI because the build is configured with --bootloaders "grub-efi,syslinux". Directory grub-pc is ignored.

Memtest rule

Do not assume live-build's built-in memtest integration is sufficient for bee. We already tried that path and regressed again on 2026-04-01: lb binary_memtest ran, but the final ISO still lacked memtest binaries and menu entries.

For this project, memtest is accepted only when the produced ISO actually contains all of the following:

  • boot/memtest86+x64.bin
  • boot/memtest86+x64.efi
  • a memtest entry in boot/grub/grub.cfg
  • a memtest entry in isolinux/live.cfg

Rules:

  • Keep explicit post-build memtest validation in build.sh.
  • Treat ISO reader success as a separate prerequisite from memtest content. If the reader cannot list or extract from the ISO, that is a validator failure, not proof that memtest is missing.
  • If built-in integration does not produce the artifacts above, use a deterministic project-owned copy/extract step instead of hoping live-build will "start working".
  • Do not switch back to built-in-only memtest without fresh build evidence from a real ISO.
  • If you reference memtest files manually, verify the exact package file list first for the target Debian release.

Known bad loops for this repository:

  • Do not retry built-in-only memtest without new evidence. We already proved that lb binary_memtest can run while the final ISO still has no memtest.
  • Do not assume fixing memtest file names is enough. Correct names did not fix the final artifact path.
  • Do not assume chroot/boot/ contains memtest payloads at the time hooks run.
  • Do not assume source grub.cfg / live.cfg.in are the final writers of ISO bootloader configs.
  • Do not assume the current config/hooks/normal/9100-memtest.hook.binary timing is late enough to patch final binary/boot/grub/grub.cfg or binary/isolinux/live.cfg; logs from 2026-04-01 showed those files were not present yet when the hook executed.
  • Do not treat a validator warning as ground truth until you have confirmed the ISO reader actually succeeded. On 2026-04-01 we misdiagnosed another memtest regression because the final ISO was correct but the validator produced a false negative.