fix(iso): recover memtest after live-build

bible-local/decisions/2026-04-01-memtest-build-strategy.md

@@ -23,6 +23,63 @@ Current evidence from the archived `easy-bee-nvidia-v3.14-amd64` logs dated 2026
 
 So the assumption "live-build built-in memtest integration is enough on this stack" is currently false for this project until proven otherwise by a real built ISO.
 
+Additional evidence from the archived `easy-bee-nvidia-v3.17-dirty-amd64` logs dated 2026-04-01:
+
+- the build now completes successfully because memtest is non-blocking by default
+- `lb binary_memtest` still runs and installs `memtest86+`
+- the project-owned hook `config/hooks/normal/9100-memtest.hook.binary` does execute
+- but it executes too early for its current target paths:
+  - `binary/boot/grub/grub.cfg` is still missing at hook time
+  - `binary/isolinux/live.cfg` is still missing at hook time
+  - memtest binaries are also still absent in `binary/boot/`
+- later in the build, live-build does create intermediate bootloader configs with memtest lines in the workdir
+- but the final ISO still lacks memtest binaries and still lacks memtest lines in extracted ISO `boot/grub/grub.cfg` and `isolinux/live.cfg`
+
+So the assumption "the current normal binary hook path is late enough to patch final memtest artifacts" is also false.
+
+## Known Failed Attempts
+
+These approaches were already tried and should not be repeated blindly:
+
+1. Built-in live-build memtest only.
+Reason it failed:
+- `lb binary_memtest` runs, but the final ISO still misses memtest binaries and menu entries.
+
+2. Fixing only the memtest file names for Debian Bookworm.
+Reason it failed:
+- correct file names alone do not make the files appear in the final ISO.
+
+3. Copying memtest from `chroot/boot/` into `binary/boot/` via a binary hook.
+Reason it failed:
+- in this stack `chroot/boot/` is often empty for memtest payloads at the relevant time.
+
+4. Fallback extraction from cached `memtest86+` `.deb`.
+Reason it failed:
+- this was explored already and was not enough to stabilize the final ISO path end-to-end.
+
+5. Restoring explicit memtest menu entries in source bootloader templates only.
+Reason it failed:
+- memtest lines in source templates or intermediate workdir configs do not guarantee the final ISO contains them.
+
+6. Patching `binary/boot/grub/grub.cfg` and `binary/isolinux/live.cfg` from the current `config/hooks/normal/9100-memtest.hook.binary`.
+Reason it failed:
+- the hook runs before those files exist, so the hook cannot patch them there.
+
+## What This Means
+
+When revisiting memtest later, start from the constraints above rather than retrying the same patterns:
+
+- do not assume the built-in memtest stage is sufficient
+- do not assume `chroot/boot/` will contain memtest payloads
+- do not assume source bootloader templates are the last writer of final ISO configs
+- do not assume the current normal binary hook timing is late enough for final patching
+
+Any future memtest fix must explicitly identify:
+
+- where the memtest binaries are reliably available at build time
+- which exact build stage writes the final bootloader configs that land in the ISO
+- and a post-build proof from a real ISO, not only from intermediate workdir files
+
 ## Decision
 
 For `bee`, memtest must be treated as an explicit ISO artifact with explicit post-build validation.
@@ -44,12 +101,17 @@ Project rules from now on:
 Current implementation direction:
 
 - keep the live-build memtest stage enabled if it helps package acquisition
-- but enforce memtest explicitly in a project-owned binary hook
+- do not rely on the current early `binary_hooks` timing for final patching
-- patch the generated `binary/boot/grub/grub.cfg` and `binary/isolinux/live.cfg` directly in the binary stage if memtest entries are missing
+- prefer a post-`lb build` recovery step in `build.sh` that:
+  - patches the fully materialized `LB_DIR/binary` tree
+  - injects memtest binaries there
+  - ensures final bootloader entries there
+  - reruns late binary stages (`binary_checksums`, `binary_iso`, `binary_zsync`) after the patch
 
 ## Consequences
 
 - Future memtest changes must begin by reading this ADR and the commits listed above.
+- Future memtest changes must also begin by reading the failed-attempt list above.
 - We should stop re-introducing "prefer built-in live-build memtest" as a default assumption without new evidence.
 - Memtest validation in `build.sh` is not optional; it is the acceptance gate that prevents another silent regression.
 - If we change memtest strategy again, we must update this ADR with the exact build evidence that justified the change.

bible-local/docs/iso-build-rules.md

@@ -39,3 +39,17 @@ Rules:
   a real ISO.
 - If you reference memtest files manually, verify the exact package file list
   first for the target Debian release.
+
+Known bad loops for this repository:
+
+- Do not retry built-in-only memtest without new evidence. We already proved
+  that `lb binary_memtest` can run while the final ISO still has no memtest.
+- Do not assume fixing memtest file names is enough. Correct names did not fix
+  the final artifact path.
+- Do not assume `chroot/boot/` contains memtest payloads at the time hooks run.
+- Do not assume source `grub.cfg` / `live.cfg.in` are the final writers of ISO
+  bootloader configs.
+- Do not assume the current `config/hooks/normal/9100-memtest.hook.binary`
+  timing is late enough to patch final `binary/boot/grub/grub.cfg` or
+  `binary/isolinux/live.cfg`; logs from 2026-04-01 showed those files were not
+  present yet when the hook executed.

iso/builder/build.sh

@@ -272,6 +272,59 @@ memtest_fail() {
     return 0
 }
 
+iso_memtest_present() {
+    iso_path="$1"
+
+    [ -f "$iso_path" ] || return 1
+
+    if command -v bsdtar >/dev/null 2>&1; then
+        :
+    elif command -v xorriso >/dev/null 2>&1; then
+        :
+    else
+        return 1
+    fi
+
+    iso_list_files "$iso_path" | grep -q '^boot/memtest86+x64\.bin$' || return 1
+    iso_list_files "$iso_path" | grep -q '^boot/memtest86+x64\.efi$' || return 1
+
+    grub_cfg="$(mktemp)"
+    isolinux_cfg="$(mktemp)"
+
+    iso_extract_file "$iso_path" boot/grub/grub.cfg > "$grub_cfg" 2>/dev/null || {
+        rm -f "$grub_cfg" "$isolinux_cfg"
+        return 1
+    }
+    iso_extract_file "$iso_path" isolinux/live.cfg > "$isolinux_cfg" 2>/dev/null || {
+        rm -f "$grub_cfg" "$isolinux_cfg"
+        return 1
+    }
+
+    grep -q 'Memory Test (memtest86+)' "$grub_cfg" || {
+        rm -f "$grub_cfg" "$isolinux_cfg"
+        return 1
+    }
+    grep -q '/boot/memtest86+x64\.efi' "$grub_cfg" || {
+        rm -f "$grub_cfg" "$isolinux_cfg"
+        return 1
+    }
+    grep -q '/boot/memtest86+x64\.bin' "$grub_cfg" || {
+        rm -f "$grub_cfg" "$isolinux_cfg"
+        return 1
+    }
+    grep -q 'Memory Test (memtest86+)' "$isolinux_cfg" || {
+        rm -f "$grub_cfg" "$isolinux_cfg"
+        return 1
+    }
+    grep -q '/boot/memtest86+x64\.bin' "$isolinux_cfg" || {
+        rm -f "$grub_cfg" "$isolinux_cfg"
+        return 1
+    }
+
+    rm -f "$grub_cfg" "$isolinux_cfg"
+    return 0
+}
+
 validate_iso_memtest() {
     iso_path="$1"
     echo "=== validating memtest in ISO ==="
@@ -335,6 +388,127 @@ validate_iso_memtest() {
     echo "=== memtest validation OK ==="
 }
 
+append_memtest_grub_entry() {
+    grub_cfg="$1"
+    [ -f "$grub_cfg" ] || return 1
+    grep -q 'Memory Test (memtest86+)' "$grub_cfg" && return 0
+    grep -q '### BEE MEMTEST ###' "$grub_cfg" && return 0
+
+    cat >> "$grub_cfg" <<'EOF'
+
+### BEE MEMTEST ###
+if [ "${grub_platform}" = "efi" ]; then
+    menuentry "Memory Test (memtest86+)" {
+        chainloader /boot/memtest86+x64.efi
+    }
+else
+    menuentry "Memory Test (memtest86+)" {
+        linux16 /boot/memtest86+x64.bin
+    }
+fi
+### /BEE MEMTEST ###
+EOF
+}
+
+append_memtest_isolinux_entry() {
+    isolinux_cfg="$1"
+    [ -f "$isolinux_cfg" ] || return 1
+    grep -q 'Memory Test (memtest86+)' "$isolinux_cfg" && return 0
+    grep -q '### BEE MEMTEST ###' "$isolinux_cfg" && return 0
+
+    cat >> "$isolinux_cfg" <<'EOF'
+
+# ### BEE MEMTEST ###
+label memtest
+    menu label ^Memory Test (memtest86+)
+    linux /boot/memtest86+x64.bin
+# ### /BEE MEMTEST ###
+EOF
+}
+
+copy_memtest_from_deb() {
+    deb="$1"
+    dst_boot="$2"
+    tmpdir="$(mktemp -d)"
+
+    dpkg-deb -x "$deb" "$tmpdir"
+    for f in memtest86+x64.bin memtest86+x64.efi; do
+        if [ -f "$tmpdir/boot/$f" ]; then
+            cp "$tmpdir/boot/$f" "$dst_boot/$f"
+        fi
+    done
+    rm -rf "$tmpdir"
+}
+
+recover_iso_memtest() {
+    lb_dir="$1"
+    iso_path="$2"
+    binary_boot="$lb_dir/binary/boot"
+    grub_cfg="$lb_dir/binary/boot/grub/grub.cfg"
+    isolinux_cfg="$lb_dir/binary/isolinux/live.cfg"
+
+    echo "=== attempting memtest recovery in binary tree ==="
+
+    mkdir -p "$binary_boot"
+
+    for root in \
+        "$lb_dir/chroot/boot" \
+        "/boot"; do
+        for f in memtest86+x64.bin memtest86+x64.efi; do
+            if [ ! -f "$binary_boot/$f" ] && [ -f "$root/$f" ]; then
+                cp "$root/$f" "$binary_boot/$f"
+                echo "memtest recovery: copied $f from $root"
+            fi
+        done
+    done
+
+    if [ ! -f "$binary_boot/memtest86+x64.bin" ] || [ ! -f "$binary_boot/memtest86+x64.efi" ]; then
+        for dir in \
+            "$lb_dir/cache/packages.binary" \
+            "$lb_dir/cache/packages.chroot" \
+            "$lb_dir/chroot/var/cache/apt/archives" \
+            "${BEE_CACHE_DIR:-${DIST_DIR}/cache}/lb-packages" \
+            "/var/cache/apt/archives"; do
+            [ -d "$dir" ] || continue
+            deb="$(find "$dir" -maxdepth 1 -type f -name 'memtest86+*.deb' 2>/dev/null | head -1)"
+            [ -n "$deb" ] || continue
+            echo "memtest recovery: extracting payload from $deb"
+            copy_memtest_from_deb "$deb" "$binary_boot"
+            break
+        done
+    fi
+
+    if [ ! -f "$binary_boot/memtest86+x64.bin" ] || [ ! -f "$binary_boot/memtest86+x64.efi" ]; then
+        tmpdl="$(mktemp -d)"
+        if (
+            cd "$tmpdl" && apt-get download memtest86+ >/dev/null 2>&1
+        ); then
+            deb="$(find "$tmpdl" -maxdepth 1 -type f -name 'memtest86+*.deb' 2>/dev/null | head -1)"
+            if [ -n "$deb" ]; then
+                echo "memtest recovery: downloaded $deb"
+                copy_memtest_from_deb "$deb" "$binary_boot"
+            fi
+        fi
+        rm -rf "$tmpdl"
+    fi
+
+    if [ -f "$grub_cfg" ]; then
+        append_memtest_grub_entry "$grub_cfg" && echo "memtest recovery: ensured GRUB entry"
+    else
+        echo "memtest recovery: WARNING: missing $grub_cfg"
+    fi
+
+    if [ -f "$isolinux_cfg" ]; then
+        append_memtest_isolinux_entry "$isolinux_cfg" && echo "memtest recovery: ensured isolinux entry"
+    else
+        echo "memtest recovery: WARNING: missing $isolinux_cfg"
+    fi
+
+    run_optional_step_sh "rebuild live-build checksums after memtest recovery" "91-lb-checksums" "lb binary_checksums 2>&1"
+    run_optional_step_sh "rebuild ISO after memtest recovery" "92-lb-binary-iso" "rm -f '$iso_path' && lb binary_iso 2>&1"
+    run_optional_step_sh "rebuild zsync after memtest recovery" "93-lb-zsync" "lb binary_zsync 2>&1"
+}
+
 AUDIT_VERSION_EFFECTIVE="$(resolve_audit_version)"
 ISO_VERSION_EFFECTIVE="$(resolve_iso_version)"
 ISO_BASENAME="easy-bee-${BEE_GPU_VENDOR}-v${ISO_VERSION_EFFECTIVE}-amd64"
@@ -451,6 +625,32 @@ run_step_sh() {
     run_step "${step_name}" "${step_slug}" sh -c "${step_script}"
 }
 
+run_optional_step_sh() {
+    step_name="$1"
+    step_slug="$2"
+    step_script="$3"
+
+    if [ "${BEE_REQUIRE_MEMTEST:-0}" = "1" ]; then
+        run_step_sh "${step_name}" "${step_slug}" "${step_script}"
+        return 0
+    fi
+
+    step_log="${LOG_DIR}/${step_slug}.log"
+    echo ""
+    echo "=== optional step: ${step_name} ==="
+    echo "=== optional step log: ${step_log} ==="
+    set +e
+    sh -c "${step_script}" > "${step_log}" 2>&1
+    step_status=$?
+    set -e
+    cat "${step_log}"
+    if [ "${step_status}" -ne 0 ]; then
+        echo "WARNING: optional step failed: ${step_name} (see ${step_log})" >&2
+    else
+        echo "=== optional step OK: ${step_name} ==="
+    fi
+}
+
 start_build_log
 
 # Auto-detect kernel ABI: refresh apt index, then query current linux-image-amd64 dependency.
@@ -857,6 +1057,10 @@ fi
 ISO_RAW="${LB_DIR}/live-image-amd64.hybrid.iso"
 if [ -f "$ISO_RAW" ]; then
     dump_memtest_debug "post-build" "${LB_DIR}" "$ISO_RAW"
+    if ! iso_memtest_present "$ISO_RAW"; then
+        recover_iso_memtest "${LB_DIR}" "$ISO_RAW"
+        dump_memtest_debug "post-recovery" "${LB_DIR}" "$ISO_RAW"
+    fi
     validate_iso_memtest "$ISO_RAW"
     cp "$ISO_RAW" "$ISO_OUT"
     echo ""