bee/bible-local/decisions/2026-04-01-memtest-build-strategy.md

Decision: Treat memtest as explicit ISO content, not as trusted live-build magic

Date: 2026-04-01 Status: active

Context

We have already iterated on memtest multiple times and kept cycling between the same ideas. The commit history shows several distinct attempts:

• f91bce8 — fixed Bookworm memtest file names to memtest86+x64.bin / memtest86+x64.efi
• 5857805 — added a binary hook to copy memtest files from the build tree into the ISO root
• f96b149 — added fallback extraction from the cached .deb when chroot/boot/ stayed empty
• d43a9ae — removed the custom hook and switched back to live-build built-in memtest integration
• 60cb8f8 — restored explicit memtest menu entries and added ISO validation
• 3dbc218 / 3869788 — added archived build logs and better memtest diagnostics

Current evidence from the archived easy-bee-nvidia-v3.14-amd64 logs dated 2026-04-01:

• lb binary_memtest does run and installs memtest86+
• but the final ISO still does not contain boot/memtest86+x64.bin
• the final ISO also does not contain memtest menu entries in boot/grub/grub.cfg or isolinux/live.cfg

So the assumption "live-build built-in memtest integration is enough on this stack" is currently false for this project until proven otherwise by a real built ISO.

Additional evidence from the archived easy-bee-nvidia-v3.17-dirty-amd64 logs dated 2026-04-01:

• the build now completes successfully because memtest is non-blocking by default
• lb binary_memtest still runs and installs memtest86+
• the project-owned hook config/hooks/normal/9100-memtest.hook.binary does execute
• but it executes too early for its current target paths:
  • binary/boot/grub/grub.cfg is still missing at hook time
  • binary/isolinux/live.cfg is still missing at hook time
  • memtest binaries are also still absent in binary/boot/
• later in the build, live-build does create intermediate bootloader configs with memtest lines in the workdir
• but the final ISO still lacks memtest binaries and still lacks memtest lines in extracted ISO boot/grub/grub.cfg and isolinux/live.cfg

So the assumption "the current normal binary hook path is late enough to patch final memtest artifacts" is also false.

Correction after inspecting the real easy-bee-nvidia-v3.20-5-g76a9100-amd64.iso artifact dated 2026-04-01:

• the final ISO does contain boot/memtest86+x64.bin
• the final ISO does contain boot/memtest86+x64.efi
• the final ISO does contain memtest menu entries in both boot/grub/grub.cfg and isolinux/live.cfg
• so v3.20-5-g76a9100 was not another real memtest regression in the shipped ISO
• the regression was in the build-time validator/debug path in build.sh

Root cause of the false alarm:

• build.sh treated "ISO reader command exists" as equivalent to "ISO reader successfully listed/extracted members"
• iso_list_files / iso_extract_file failures were collapsed into the same observable output as "memtest content missing"
• this made a reader failure look identical to a missing memtest payload
• as a result, we re-entered the same memtest investigation loop even though the real ISO was already correct

Known Failed Attempts

These approaches were already tried and should not be repeated blindly:

1. Built-in live-build memtest only. Reason it failed:

• lb binary_memtest runs, but the final ISO still misses memtest binaries and menu entries.

2. Fixing only the memtest file names for Debian Bookworm. Reason it failed:

• correct file names alone do not make the files appear in the final ISO.

3. Copying memtest from chroot/boot/ into binary/boot/ via a binary hook. Reason it failed:

• in this stack chroot/boot/ is often empty for memtest payloads at the relevant time.

4. Fallback extraction from cached memtest86+ .deb. Reason it failed:

• this was explored already and was not enough to stabilize the final ISO path end-to-end.

5. Restoring explicit memtest menu entries in source bootloader templates only. Reason it failed:

• memtest lines in source templates or intermediate workdir configs do not guarantee the final ISO contains them.

6. Patching binary/boot/grub/grub.cfg and binary/isolinux/live.cfg from the current config/hooks/normal/9100-memtest.hook.binary. Reason it failed:

• the hook runs before those files exist, so the hook cannot patch them there.

What This Means

When revisiting memtest later, start from the constraints above rather than retrying the same patterns:

• do not assume the built-in memtest stage is sufficient
• do not assume chroot/boot/ will contain memtest payloads
• do not assume source bootloader templates are the last writer of final ISO configs
• do not assume the current normal binary hook timing is late enough for final patching

Any future memtest fix must explicitly identify:

• where the memtest binaries are reliably available at build time
• which exact build stage writes the final bootloader configs that land in the ISO
• and a post-build proof from a real ISO, not only from intermediate workdir files
• whether the ISO inspection step itself succeeded, rather than merely whether the validator printed a memtest warning

Decision

For bee, memtest must be treated as an explicit ISO artifact with explicit post-build validation.

Project rules from now on:

• Do not trust --memtest memtest86+ by itself.
• A memtest implementation is considered valid only if the produced ISO actually contains:
  • boot/memtest86+x64.bin
  • boot/memtest86+x64.efi
  • a GRUB menu entry
  • an isolinux menu entry
• If live-build built-in integration does not produce those artifacts, use an explicit project-owned mechanism such as:
  • a binary hook copying files into binary/boot/
  • extraction from the cached memtest86+ .deb
  • another deterministic build-time copy step
• Do not remove such explicit logic later unless a fresh real ISO build proves that built-in integration alone produces all required files and menu entries.

Current implementation direction:

• keep the live-build memtest stage enabled if it helps package acquisition
• do not rely on the current early binary_hooks timing for final patching
• prefer a post-lb build recovery step in build.sh that:
  • patches the fully materialized LB_DIR/binary tree
  • injects memtest binaries there
  • ensures final bootloader entries there
  • reruns late binary stages (binary_checksums, binary_iso, binary_zsync) after the patch
• also treat ISO validation tooling as part of the critical path:
  • install a stable ISO reader in the builder image
  • fail with an explicit reader error if ISO listing/extraction fails
  • do not treat reader failure as evidence that memtest is missing

Consequences

• Future memtest changes must begin by reading this ADR and the commits listed above.
• Future memtest changes must also begin by reading the failed-attempt list above.
• We should stop re-introducing "prefer built-in live-build memtest" as a default assumption without new evidence.
• Memtest validation in build.sh is not optional; it is the acceptance gate that prevents another silent regression.
• But validation output is only trustworthy if ISO reading itself succeeded. A "missing memtest" warning without a successful ISO read is not evidence.
• If we change memtest strategy again, we must update this ADR with the exact build evidence that justified the change.