# Decision: Treat memtest as explicit ISO content, not as trusted live-build magic

**Date:** 2026-04-01
**Status:** resolved

## Context

We have already iterated on `memtest` multiple times and kept cycling between the same ideas.
The commit history shows several distinct attempts:

- `f91bce8` — fixed Bookworm memtest file names to `memtest86+x64.bin` / `memtest86+x64.efi`
- `5857805` — added a binary hook to copy memtest files from the build tree into the ISO root
- `f96b149` — added fallback extraction from the cached `.deb` when `chroot/boot/` stayed empty
- `d43a9ae` — removed the custom hook and switched back to live-build built-in memtest integration
- `60cb8f8` — restored explicit memtest menu entries and added ISO validation
- `3dbc218` / `3869788` — added archived build logs and better memtest diagnostics

Current evidence from the archived `easy-bee-nvidia-v3.14-amd64` logs dated 2026-04-01:

- `lb binary_memtest` does run and installs `memtest86+`
- but the final ISO still does **not** contain `boot/memtest86+x64.bin`
- the final ISO also does **not** contain memtest menu entries in `boot/grub/grub.cfg` or `isolinux/live.cfg`

So the assumption "live-build built-in memtest integration is enough on this stack" is currently false for this project until proven otherwise by a real built ISO.

Additional evidence from the archived `easy-bee-nvidia-v3.17-dirty-amd64` logs dated 2026-04-01:

- the build now completes successfully because memtest is non-blocking by default
- `lb binary_memtest` still runs and installs `memtest86+`
- the project-owned hook `config/hooks/normal/9100-memtest.hook.binary` does execute
- but it executes too early for its current target paths:
  - `binary/boot/grub/grub.cfg` is still missing at hook time
  - `binary/isolinux/live.cfg` is still missing at hook time
  - memtest binaries are also still absent in `binary/boot/`
- later in the build, live-build does create intermediate bootloader configs with memtest lines in the workdir
- but the final ISO still lacks memtest binaries and still lacks memtest lines in extracted ISO `boot/grub/grub.cfg` and `isolinux/live.cfg`

So the assumption "the current normal binary hook path is late enough to patch final memtest artifacts" is also false.

Correction after inspecting the real `easy-bee-nvidia-v3.20-5-g76a9100-amd64.iso`
artifact dated 2026-04-01:

- the final ISO does contain `boot/memtest86+x64.bin`
- the final ISO does contain `boot/memtest86+x64.efi`
- the final ISO does contain memtest menu entries in both `boot/grub/grub.cfg`
  and `isolinux/live.cfg`
- so `v3.20-5-g76a9100` was **not** another real memtest regression in the
  shipped ISO
- the regression was in the build-time validator/debug path in `build.sh`

Root cause of the false alarm:

- `build.sh` treated "ISO reader command exists" as equivalent to "ISO reader
  successfully listed/extracted members"
- `iso_list_files` / `iso_extract_file` failures were collapsed into the same
  observable output as "memtest content missing"
- this made a reader failure look identical to a missing memtest payload
- as a result, we re-entered the same memtest investigation loop even though
  the real ISO was already correct

Additional correction from the subsequent `v3.21` build logs dated 2026-04-01:

- once ISO reading was fixed, the post-build debug correctly showed the raw ISO
  still carried live-build's default memtest layout (`live/memtest.bin`,
  `live/memtest.efi`, `boot/grub/memtest.cfg`, `isolinux/memtest.cfg`)
- that mismatch is expected to trigger project recovery, because `bee` requires
  `boot/memtest86+x64.bin` / `boot/memtest86+x64.efi` plus matching menu paths
- however, `build.sh` exited before recovery because `set -e` treated a direct
  `iso_memtest_present` return code of `1` as fatal
- so the next repeated loop was caused by shell control flow, not by proof that
  the recovery design itself was wrong

## Known Failed Attempts

These approaches were already tried and should not be repeated blindly:

1. Built-in live-build memtest only.
Reason it failed:
- `lb binary_memtest` runs, but the final ISO still misses memtest binaries and menu entries.

2. Fixing only the memtest file names for Debian Bookworm.
Reason it failed:
- correct file names alone do not make the files appear in the final ISO.

3. Copying memtest from `chroot/boot/` into `binary/boot/` via a binary hook.
Reason it failed:
- in this stack `chroot/boot/` is often empty for memtest payloads at the relevant time.

4. Fallback extraction from cached `memtest86+` `.deb`.
Reason it failed:
- this was explored already and was not enough to stabilize the final ISO path end-to-end.

5. Restoring explicit memtest menu entries in source bootloader templates only.
Reason it failed:
- memtest lines in source templates or intermediate workdir configs do not guarantee the final ISO contains them.

6. Patching `binary/boot/grub/grub.cfg` and `binary/isolinux/live.cfg` from the current `config/hooks/normal/9100-memtest.hook.binary`.
Reason it failed:
- the hook runs before those files exist, so the hook cannot patch them there.

## What This Means

When revisiting memtest later, start from the constraints above rather than retrying the same patterns:

- do not assume the built-in memtest stage is sufficient
- do not assume `chroot/boot/` will contain memtest payloads
- do not assume source bootloader templates are the last writer of final ISO configs
- do not assume the current normal binary hook timing is late enough for final patching

Any future memtest fix must explicitly identify:

- where the memtest binaries are reliably available at build time
- which exact build stage writes the final bootloader configs that land in the ISO
- and a post-build proof from a real ISO, not only from intermediate workdir files
- whether the ISO inspection step itself succeeded, rather than merely whether
  the validator printed a memtest warning
- whether a non-zero probe is intentionally handled inside an `if` / `case`
  context rather than accidentally tripping `set -e`

## Decision

For `bee`, memtest must be treated as an explicit ISO artifact with explicit post-build validation.

Project rules from now on:

- Do **not** trust `--memtest memtest86+` by itself.
- A memtest implementation is considered valid only if the produced ISO actually contains:
  - `boot/memtest86+x64.bin`
  - `boot/memtest86+x64.efi`
  - a GRUB menu entry
  - an isolinux menu entry
- If live-build built-in integration does not produce those artifacts, use an explicit project-owned mechanism such as:
  - a binary hook copying files into `binary/boot/`
  - extraction from the cached `memtest86+` `.deb`
  - another deterministic build-time copy step
- Do **not** remove such explicit logic later unless a fresh real ISO build proves that built-in integration alone produces all required files and menu entries.

Current implementation direction:

- keep the live-build memtest stage enabled if it helps package acquisition
- do not rely on the current early `binary_hooks` timing for final patching
- prefer a post-`lb build` recovery step in `build.sh` that:
  - patches the fully materialized `LB_DIR/binary` tree
  - injects memtest binaries there
  - ensures final bootloader entries there
  - reruns late binary stages (`binary_checksums`, `binary_iso`, `binary_zsync`) after the patch
- also treat ISO validation tooling as part of the critical path:
  - install a stable ISO reader in the builder image
  - fail with an explicit reader error if ISO listing/extraction fails
  - do not treat reader failure as evidence that memtest is missing
  - do not call a probe that may return "needs recovery" as a bare command under
    `set -e`; wrap it in explicit control flow

## Consequences

- Future memtest changes must begin by reading this ADR and the commits listed above.
- Future memtest changes must also begin by reading the failed-attempt list above.
- We should stop re-introducing "prefer built-in live-build memtest" as a default assumption without new evidence.
- Memtest validation in `build.sh` is not optional; it is the acceptance gate that prevents another silent regression.
- But validation output is only trustworthy if ISO reading itself succeeded. A
  "missing memtest" warning without a successful ISO read is not evidence.
- If we change memtest strategy again, we must update this ADR with the exact build evidence that justified the change.

## Working Solution (confirmed 2026-04-01, commits 76a9100 → 2baf3be)

This approach was confirmed working in ISO `easy-bee-nvidia-v3.20-5-g76a9100-amd64.iso`
and validated again in subsequent builds. The final ISO contains all required memtest artifacts.

### Components

**1. Binary hook `config/hooks/normal/9100-memtest.hook.binary`**

Runs inside the live-build binary phase. Does not patch bootloader files at hook time —
those files may not exist yet. Instead:

- Tries to copy `memtest86+x64.bin` / `memtest86+x64.efi` from `chroot/boot/` first.
- Falls back to extracting from the cached `.deb` (via `dpkg-deb -x`) if `chroot/boot/` is empty.
- It does not edit bootloader menus. The complete menus are rendered later
  from the two canonical project templates.

Controlled by `BEE_REQUIRE_MEMTEST=1` env var to turn warnings into hard errors when needed.

**2. Post-`lb build` recovery step in `build.sh`**

After `lb build` completes, `build.sh` checks whether the fully materialized `binary/` tree
contains all required memtest artifacts. If not:

- Copies/extracts memtest binaries into `binary/boot/`.
- Calls `enforce_live_build_bootloader_assets`, which renders the complete
  GRUB and isolinux configs from the canonical project templates.
- Reruns the late binary stages (`binary_checksums`, `binary_iso`, `binary_zsync`) to rebuild
  the ISO with the patched tree.

This is the deterministic safety net: even if the hook runs at the wrong time, the recovery
step handles the final `binary/` tree after live-build has written all bootloader configs.

**3. ISO validation hardening**

The memtest probe in `build.sh` is wrapped in explicit `if` / `case` control flow, not called
as a bare command under `set -e`. A non-zero probe return (needs recovery) is intentional and
handled — it does not abort the build prematurely.

ISO reading (`xorriso -indev -ls` / extraction) is treated as a separate prerequisite.
If the reader fails, the validator reports a reader error explicitly, not a memtest warning.
This prevents the false-negative loop that burned 2026-04-01 v3.14–v3.19.

### Why this works when earlier attempts did not

The earlier patterns all shared a single flaw: they assumed a single build-time point
(hook or source template) would be the last writer of bootloader configs and memtest payloads.
In live-build on Debian Bookworm that assumption is false — live-build continues writing
bootloader files after custom hooks run, and `chroot/boot/` does not reliably hold memtest payloads.

The recovery step sidesteps the ordering problem entirely: it acts on the fully materialized
`binary/` tree after `lb build` finishes, then rebuilds the ISO from that patched tree.
There is no ordering dependency to get wrong.

### Do not revert

Do not remove the recovery step or the hook without a fresh real ISO build proving
live-build alone produces all four required artifacts:
- `boot/memtest86+x64.bin`
- `boot/memtest86+x64.efi`
- memtest entry in `boot/grub/grub.cfg`
- memtest entry in `isolinux/live.cfg`