bee/bible-local/decisions/2026-07-28-pci-realloc-out-of-default-boot-entries.md

Decision: Move pci=realloc out of the default/toram/no-GUI GRUB entries

Date: 2026-07-28 Status: active

Context

pci=realloc was added to the default, toram, and no-GUI/gsp-off GRUB menu entries by commit 8bf8dfa ("default to KMS + pci=realloc, drop nomodeset from main entries") as a workaround for GPU BAR assignment failures on servers where the BIOS lacks "Above 4G Decoding". It forces the Linux kernel to fully reallocate all PCI resources instead of trusting the BIOS-assigned addresses.

On a dual-socket AMD EPYC 9655 server (MSI CG480-S6053, 6x GPU + 8x NVMe behind 2-port hot-plug-capable PCIe switches), this same flag caused a different failure: the OS only exposed 2 of 8 NVMe drives, even though lspci and the BIOS both enumerated all 6 controllers actually present in the affected chassis. dmesg showed the kernel's bridge-window reallocation pass exhausting available MMIO32 address space:

pci 0000:f2:03.0: bridge window [mem size 0x00200000 64bit pref]: can't assign; no space
pci 0000:f7:00.0: BAR 0 [mem size 0x00004000 64bit]: can't assign; no space

The BIOS had already assigned working BARs to all 6 NVMe controllers at boot; pci=realloc threw those away and failed to re-assign 4 of them, so the nvme driver never bound to those 4 (no nvme nvmeX: pci function line), and they never became block devices. The dense NVMe backplane, not the GPUs, hit the address-space ceiling that pci=realloc re-triggered on every device in the topology, not just the GPUs it was meant to fix.

pci=realloc is a blanket flag: it was applied globally to the default and toram entries (the ones almost every operator actually boots), so any server with a BIOS whose GPU BARs are fine, but whose PCIe topology is dense enough elsewhere (many NVMe bays, many switches), pays this cost unconditionally.

Decision

pci=realloc is removed from the default and toram menu entries in both iso/builder/build.sh (write_canonical_grub_cfg) and the static iso/builder/config/bootloaders/grub-efi/grub.cfg template. Rather than adding a new single-purpose menu entry for this one workaround, it is folded into the existing no-GUI/gsp-off entry, which is already the troubleshooting/fail-safe-style option operators reach for when the default entry does not work cleanly:

EASY-BEE v<version> -- no GUI / troubleshooting (gsp-off, pci=realloc)

An operator who hits either a GSP-related GPU issue or a GPU BAR assignment failure (BIOS lacking Above 4G Decoding) selects this single entry manually, instead of the menu growing one dedicated item per individual failure mode. This keeps the total entry count in grub.cfg unchanged from before the incident (still 3 real boot entries plus wipe/memtest/firmware-settings), rather than letting it grow linearly with every workaround discovered.

The isolinux (BIOS/legacy boot) menu in iso/builder/config/bootloaders/isolinux/live.cfg.in already had no pci=realloc entry at all (pre-existing asymmetry with grub-efi, not introduced by this change) -- not addressed here since no incident has been observed via legacy boot.

Consequences

• Servers with dense PCIe topologies (many NVMe bays, many GPUs) no longer silently lose drives to a workaround they didn't need, by default.
• Servers that genuinely need the GPU BAR workaround (BIOS lacking Above 4G Decoding) now require the operator to notice GPU BAR failures and deliberately pick the workaround entry, instead of getting it for free -- document this tradeoff in the support runbook / vendor-facing material if GPU BAR failures turn out to be common enough to warrant it.
• DO NOT move pci=realloc back into the default/toram entries without first checking whether the affected server's topology can tolerate a full PCI reallocation pass -- this was the exact failure mode of the incident that prompted this decision.