Mikhail ChusavitinandClaude Sonnet 5
e036a9c7b8
pci=realloc was forcing a full PCI resource reallocation on every default and toram boot, added as a GPU BAR workaround for BIOS without Above 4G Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS and lspci both saw them fine. It now only applies to the no-GUI/gsp-off troubleshooting entry, alongside a bible-local ADR for the two prior rounds of SAT/blackbox durability work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
80 lines
3.8 KiB
Markdown
80 lines
3.8 KiB
Markdown
# Decision: Move pci=realloc out of the default/toram/no-GUI GRUB entries
|
|
|
|
**Date:** 2026-07-28
|
|
**Status:** active
|
|
|
|
## Context
|
|
|
|
`pci=realloc` was added to the default, toram, and no-GUI/gsp-off GRUB menu
|
|
entries by commit 8bf8dfa ("default to KMS + pci=realloc, drop nomodeset from
|
|
main entries") as a workaround for GPU BAR assignment failures on servers
|
|
where the BIOS lacks "Above 4G Decoding". It forces the Linux kernel to
|
|
fully reallocate all PCI resources instead of trusting the BIOS-assigned
|
|
addresses.
|
|
|
|
On a dual-socket AMD EPYC 9655 server (MSI CG480-S6053, 6x GPU + 8x NVMe
|
|
behind 2-port hot-plug-capable PCIe switches), this same flag caused a
|
|
different failure: the OS only exposed 2 of 8 NVMe drives, even though
|
|
`lspci` and the BIOS both enumerated all 6 controllers actually present in
|
|
the affected chassis. `dmesg` showed the kernel's bridge-window reallocation
|
|
pass exhausting available MMIO32 address space:
|
|
|
|
```
|
|
pci 0000:f2:03.0: bridge window [mem size 0x00200000 64bit pref]: can't assign; no space
|
|
pci 0000:f7:00.0: BAR 0 [mem size 0x00004000 64bit]: can't assign; no space
|
|
```
|
|
|
|
The BIOS had already assigned working BARs to all 6 NVMe controllers at
|
|
boot; `pci=realloc` threw those away and failed to re-assign 4 of them,
|
|
so the `nvme` driver never bound to those 4 (no `nvme nvmeX: pci function`
|
|
line), and they never became block devices. The dense NVMe backplane, not
|
|
the GPUs, hit the address-space ceiling that `pci=realloc` re-triggered on
|
|
every device in the topology, not just the GPUs it was meant to fix.
|
|
|
|
`pci=realloc` is a blanket flag: it was applied globally to the default and
|
|
toram entries (the ones almost every operator actually boots), so any
|
|
server with a BIOS whose GPU BARs are fine, but whose PCIe topology is
|
|
dense enough elsewhere (many NVMe bays, many switches), pays this cost
|
|
unconditionally.
|
|
|
|
## Decision
|
|
|
|
`pci=realloc` is removed from the default and toram menu entries in both
|
|
`iso/builder/build.sh` (`write_canonical_grub_cfg`) and the static
|
|
`iso/builder/config/bootloaders/grub-efi/grub.cfg` template. Rather than
|
|
adding a new single-purpose menu entry for this one workaround, it is
|
|
folded into the existing no-GUI/gsp-off entry, which is already the
|
|
troubleshooting/fail-safe-style option operators reach for when the
|
|
default entry does not work cleanly:
|
|
|
|
```
|
|
EASY-BEE v<version> -- no GUI / troubleshooting (gsp-off, pci=realloc)
|
|
```
|
|
|
|
An operator who hits either a GSP-related GPU issue or a GPU BAR
|
|
assignment failure (BIOS lacking Above 4G Decoding) selects this single
|
|
entry manually, instead of the menu growing one dedicated item per
|
|
individual failure mode. This keeps the total entry count in
|
|
`grub.cfg` unchanged from before the incident (still 3 real boot
|
|
entries plus wipe/memtest/firmware-settings), rather than letting it
|
|
grow linearly with every workaround discovered.
|
|
|
|
The isolinux (BIOS/legacy boot) menu in `iso/builder/config/bootloaders/isolinux/live.cfg.in`
|
|
already had no `pci=realloc` entry at all (pre-existing asymmetry with
|
|
grub-efi, not introduced by this change) -- not addressed here since no
|
|
incident has been observed via legacy boot.
|
|
|
|
## Consequences
|
|
|
|
- Servers with dense PCIe topologies (many NVMe bays, many GPUs) no longer
|
|
silently lose drives to a workaround they didn't need, by default.
|
|
- Servers that genuinely need the GPU BAR workaround (BIOS lacking Above 4G
|
|
Decoding) now require the operator to notice GPU BAR failures and
|
|
deliberately pick the workaround entry, instead of getting it for free --
|
|
document this tradeoff in the support runbook / vendor-facing material
|
|
if GPU BAR failures turn out to be common enough to warrant it.
|
|
- DO NOT move `pci=realloc` back into the default/toram entries without
|
|
first checking whether the affected server's topology can tolerate a
|
|
full PCI reallocation pass -- this was the exact failure mode of the
|
|
incident that prompted this decision.
|