fix(nvidia): load ib_umad and install nvlsm/libnvidia-nscq for NVL5 fabric

The NVSwitch-detection fix (983f41a) was necessary but not sufficient on
HGX B200 (Kaytus KR9288-X3): ExecCondition now correctly runs fabricmanager,
but nvidia-fabricmanager-start.sh then aborts with "Kernel module ib_umad
has not been loaded" on this NVL5+ board, leaving the fabric stuck in
"In Progress" and cascading into failed dcgmi diag/NVBandwidth/NCCL/Validate
GPU SAT tasks (confirmed via support bundle 20260729-165331).

Neither nvidia-fabricmanager nor DCGM declare the NVLink5 support packages
as apt dependencies (checked against the cuda-repo Packages index directly):
ib_umad must be modprobed before FM starts, nvlsm (InfiniBand Subnet Manager)
is launched internally by FM's own start script once present, and DCGM
dlopens libnvidia-nscq at runtime for NVSwitch health monitoring.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-07-30 10:34:11 +03:00
co-authored by Claude Sonnet 5
parent 983f41a1f0
commit 6c8be629d2
2 changed files with 25 additions and 0 deletions
@@ -16,3 +16,21 @@ datacenter-gpu-manager-4-proprietary=1:%%DCGM_VERSION%%
datacenter-gpu-manager-4-proprietary-cuda13=1:%%DCGM_VERSION%%
ocl-icd-libopencl1
clinfo
# 4th-gen NVSwitch (NVLink5) support for HGX/DGX B100/B200/B300. Neither
# nvidia-fabricmanager nor datacenter-gpu-manager declare these as apt
# dependencies (confirmed against the cuda-repo Packages index), so the
# fabricmanager start script and DCGM's NVSwitch backend silently fail
# without them:
# - libibumad3: userspace lib nvidia-fabricmanager-start.sh needs to poll
# the CX7 management NIC (mlx5_*) before training the NVSwitch fabric.
# Comes from Debian's rdma-core, not the cuda-repo.
# - nvlsm: InfiniBand-compliant Subnet Manager. FM's own start script
# launches it internally on NVL5+ systems once present — no separate
# systemd unit to enable. Versioned independently of the driver branch.
# - libnvidia-nscq: NVSwitch query library DCGM dlopens at runtime for
# `dcgmi diag`/health monitoring on NVSwitch systems; must match the
# driver branch like nvidia-fabricmanager does.
libibumad3
nvlsm
libnvidia-nscq=%%NVIDIA_FABRICMANAGER_VERSION%%
@@ -7,3 +7,10 @@ After=bee-nvidia.service
# Skip fabricmanager on systems without NVSwitch hardware.
# ExecCondition exits 1-254 → unit is silently skipped (inactive, not failed).
ExecCondition=/usr/local/bin/bee-check-nvswitch
# On NVL5+ systems (e.g. HGX B200) nvidia-fabricmanager-start.sh polls for an
# Infiniband device before starting FM and refuses to start if "ib_umad" isn't
# loaded, even though mlx5_ib/ib_core/ib_uverbs already are — nothing else on
# the live ISO triggers its autoload. Load it explicitly; "-" keeps this from
# failing the unit on older HGX generations without an ib_umad module.
ExecStartPre=-/sbin/modprobe ib_umad