From 6c8be629d2cf55f685b5cf58c8271fae9b51f7dc Mon Sep 17 00:00:00 2001 From: Mikhail Chusavitin Date: Thu, 30 Jul 2026 10:34:11 +0300 Subject: [PATCH] fix(nvidia): load ib_umad and install nvlsm/libnvidia-nscq for NVL5 fabric The NVSwitch-detection fix (983f41a) was necessary but not sufficient on HGX B200 (Kaytus KR9288-X3): ExecCondition now correctly runs fabricmanager, but nvidia-fabricmanager-start.sh then aborts with "Kernel module ib_umad has not been loaded" on this NVL5+ board, leaving the fabric stuck in "In Progress" and cascading into failed dcgmi diag/NVBandwidth/NCCL/Validate GPU SAT tasks (confirmed via support bundle 20260729-165331). Neither nvidia-fabricmanager nor DCGM declare the NVLink5 support packages as apt dependencies (checked against the cuda-repo Packages index directly): ib_umad must be modprobed before FM starts, nvlsm (InfiniBand Subnet Manager) is launched internally by FM's own start script once present, and DCGM dlopens libnvidia-nscq at runtime for NVSwitch health monitoring. Co-Authored-By: Claude Sonnet 5 --- .../package-lists/bee-nvidia.list.chroot | 18 ++++++++++++++++++ .../bee-nvswitch-check.conf | 7 +++++++ 2 files changed, 25 insertions(+) diff --git a/iso/builder/config/package-lists/bee-nvidia.list.chroot b/iso/builder/config/package-lists/bee-nvidia.list.chroot index 0dfe712..95bfd18 100644 --- a/iso/builder/config/package-lists/bee-nvidia.list.chroot +++ b/iso/builder/config/package-lists/bee-nvidia.list.chroot @@ -16,3 +16,21 @@ datacenter-gpu-manager-4-proprietary=1:%%DCGM_VERSION%% datacenter-gpu-manager-4-proprietary-cuda13=1:%%DCGM_VERSION%% ocl-icd-libopencl1 clinfo + +# 4th-gen NVSwitch (NVLink5) support for HGX/DGX B100/B200/B300. Neither +# nvidia-fabricmanager nor datacenter-gpu-manager declare these as apt +# dependencies (confirmed against the cuda-repo Packages index), so the +# fabricmanager start script and DCGM's NVSwitch backend silently fail +# without them: +# - libibumad3: userspace lib nvidia-fabricmanager-start.sh needs to poll +# the CX7 management NIC (mlx5_*) before training the NVSwitch fabric. +# Comes from Debian's rdma-core, not the cuda-repo. +# - nvlsm: InfiniBand-compliant Subnet Manager. FM's own start script +# launches it internally on NVL5+ systems once present — no separate +# systemd unit to enable. Versioned independently of the driver branch. +# - libnvidia-nscq: NVSwitch query library DCGM dlopens at runtime for +# `dcgmi diag`/health monitoring on NVSwitch systems; must match the +# driver branch like nvidia-fabricmanager does. +libibumad3 +nvlsm +libnvidia-nscq=%%NVIDIA_FABRICMANAGER_VERSION%% diff --git a/iso/overlay/etc/systemd/system/nvidia-fabricmanager.service.d/bee-nvswitch-check.conf b/iso/overlay/etc/systemd/system/nvidia-fabricmanager.service.d/bee-nvswitch-check.conf index fbd5e04..82154e8 100644 --- a/iso/overlay/etc/systemd/system/nvidia-fabricmanager.service.d/bee-nvswitch-check.conf +++ b/iso/overlay/etc/systemd/system/nvidia-fabricmanager.service.d/bee-nvswitch-check.conf @@ -7,3 +7,10 @@ After=bee-nvidia.service # Skip fabricmanager on systems without NVSwitch hardware. # ExecCondition exits 1-254 → unit is silently skipped (inactive, not failed). ExecCondition=/usr/local/bin/bee-check-nvswitch + +# On NVL5+ systems (e.g. HGX B200) nvidia-fabricmanager-start.sh polls for an +# Infiniband device before starting FM and refuses to start if "ib_umad" isn't +# loaded, even though mlx5_ib/ib_core/ib_uverbs already are — nothing else on +# the live ISO triggers its autoload. Load it explicitly; "-" keeps this from +# failing the unit on older HGX generations without an ib_umad module. +ExecStartPre=-/sbin/modprobe ib_umad