fix(nvidia): detect NVSwitch fabric via driver, not host-PCIe
On HGX B200 (Kaytus KR9288-X3) NVSwitch is managed via HMC/ERoT and never appears on host lspci as 10de:0680, so the old check always returned "not present" and silently skipped fabricmanager — leaving the fabric untrained and CUDA/NCCL/dcgmi failing with "system not yet initialized" on otherwise-healthy hardware. bee-check-nvswitch now checks, in order, nvidia-smi's GPU Fabric GUID, /proc/driver/nvidia-nvswitch/devices, and finally the legacy host-PCIe scan for older HGX generations.
This commit is contained in:
@@ -194,6 +194,26 @@ current-state answer.
|
||||
| `_services/bee-*.log` | Per-service startup/runtime logs for each bee-owned systemd unit. |
|
||||
| `_services/<service>.status.txt`, `.journal.log`, `combined.journal.log` | Captured fresh at bundle-build time, for every bee-owned service (`bee-audit`, `bee-web`, `bee-network`, `bee-nvidia`, `bee-preflight`, `bee-selfheal`, `bee-sshsetup`, `bee-blackbox`). |
|
||||
|
||||
## Why a bee service/check might have failed
|
||||
|
||||
Before treating a failed/inactive unit as a hardware fault, check whether
|
||||
its own logic is *designed* to skip or fail under specific benign
|
||||
conditions — several of these look like failures in `systemctl status` but
|
||||
are expected behavior on that particular machine.
|
||||
|
||||
| Service / check | What normal failure/skip looks like and why |
|
||||
|---|---|
|
||||
| `bee-blackbox.service` | Restarts forever (`Restart=always`) mirroring `export/` to removable media; only a real problem if `_state/blackbox-state.json`'s last-sync timestamp is stale — no USB media present is a common, harmless cause. |
|
||||
| `bee-nvidia.service` | Exits 0 and skips entirely if no `10de:0300`/`0302` (NVIDIA display/3D) PCI function is present, or if `/usr/local/bin/bee-nvidia-load` is absent (non-NVIDIA build) — not a failure. A real failure is `insmod` erroring, or (SXM→PCIe conversion boards) GSP firmware entering a crash/reload loop that never populates `/proc/devices` with `nvidiactl` within 90s; the script then tries `rmmod` to fall back to `NVreg_EnableGpuFirmware=0` — if `rmmod` also fails (`EBUSY`, module stuck), it logs `gsp-stuck` to `/run/bee-nvidia-mode` and the machine must be rebooted with `bee.nvidia.mode=gsp-off` (or `nomsi` if the dmesg tail shows `Failed to enable MSI-X`) at the boot menu. See `export/gpu/systemctl-nvidia-units.txt` and this service's `dmesg:` lines in its log for the exact symptom. |
|
||||
| `nvidia-fabricmanager.service` (`bee-check-nvswitch` `ExecCondition`) | **Not a failure when inactive.** `ExecCondition=/usr/local/bin/bee-check-nvswitch` decides whether this SKU has an NVSwitch fabric, in order: (1) `nvidia-smi -q`'s `GPU Fabric GUID` field has a real value (not `N/A`) — the primary signal, since on platforms like HGX B200 (Kaytus KR9288-X3) NVSwitch is managed via HMC/ERoT and never appears on host-PCIe at all; (2) `/proc/driver/nvidia-nvswitch/devices` is non-empty; (3) legacy `lspci -Dn` scan for a vendor `10de` class `0680` (Bridge, Other) device, for older HGX generations where NVSwitch *is* host-PCIe-visible. If all three say no (plain PCIe GPUs / direct NVLink bridges with no switch), the check exits 1 and systemd marks the unit `inactive (dead)` with condition unmet — never `failed`. Confirm via `systemctl status nvidia-fabricmanager.service` showing `ConditionResult=no`, plus `export/gpu/nvidia-smi-topo.txt` showing all-`PIX` topology (see "GPU topology" above) — same signature as `nvlink_pairs_checked=0`. **A stale build with the pre-driver-level lspci-only check will misfire on host-PCIe-invisible NVSwitch platforms** (e.g. HGX B200): condition (3) alone always says no, FM is always skipped, and `cuInit`/NCCL/`dcgmi diag` then fail with "system not yet initialized" on otherwise-healthy hardware — check `bee_version` in `manifest.txt` against the fix if this pattern shows up. An actual fabric-manager fault (real NVSwitch present, condition passes, but `ExecStart` errors) instead shows up as `failed`/`activating (auto-restart)` with training errors in `export/gpu/fabricmanager.log`/`nvlsm.log`. |
|
||||
| `nvidia-dcgm.service` | Ordered `After=bee-nvidia.service`; if it starts before device nodes exist, `nv-hostengine` enumerates zero GPUs and never rescans — `bee-nvidia-load` works around this by force-restarting it once modules are confirmed loaded, so a stale empty inventory here usually means that restart itself failed or timed out (see `bee-nvidia.log`'s `WARN: failed to restart nvidia-dcgm.service` / `timed out after 60s` lines) rather than a DCGM bug. |
|
||||
| `bee-audit.service` / `bee-preflight.service` | `bee-preflight` deliberately never fails the unit (`; exit 0` regardless of the underlying `bee preflight` return code) — check `runtime-health.log`'s own `WARN: preflight exited with rc=N` line, not unit status, for a real preflight problem. `bee-audit` failing outright usually means the underlying `bee audit` binary hit an unrecoverable collection error; check `export/bee-audit.log`, not just `systemctl status`. |
|
||||
| `bee-network.service` | Also structured to exit 0 unconditionally — `list_interfaces` filters out virtual/loopback interfaces, then DHCP is attempted per-NIC with a 300s timeout; a per-interface DHCP failure or timeout is logged (`DHCP timed out on <iface>` / `DHCP failed on <iface>`) but does not fail the unit, since a lease-less NIC (no cable, no DHCP server on that segment) is common and not itself a hardware defect. Check `bee-network.log`'s `leases acquired: N` vs `interfaces scanned: N` line for the real outcome. |
|
||||
| `bee-sshsetup.service` | Fails only on `useradd`/`chpasswd`/file-write errors (e.g. read-only `/etc`), which would be a filesystem problem, not an SSH-specific one. Whether it configures key-only auth or the `bee`/`eeb` password fallback is a deliberate branch on `/etc/bee-ssh-password-fallback`'s presence, not a fault either way — check which branch its log took before assuming misconfiguration. |
|
||||
| `bee-selfheal.service`/`.timer` | A oneshot that runs on a timer and restarts other bee services when it detects specific bad states — its own "failure" is rare (only if the lock dir can't be created, or the restart calls themselves fail) but its *log content* (`_services/bee-selfheal.log`) is often the fastest place to see which downstream service was unhealthy and had to be kicked: `/dev/nvidia0` missing while a GPU is present, an interrupted `.tmp` write for `bee-audit.json`/`runtime-health.json`, `bee-web.service` inactive, or `bee-web`'s own `/healthz` endpoint not answering on `127.0.0.1:80`. |
|
||||
| `bee-web.service` | `Restart=always`, no `MemoryMax` (deliberately — it spawns multi-GB GPU stress subprocesses like `dcgmproftester` that a cgroup limit would OOM-kill); a genuine crash-loop here is unusual and worth its full `journal.log`, not just the latest restart. |
|
||||
| `bee-hpc-tuning.service` | Best-effort CPU governor/C-state tuning ordered before `bee-nvidia`/`bee-audit`; failure here (e.g. governor sysfs knobs absent on this CPU/BIOS combination) does not block the rest of boot but can explain unexpectedly poor benchmark numbers if not caught. |
|
||||
|
||||
## `livecd/` in detail
|
||||
|
||||
Only relevant to this particular live-boot/kiosk session — not the physical
|
||||
|
||||
Regular → Executable
+29
-3
@@ -1,4 +1,30 @@
|
||||
#!/bin/sh
|
||||
# Exit 0 if NVSwitch hardware is detected; exit 1 to skip fabricmanager on non-NVSwitch systems.
|
||||
# NVSwitch appears in lspci as vendor 10de, class 0680 (Bridge, Other).
|
||||
lspci -Dn 2>/dev/null | awk '$2 == "0680:" && $3 ~ /^10de:/ { found=1; exit } END { exit(found ? 0 : 1) }'
|
||||
# ExecCondition для nvidia-fabricmanager.service.
|
||||
# exit 0 -> запустить FabricManager (система с NVSwitch-фабрикой)
|
||||
# exit 1 -> пропустить FabricManager (не-NVSwitch система)
|
||||
#
|
||||
# На HGX B200 (Kaytus KR9288-X3) NVSwitch управляется через HMC/ERoT и НЕ виден
|
||||
# на host-PCIe как 10de class 0680 — старый lspci-детект всегда давал ложное "нет"
|
||||
# и FM ошибочно пропускался (CUDA: "system not yet initialized"). Детектим фабрику
|
||||
# через драйвер NVIDIA. Драйвер уже загружен (After=bee-nvidia.service).
|
||||
|
||||
command -v nvidia-smi >/dev/null 2>&1 || exit 1 # нет драйвера/GPU -> пропустить
|
||||
|
||||
# (1) Основной признак: GPU, привязанный к NVSwitch-фабрике, отдаёт GPU Fabric GUID
|
||||
# (реальное значение, не N/A). Читается ещё до инициализации FM.
|
||||
if timeout 15 nvidia-smi -q 2>/dev/null | grep -i 'GPU Fabric GUID' | grep -qv 'N/A'; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# (2) Fallback: NVSwitch'и, поднятые kernel-драйвером nvidia-nvswitch.
|
||||
if [ -d /proc/driver/nvidia-nvswitch/devices ] &&
|
||||
[ -n "$(ls -A /proc/driver/nvidia-nvswitch/devices 2>/dev/null)" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# (3) Legacy-совместимость: старые HGX, где NVSwitch виден на host-PCIe (10de:0680).
|
||||
if lspci -Dn 2>/dev/null | awk '$2=="0680:" && $3 ~ /^10de:/ {f=1} END{exit(f?0:1)}'; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
exit 1
|
||||
|
||||
Reference in New Issue
Block a user