Each fan tile carries data-fan / data-ceil / data-measured and a small
script polls /api/metrics/latest every 5s — the metrics collector's own
sampling period, served from memory with no extra BMC call — updating
each tile's spin rate, duty fill and tooltip in place. Polling faster
would only re-read identical numbers; sampling the BMC faster would
choke it under load. FanReading gains json tags for the endpoint.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
It is a platform cooling test (CPU + memory always; GPU only as extra
heat when present), not a GPU test — so it belongs with the other
whole-box checks, not in the NVIDIA section. Still Load-page only, via
the new renderLoadOnlySATCard (inverse of renderCheckOnlySATCard).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
- GPU load switches from bee-gpu-burn (compute burn, ~88% TDP) to
dcgmproftester -t 1004 / targeted_power via
resolveBenchmarkPowerLoadCommand — the same engine Power/Thermal Fit
uses and the hottest sustained NVIDIA load we have, so fans are
actually pushed toward their ceiling.
- Sample loop is now IPMI-hang-proof: every ipmitool read is time-boxed
in an abandonable goroutine, and the poll interval backs off
geometrically (1s→30s) when reads are slow, tightening again on
recovery. A plateau is only trusted while telemetry is healthy;
degraded runs ride out to MaxLoadSec. Summary gains fan_samples /
telemetry_degraded. Drops the per-second nvidia-smi+power+cpu-temp
sampling from the hot loop.
- Dead code removed: FanStressRow, GPUStressMetric, sampleFanStressRow,
sampleGPUStressMetrics, WriteFanStressCSV/WriteFanSensorsCSV,
analyzeMaxTemp, sampleSystemPowerResolved.
Topology fan tiles:
- size encodes the fan's ceiling RPM (its class), not current speed;
coloured fill rising from the bottom encodes live duty cycle
(current / ceiling), shown only when the ceiling was measured.
- glyph spin rate now maps absolute RPM into a human-perceptible band
(fanSpinPeriodSec: 2.2s/turn at <=1000 RPM, 0.35s at >=13000).
- the "N fans · N OK · tile size ∝ …" caption line is gone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
Repurpose the previously-unwired RunFanStressTest into RunFanCheck, a
Load-tier SAT test that drives stressapptest (CPU+memory) and, when a GPU
is present, a GPU burn to 100% simultaneously, then watches every fan
until none has climbed for ~60s. The observed peak RPM per fan is the
"ceiling"; it is persisted through the existing fan-observation store.
MSI G4201 / AMI MegaRAC exposes no host-side fan force (every OEM IPMI
command returns 0xc1; Redfish Thermal is GET-only), so load-driven ramp
is the closest safe equivalent. See
bible-local/decisions/2026-09-04-fan-ceiling-check.md.
- platform.ResolveFanMaxRPM: per-fan max with fallback (persisted peak ->
peer peak -> current RPM), resolved in platform, not the view.
- platform.ErrTestNotApplicable: no load source or no fan sensors ->
task lands as cancelled ("not applicable"), never failed, so an
engineer never sees a false red. executeTaskWithOptions maps the
sentinel; finalizeTaskForResult honours a pre-set TaskCancelled.
- Verdict FAIL only for a fan at 0 RPM / IPMI cr-nr under load.
- /topo: one small spinning square per fan, sized by RPM / resolved max,
clickable through to a new "fan" component-detail type; per-fan status
recorded to the component-status DB from the fan SAT summary.
- Wiring: /api/sat/fan/run route, "fan" task target, Load-page card,
stress-mode Run All.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
Every GPU-selection picker (Load/SAT, Burn, Benchmark) now renders the
card serial: "GPU N — <model> · <mem> MiB · sn: <serial>". The serial is
monospace and the digits that differ across the listed GPUs (common
prefix/suffix stripped) are emphasised so operators can tell cards apart.
Row markup was duplicated across three pages (and twice within
page_validate.go). Consolidated into a single module,
internal/webui/gpu_picker.go: beeGpuPicker.render({...}) builds every row;
gpuPickerCSS/gpuPickerJS are injected once by layoutHead/renderPage. Pages
keep their own selection-note text and multi-GPU toggles but no longer
hand-build <label> markup.
ListNvidiaGPUs() adds the serial via nvidia-smi --query-gpu=...,serial;
N/A is normalised to empty. Serial flows to the client unchanged through
the existing /api/gpu/nvidia JSON response.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AudE3Q2nd9kxxVuxPKcTho
Every sample was stamped with time.Now(), so a timezone switch or NTP
step punched a multi-hour gap into the series: old points collapsed to
the left edge, new points bunched at the right, joined by one diagonal.
Stamp rows from a monotonic seqClock instead — seeded once from the wall
clock (or the newest persisted row) and thereafter advanced only by the
monotonic elapsed time between writes. Rebase Downsample/Prune on the
newest sample rather than time.Now() so a clock step just before the
hourly compaction can't drop fresh data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GofKhuF9xQHaz3UFfncR6D
The NVIDIA GSP firmware was staged into the overlay at a bare
lib/firmware/nvidia/<ver>/. On a merged-usr Debian root /lib is a symlink
to usr/lib, so the fast-path repack's `rsync -a` (overlay stage -> the
unsquashfs'd tree) replaced that symlink with a plain directory holding
only the firmware. Everything else reachable only through /lib
(/lib/systemd/systemd, /lib/x86_64-linux-gnu/ld-linux + libc, ...) was
orphaned. The squashfs still packed and mounted, but boot panicked:
run-init: can't execute '/sbin/init': No such file or directory
run-init: can't execute '/bin/sh': No such file or directory
Kernel panic - not syncing: Attempted to kill init
Full `lb build` was unaffected (live-build copies includes.chroot with
cp -a, which follows the /lib symlink), so only fast-path ISOs were bad.
Three fixes:
- build.sh: stage GSP firmware under usr/lib/firmware/nvidia/<ver>/, the
canonical merged-usr path. Changing the overlay path also makes
overlay_paths_were_removed() force one full build on the next run.
- fast-path.sh: add `rsync --keep-dirlinks` so a real directory in the
overlay stage can never again replace a symlink-to-directory in the
root. Verified: firmware lands in usr/lib/firmware, /lib stays a symlink.
- iso-validation.sh: new validate_iso_rootfs_layout, run for every variant
on both build paths. Fails the build if the squashfs has a plain-dir
/bin|/sbin|/lib|/lib64, or if /usr/sbin/init is present without a
resolvable /usr/lib/systemd/systemd and /lib symlink. Verified it flags
the broken v14.02-1-g642e686 ISO and passes a correct layout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H2LLuid8PFhBqBPcxXxkQU
On hosts where the boot medium is BMC/IPMI virtual media, the ~2.8 GB
squashfs read often drops mid-transfer: rsync dies with "Input/output
error (5)", every file "has vanished", and live-boot moves the partial
RAM copy over the medium, ending the boot with "No supported filesystem
images found at /live".
Add hook 9013-toram-retry, which patches live-boot's 9990-toram-todisk.sh
so the whole-medium copy runs in a retry loop: up to 8 attempts with a
geometrically growing pause (15s -> 900s cap), unmounting/waiting/remounting
the medium between tries and honouring rsync's exit code (upstream ignores
it). rsync resumes from where it stopped, so a retry needing only the tail
finishes fast.
Verified against live-boot 1:20230131+deb12u1: substitution applies,
sh -n is clean, hook is idempotent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018JA6Cj84yxUfk8N3ppw6mr
needs_full_build() decided the fast path was safe by hashing only config
inputs (VERSIONS, package lists, hooks, archives, auto/config, Dockerfile).
It never hashed the build logic itself, so a change to build.sh or
lib/fast-path.sh counted as a "light" file and the next build silently
reused a squashfs produced by the old code. The merged-usr /lib fix in the
previous commit only forced a full rebuild by accident (the overlay
manifest saw firmware move from lib/ to usr/lib/).
hash_heavy_config now covers every *.sh under iso/builder (build.sh, the
build-*.sh helpers, lib/*.sh) plus auto/. A change to build logic is now
as heavy as a package-list change. config/bootloaders stays excluded (the
fast path regenerates the outer ISO layer from it every time). Hash is
deterministic (LC_ALL=C sort).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H2LLuid8PFhBqBPcxXxkQU
The techdump — the only place the IPMI FRU / Elabel identity (Product Name
= server model, serials, part numbers) is recorded — only ran under LiveCD.
Run it on every audit so `bee audit` on an installed host captures it too.
Also add `ipmitool mc info` and `ipmitool lan print` to the techdump, and
pin `ipmitool fru print 0` to FRU device 0 to match the Export Tools
"FRU / Elabel" editor exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rework the /topo diagram from side-by-side stacked cards into one tall
vertical bar per CPU socket with everything attached to it branching off
sideways (socket 0 left/branches right, socket 1 right/branches left).
Disks are now parented under the storage controller they physically hang
off (SATA/AHCI, SAS HBA, RAID) — itself a NUMA-affine PCIe device under one
socket — instead of a synthetic catch-all node. The disk->controller link
is read from a new storage-controllers.txt techdump
(platform.StorageControllerMapScript, a /sys/block walk); disks with no
resolvable controller fall back to an "Other" bar. No board/root node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYjrG6bVmeG1Z2Wmc8kg3o
The bundle README is the bootstrap for an agent that knows nothing about
the layout; it listed neither the pcie-link (whole-bus link-retrain) nor
the tpm (read-only) SAT target, so a reader had no pointer to
tasks/bee-sat/pcie-link-* / tpm-* or their summary keys. Add a Fastest-
answers entry for each, refresh the component-status key examples, and add
a Signatures line for the bridge-side link-retrain false positive (bridge
DEGRADED at before==after==max while its downstream endpoint reads OK =
link at the device's native max, not a fault).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mx6AvuXcJNT7jx3jjWw4hc
The downstream-capability adjustment lowered f.MaxSpeed/f.MaxWidth to the
rate the endpoint supports, but the Degraded verdict still compared the
retrained speed against the stale local maxSpeed holding the bridge's own
uncapped capability. A Gen5 root port feeding a Gen4 HBA (or a Gen2 NIC)
was flagged DEGRADED even though the link ran at the fastest rate the pair
supports, failing the PCIe link check on healthy hardware.
Keep the local maxSpeed in sync with the peer-capped f.MaxSpeed so both
the pre-retrain and post-retrain verdicts use the real target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mx6AvuXcJNT7jx3jjWw4hc
pcie-link's retrain check compared post-retrain speed against a port's
max capability without checking whether anything was actually seated
downstream. A switch downstream port with no card plugged in retrains
to zero lanes but still reports a reset-state speed, so it was
misreported as "retrained to Gen1, capable of Gen5" — indistinguishable
from a real degraded link.
Now a port that was already empty before the retrain is skipped (same
treatment as a disabled device: legitimately unpopulated slots are
common and shouldn't fail the check), while a device that answered
before the retrain and disappeared right after it still fails, with an
honest "no device detected downstream" message instead of a fabricated
speed reading.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
mirror.mephi.ru was timing out during lb binary_rootfs. Default to
deb.debian.org (Fastly-backed, geo-balanced) and allow overriding via
--mirror on build-in-container.sh/build.sh, or BEE_MIRROR in auto/config.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pcie-nvidia-link.txt, pcie-nvidia-link-under-load.txt, and
kernel-aer-nvidia.txt only ever lived in supportBundleCommands, which
exclusively runs inside the on-demand "Download Support Bundle" web UI
action. The blackbox USB auto-sync worker never calls that function -
it only mirrors whatever CaptureTechnicalDump already wrote into the
live export tree at boot. So these three files were structurally
unreachable from a blackbox pull no matter how fresh the build was;
earlier analysis of a real blackbox misattributed their absence to
build/version drift instead.
Move the underlying scripts into shared exported constants
(platform.PCIeNvidiaLinkScript, PCIeNvidiaLinkUnderLoadScript,
KernelAERNvidiaScript) and add them to techDumpNvidiaCommands, so
CaptureTechnicalDump captures all three once at boot (bee-audit is
oneshot, so the ~8s bee-gpu-burn cost for the under-load sample is a
one-time boot cost, not a per-sync-cycle one) alongside the existing
nvidia-smi-* dumps. supportBundleCommands still re-runs the same
scripts on demand for a fresher sample - that's intentional, not a
duplicate to clean up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power
saving, and applyPCIeLinkSpeedWarning fired on every idle collector
pass regardless - since component-status DB records never downgrade
(Record() only ever raises severity), one boot-time idle sample
permanently pinned pcie:gpu:nvidia to Warning for the rest of the
session even after every load-bearing GPU SAT test passed clean. Two
prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag)
didn't hold up against this hardware/driver combination - see
bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the
full history.
Rather than add a downgrade path, stop writing an unverified status in
the first place: parseLspciDevice no longer calls
applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay
populated as plain descriptive fields; only a verified-under-load
caller may now turn them into a status verdict.
Two new SAT targets provide that verified signal:
- pcie-link (platform/pcie_link_check.go): forces every enabled PCIe
device - not just GPUs - to retrain via the PCIe spec's Link Control
"Retrain Link" bit, then compares the negotiated speed against the
device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn
equivalent load tool. Classifies by PCI class code + vendor ID, not
name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into
their own component-status keys so a degraded NIC never reads as a
GPU fault.
- nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives
real host<->device traffic via dcgmi diag -r nvbandwidth and
resamples link speed immediately after, independent of nvbandwidth's
own pass/fail.
Both wired into the task queue/webui the same way as nvidia-config
(routes, dispatch, priority, Validate page cards, Run All Check SAT).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`ipmitool fru edit` frequently exits non-zero after a field resize
(shrinking/growing a string shifts every later offset in the FRU
record) even though the write itself landed correctly on the BMC -
the failure is in ipmitool's own post-write re-parse, not the write.
bee was treating that exit code as authoritative and marking the task
failed even when the field was actually written.
runIPMIFRUWriteTask now re-reads the FRU on a non-zero exit and
compares the target field's actual value before deciding the task
failed, logging which path it took either way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A CPU that had thermally throttled (sysfs thermal_throttle counter >
0) still reported status "OK" everywhere: dmidecode-derived CPU status
only distinguishes populated/enabled/disabled and never looked at the
throttle flag the collector already recorded next to it, and neither
SAT path meant to catch this actually could:
- The routine "cpu" SAT pack (RunCPUAcceptancePack) only checked
lscpu/sensors/stress-ng exit codes — stress-ng exits 0 whether or
not the CPU throttled while running it, so an 89°C/throttled CPU
right after a "successful" run still showed cpu:all as OK in
component-status.json.
- The more thorough platform-stress test already detected throttling
and fan-spindown correctly, but wrote its verdict as "Overall: FAIL
— ..." with no "=", which parseSATKV can't parse — so even a real
detected throttle event never reached the component-status DB.
Fixes:
- cpu_telemetry.go: escalate a CPU's status to Warning (only-escalate,
same severity ranking already used elsewhere) when Throttled is set.
- sat.go: add a before/after thermal-throttle-counter check job around
the "cpu" pack's stress-ng run, so a throttle event during the run
fails that job and (via the existing FAILED->Warning DB mapping)
flips cpu:all to Warning.
- platform_stress.go: emit a machine-readable overall_status= line
alongside the human-readable verdict so platform-stress results
actually reach ApplySATResultToDB.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dmesgSeverity classified a kernel log line as Critical via plain
strings.Contains, without word boundaries, so common words that merely
contain a keyword as a substring false-positive — "disabled by
default" was flagged Critical because "default" contains "fault"
(de-fault). Found in a support bundle where 140/238 event_logs entries
came back Critical, most of them harmless boot messages (module load
notices, "... is initialized", "disabled by default"), drowning out
genuinely critical entries (Xid, AER, ECC) in the same list.
Switch to the same \b-bounded regexes already used for the capture
patterns above it in this file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
An NVMe SSD passed through a RAID/HBA as /dev/sdX (not /dev/nvmeX) gets
smartctl'd over the SCSI protocol, which reports its model in
scsi_model_name/scsi_vendor/scsi_product instead of the ATA/NVMe
model_name field. smartctlInfo only mapped model_name, so the drive's
model was silently dropped from reanimator.json even though smartctl
ran successfully and the serial number came through fine — caught via
a support bundle where lsblk showed "SSSTC CA6-8D1024" for a drive but
reanimator.json's storage entry had no model at all.
Add scsi_model_name/scsi_vendor/scsi_product to smartctlInfo and fall
back to them, in that order, when model_name is empty.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A wedged GPU (fallen off the bus, needs a physical power-cycle) makes
nvidia-smi/nvidia-bug-report.sh/dcgmi hang indefinitely, so building a
support bundle on such a host never finished and the "Download Support
Bundle" button spun on "Building..." forever with no way to recover
short of restarting the service.
Give every subprocess this file shells out to a bounded context.
nvidia-bug-report.sh gets a longer timeout since it legitimately takes
tens of seconds on multi-GPU boxes even when healthy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
statusSeverity was byte-for-byte duplicated in collector/contract.go and
app/sat_overlay.go (same switch, same comment). Export it as
collector.StatusSeverity and drop the app-package copy.
page_topo.go inlined the same GPU/NIC/RAID class switch as its own
pcieDeviceKind function in one spot — call the function instead.
page_validate.go had its own copy of pages.go's firstNonEmpty (all call
sites already pre-trim their inputs via validateTrimPtr, so the extra
TrimSpace was redundant) — drop the duplicate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
matchesGPUVendor (sat_overlay.go) matched any PCIe DeviceClass containing
the substring "Controller" instead of the actual GPU classes, so a
same-vendor non-GPU device (e.g. an NVIDIA-branded NIC/storage controller)
with a degraded PCIe link surfaced as a pcie:gpu:<vendor> alarm on the
Hardware Summary/webui.
isGPUDevice (app_format.go) had the same bug via a different path: an
"any NVIDIA-vendor device is a GPU" fallback that could inflate the
"GPU: N x <model>" audit summary line with non-GPU companion devices
(NVSwitch/NVLink bridges, etc).
Both now defer to collector.IsGPUClass (exported from the previously
unexported isGPUClass), the canonical exact-match classifier already used
by amdgpu.go — removing the duplicated/incorrect class checks instead of
reimplementing them. Also dedup two more locally-duplicated classifiers
within webui (which doesn't import collector by convention): NIC-class
matching (pages.go/page_topo.go) and GPU-class matching in
page_validate.go now share one copy per package instead of being
reimplemented per file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
buildNvidiaTaskSet only treated ParallelGPUs as a reason to keep all
selected GPUs in one task, so ramp-up (StaggerGPUStart) fell through
to per-model task splitting. When that split left a single GPU in a
task, resolveNvidiaRampPlan silently returned StaggerSeconds=0,
making the ramp-up toggle a no-op.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pcie:gpu:<vendor> in the component-status DB (what the Hardware
Summary/webui "check passed" status reads) was only ever written by SAT
diag results, and none of the nvidia/nvidia-config/nvidia-interconnect/
nvidia-bandwidth SAT jobs check PCIe link speed. So a real Gen1/Gen4
degradation the collector already flagged as Warning in the hardware
snapshot never reached the DB-backed status, and the audit kept
reporting "OK" despite GPUs training at Gen1 with BMC showing x16.
Add writePCIeGPUStatusesToDB, mirroring the existing PSU write-through,
so RunAudit pushes the collector's PCIe status into the DB alongside
SAT results.
Also add export/gpu/pcie-nvidia-link-under-load.txt to the support
bundle: NVIDIA drivers deliberately downclock PCIe at idle to save
power and re-train to full speed under load, so an idle Gen1 reading
alone can't distinguish real hardware/riser degradation from normal
power management. Resample the same sysfs link attributes while
bee-gpu-burn is actively loading the GPUs so both bundles ship
together.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds POST /api/system/time-sync so the dashboard's "Sync with this
browser" button can set the host's date, time, and IANA timezone from
the client's browser — useful for appliances with no NTP/network
access to keep their own clock straight. Disables NTP before writing
the manual time so it doesn't get immediately overwritten.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nvidia-bug-report.sh appends .gz to --output-file when gzip is available,
which silently produced empty nvidia-bug-report.txt in support bundles
(cat looked for the uncompressed name that never existed).
Also: a GPU that falls off the PCIe/NVLink bus (Xid 79) or gets flagged
for Node Reboot Required (Xid 154) mid-SAT-run left every downstream test
failing with generic, unrelated-looking errors (CUDA "unknown error",
"unable to determine device handle") with no indication the GPU needed a
physical power-cycle to recover. Detect these codes from SAT run logs and
surface a plain-English "physical reboot required" message in the task's
failure detail, the persisted component-status DB, a dashboard banner on
the Hardware Summary card, and topology diagram GPU-node severity.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
enrichPCIeWithNVIDIAData unconditionally overwrote dev.Status after
collectPCIe() had already flagged a Warning/Critical (e.g. PCIe link
speed degraded), so a clean ECC/remap/reset readout silently downgraded
that finding back to OK while leaving the stale ErrorDescription behind.
Add a severity-ordered merge (OK/Unknown < Warning < Critical) shared
via mergeDeviceStatus in contract.go, and route both the NVIDIA status
calculation and the driver-unavailable fallback through it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The NVSwitch-detection fix (983f41a) was necessary but not sufficient on
HGX B200 (Kaytus KR9288-X3): ExecCondition now correctly runs fabricmanager,
but nvidia-fabricmanager-start.sh then aborts with "Kernel module ib_umad
has not been loaded" on this NVL5+ board, leaving the fabric stuck in
"In Progress" and cascading into failed dcgmi diag/NVBandwidth/NCCL/Validate
GPU SAT tasks (confirmed via support bundle 20260729-165331).
Neither nvidia-fabricmanager nor DCGM declare the NVLink5 support packages
as apt dependencies (checked against the cuda-repo Packages index directly):
ib_umad must be modprobed before FM starts, nvlsm (InfiniBand Subnet Manager)
is launched internally by FM's own start script once present, and DCGM
dlopens libnvidia-nscq at runtime for NVSwitch health monitoring.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
On HGX B200 (Kaytus KR9288-X3) NVSwitch is managed via HMC/ERoT and never
appears on host lspci as 10de:0680, so the old check always returned
"not present" and silently skipped fabricmanager — leaving the fabric
untrained and CUDA/NCCL/dcgmi failing with "system not yet initialized"
on otherwise-healthy hardware. bee-check-nvswitch now checks, in order,
nvidia-smi's GPU Fabric GUID, /proc/driver/nvidia-nvswitch/devices, and
finally the legacy host-PCIe scan for older HGX generations.
- page_scenario.go: onclick built via JSON.stringify() embedded raw double
quotes inside a double-quoted HTML attribute, truncating the attribute so
the click handler never compiled; pass the name through an escaped
data-scenario-name attribute instead.
- build.sh: overlay staging rsyncs (OVERLAY_DIR->stage, stage->includes.chroot)
ran without --delete, so a scenario removed from the repo (a9924b0) stayed
baked into every ISO built from the persistent stage cache since — the
"second script" in the Scenario page's list.
- blackbox: rewritten around a deterministic local zip + incremental
patch-the-changed-suffix onto removable media, instead of walking/copying
~90 files through a synchronous ntfs-3g FUSE mount every cycle. journalctl
captures are now "--since last sync" (were "--since boot", growing with
uptime) and metrics.db is excluded (was copied whole every cycle).
- scenario: nvbandwidth-acs-ab now escalates GPU count (same-socket pair,
other socket's pair, one cross-socket pair, all GPUs) under each ACS state
instead of always running all 6 GPUs at once, using a new `bee
gpu-bandwidth-groups` subcommand that discovers socket layout from
`nvidia-smi topo -m` at runtime — gpu_indices is host-specific, so this
can't be baked into the scenario file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bee.nvidia.mode=gsp-off loads the driver with NVreg_EnableGpuFirmware=0 and
skips nvidia-uvm/nvidia-modeset, so CUDA (nvbandwidth/dcgm/nccl) can't run —
and on Hopper H200 GSP is effectively mandatory anyway, so the mode is dead
weight in this test image. Remove it from every boot entry. fail-safe is now
bee.nvidia.mode=normal pci=realloc iommu=pt, so the GPUs come up with uvm and
the GPU test actually runs under iommu=pt (the point of the entry). All four
sources (grub.cfg + live.cfg.in templates, build.sh write_canonical_*) match.
The internal GSP auto-fallback in bee-nvidia-load (normal mode, GSP init
hangs >90s → retry with EnableGpuFirmware=0, then still load uvm) is left
intact — it serves converted SXM->PCIe cards on other test targets and keeps
CUDA working, unlike the removed explicit menu mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
no-GUI is now just the normal entry minus the GUI: bee.gui=off with
bee.nvidia.mode=normal (drops the previous gsp-off/pci=realloc it had picked
up). fail-safe is now simply the GPU/PCIe-workaround boot —
bee.nvidia.mode=gsp-off pci=realloc iommu=pt over the standard base — instead
of the old noapic/nodma/nomce/nolapic/nosmp minimal mode, so it can actually
run GPU tests (nvbandwidth) to verify the iommu=pt half of the vendor fix.
Applied identically across all four sources (grub.cfg + live.cfg.in templates
and build.sh write_canonical_* generators).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The GRUB and isolinux menus (and their two sources of truth — the staged
templates and build.sh's write_canonical_* generators that rewrite the built
menu) had drifted: GRUB shipped a lean set, isolinux carried extra
display-debug entries (gsp-off, kms, kms-gsp-off) and lacked a WIPE entry in
the canonical writer. Collapse all four to one set:
normal, load-to-RAM (default), no-GUI/troubleshooting (gsp-off, pci=realloc),
fail-safe, WIPE ALL DISKS, Memory Test, + UEFI Firmware Settings (EFI only)
The fail-safe entry gains iommu=pt (AMD-Vi passthrough) — the OS-side half of
the vendor fix for the MSI CG480-S6053 cross-socket nvbandwidth reboot; ACS
can't be toggled from a stock-kernel cmdline so that stays a BIOS setting.
default stays index 1 = toram (config.cfg set default=1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the old nvbandwidth-all-gpu-power-watch scenario (dropped: the
power/IPMI-sampler framing was a dead-end — the blackbox shows every other
load test, incl. NCCL all-reduce across all 6 GPUs, passes with power fully
connected; only nvbandwidth on cross-socket GPU P2P reboots the box).
nvbandwidth-acs-ab is a two-phase reproduction for the MSI CG480-S6053 case.
Phase 1 disables PCIe ACS redirect at runtime via setpci (saving each
bridge's original ACS control word to /run/bee-acs-orig) then runs
nvbandwidth across all 6 GPUs — expected to survive. Phase 2 restores the
saved ACS values and repeats — expected to reproduce the reboot. Safe phase
first so its full blackbox reaches the USB stick before the risky phase can
reset the box (sync-bracket before the final job flushes phase-1 evidence).
ACS mutation runs via `bash -c` since scenario cmd is exec'd directly with no
shell. Both shipped copies (repo scenarios/ and image overlay) identical;
verified both parse via ParseScenarioJSON (4 jobs).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
