pcie-nvidia-link.txt, pcie-nvidia-link-under-load.txt, and
kernel-aer-nvidia.txt only ever lived in supportBundleCommands, which
exclusively runs inside the on-demand "Download Support Bundle" web UI
action. The blackbox USB auto-sync worker never calls that function -
it only mirrors whatever CaptureTechnicalDump already wrote into the
live export tree at boot. So these three files were structurally
unreachable from a blackbox pull no matter how fresh the build was;
earlier analysis of a real blackbox misattributed their absence to
build/version drift instead.
Move the underlying scripts into shared exported constants
(platform.PCIeNvidiaLinkScript, PCIeNvidiaLinkUnderLoadScript,
KernelAERNvidiaScript) and add them to techDumpNvidiaCommands, so
CaptureTechnicalDump captures all three once at boot (bee-audit is
oneshot, so the ~8s bee-gpu-burn cost for the under-load sample is a
one-time boot cost, not a per-sync-cycle one) alongside the existing
nvidia-smi-* dumps. supportBundleCommands still re-runs the same
scripts on demand for a fresher sample - that's intentional, not a
duplicate to clean up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power
saving, and applyPCIeLinkSpeedWarning fired on every idle collector
pass regardless - since component-status DB records never downgrade
(Record() only ever raises severity), one boot-time idle sample
permanently pinned pcie:gpu:nvidia to Warning for the rest of the
session even after every load-bearing GPU SAT test passed clean. Two
prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag)
didn't hold up against this hardware/driver combination - see
bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the
full history.
Rather than add a downgrade path, stop writing an unverified status in
the first place: parseLspciDevice no longer calls
applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay
populated as plain descriptive fields; only a verified-under-load
caller may now turn them into a status verdict.
Two new SAT targets provide that verified signal:
- pcie-link (platform/pcie_link_check.go): forces every enabled PCIe
device - not just GPUs - to retrain via the PCIe spec's Link Control
"Retrain Link" bit, then compares the negotiated speed against the
device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn
equivalent load tool. Classifies by PCI class code + vendor ID, not
name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into
their own component-status keys so a degraded NIC never reads as a
GPU fault.
- nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives
real host<->device traffic via dcgmi diag -r nvbandwidth and
resamples link speed immediately after, independent of nvbandwidth's
own pass/fail.
Both wired into the task queue/webui the same way as nvidia-config
(routes, dispatch, priority, Validate page cards, Run All Check SAT).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`ipmitool fru edit` frequently exits non-zero after a field resize
(shrinking/growing a string shifts every later offset in the FRU
record) even though the write itself landed correctly on the BMC -
the failure is in ipmitool's own post-write re-parse, not the write.
bee was treating that exit code as authoritative and marking the task
failed even when the field was actually written.
runIPMIFRUWriteTask now re-reads the FRU on a non-zero exit and
compares the target field's actual value before deciding the task
failed, logging which path it took either way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A CPU that had thermally throttled (sysfs thermal_throttle counter >
0) still reported status "OK" everywhere: dmidecode-derived CPU status
only distinguishes populated/enabled/disabled and never looked at the
throttle flag the collector already recorded next to it, and neither
SAT path meant to catch this actually could:
- The routine "cpu" SAT pack (RunCPUAcceptancePack) only checked
lscpu/sensors/stress-ng exit codes — stress-ng exits 0 whether or
not the CPU throttled while running it, so an 89°C/throttled CPU
right after a "successful" run still showed cpu:all as OK in
component-status.json.
- The more thorough platform-stress test already detected throttling
and fan-spindown correctly, but wrote its verdict as "Overall: FAIL
— ..." with no "=", which parseSATKV can't parse — so even a real
detected throttle event never reached the component-status DB.
Fixes:
- cpu_telemetry.go: escalate a CPU's status to Warning (only-escalate,
same severity ranking already used elsewhere) when Throttled is set.
- sat.go: add a before/after thermal-throttle-counter check job around
the "cpu" pack's stress-ng run, so a throttle event during the run
fails that job and (via the existing FAILED->Warning DB mapping)
flips cpu:all to Warning.
- platform_stress.go: emit a machine-readable overall_status= line
alongside the human-readable verdict so platform-stress results
actually reach ApplySATResultToDB.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dmesgSeverity classified a kernel log line as Critical via plain
strings.Contains, without word boundaries, so common words that merely
contain a keyword as a substring false-positive — "disabled by
default" was flagged Critical because "default" contains "fault"
(de-fault). Found in a support bundle where 140/238 event_logs entries
came back Critical, most of them harmless boot messages (module load
notices, "... is initialized", "disabled by default"), drowning out
genuinely critical entries (Xid, AER, ECC) in the same list.
Switch to the same \b-bounded regexes already used for the capture
patterns above it in this file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
An NVMe SSD passed through a RAID/HBA as /dev/sdX (not /dev/nvmeX) gets
smartctl'd over the SCSI protocol, which reports its model in
scsi_model_name/scsi_vendor/scsi_product instead of the ATA/NVMe
model_name field. smartctlInfo only mapped model_name, so the drive's
model was silently dropped from reanimator.json even though smartctl
ran successfully and the serial number came through fine — caught via
a support bundle where lsblk showed "SSSTC CA6-8D1024" for a drive but
reanimator.json's storage entry had no model at all.
Add scsi_model_name/scsi_vendor/scsi_product to smartctlInfo and fall
back to them, in that order, when model_name is empty.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A wedged GPU (fallen off the bus, needs a physical power-cycle) makes
nvidia-smi/nvidia-bug-report.sh/dcgmi hang indefinitely, so building a
support bundle on such a host never finished and the "Download Support
Bundle" button spun on "Building..." forever with no way to recover
short of restarting the service.
Give every subprocess this file shells out to a bounded context.
nvidia-bug-report.sh gets a longer timeout since it legitimately takes
tens of seconds on multi-GPU boxes even when healthy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
statusSeverity was byte-for-byte duplicated in collector/contract.go and
app/sat_overlay.go (same switch, same comment). Export it as
collector.StatusSeverity and drop the app-package copy.
page_topo.go inlined the same GPU/NIC/RAID class switch as its own
pcieDeviceKind function in one spot — call the function instead.
page_validate.go had its own copy of pages.go's firstNonEmpty (all call
sites already pre-trim their inputs via validateTrimPtr, so the extra
TrimSpace was redundant) — drop the duplicate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
matchesGPUVendor (sat_overlay.go) matched any PCIe DeviceClass containing
the substring "Controller" instead of the actual GPU classes, so a
same-vendor non-GPU device (e.g. an NVIDIA-branded NIC/storage controller)
with a degraded PCIe link surfaced as a pcie:gpu:<vendor> alarm on the
Hardware Summary/webui.
isGPUDevice (app_format.go) had the same bug via a different path: an
"any NVIDIA-vendor device is a GPU" fallback that could inflate the
"GPU: N x <model>" audit summary line with non-GPU companion devices
(NVSwitch/NVLink bridges, etc).
Both now defer to collector.IsGPUClass (exported from the previously
unexported isGPUClass), the canonical exact-match classifier already used
by amdgpu.go — removing the duplicated/incorrect class checks instead of
reimplementing them. Also dedup two more locally-duplicated classifiers
within webui (which doesn't import collector by convention): NIC-class
matching (pages.go/page_topo.go) and GPU-class matching in
page_validate.go now share one copy per package instead of being
reimplemented per file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
buildNvidiaTaskSet only treated ParallelGPUs as a reason to keep all
selected GPUs in one task, so ramp-up (StaggerGPUStart) fell through
to per-model task splitting. When that split left a single GPU in a
task, resolveNvidiaRampPlan silently returned StaggerSeconds=0,
making the ramp-up toggle a no-op.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pcie:gpu:<vendor> in the component-status DB (what the Hardware
Summary/webui "check passed" status reads) was only ever written by SAT
diag results, and none of the nvidia/nvidia-config/nvidia-interconnect/
nvidia-bandwidth SAT jobs check PCIe link speed. So a real Gen1/Gen4
degradation the collector already flagged as Warning in the hardware
snapshot never reached the DB-backed status, and the audit kept
reporting "OK" despite GPUs training at Gen1 with BMC showing x16.
Add writePCIeGPUStatusesToDB, mirroring the existing PSU write-through,
so RunAudit pushes the collector's PCIe status into the DB alongside
SAT results.
Also add export/gpu/pcie-nvidia-link-under-load.txt to the support
bundle: NVIDIA drivers deliberately downclock PCIe at idle to save
power and re-train to full speed under load, so an idle Gen1 reading
alone can't distinguish real hardware/riser degradation from normal
power management. Resample the same sysfs link attributes while
bee-gpu-burn is actively loading the GPUs so both bundles ship
together.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds POST /api/system/time-sync so the dashboard's "Sync with this
browser" button can set the host's date, time, and IANA timezone from
the client's browser — useful for appliances with no NTP/network
access to keep their own clock straight. Disables NTP before writing
the manual time so it doesn't get immediately overwritten.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nvidia-bug-report.sh appends .gz to --output-file when gzip is available,
which silently produced empty nvidia-bug-report.txt in support bundles
(cat looked for the uncompressed name that never existed).
Also: a GPU that falls off the PCIe/NVLink bus (Xid 79) or gets flagged
for Node Reboot Required (Xid 154) mid-SAT-run left every downstream test
failing with generic, unrelated-looking errors (CUDA "unknown error",
"unable to determine device handle") with no indication the GPU needed a
physical power-cycle to recover. Detect these codes from SAT run logs and
surface a plain-English "physical reboot required" message in the task's
failure detail, the persisted component-status DB, a dashboard banner on
the Hardware Summary card, and topology diagram GPU-node severity.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
enrichPCIeWithNVIDIAData unconditionally overwrote dev.Status after
collectPCIe() had already flagged a Warning/Critical (e.g. PCIe link
speed degraded), so a clean ECC/remap/reset readout silently downgraded
that finding back to OK while leaving the stale ErrorDescription behind.
Add a severity-ordered merge (OK/Unknown < Warning < Critical) shared
via mergeDeviceStatus in contract.go, and route both the NVIDIA status
calculation and the driver-unavailable fallback through it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The NVSwitch-detection fix (983f41a) was necessary but not sufficient on
HGX B200 (Kaytus KR9288-X3): ExecCondition now correctly runs fabricmanager,
but nvidia-fabricmanager-start.sh then aborts with "Kernel module ib_umad
has not been loaded" on this NVL5+ board, leaving the fabric stuck in
"In Progress" and cascading into failed dcgmi diag/NVBandwidth/NCCL/Validate
GPU SAT tasks (confirmed via support bundle 20260729-165331).
Neither nvidia-fabricmanager nor DCGM declare the NVLink5 support packages
as apt dependencies (checked against the cuda-repo Packages index directly):
ib_umad must be modprobed before FM starts, nvlsm (InfiniBand Subnet Manager)
is launched internally by FM's own start script once present, and DCGM
dlopens libnvidia-nscq at runtime for NVSwitch health monitoring.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ScenarioSpec gains an optional "description" field. Listing (both
ListLocalScenarioFiles and ListScenarioFilesOnRemovableMedia, via the new
scenarioDescription helper) reads it out of each file without requiring
full ParseScenarioJSON validation to succeed, so a listing never hides a
scenario over an unrelated validation issue. The webui Scenario page now
renders Name/Description/Found-on/Run instead of just Name/Found-on — a
bare filename rarely tells anyone but the author what a scenario actually
does.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
platform.LocalScenariosDir (/usr/share/bee/scenarios, populated from
iso/overlay/usr/share/bee/scenarios/ by build.sh's overlay rsync) is now
checked before removable media for both `bee run <name>` and the "6.
Scenario" page — a scenario shipped with the image works with no USB
stick required. ReadScenario/ListAvailableScenarios merge local + USB;
the removable-media-only functions from the previous commit are kept
as-is (still used directly where that's actually what's wanted) rather
than renamed out from under existing callers/tests.
iso/overlay/usr/share/bee/scenarios/nvbandwidth-all-gpu-power-watch.json
is a copy of scenarios/nvbandwidth-all-gpu-power-watch.json — the two
aren't auto-synced (documented in scenarios/README.md), so shipping a
scenario baked-in means checking it into both places.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Exposes the scenario engine (platform.System.RunScenario, added earlier)
in the web UI instead of only the `bee run` CLI: a new nav item lists every
scenarios/*.json found on mounted removable media (GET /api/scenario/list)
and runs one with a click (POST /api/scenario/run), enqueued as a normal
Task with target "scenario" — progress/logs live in Tasks like any other
SAT pack, no separate live-output UI needed.
- app.go: satRunner gains RunScenario, exportManager gains
ListScenarioFilesOnRemovableMedia/ReadScenarioFromRemovableMedia — both
already implemented on platform.System, just newly exposed through App.
- webui/tasks.go: taskParams.ScenarioName; runTask's "scenario" case reads
the file from removable media, parses it, and runs it.
- webui/page_scenario.go: the page itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Investigating the CG480-S6053 reboot needed a way to run an ad-hoc load
(nvbandwidth across a specific GPU set) while sampling IPMI/nvidia-smi
telemetry in the background — without hardcoding a one-off test into the
SAT pack code for a single investigation.
- audit/internal/platform/scenario.go: ScenarioSpec/ScenarioJob (JSON,
no new dependency) + System.RunScenario. "command" jobs run sequential
or parallel (per-job "parallel" flag); "sampler" jobs run concurrently
in the background on their own interval until every command job
finishes or the scenario's timeout elapses. "{{gpus}}" in a command's
cmd is substituted from that job's gpu_indices. Command jobs are wired
through the same satJobBoundaryHook/satSyncBracketHook seams the SAT
job runner uses, so a scenario run gets the same durability treatment
(evidence that a risky command started/finished reaches blackbox before
a possible crash, not just whatever streamed to the RAM-backed export
dir).
- export.go: ReadScenarioFromRemovableMedia mounts each removable target
looking for scenarios/<name>.json — an air-gapped engineer can author a
scenario elsewhere, drop it under scenarios/ on the same USB stick
already plugged in for blackbox, and run it with no network path onto
the host.
- cmd/bee: new `bee run <file.json|name>` (bare name = looked up on
removable media); `bee scenario run <arg>` kept as a longer alias.
- scenarios/nvbandwidth-all-gpu-power-watch.json: the scenario that
reproduced the actual reboot (full nvbandwidth across all GPUs, which
crashed, vs. clean per-socket passes), with IPMI sensor + GPU power/temp
sampling for a power-delivery correlation check.
Also: webui/page_topo.go — the /topo page's component-status-detail modal
(GET /api/component-detail/{type}) showed "No status data recorded yet"
for any component type ComponentStatusDB has no history for yet (e.g. GPU
before a SAT run this boot), even though the topology card for the same
component already showed "N OK" from the audit inventory snapshot.
inventoryFallbackRecords now synthesizes records from that same inventory
snapshot when StatusDB is empty, using the same device classifiers
(isGPUDeviceClass etc.) and severity mapping (classifyTopoSeverity) the
topology card itself uses, so the two views never disagree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
copyPath/copyPathFiltered treated any os.Stat/os.ReadDir/os.Open error on a
source entry as fatal, aborting the entire tree copy. A source path is read
from the live export dir while bee's own task runner concurrently renames
task directories (e.g. "_pending" -> "_done") — an entry present in the
parent's os.ReadDir a moment ago disappearing by the time it's individually
Stat'd/Open'd is an expected race, not a real failure.
Seen on a real crash bundle: blackbox got stuck in status "degraded" from
early in the run (first hit during the CPU pack, well before the GPU tests)
after exactly this race, and every syncBracket wait then timed out for the
rest of the run — the discovery/wait plumbing from the previous fix works,
but had nothing working under it to wait on. The target also accumulated
stale "_pending" copies alongside "_done" ones with no cleanup, though
fixing that dedup is left for a follow-up.
os.IsNotExist(err) now skips the vanished entry instead of propagating.
Added regression tests simulating the race directly (copy_path_race_test.go).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the flat export/ + techdump/ + system/ + systemd/ layout (two
same-named-but-different techdump/ dirs, disk reports duplicated between a
SAT run dir and techdump/) with one shared categorizeExportTree used by both
BuildSupportBundle and the blackbox USB mirror:
- export/{cpu,memory,storage,gpu,network,platform}/ — raw vendor-tool
output grouped by device type, plus export/reanimator.json ready to POST
to Reanimator's /ingest/hardware endpoint.
- status/ — computed diagnosis (component-status.json, runtime-health.*,
and metrics.db, previously missing from every bundle entirely).
- tasks/ — bee's own task-run bookkeeping (bee-sat/, bee-bench/, task
reports, service logs) where duplication with export/ is expected.
- livecd/{gui,host}/ — live-boot/kiosk-session-only diagnostics, kept out
of the hardware-facing tree.
Drops the redundant disk-report mirror write in RunStorageAcceptancePack
and the now-dead syncDirectoryTree/removeMissingPaths; adds a size+mtime
skip in copyPath so the blackbox mirror doesn't rewrite unchanged files
every cycle. README.md rewritten to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- gpuBandwidthSocketGroups: a single GPU whose NUMA node fails to resolve
no longer collapses the whole per-socket nvbandwidth split into one
fallback pass — it now folds into the last resolved group instead,
preserving isolation for the sockets that did resolve.
- blackbox discoverMarkedTargets: skip mounting/unmounting devices that
already have a running worker on every 2s discovery tick. This was
observed hammering the same USB target continuously (mount+unmount
every ~2s for the whole session) and contending with the worker's own
sync cycle, plausibly explaining multi-minute sync cycles seen on a
real crash bundle.
- syncFilesystem now calls syscall.Sync() directly instead of spawning
/bin/sync per copied file; blackbox mounts removable targets with
-o sync so writes are durable without relying on the app-level sync as
the primary mechanism.
- New platform.SetSyncBracketHook / satJob.syncBracket: blocks (with a
bounded timeout) on blackbox actually reaching removable media right
before and right after a diagnostic's real load step (nvbandwidth,
memtester, stress-ng, dcgmi diag, nccl, smartctl/nvme self-test...),
instead of only firing a fire-and-forget kick after the job's own log
file is written. A crash mid-load now has durable evidence the load
started, not just whatever streamed to the RAM-backed export dir before
blackbox's next scheduled cycle.
Found investigating a real support bundle where blackbox's last
successful sync (19:55:25) predated both the previous job finishing and
the crashing nvbandwidth job starting (19:56:57) — none of the crash
window ever reached durable media.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds BEE_HOSTNAME, derived with the same easy-bee-<variant>-v<version>
scheme already used for ISO_BASENAME/SQUASHFS_FILENAME/BEE_ISO_VOLUME, and
wires it into auto/config via lb config --hostname. Documented the shared
naming scheme as a runtime-flows.md invariant.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pci=realloc was forcing a full PCI resource reallocation on every default
and toram boot, added as a GPU BAR workaround for BIOS without Above 4G
Decoding. On a dense NVMe-backplane server it exhausted MMIO32 space and
silently dropped 4 of 6 NVMe controllers from the OS even though the BIOS
and lspci both saw them fine. It now only applies to the no-GUI/gsp-off
troubleshooting entry, alongside a bible-local ADR for the two prior
rounds of SAT/blackbox durability work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A crash mid-command (e.g. the nvbandwidth reboot) previously lost that
job's entire output: streamExecOutput only buffered stdout/stderr in
memory and the job's log file was written once, after the process
exited. It now also streams each line straight to that file as it
arrives, so whatever printed before a crash survives.
Root filesystem here is a tmpfs overlay (toram boot), so the only real
persistence boundary is blackbox's mirror to removable media, not the
local write itself. platform.SetJobBoundaryHook lets app wire a touch
of a small kick-file after each job's output is written; blackboxWorker
now polls that file's mtime alongside its normal adaptive timer and
syncs immediately on a kick instead of waiting out the current flush
period (up to 30s).
On multi-socket systems, run the NVIDIA bandwidth diagnostic once per
CPU socket before the all-GPU pass, so a crash confined to the all-GPU
run (with clean per-socket passes preceding it) isolates a cross-socket
peer-to-peer fault instead of leaving it conflated with a general
GPU/PCIe issue. Single-socket systems keep the original one-pass shape.
Also expand the support-bundle README with reference notes distilled
from a real analysis pass (BMC clock drift, "0/empty" tool output
meaning absent hardware rather than a fault, timestamp-matching before
assigning causality, and a normal-power-cycle SEL signature), plus a
step-by-step recipe for diagnosing an unexpected reboot/crash during a
specific test.
Memory used to render as one unattached row below the whole diagram
regardless of which socket it belonged to. schema.HardwareMemory has no
NUMANode field (unlike PCIe devices), so CPU affinity is instead read out of
the DIMM's own Locator string: either a CPU number encoded directly in it
("CPU0_DIMM_A1"), or — when the Locator has no CPU number of its own, e.g.
"DIMM000(A)" — a node number from Bank Locator ("_Node1_Channel0_Dimm0"),
read from the persisted dmidecode-type17.txt techdump the same way the
NVLink card already reads extra techdump for visualization only.
DIMMs that can't be attached to a column via either heuristic still fall
back to the old unattached "Memory" row so nothing silently disappears.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Storage acceptance runs already write disk-<prefix>-report.txt (per-drive
health + pseudographic resource bars) into the timestamped bee-sat run dir.
Also mirror it into techdump, a sibling of baseDir under the same export
dir, so a support bundle surfaces these reports alongside the rest of the
diagnostics without anyone having to dig into a specific bee-sat run dir.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every place that surfaced a FAILED SAT result — task error messages,
component-status.json detail, and the hardware snapshot's ErrorDescription/
StatusHistory — used to say only "SAT overall_status=FAILED (see
summary.txt)" or "<label> failed", forcing an engineer to go dig through the
run directory to find out what actually broke.
nvidia-config's summary.txt now carries a "warnings" field with the specific
GPU/NVLink finding. A new SATFailureDetail/satFailureDetailFromKV in
component_status_db.go reads that field, or falls back to naming whichever
generic SAT sub-job(s) reported non-OK/UNSUPPORTED status along with their
exit code. This feeds both the task-runner error message and the component
status DB. sat_overlay.go's satKeyStatus (which drives ErrorDescription on
the exported hardware snapshot) now does the same, with storage kept
per-device so one drive's rc doesn't get attributed to another's card.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
An agent handed a bundle path had no way to know its layout without
grepping through random logs first — confirmed by watching a separate
session read bee-nvidia.log before anything else on a real bundle. Embeds
a single README.md (bee-embed, internal/app/assets/) explaining what bee
is and giving direct answers to the questions someone analyzing a bundle
is most likely to ask (did the tests pass, what hardware is this, is a
service healthy, RAID/GPU/NVLink state, etc), written at the bundle root
by both BuildSupportBundle (support-bundle archive root, sibling of
manifest.txt) and blackboxWorker.syncCycle (removable-media boot-folder
root, which otherwise has no manifest.txt-equivalent pointing anywhere).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The standalone "confidential-computing" SAT target only ever checked CC
readiness, which most fleets never opt into (a NOT_READY verdict there
isn't a fault). Meanwhile DCGM diag never asserts GPU config compliance
(ECC/MIG/power-limit vs factory default) or NVLink topology (per NVIDIA's
own DGX BasePOD deployment guide, this needs a separate validation step)
— gaps confirmed against public DCGM docs and a real NV17-vs-expected-NV18
bonded pair found on a live bundle.
Repurposes the routine into "nvidia-config": reuses the existing
ListNvidiaGPUSettings() (already backing the GPU-settings page) to flag
ECC disabled, a MIG mode change stuck pending a reset/reboot, and a power
limit capped >5% below default; parses "nvidia-smi topo -m" bonded pairs
against "nvlink -s/-e" to flag any inactive lane or nonzero replay/
recovery/CRC counter on an otherwise-active bond. CC readiness is folded
in as one informational field (does not gate overall_status) rather than
a dedicated test. Reports under the same pcie:gpu:nvidia severity key as
every other nvidia-* SAT target instead of an isolated key, so a
config/NVLink FAILED result isn't invisible next to stress-test results.
Also fixes ApplySATResultToDB silently dropping any target with no
matching switch case (exactly what the old confidential-computing target
did) with a new coverage test enumerating every real SAT target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The long-lived bee-web process (writing PSU/kmsg watchdog records ~every
60s) and each short-lived "bee bee-worker" SAT-task subprocess each held
an independent in-memory copy of component-status.json. Whichever saved
last won outright, silently erasing whatever the other had just written —
e.g. a GPU SAT task's pcie:gpu:nvidia result vanishing the next time the
PSU watchdog ticked. ComponentStatusDB.Record now reloads on-disk state
(keyed by newer LastCheckedAt) before merging its own update.
Also stops re-logging identical repeat observations to History: the
ingest contract defines status_history as a transition log ("История
переходов статусов"), not a per-poll journal, but Record appended one
entry per call regardless — a continuously-polled PSU grew an unbounded
run of identical "still OK" entries. Now only appends when a source's
last recorded status for a key actually changes.
The PSU watchdog itself now backs off (60s -> doubling, capped at 30min)
while steady and resets to 60s the moment any PSU's status changes, cutting
ipmitool shellouts and file writes for a fleet that's been stable for a
while.
Also adds the missing "raid" case to the component-detail API (was
returning 404 for any RAID card's detail click on /topo).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PSU and firmware (BMC/BIOS) boxes were absolutely-positioned SVG rects in a
single fixed-width row with no wrap, so an arbitrary/larger count piled up
and overlapped once a board had more of either than fit in that row. Move
them to plain flex-wrap HTML below the diagram (Firmware row first, then
Power Supplies), which reflows naturally for any count. The main CPU/PCIe/
GPU diagram now scrolls horizontally (overflow-x:auto) instead of being
squashed to fit narrow viewports, matching the wide-table convention used
elsewhere in webui.
Also fixes a crash: renderTopoMainDiagram forced numCols to 1 for layout
purposes when a snapshot has zero CPUs, then unconditionally indexed
hw.CPUs[0], panicking the whole /topo page on any audit without CPU data.
Same-kind/same-column components (e.g. 4 GPUs in one NUMA node) now render
as one stacked card summarizing worst-case status plus a tally line ("3 OK,
1 Warning") instead of one box per component, and card severity coloring
uses real fill/stroke vars instead of HTML badge classes that don't apply
any style to SVG shapes.
nvidia-smi exposes reset_status.reset_required and remapped_rows.* only on
newer drivers for Ampere+ GPUs; queried via a separate exec call since an
unrecognized field name fails the whole --query-gpu command and would have
wiped out unrelated telemetry (temp/ECC/power) on older drivers otherwise.
Also refines Xid severity in both the ingest dmesg collector and the
always-on kmsg watcher: Xid 64 (row-remap InfoROM write failure) now
escalates to Critical instead of the generic warning every other Xid got,
and fixes the SAT-window flush path which previously hardcoded "Warning"
and ignored pattern severity entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The bundled storcli2 (008.0005.0000.0010, Feb 2023) reports zero
controllers on a live SAS3808-iMR system even though storcli64 and
lspci both see it fine (confirmed the PCI device is correctly bound to
megaraid_sas — mpi3mr doesn't even list it as a candidate driver, so
this isn't a driver-binding issue storcli2 could route around).
Swapping in Broadcom's newer 08.017.000.0008 release on the chance it
recognizes this controller/firmware combo where the older build
doesn't; storcli64's drive-listing path (already fixed to parse the
real per-slot JSON shape) remains the primary, confirmed-working
source regardless of whether this update changes storcli2's behavior
for this card.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
needs_full_build() compared mtimes of VERSIONS/package-lists/hooks against
a marker file, which is fragile: a failed build attempt never touches the
marker, and git checkouts/rsync can reorder mtimes relative to real edit
order. This let the fast-path silently reuse a stale squashfs (baked
against old package pins) with no error — root-caused after a driver/DCGM
pin change (590/DCGM 4.6.0 -> 580/DCGM 4.5.2) produced a ~290MB larger ISO
that only shrank back down with --clean-build.
Switch to a content hash (VERSIONS + package-lists + hooks + archives +
bootloaders + auto/config + Dockerfile) instead of mtimes, and add a real
version check in validate_iso_nvidia_runtime: extract dpkg status from the
squashfs and confirm the installed datacenter-gpu-manager-4-core version
matches DCGM_VERSION. The previous check only confirmed dcgmi/nv-hostengine
/dcgmproftester exist, which stays true across DCGM versions and can't
catch this class of staleness.
RAID Controller Management showed "No drives detected" for a live
SAS3808-iMR controller even though storcli64 clearly enumerates its
drives. Root cause: the "eall/sall show all J" drive-listing parser
(collector/raid.go and its webui/raid_mgmt.go duplicate) assumed
drives are reported as a single "Drive Information" array, but real
storcli64/storcli2 output nests each drive under its own dynamically
named key ("Drive /c0/e69/s0", paired with a "... - Detailed
Information" key) — confirmed against a live techdump/storcli64-drives.json
capture. The assumed shape was never actually produced by the tool, so
this affected every storcli64/storcli2 controller, not just Tri-Mode
ones (the storcli2 fallback added in b7f015c never got a chance to
mask it in practice, since storcli2 itself reports zero controllers on
this hardware — a separate, unrelated tool-side gap).
Both parsers now read "Response Data" as a raw key map and pull drives
from either shape, so older storcli output using the array form still
works alongside the real per-slot form.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
/topo blocked the HTTP request on live nvidia-smi calls with no timeout
(topo -m, nvlink -s/-e run on every page load), so a wedged driver hung
the page indefinitely. CaptureTechnicalDump now persists these dumps
once per audit cycle; the page reads them from techdump/ instead.
buildSocketIndex mapped NUMA node number to CPU by treating dmidecode's
Socket Designation (often 1-indexed, "CPU1"/"CPU2") as equal to the
NUMA node number (always 0-indexed) — GPUs/NICs on NUMA node 0 fell
into the "unknown" column, others attached to the wrong CPU box. Now
ranks CPUs by Socket value instead of assuming a shared numbering base.
Fixed a bug in ComponentStatusDB/applyComponentStatusDB where GPU SAT
results were keyed per-target ("pcie:gpu:nvidia-stress") instead of
per-vendor, which both broke cross-tier severity tracking (a later
clean "2. Check" run and an earlier failing "3. Load" run never
compared severities) and silently failed to match any real BDF, so the
DB overlay never reached the topology graph at all. GPU keys are now
normalized to vendor ("pcie:gpu:nvidia"/"pcie:gpu:amd"). Also skip
writing to the DB when a SAT task was aborted by the user (ctx
canceled), so a partial run can't stomp a previously recorded status.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers the sat.go fix from 2599d9c: a failing "dcgmi discovery -l"
preflight job must not flip the pack's overall status, and jobs marked
with retries should recover from a transient first-attempt failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a GPU Settings card to /tools for the settings that actually
persist on NVIDIA data-center GPUs: ECC mode, MIG mode, and
Confidential Computing mode (all stored in the GPU's inforom/firmware,
take effect after a GPU reset or reboot) plus power limit (does not
persist — reapplied on demand). Includes a one-click "Reset All to
Defaults" that restores factory settings across every visible GPU,
touching only whatever has actually drifted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dcgmi discovery -l is a preflight/metadata step ahead of the real DCGM
diag jobs; a transient failure racing nv-hostengine startup shouldn't
flip the whole pack's status, so it's now marked informational with a
couple of retries. Separately, bound the fabricmanager/nvidia-dcgm
systemctl restart/start calls in bee-nvidia-load with a timeout so a
wedged unit (e.g. fabric training stuck on a bad NVSwitch fabric)
can't hang bee-nvidia.service forever and block dcgm from ever
starting.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
storcli64 enumerates these controllers but reports zero drives; storcli2
is the tool Broadcom ships for Tri-Mode/MegaRAID8 hardware and uses a
compatible JSON schema for drive listing. Wires storcli2 into both the
collector (structured drive data) and the webui RAID Management page
(dedup so a controller isn't double-listed if storcli64 already sees it
with zero drives), plus techdump raw collection and the ISO vendor-tool
build step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New read-only visualization page: CPU sockets as anchor nodes, PCIe
devices (GPU/NIC/RAID) linked to their NUMA-affine socket with edge
color derived strictly from link_speed vs max_link_speed (not from the
device's own Status, which can also be overwritten by SAT-test results
on the same field), PSU/BMC as standalone boxes with no connecting
line, and a separate NVLink Topology card (live nvidia-smi topo -m /
nvlink -s/-e queries, not persisted to any contract).
GPU-GPU edges are drawn strictly from the actual bonded-pair list
parsed out of "nvidia-smi topo -m" (parseGPUPairAdjacency), not from
adjacent box position in the layout — an earlier ASCII mockup drew a
"chain" through unrelated GPUs, which a dedicated regression test now
guards against. A bonded pair spanning two different NUMA nodes is
flagged Warning on the edge and on both GPU boxes, per project
decision that this is an anomaly worth surfacing, not a neutral fact.
Zero changes to the ingest contract: this reverts the HardwareNVLinkPort/
HardwarePCIeDevice.NVLinks field shipped in v11.55 (33d6eee) along with
its collector/nvidia.go enrichment — that field risked a 400 from
Reanimator Core's strict decoder without an RFC, and isn't needed since
the topo page queries nvidia-smi directly instead of reading it from
audit.json. The v11.55 systemd fix (bee-nvidia.service ordering,
nv-hostengine restart) and the nvlink-status/-errors/dcgmi dumps in the
support bundle are untouched.
Also reorganizes support bundle collection per project convention:
system/ is now LiveCD-operational logs only (Xorg, services, console,
network/FS of the host itself); all server-hardware dumps (lspci,
NVIDIA/NVLink/DCGM, fabric manager, PCIe AER, ethtool, mstflint) move
to techdump/, deduplicating two entries already produced by
platform/techdump.go. Adds nvidia-bug-report.sh (previously only
collected inside the NVIDIA SAT pack) and lscpu to the always-on dump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add NVLink port status (nvidia-smi nvlink -s), error counters
(nvidia-smi nvlink -e), and dcgmi nvlink status to the support bundle,
and enrich HardwarePCIeDevice entries with per-link telemetry.
Replace the raw nv-hostengine pkill/restart dance in bee-nvidia-load
with systemctl restart/start of nvidia-dcgm.service, and order
bee-nvidia.service Before= nvidia-dcgm.service and
nvidia-fabricmanager.service so modules/device nodes exist before
those units start.
dcgmi diag's failures only surface a terse wrapper message ("Detected
driver major version 0 is not between the required versions 346 and
2000"), which doesn't say why. Reading DCGM's own nvvs source
(NvidiaValidationSuite.cpp::CheckDriverVersion) shows this fires when
dcgmSystem.GetDeviceAttributes(0, ...) comes back empty — the version
string is never actually malformed. Diagnosing further requires info
bee never collected: whether dcgmi discovery even sees GPU 0/1
correctly, and nvvs's own internal debug log (normally
/var/log/nvidia-dcgm/nvvs.log, which bee never captured).
Add a "dcgmi discovery -l" job before every DCGM diag invocation
(check-gpu-dcgm-l2, nvbandwidth, targeted_power, pulse_test,
targeted_stress), and route dcgmi diag through
"-v -d DEBUG --debugLogFile {{run_dir}}/..." so the full nvvs debug
log lands in the SAT run dir and gets picked up by the support bundle
automatically, instead of needing a live SSH session on hardware we
usually can't get a second look at.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.52's ISO build failed at chroot_install-packages:
datacenter-gpu-manager-4-cuda13 : Depends: datacenter-gpu-manager-4-core
(= 1:4.5.2-1) but 1:4.6.0-1 is to be installed
cuda13/proprietary/proprietary-cuda13 all declare an exact-version
dependency on datacenter-gpu-manager-4-core, but core itself was never
listed, so apt's solver picked the newest core in the repo (4.6.0-1)
for the transaction and then reported the pinned variants as unmet
instead of resolving everything to 4.5.2-1. Pin core explicitly so
there's no ambiguity for the solver.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.51 pinned 580.126.20 (DGX OS 7.5.0's validated combo), but
build-nvidia-module.sh downloads the .run installer from
download.nvidia.com/XFree86/Linux-x86_64/<version>/, a separate catalog
from the CUDA apt repo checked previously. 580.126.20 404s there —
same failure class as the original 550.54.15 CDN-404 incident.
Checked all 580.x point releases on that CDN; only some are mirrored
(580.126.09, 580.159.03/.04, 580.173.02 are 200; 580.126.16, 580.126.20,
580.167.08 are 404). Switched to 580.159.03: confirmed downloadable,
confirmed present as .deb in the apt repo, and it's the exact driver
version running on the real DGX Spark.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
590.48.01/DCGM 4.6.0-1 was never a deliberate compatibility choice — the
driver version traces back to a CDN 404 on the originally pinned
550.54.15 (commit 559fc29), and DCGM was bumped twice since purely
because NVIDIA pulled the previously-pinned core package from the repo
(6dee8f3, b2e177a). The 590 branch only has two point releases so far,
and its nvvs plugin fails dcgmi diag with "Detected driver major
version 0 is not between the required versions 346 and 2000" on H100 —
confirmed via a staging support bundle.
Switch to 580.126.20 / DCGM 4.5.2-1, the combo NVIDIA's own DGX OS 7.5.0
release notes validate for H100. Both packages confirmed present in the
NVIDIA CUDA debian12/x86_64 apt repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
extractArchivePath only stripped the "Archive written to " prefix when
the string ended in ".tar.gz", but SAT packs write bare run directories,
never actual tar.gz archives. Any task routed through an
ActionResult-wrapping pack function (e.g. RunNvidiaAcceptancePackWithOptions,
used by the DCGM L1-L4 diag task) passed the whole prefixed string into
ReadSATOverallStatus, which then failed to find summary.txt and silently
returned "", masking real job failures as task-level "done".
Strip the prefix directly instead of gating on a suffix that SAT run
dirs never have.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Flags GPUs that reach one or more peers only via a SYS-class PCIe hop
(crossing the CPU/NUMA-node boundary) in nvidia-smi topo -m. On servers
where GPUs are only bridged pairwise via NVLink bridge (no switched
NVLink fabric), this is the exact path that traffic between different
bridge pairs has to cross, and can cut multi-GPU throughput by 2x+ for
workloads spanning more than one pair.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New read-only "Check" step reports whether this server can run NVIDIA
Confidential Computing: CPU TEE support (Intel TDX / AMD SEV-SNP, via
dmesg and kvm_amd sysfs params) and GPU firmware CC capability (via
`nvidia-smi conf-compute -q`). Also collect that command's output into
the techdump export bundle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ramp Sequence table's Run 1 row showed "—" for GPU power because the
step-1 fast path (reusing single-card calibration) never populated
PerGPUTelemetry like steps 2+ do. Also add GPU total W / Server itself W
columns and an idle baseline row so server-vs-GPU consumption is visible
per ramp step.
Product Asset Tag (p 5) and the repeated custom "Extra" fields (Product
Extra p 7, Board Extra b 5/6/7, Chassis Extra c 2/3) from the Inspur FRU
field doc weren't writable — ipmitool prints identically-named lines for
each custom field with no index of its own, so a plain name lookup
couldn't tell them apart. parseFRUOutput now counts occurrences per area
to recover the real index, and the existing area/index round-trip in the
FRU editor write path picks it up automatically. Out-of-band (-H/-U/-P)
writing remains out of scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
smartctl -t short only launches the self-test and returns immediately
("Testing has begun"); unlike nvme device-self-test --wait, it has no
blocking mode. Validate/Load runs closed the task and produced reports
before the drive actually finished the test. Now poll smartctl -a until
the test completes (or times out) and report the real result.
Also add a per-disk "Resource" section with pseudographic progress bars
for uptime (vs 5y design life), bytes written (vs 1 DWPD x 5y budget),
and bytes read (percent from SMART attribute 242), all rendered in
human-scaled units (days/years, TB/PB) instead of raw hour/byte counts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RAID Controller Management previously hid any LSI drive that wasn't
already Frgn/UGood/JBOD, and scoped VROC "free drives" from all system
disks instead of the ones actually wired to the VROC controller's
ports - drives attached directly to the CPU or another HBA could leak
in. Now every drive is listed per its own controller, and LSI drives
not already ready for array creation get a "Prepare" button that
forces them to Unconfigured Good via storcli.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Check mode: read-only SMART/NVMe data collection, no self-test.
Load mode: same collection + short self-test (nvme device-self-test -s 1,
smartctl -t short). Card descriptions updated accordingly.
After each storage SAT run, a disk-N-devname-report.txt is written
per device into the runDir (auto-included in support bundles).
Web UI task page renders one card per disk directly below Task Report.
Also fixes pre-existing TestDashboardRendersRuntimeHealthTable failure:
test fixture used "inactive" status but code now treats inactive as OK
for completed oneshot services; updated to "failed" to match intent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces separate IPMI FRU and SAA DMI cards with a single FRU / Elabel
card that reads all available sources in parallel and shows each field
with a color-coded source chip (IPMI FRU / Huawei iBMC / SAA DMI).
Huawei elabel fields are read/written via OEM IPMI raw commands
(NetFn 0x30, cmd 0x90) with 19-byte chunking protocol, matching
the FusionServer ElabelTool V511 wire format. Covers DeviceName,
DeviceSerialNumber, ProductName, ProductSerialNumber, ProductAssetTag,
ProductManufacturer, MainboardManufacturer, BoardProductName,
ChassisPartnumber, ChassisType (read-only), IOChassisSerial,
IOChassisAssetTag, and GUID (read-only via standard 0x06 0x08).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- sat.go: DetectGPUVendor lspci fallback now checks GPU device classes
([0300]/[0302]/[0380]) per line instead of scanning the whole output for
vendor name; AMD EPYC servers have dozens of AMD-branded PCIe entries
(Root Complex, IOMMU, Host Bridge) that were triggering the old check
- blackbox.go: fix deadlock in finishCycle — it held w.mu while calling
persistState(), which acquires rt.mu then re-acquires w.mu inside
persistStateLocked(); now w.mu is released before persistState()
- build.sh: remove NVIDIA-specific overlay files (bee-gpu-burn,
bee-john-gpu-stress, bee-nccl-gpu-stress, bee-nvidia-recover,
bee-dcgmproftester-staggered, bee-check-nvswitch,
nvidia-fabricmanager.service.d/) for non-nvidia build variants
- bee-selfheal: gate NVIDIA recovery on BEE_GPU_VENDOR=nvidia so the
script does not attempt to restart bee-nvidia.service on NOGPU builds
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace global Save button with per-row ✓ (save) / ✗ (cancel) buttons
that appear only when a field is changed
- All fields shown as editable inputs; server rejects unknown fields
with a clear error message instead of hiding them in the UI
- Monospace font and 1.5px border for all value inputs
- Server-side name→area/index lookup for fields sent without area
- SAA DMI card: same per-row UX, confirm dialog kept (requires reboot)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ipmitool fru print on some BMC implementations returns short names
("Chassis Serial", "Board Mfg", "Board Product", "Board Serial",
"Product Serial") instead of the full names in the vendor doc.
Add both variants to fruEditableFields so all fields are editable
regardless of which naming convention the BMC uses.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
POST /api/system/reboot → systemctl reboot
POST /api/system/shutdown → systemctl poweroff
Both require confirm() before executing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The USB Black-Box card already provides enable/disable per device.
The standalone Blackbox Logging card was non-functional and redundant.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Unified card for LSI/Broadcom and Intel VROC controllers: auto-detects
foreign configurations and warns the operator with Import/Clear actions;
allows creating RAID 1 mirrors from unconfigured drives regardless of
controller type. Live output streams via SSE into an inline terminal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- audit JSON: IPMI sensor readings (ipmitool sensor) merged into hardware.sensors alongside lm-sensors data
- audit JSON: IPMI SEL entries (ipmitool sel list) in hardware.event_logs with source "ipmi-sel"
- audit JSON: dmesg error/warning lines in hardware.event_logs with source "dmesg" (filtered by error/warn/AER/Xid/NVRM/ECC/panic patterns)
- support bundle: added ipmitool-sensor.txt, ipmitool-sel.txt, ipmitool-sel-time.txt to techdump
- saa_dmi.go: fix dmiItemRE to accept SHN with parentheses (e.g. PS(4)LC for PSU fields)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- saa_dmi.go: set cmd.Dir=/usr/local/bin on all saa exec calls so
acpica_bin/acpidump is found relative to correct working directory
- build.sh: copy all saa companion dirs (acpica_bin, ExternalData,
tool, stunnel, GO_SNMP) to /usr/local/bin/ preserving structure
- iso/vendor: add acpica_bin/acpiexec, ExternalData/, tool/gpu/nVidia/x64/,
tool/USBController/, stunnel/, GO_SNMP/ from SAA 1.5.0 release package
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Check (2): validate mode only — no mode switcher, no stress-only cards
(nvidia-targeted-stress, nvidia-targeted-power, nvidia-pulse hidden)
- Load (3): stress mode only — no mode switcher, all cards shown
- satStressMode() hardcoded per page; satModeChanged() removed
- Profile card with radio buttons removed from both pages
- Replaced with simple Run All button + est. time
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove "6." / "7." prefixes from Tools and Settings nav labels and page titles
- Add a horizontal separator (nav-sep) before the Tools/Settings group
- Move Tasks into the nav as a regular nav-item after the separator,
replacing the separate tasks-nav-btn at the sidebar bottom
- Tasks item retains the active-count badge (tasks-nav-count)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tools page now contains only NVMe Block Format and Supermicro - DMI.
Moved to Settings (7):
- System Install (Install to RAM + Install to Disk)
- Support Bundle + USB Black-Box
- Tool Check
- NVIDIA Self Heal (replaces simple NVIDIA Recovery card)
- Network
- Services
Update TestToolsPageRendersNvidiaSelfHealSection to assert the moved
cards on /settings instead of /tools.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extract saa 1.5.0 (Linux x86_64) into iso/vendor/saa — baked into ISO
at /usr/local/bin/saa via the existing vendor loop in build.sh
- Add saa to the vendor tool loop in iso/builder/build.sh
- Rename the web UI card from "SAA - DMI" to "Supermicro - DMI"
- Remove the redundant description hint about saa on PATH
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds board and memory parser test fixtures based on real dmidecode output
from Dell PowerEdge R740xd, HPE ProLiant DL380 Gen10, and Supermicro
SYS-6028R-WTR sourced from the linuxhw/DMI dataset. Extends cleanDMIValue
with four additional vendor placeholder strings found in the dataset:
"0123456789", "1234567890", "NOT AVAILABLE", and "TO BE FILLED BY O.E.M"
(without trailing dot). Adds memory_test.go covering mixed populated/empty
DIMM slots and both GB and MB size formats.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the guessed pipe/key=value parser with the correct format
documented in SAA User Guide 4.8.1:
[Section]
Item Name {SHN} = "value" // comment
Handles string values (strips surrounding quotes), non-string values
(UUID, hex), section headers for display names, version line, and
// comments. Verified against the SAA 1.5.0 User Guide sample.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a new card to the web UI Tools page for reading and editing DMI
fields via SAA (In-Band). Reads current DMI configuration with GetDmiInfo,
displays all fields as an editable table, and applies only the changed
fields via EditDmiInfo + ChangeDmiInfo. Backs up the original DMI file to
dmi-backups/ before any write, making it available in the support bundle
for rollback. Also adds "saa" to the standard tool check list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NVIDIA removed datacenter-gpu-manager-4-core 1:4.5.3-1 from the
repository and published 1:4.6.0-1. The cuda13 and proprietary
packages still declared an exact-version dependency on 4.5.3-1 core,
making the old pin unresolvable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the flat menu (Dashboard, Audit, Validate, Burn, Benchmark,
Tasks, Tools) with a numbered progression that guides engineers through
a logical acceptance workflow:
Dashboard (landing) → 1. Audit → 2. Check → 3. Load → 4. Speed
→ 5. Endurance → 6. Tools → 7. Settings
Key changes:
- layout.go: numbered nav labels, new hrefs, Tasks removed from nav
and replaced with a persistent sidebar badge (polls /api/tasks every
5 s, highlights amber when tasks are active)
- server.go: 301 redirects from /validate→/check, /burn→/load,
/benchmark→/speed for backward compatibility
- pages.go: dispatch cases for all new routes; old routes kept as
fallbacks
- page_validate.go: add renderCheck() — non-destructive check page
with validate-mode tests only (no stress toggle, no targeted-stress/
targeted-power/pulse cards)
- page_burn.go: add renderLoad() wrapper; update scope alert to
reference /check instead of /validate
- page_benchmark.go: add renderSpeed() (performance focus) and
renderEndurance() (stability/overnight focus) wrappers
- page_settings.go: new Settings page with blackbox logging toggle,
NVIDIA driver reset, and build info
- server_test.go: update five tests to use new route names and
content expectations
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bee-nvidia.service loads NVIDIA kernel modules; without After=bee-nvidia.service
fabricmanager starts before /dev/nvidiactl is ready, fails, and relies on
systemd restart to recover (~38s delay on affected systems).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
smartpqi uses scsi_transport_sas but does not register a sas_host
object, so /sys/class/sas_host/host14 does not exist and the existing
SAS detection check passes right through. Writing to host14/scan then
calls sas_user_scan which blocks indefinitely on scsi_scan_target's
mutex (confirmed by kernel hung-task traces in the field).
Add a second detection path via /sys/class/scsi_host/hostX/proc_name:
skip hosts whose driver is "smartpqi" or "hpsa" (HPE Smart Array
predecessors that exhibit the same behaviour).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
EASY_BEE_NVIDIA_LEGACY_V<date> is 33 characters; ISO 9660 volid is
limited to 32. Compute the maximum token length dynamically from the
prefix length and trim ISO_VERSION_LABEL_TOKEN with cut before
assembling BEE_ISO_VOLUME. All four variants now fit within the limit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add nvidia-aer-correctable and pcie-aer-correctable patterns to catch
"bus correctable error" events seen in SEL (Critical Interrupt / offset 7).
Both patterns carry severity "warning" — correctable errors are
hardware-recovered and should not flag a card as failed.
Fix kmsg_watcher routing: GPU-category events were keyed as pcie:<BDF>
but the UI queries for pcie:gpu: prefix. Split the switch so "gpu" →
pcie:gpu:<BDF> and "pcie" → pcie:<BDF>. This applies to both
flushWindow (SAT-window path) and flushImmediate (always-on path).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
\\$1 in a double-quoted string expands as literal backslash + $1 (the
script's first positional arg). With set -u and no CLI args (IP entered
via read), this fails. \$1 correctly escapes the dollar sign, producing
a literal $1 for awk on the remote host.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HTMX was never loaded on the page, so hx-get on the component label
spans was dead code — the dialog opened empty. Replace with a plain
openComponentDetail() fetch call. Also fix dialog positioning broken
by the CSS reset (*{margin:0} overrode the UA margin:auto that centers
<dialog>). Replace card hx-trigger polling with a setInterval.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All intermediate build artifacts (binaries, live-build work dirs, overlay
stages, NVIDIA/NCCL/cuBLAS/john caches) now live under dist/cache/.
Final ISOs go to dist/release/ instead of scattered dist/easy-bee-v*/ and iso/out/.
dist/ is already gitignored, iso/out/ entry removed as redundant.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Auto-detects build mode: remote VM if BUILDER_HOST is set in .env,
local Docker otherwise. Cache hardcoded to dist/container-cache (gitignored).
All flags forwarded to build-in-container.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- kmsg watcher now records kernel errors (GPU Xid, MCE, EDAC, storage I/O) at all times,
not only during SAT tasks; flushImmediate writes directly to ComponentStatusDB
- New health_poller: polls ipmitool sdr every 60s for PSU health (watchdog:psu source)
- Hardware Summary card auto-refreshes every 30s via htmx without page reload
- Component rows (CPU/Memory/Storage/GPU/PSU) are now clickable -- opens a modal
with per-component status, source, timestamp and last 20 history entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Installs a local-premount initramfs hook that intercepts bee.wipe=all before
squashfs is mounted. Shows a numbered disk selection TUI (pure POSIX sh), wipes
selected disks (nvme format / blkdiscard / dd fallback), syncs, and reboots.
Works even when squashfs fails to mount.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a "WIPE ALL DISKS" entry to both GRUB and isolinux menus (bee.wipe=all).
Includes bee-wipe-disks for manual use from a running live system.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Squashfs versioning:
- ISO now contains filesystem-v<VERSION>.squashfs instead of the generic
filesystem.squashfs, making it immediately visible which build is
running (visible in /run/live/medium/live/ at boot time).
- Full build path: rename filesystem.squashfs → filesystem-v*.squashfs
after lb build, before lb binary_checksums/binary_iso.
- Fast path: find and unpack whatever filesystem*.squashfs exists, repack
as the new versioned name, remove the old file, update the ISO.
- needs_full_build: accept any filesystem*.squashfs so version changes
alone don't force a full rebuild.
Media selection hardening:
- Add live-media=/dev/disk/by-label/<LABEL> to the kernel boot line in
addition to the existing live-media-label=<LABEL>. live-boot will now
open exactly the labeled device rather than scanning all block devices,
preventing accidental use of squashfs files from local disks or
stale virtual media attached via IPMI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mksquashfs 4.5.1 (bookworm) writes a non-SQUASHFS_INVALID_BLK value for
xattr_id_table_start in the superblock even when -no-xattrs is passed, if
the source chroot contains POSIX ACL xattrs set by dpkg at install time.
Linux 6.1 squashfs driver then fails with "unable to read xattr id index
table" and refuses to mount the filesystem.
Strip all xattrs from the chroot via Python3 (already present) immediately
before mksquashfs runs. With an xattr-free source tree the resulting
squashfs is guaranteed to have SQUASHFS_INVALID_BLK in the xattr field.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--chroot-squashfs-compression-options does not exist in live-build
bookworm (1:20230502). The correct mechanism is the MKSQUASHFS_OPTIONS
environment variable read by binary_rootfs.
Export MKSQUASHFS_OPTIONS="-no-xattrs" before lb build so live-build's
binary_rootfs picks it up, and add -no-xattrs explicitly to every
direct mksquashfs call in build.sh (fast-path repack and the dormant
split-layers function). Remove the invalid lb config option.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--chroot-squashfs-options is not a valid lb_config option; the correct
name is --chroot-squashfs-compression-options. Without this fix lb config
aborts immediately, so the -no-xattrs flag (which prevents the
"unable to read xattr id index table" boot failure) was never applied.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Kernel squashfs driver fails with "unable to read xattr id index table"
when the squashfs contains POSIX ACL xattrs (system.posix_acl_*) written
by mksquashfs as unrecognised entries. This caused every built ISO to
drop to an initramfs shell at boot.
Add -no-xattrs to mksquashfs options so xattrs are stripped at build
time. xattrs are not needed in a live read-only rootfs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
split_live_squashfs_layers moved /usr out of filesystem.squashfs into a
separate 10-usr.squashfs, leaving a rootfs skeleton that live-boot
(1:20230131+deb12u1) cannot mount: the initramfs panics with
"Can not mount /dev/loop0 ... filesystem.squashfs".
live-boot in bookworm expects a single self-contained filesystem.squashfs.
Revert to the standard single-squashfs layout and remove the dead
multi-squashfs guard in needs_full_build().
The split_live_squashfs_layers function is kept for future reference.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
trap RETURN is a bash extension not supported by /bin/sh on Debian.
With set -e active the unsupported trap call exited the build immediately
after lb build, before bootloader sync and ISO copy steps ran.
Remove both trap RETURN lines — explicit rm -rf at the end of the
function is sufficient for cleanup on the happy path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
