The fan-ceiling check, the webui metrics collector (every 5s) and the PSU
health poller each shelled out to ipmitool independently. The BMC's KCS
interface serializes those calls anyway, so under load the concurrent
`ipmitool sdr`/`dcmi power reading` invocations just queued behind each
other — that's what produced "IPMI slow" backoff during a fan-ceiling run in
a blackbox dump, while the dashboard looked fine only because it was reading
its own, separately-stale data from a different ipmitool call.
hw_telemetry.go is now the sole recurring poller (fan RPM, temperature, PSU
power/status, DCMI system power), with the adaptive 1s-30s backoff that used
to be duplicated inside the fan check. Every hot-path consumer reads the
shared cache (hwSnapshot / platform.HardwareSDRSnapshot) instead of calling
ipmitool itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The passive collector records PSU draw at idle too, so right after boot
the "observed peak" is just idle wattage and a load % off it is
meaningless. Require the observed peak to sit >=25% above current draw
before using it as a load scale; until a real full-load run bumps it,
show plain watts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
Extract the fan peak-tracking into observedPeakStore (observe max under
load, hold >= minHold to reject spikes, round, persist JSON) and add a
second instance for PSU draw (psu-observation.json, keyed by PSU
ordinal). Fed from samplePSUPower like fans are from sampleFanSpeeds, so
any full-load run refines it — the Fan Ceiling Check (which also samples
PSU power at a slow cadence off its loop and writes psu_<i>_peak_w), a
burn, thermal cycling, and the 5s web metrics collector.
/topo PSU cards now scale the load fill by wattage_w when the BMC
reports it, else by the observed peak draw — marked "~N% load". This
MSI stand's BMC gives only instantaneous input power, so the observed
peak is the only capacity figure available.
Fan behaviour is unchanged (tests exercise updateFanObservation /
estimateFanDutyCyclePctFromObservation / ResolveFanMaxRPM through the
new store).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
Same idea as the fan duty-cycle fill: a bar rising from the bottom of
each PSU card for draw / nameplate rating, live-updated on the shared
5s /api/metrics/latest poll. Drawn only when the rating (wattage_w) is
known — a raw wattage with nothing to scale against is not a load
figure, so BMCs that report only input power (e.g. this MSI stand) keep
showing the plain watts. Detail line becomes "<v> V · <draw> / <rating> W
· <n>% load" when both are present.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
Each PSU is its own card, coloured by its own status — a failed unit
goes red on its own instead of dragging a single grouped card down —
and shows input voltage + draw (measured output/input, else nameplate
rating). Cards click through to the PSU detail modal and carry data-psu
so the shared topoLiveScript refreshes their wattage from
/api/metrics/latest on the same 5s poll as the fan tiles.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
Each fan tile carries data-fan / data-ceil / data-measured and a small
script polls /api/metrics/latest every 5s — the metrics collector's own
sampling period, served from memory with no extra BMC call — updating
each tile's spin rate, duty fill and tooltip in place. Polling faster
would only re-read identical numbers; sampling the BMC faster would
choke it under load. FanReading gains json tags for the endpoint.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
It is a platform cooling test (CPU + memory always; GPU only as extra
heat when present), not a GPU test — so it belongs with the other
whole-box checks, not in the NVIDIA section. Still Load-page only, via
the new renderLoadOnlySATCard (inverse of renderCheckOnlySATCard).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
- GPU load switches from bee-gpu-burn (compute burn, ~88% TDP) to
dcgmproftester -t 1004 / targeted_power via
resolveBenchmarkPowerLoadCommand — the same engine Power/Thermal Fit
uses and the hottest sustained NVIDIA load we have, so fans are
actually pushed toward their ceiling.
- Sample loop is now IPMI-hang-proof: every ipmitool read is time-boxed
in an abandonable goroutine, and the poll interval backs off
geometrically (1s→30s) when reads are slow, tightening again on
recovery. A plateau is only trusted while telemetry is healthy;
degraded runs ride out to MaxLoadSec. Summary gains fan_samples /
telemetry_degraded. Drops the per-second nvidia-smi+power+cpu-temp
sampling from the hot loop.
- Dead code removed: FanStressRow, GPUStressMetric, sampleFanStressRow,
sampleGPUStressMetrics, WriteFanStressCSV/WriteFanSensorsCSV,
analyzeMaxTemp, sampleSystemPowerResolved.
Topology fan tiles:
- size encodes the fan's ceiling RPM (its class), not current speed;
coloured fill rising from the bottom encodes live duty cycle
(current / ceiling), shown only when the ceiling was measured.
- glyph spin rate now maps absolute RPM into a human-perceptible band
(fanSpinPeriodSec: 2.2s/turn at <=1000 RPM, 0.35s at >=13000).
- the "N fans · N OK · tile size ∝ …" caption line is gone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
Repurpose the previously-unwired RunFanStressTest into RunFanCheck, a
Load-tier SAT test that drives stressapptest (CPU+memory) and, when a GPU
is present, a GPU burn to 100% simultaneously, then watches every fan
until none has climbed for ~60s. The observed peak RPM per fan is the
"ceiling"; it is persisted through the existing fan-observation store.
MSI G4201 / AMI MegaRAC exposes no host-side fan force (every OEM IPMI
command returns 0xc1; Redfish Thermal is GET-only), so load-driven ramp
is the closest safe equivalent. See
bible-local/decisions/2026-09-04-fan-ceiling-check.md.
- platform.ResolveFanMaxRPM: per-fan max with fallback (persisted peak ->
peer peak -> current RPM), resolved in platform, not the view.
- platform.ErrTestNotApplicable: no load source or no fan sensors ->
task lands as cancelled ("not applicable"), never failed, so an
engineer never sees a false red. executeTaskWithOptions maps the
sentinel; finalizeTaskForResult honours a pre-set TaskCancelled.
- Verdict FAIL only for a fan at 0 RPM / IPMI cr-nr under load.
- /topo: one small spinning square per fan, sized by RPM / resolved max,
clickable through to a new "fan" component-detail type; per-fan status
recorded to the component-status DB from the fan SAT summary.
- Wiring: /api/sat/fan/run route, "fan" task target, Load-page card,
stress-mode Run All.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
Every GPU-selection picker (Load/SAT, Burn, Benchmark) now renders the
card serial: "GPU N — <model> · <mem> MiB · sn: <serial>". The serial is
monospace and the digits that differ across the listed GPUs (common
prefix/suffix stripped) are emphasised so operators can tell cards apart.
Row markup was duplicated across three pages (and twice within
page_validate.go). Consolidated into a single module,
internal/webui/gpu_picker.go: beeGpuPicker.render({...}) builds every row;
gpuPickerCSS/gpuPickerJS are injected once by layoutHead/renderPage. Pages
keep their own selection-note text and multi-GPU toggles but no longer
hand-build <label> markup.
ListNvidiaGPUs() adds the serial via nvidia-smi --query-gpu=...,serial;
N/A is normalised to empty. Serial flows to the client unchanged through
the existing /api/gpu/nvidia JSON response.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AudE3Q2nd9kxxVuxPKcTho
Every sample was stamped with time.Now(), so a timezone switch or NTP
step punched a multi-hour gap into the series: old points collapsed to
the left edge, new points bunched at the right, joined by one diagonal.
Stamp rows from a monotonic seqClock instead — seeded once from the wall
clock (or the newest persisted row) and thereafter advanced only by the
monotonic elapsed time between writes. Rebase Downsample/Prune on the
newest sample rather than time.Now() so a clock step just before the
hourly compaction can't drop fresh data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GofKhuF9xQHaz3UFfncR6D
Rework the /topo diagram from side-by-side stacked cards into one tall
vertical bar per CPU socket with everything attached to it branching off
sideways (socket 0 left/branches right, socket 1 right/branches left).
Disks are now parented under the storage controller they physically hang
off (SATA/AHCI, SAS HBA, RAID) — itself a NUMA-affine PCIe device under one
socket — instead of a synthetic catch-all node. The disk->controller link
is read from a new storage-controllers.txt techdump
(platform.StorageControllerMapScript, a /sys/block walk); disks with no
resolvable controller fall back to an "Other" bar. No board/root node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYjrG6bVmeG1Z2Wmc8kg3o
NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power
saving, and applyPCIeLinkSpeedWarning fired on every idle collector
pass regardless - since component-status DB records never downgrade
(Record() only ever raises severity), one boot-time idle sample
permanently pinned pcie:gpu:nvidia to Warning for the rest of the
session even after every load-bearing GPU SAT test passed clean. Two
prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag)
didn't hold up against this hardware/driver combination - see
bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the
full history.
Rather than add a downgrade path, stop writing an unverified status in
the first place: parseLspciDevice no longer calls
applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay
populated as plain descriptive fields; only a verified-under-load
caller may now turn them into a status verdict.
Two new SAT targets provide that verified signal:
- pcie-link (platform/pcie_link_check.go): forces every enabled PCIe
device - not just GPUs - to retrain via the PCIe spec's Link Control
"Retrain Link" bit, then compares the negotiated speed against the
device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn
equivalent load tool. Classifies by PCI class code + vendor ID, not
name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into
their own component-status keys so a degraded NIC never reads as a
GPU fault.
- nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives
real host<->device traffic via dcgmi diag -r nvbandwidth and
resamples link speed immediately after, independent of nvbandwidth's
own pass/fail.
Both wired into the task queue/webui the same way as nvidia-config
(routes, dispatch, priority, Validate page cards, Run All Check SAT).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`ipmitool fru edit` frequently exits non-zero after a field resize
(shrinking/growing a string shifts every later offset in the FRU
record) even though the write itself landed correctly on the BMC -
the failure is in ipmitool's own post-write re-parse, not the write.
bee was treating that exit code as authoritative and marking the task
failed even when the field was actually written.
runIPMIFRUWriteTask now re-reads the FRU on a non-zero exit and
compares the target field's actual value before deciding the task
failed, logging which path it took either way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
statusSeverity was byte-for-byte duplicated in collector/contract.go and
app/sat_overlay.go (same switch, same comment). Export it as
collector.StatusSeverity and drop the app-package copy.
page_topo.go inlined the same GPU/NIC/RAID class switch as its own
pcieDeviceKind function in one spot — call the function instead.
page_validate.go had its own copy of pages.go's firstNonEmpty (all call
sites already pre-trim their inputs via validateTrimPtr, so the extra
TrimSpace was redundant) — drop the duplicate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
matchesGPUVendor (sat_overlay.go) matched any PCIe DeviceClass containing
the substring "Controller" instead of the actual GPU classes, so a
same-vendor non-GPU device (e.g. an NVIDIA-branded NIC/storage controller)
with a degraded PCIe link surfaced as a pcie:gpu:<vendor> alarm on the
Hardware Summary/webui.
isGPUDevice (app_format.go) had the same bug via a different path: an
"any NVIDIA-vendor device is a GPU" fallback that could inflate the
"GPU: N x <model>" audit summary line with non-GPU companion devices
(NVSwitch/NVLink bridges, etc).
Both now defer to collector.IsGPUClass (exported from the previously
unexported isGPUClass), the canonical exact-match classifier already used
by amdgpu.go — removing the duplicated/incorrect class checks instead of
reimplementing them. Also dedup two more locally-duplicated classifiers
within webui (which doesn't import collector by convention): NIC-class
matching (pages.go/page_topo.go) and GPU-class matching in
page_validate.go now share one copy per package instead of being
reimplemented per file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
buildNvidiaTaskSet only treated ParallelGPUs as a reason to keep all
selected GPUs in one task, so ramp-up (StaggerGPUStart) fell through
to per-model task splitting. When that split left a single GPU in a
task, resolveNvidiaRampPlan silently returned StaggerSeconds=0,
making the ramp-up toggle a no-op.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds POST /api/system/time-sync so the dashboard's "Sync with this
browser" button can set the host's date, time, and IANA timezone from
the client's browser — useful for appliances with no NTP/network
access to keep their own clock straight. Disables NTP before writing
the manual time so it doesn't get immediately overwritten.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nvidia-bug-report.sh appends .gz to --output-file when gzip is available,
which silently produced empty nvidia-bug-report.txt in support bundles
(cat looked for the uncompressed name that never existed).
Also: a GPU that falls off the PCIe/NVLink bus (Xid 79) or gets flagged
for Node Reboot Required (Xid 154) mid-SAT-run left every downstream test
failing with generic, unrelated-looking errors (CUDA "unknown error",
"unable to determine device handle") with no indication the GPU needed a
physical power-cycle to recover. Detect these codes from SAT run logs and
surface a plain-English "physical reboot required" message in the task's
failure detail, the persisted component-status DB, a dashboard banner on
the Hardware Summary card, and topology diagram GPU-node severity.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- page_scenario.go: onclick built via JSON.stringify() embedded raw double
quotes inside a double-quoted HTML attribute, truncating the attribute so
the click handler never compiled; pass the name through an escaped
data-scenario-name attribute instead.
- build.sh: overlay staging rsyncs (OVERLAY_DIR->stage, stage->includes.chroot)
ran without --delete, so a scenario removed from the repo (a9924b0) stayed
baked into every ISO built from the persistent stage cache since — the
"second script" in the Scenario page's list.
- blackbox: rewritten around a deterministic local zip + incremental
patch-the-changed-suffix onto removable media, instead of walking/copying
~90 files through a synchronous ntfs-3g FUSE mount every cycle. journalctl
captures are now "--since last sync" (were "--since boot", growing with
uptime) and metrics.db is excluded (was copied whole every cycle).
- scenario: nvbandwidth-acs-ab now escalates GPU count (same-socket pair,
other socket's pair, one cross-socket pair, all GPUs) under each ACS state
instead of always running all 6 GPUs at once, using a new `bee
gpu-bandwidth-groups` subcommand that discovers socket layout from
`nvidia-smi topo -m` at runtime — gpu_indices is host-specific, so this
can't be baked into the scenario file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Scenario page's Run button enqueued a task with target "scenario" that
never launched any load: the queue runs each task in an external
bee-worker subprocess (RunPersistedTask -> executeTaskWithOptions in
task_runner.go), whose target switch had no "scenario" case, so the task
died with "unknown target: scenario". The case had only been added to
taskQueue.runTask's switch in tasks.go — a stale copy exercised solely by
unit tests, which is why the tests passed while the button did nothing.
Add the "scenario" case (ReadScenario -> ParseScenarioJSON -> RunScenario)
to executeTaskWithOptions, and collapse runTask into a thin delegate to it
so there is a single target dispatch. The old runTask switch had already
drifted into a subset (missing nvme-format, *-write, raid-*, nvidia-config,
...); removing it eliminates the divergence that hid this bug. Also drop the
now-orphaned taskQueue.statusDB helper.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ScenarioSpec gains an optional "description" field. Listing (both
ListLocalScenarioFiles and ListScenarioFilesOnRemovableMedia, via the new
scenarioDescription helper) reads it out of each file without requiring
full ParseScenarioJSON validation to succeed, so a listing never hides a
scenario over an unrelated validation issue. The webui Scenario page now
renders Name/Description/Found-on/Run instead of just Name/Found-on — a
bare filename rarely tells anyone but the author what a scenario actually
does.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
platform.LocalScenariosDir (/usr/share/bee/scenarios, populated from
iso/overlay/usr/share/bee/scenarios/ by build.sh's overlay rsync) is now
checked before removable media for both `bee run <name>` and the "6.
Scenario" page — a scenario shipped with the image works with no USB
stick required. ReadScenario/ListAvailableScenarios merge local + USB;
the removable-media-only functions from the previous commit are kept
as-is (still used directly where that's actually what's wanted) rather
than renamed out from under existing callers/tests.
iso/overlay/usr/share/bee/scenarios/nvbandwidth-all-gpu-power-watch.json
is a copy of scenarios/nvbandwidth-all-gpu-power-watch.json — the two
aren't auto-synced (documented in scenarios/README.md), so shipping a
scenario baked-in means checking it into both places.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Exposes the scenario engine (platform.System.RunScenario, added earlier)
in the web UI instead of only the `bee run` CLI: a new nav item lists every
scenarios/*.json found on mounted removable media (GET /api/scenario/list)
and runs one with a click (POST /api/scenario/run), enqueued as a normal
Task with target "scenario" — progress/logs live in Tasks like any other
SAT pack, no separate live-output UI needed.
- app.go: satRunner gains RunScenario, exportManager gains
ListScenarioFilesOnRemovableMedia/ReadScenarioFromRemovableMedia — both
already implemented on platform.System, just newly exposed through App.
- webui/tasks.go: taskParams.ScenarioName; runTask's "scenario" case reads
the file from removable media, parses it, and runs it.
- webui/page_scenario.go: the page itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Investigating the CG480-S6053 reboot needed a way to run an ad-hoc load
(nvbandwidth across a specific GPU set) while sampling IPMI/nvidia-smi
telemetry in the background — without hardcoding a one-off test into the
SAT pack code for a single investigation.
- audit/internal/platform/scenario.go: ScenarioSpec/ScenarioJob (JSON,
no new dependency) + System.RunScenario. "command" jobs run sequential
or parallel (per-job "parallel" flag); "sampler" jobs run concurrently
in the background on their own interval until every command job
finishes or the scenario's timeout elapses. "{{gpus}}" in a command's
cmd is substituted from that job's gpu_indices. Command jobs are wired
through the same satJobBoundaryHook/satSyncBracketHook seams the SAT
job runner uses, so a scenario run gets the same durability treatment
(evidence that a risky command started/finished reaches blackbox before
a possible crash, not just whatever streamed to the RAM-backed export
dir).
- export.go: ReadScenarioFromRemovableMedia mounts each removable target
looking for scenarios/<name>.json — an air-gapped engineer can author a
scenario elsewhere, drop it under scenarios/ on the same USB stick
already plugged in for blackbox, and run it with no network path onto
the host.
- cmd/bee: new `bee run <file.json|name>` (bare name = looked up on
removable media); `bee scenario run <arg>` kept as a longer alias.
- scenarios/nvbandwidth-all-gpu-power-watch.json: the scenario that
reproduced the actual reboot (full nvbandwidth across all GPUs, which
crashed, vs. clean per-socket passes), with IPMI sensor + GPU power/temp
sampling for a power-delivery correlation check.
Also: webui/page_topo.go — the /topo page's component-status-detail modal
(GET /api/component-detail/{type}) showed "No status data recorded yet"
for any component type ComponentStatusDB has no history for yet (e.g. GPU
before a SAT run this boot), even though the topology card for the same
component already showed "N OK" from the audit inventory snapshot.
inventoryFallbackRecords now synthesizes records from that same inventory
snapshot when StatusDB is empty, using the same device classifiers
(isGPUDeviceClass etc.) and severity mapping (classifyTopoSeverity) the
topology card itself uses, so the two views never disagree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Memory used to render as one unattached row below the whole diagram
regardless of which socket it belonged to. schema.HardwareMemory has no
NUMANode field (unlike PCIe devices), so CPU affinity is instead read out of
the DIMM's own Locator string: either a CPU number encoded directly in it
("CPU0_DIMM_A1"), or — when the Locator has no CPU number of its own, e.g.
"DIMM000(A)" — a node number from Bank Locator ("_Node1_Channel0_Dimm0"),
read from the persisted dmidecode-type17.txt techdump the same way the
NVLink card already reads extra techdump for visualization only.
DIMMs that can't be attached to a column via either heuristic still fall
back to the old unattached "Memory" row so nothing silently disappears.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every place that surfaced a FAILED SAT result — task error messages,
component-status.json detail, and the hardware snapshot's ErrorDescription/
StatusHistory — used to say only "SAT overall_status=FAILED (see
summary.txt)" or "<label> failed", forcing an engineer to go dig through the
run directory to find out what actually broke.
nvidia-config's summary.txt now carries a "warnings" field with the specific
GPU/NVLink finding. A new SATFailureDetail/satFailureDetailFromKV in
component_status_db.go reads that field, or falls back to naming whichever
generic SAT sub-job(s) reported non-OK/UNSUPPORTED status along with their
exit code. This feeds both the task-runner error message and the component
status DB. sat_overlay.go's satKeyStatus (which drives ErrorDescription on
the exported hardware snapshot) now does the same, with storage kept
per-device so one drive's rc doesn't get attributed to another's card.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The standalone "confidential-computing" SAT target only ever checked CC
readiness, which most fleets never opt into (a NOT_READY verdict there
isn't a fault). Meanwhile DCGM diag never asserts GPU config compliance
(ECC/MIG/power-limit vs factory default) or NVLink topology (per NVIDIA's
own DGX BasePOD deployment guide, this needs a separate validation step)
— gaps confirmed against public DCGM docs and a real NV17-vs-expected-NV18
bonded pair found on a live bundle.
Repurposes the routine into "nvidia-config": reuses the existing
ListNvidiaGPUSettings() (already backing the GPU-settings page) to flag
ECC disabled, a MIG mode change stuck pending a reset/reboot, and a power
limit capped >5% below default; parses "nvidia-smi topo -m" bonded pairs
against "nvlink -s/-e" to flag any inactive lane or nonzero replay/
recovery/CRC counter on an otherwise-active bond. CC readiness is folded
in as one informational field (does not gate overall_status) rather than
a dedicated test. Reports under the same pcie:gpu:nvidia severity key as
every other nvidia-* SAT target instead of an isolated key, so a
config/NVLink FAILED result isn't invisible next to stress-test results.
Also fixes ApplySATResultToDB silently dropping any target with no
matching switch case (exactly what the old confidential-computing target
did) with a new coverage test enumerating every real SAT target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The long-lived bee-web process (writing PSU/kmsg watchdog records ~every
60s) and each short-lived "bee bee-worker" SAT-task subprocess each held
an independent in-memory copy of component-status.json. Whichever saved
last won outright, silently erasing whatever the other had just written —
e.g. a GPU SAT task's pcie:gpu:nvidia result vanishing the next time the
PSU watchdog ticked. ComponentStatusDB.Record now reloads on-disk state
(keyed by newer LastCheckedAt) before merging its own update.
Also stops re-logging identical repeat observations to History: the
ingest contract defines status_history as a transition log ("История
переходов статусов"), not a per-poll journal, but Record appended one
entry per call regardless — a continuously-polled PSU grew an unbounded
run of identical "still OK" entries. Now only appends when a source's
last recorded status for a key actually changes.
The PSU watchdog itself now backs off (60s -> doubling, capped at 30min)
while steady and resets to 60s the moment any PSU's status changes, cutting
ipmitool shellouts and file writes for a fleet that's been stable for a
while.
Also adds the missing "raid" case to the component-detail API (was
returning 404 for any RAID card's detail click on /topo).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PSU and firmware (BMC/BIOS) boxes were absolutely-positioned SVG rects in a
single fixed-width row with no wrap, so an arbitrary/larger count piled up
and overlapped once a board had more of either than fit in that row. Move
them to plain flex-wrap HTML below the diagram (Firmware row first, then
Power Supplies), which reflows naturally for any count. The main CPU/PCIe/
GPU diagram now scrolls horizontally (overflow-x:auto) instead of being
squashed to fit narrow viewports, matching the wide-table convention used
elsewhere in webui.
Also fixes a crash: renderTopoMainDiagram forced numCols to 1 for layout
purposes when a snapshot has zero CPUs, then unconditionally indexed
hw.CPUs[0], panicking the whole /topo page on any audit without CPU data.
Same-kind/same-column components (e.g. 4 GPUs in one NUMA node) now render
as one stacked card summarizing worst-case status plus a tally line ("3 OK,
1 Warning") instead of one box per component, and card severity coloring
uses real fill/stroke vars instead of HTML badge classes that don't apply
any style to SVG shapes.
- nvidia-smi underlines the topo -m header with ANSI CSI codes even when
writing to a file; both NVLink matrix parsers failed to find the header
and /topo showed "No NVLink-bonded GPU pairs found" on bonded systems.
- raid-lsi-create-mirror failed with "resources already in use" (exit 11)
on JBOD drives; the task now reads drive states and auto-converts
JBOD/UBad (set good force), releases hotspares, refuses Frgn/Onln with
actionable messages, and dumps preservedcache info when add vd fails.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nvidia-smi exposes reset_status.reset_required and remapped_rows.* only on
newer drivers for Ampere+ GPUs; queried via a separate exec call since an
unrecognized field name fails the whole --query-gpu command and would have
wiped out unrelated telemetry (temp/ECC/power) on older drivers otherwise.
Also refines Xid severity in both the ingest dmesg collector and the
always-on kmsg watcher: Xid 64 (row-remap InfoROM write failure) now
escalates to Critical instead of the generic warning every other Xid got,
and fixes the SAT-window flush path which previously hardcoded "Warning"
and ignored pattern severity entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RAID Controller Management showed "No drives detected" for a live
SAS3808-iMR controller even though storcli64 clearly enumerates its
drives. Root cause: the "eall/sall show all J" drive-listing parser
(collector/raid.go and its webui/raid_mgmt.go duplicate) assumed
drives are reported as a single "Drive Information" array, but real
storcli64/storcli2 output nests each drive under its own dynamically
named key ("Drive /c0/e69/s0", paired with a "... - Detailed
Information" key) — confirmed against a live techdump/storcli64-drives.json
capture. The assumed shape was never actually produced by the tool, so
this affected every storcli64/storcli2 controller, not just Tri-Mode
ones (the storcli2 fallback added in b7f015c never got a chance to
mask it in practice, since storcli2 itself reports zero controllers on
this hardware — a separate, unrelated tool-side gap).
Both parsers now read "Response Data" as a raw key map and pull drives
from either shape, so older storcli output using the array form still
works alongside the real per-slot form.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
/topo blocked the HTTP request on live nvidia-smi calls with no timeout
(topo -m, nvlink -s/-e run on every page load), so a wedged driver hung
the page indefinitely. CaptureTechnicalDump now persists these dumps
once per audit cycle; the page reads them from techdump/ instead.
buildSocketIndex mapped NUMA node number to CPU by treating dmidecode's
Socket Designation (often 1-indexed, "CPU1"/"CPU2") as equal to the
NUMA node number (always 0-indexed) — GPUs/NICs on NUMA node 0 fell
into the "unknown" column, others attached to the wrong CPU box. Now
ranks CPUs by Socket value instead of assuming a shared numbering base.
Fixed a bug in ComponentStatusDB/applyComponentStatusDB where GPU SAT
results were keyed per-target ("pcie:gpu:nvidia-stress") instead of
per-vendor, which both broke cross-tier severity tracking (a later
clean "2. Check" run and an earlier failing "3. Load" run never
compared severities) and silently failed to match any real BDF, so the
DB overlay never reached the topology graph at all. GPU keys are now
normalized to vendor ("pcie:gpu:nvidia"/"pcie:gpu:amd"). Also skip
writing to the DB when a SAT task was aborted by the user (ctx
canceled), so a partial run can't stomp a previously recorded status.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a GPU Settings card to /tools for the settings that actually
persist on NVIDIA data-center GPUs: ECC mode, MIG mode, and
Confidential Computing mode (all stored in the GPU's inforom/firmware,
take effect after a GPU reset or reboot) plus power limit (does not
persist — reapplied on demand). Includes a one-click "Reset All to
Defaults" that restores factory settings across every visible GPU,
touching only whatever has actually drifted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
storcli64 enumerates these controllers but reports zero drives; storcli2
is the tool Broadcom ships for Tri-Mode/MegaRAID8 hardware and uses a
compatible JSON schema for drive listing. Wires storcli2 into both the
collector (structured drive data) and the webui RAID Management page
(dedup so a controller isn't double-listed if storcli64 already sees it
with zero drives), plus techdump raw collection and the ISO vendor-tool
build step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New read-only visualization page: CPU sockets as anchor nodes, PCIe
devices (GPU/NIC/RAID) linked to their NUMA-affine socket with edge
color derived strictly from link_speed vs max_link_speed (not from the
device's own Status, which can also be overwritten by SAT-test results
on the same field), PSU/BMC as standalone boxes with no connecting
line, and a separate NVLink Topology card (live nvidia-smi topo -m /
nvlink -s/-e queries, not persisted to any contract).
GPU-GPU edges are drawn strictly from the actual bonded-pair list
parsed out of "nvidia-smi topo -m" (parseGPUPairAdjacency), not from
adjacent box position in the layout — an earlier ASCII mockup drew a
"chain" through unrelated GPUs, which a dedicated regression test now
guards against. A bonded pair spanning two different NUMA nodes is
flagged Warning on the edge and on both GPU boxes, per project
decision that this is an anomaly worth surfacing, not a neutral fact.
Zero changes to the ingest contract: this reverts the HardwareNVLinkPort/
HardwarePCIeDevice.NVLinks field shipped in v11.55 (33d6eee) along with
its collector/nvidia.go enrichment — that field risked a 400 from
Reanimator Core's strict decoder without an RFC, and isn't needed since
the topo page queries nvidia-smi directly instead of reading it from
audit.json. The v11.55 systemd fix (bee-nvidia.service ordering,
nv-hostengine restart) and the nvlink-status/-errors/dcgmi dumps in the
support bundle are untouched.
Also reorganizes support bundle collection per project convention:
system/ is now LiveCD-operational logs only (Xorg, services, console,
network/FS of the host itself); all server-hardware dumps (lspci,
NVIDIA/NVLink/DCGM, fabric manager, PCIe AER, ethtool, mstflint) move
to techdump/, deduplicating two entries already produced by
platform/techdump.go. Adds nvidia-bug-report.sh (previously only
collected inside the NVIDIA SAT pack) and lscpu to the always-on dump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New read-only "Check" step reports whether this server can run NVIDIA
Confidential Computing: CPU TEE support (Intel TDX / AMD SEV-SNP, via
dmesg and kvm_amd sysfs params) and GPU firmware CC capability (via
`nvidia-smi conf-compute -q`). Also collect that command's output into
the techdump export bundle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Product Asset Tag (p 5) and the repeated custom "Extra" fields (Product
Extra p 7, Board Extra b 5/6/7, Chassis Extra c 2/3) from the Inspur FRU
field doc weren't writable — ipmitool prints identically-named lines for
each custom field with no index of its own, so a plain name lookup
couldn't tell them apart. parseFRUOutput now counts occurrences per area
to recover the real index, and the existing area/index round-trip in the
FRU editor write path picks it up automatically. Out-of-band (-H/-U/-P)
writing remains out of scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RAID Controller Management previously hid any LSI drive that wasn't
already Frgn/UGood/JBOD, and scoped VROC "free drives" from all system
disks instead of the ones actually wired to the VROC controller's
ports - drives attached directly to the CPU or another HBA could leak
in. Now every drive is listed per its own controller, and LSI drives
not already ready for array creation get a "Prepare" button that
forces them to Unconfigured Good via storcli.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Check mode: read-only SMART/NVMe data collection, no self-test.
Load mode: same collection + short self-test (nvme device-self-test -s 1,
smartctl -t short). Card descriptions updated accordingly.
After each storage SAT run, a disk-N-devname-report.txt is written
per device into the runDir (auto-included in support bundles).
Web UI task page renders one card per disk directly below Task Report.
Also fixes pre-existing TestDashboardRendersRuntimeHealthTable failure:
test fixture used "inactive" status but code now treats inactive as OK
for completed oneshot services; updated to "failed" to match intent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Show an explicit per-source status line after "Read All" instead of hiding
failed/blocked sources in a "(skipped: …)" tail. Sources blocked by a missing
Supermicro license (SFT-OOB-LIC / SFT-DCMS-SINGLE) are flagged in red with an
actionable message, so engineers see that SAA DMI is gated rather than silently
falling back to the futile ipmitool FRU path (BIOS re-syncs FRU from DMI on boot).
Also fix TestDashboardRendersRuntimeHealthTable, stale since 4f6579e moved
"inactive" to the OK service states: the fixture now uses a failed service and
the assertion matches the current contract (failed flagged, inactive not).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces separate IPMI FRU and SAA DMI cards with a single FRU / Elabel
card that reads all available sources in parallel and shows each field
with a color-coded source chip (IPMI FRU / Huawei iBMC / SAA DMI).
Huawei elabel fields are read/written via OEM IPMI raw commands
(NetFn 0x30, cmd 0x90) with 19-byte chunking protocol, matching
the FusionServer ElabelTool V511 wire format. Covers DeviceName,
DeviceSerialNumber, ProductName, ProductSerialNumber, ProductAssetTag,
ProductManufacturer, MainboardManufacturer, BoardProductName,
ChassisPartnumber, ChassisType (read-only), IOChassisSerial,
IOChassisAssetTag, and GUID (read-only via standard 0x06 0x08).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace global Save button with per-row ✓ (save) / ✗ (cancel) buttons
that appear only when a field is changed
- All fields shown as editable inputs; server rejects unknown fields
with a clear error message instead of hiding them in the UI
- Monospace font and 1.5px border for all value inputs
- Server-side name→area/index lookup for fields sent without area
- SAA DMI card: same per-row UX, confirm dialog kept (requires reboot)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
