The fan-ceiling check, the webui metrics collector (every 5s) and the PSU
health poller each shelled out to ipmitool independently. The BMC's KCS
interface serializes those calls anyway, so under load the concurrent
`ipmitool sdr`/`dcmi power reading` invocations just queued behind each
other — that's what produced "IPMI slow" backoff during a fan-ceiling run in
a blackbox dump, while the dashboard looked fine only because it was reading
its own, separately-stale data from a different ipmitool call.
hw_telemetry.go is now the sole recurring poller (fan RPM, temperature, PSU
power/status, DCMI system power), with the adaptive 1s-30s backoff that used
to be duplicated inside the fan check. Every hot-path consumer reads the
shared cache (hwSnapshot / platform.HardwareSDRSnapshot) instead of calling
ipmitool itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The long-lived bee-web process (writing PSU/kmsg watchdog records ~every
60s) and each short-lived "bee bee-worker" SAT-task subprocess each held
an independent in-memory copy of component-status.json. Whichever saved
last won outright, silently erasing whatever the other had just written —
e.g. a GPU SAT task's pcie:gpu:nvidia result vanishing the next time the
PSU watchdog ticked. ComponentStatusDB.Record now reloads on-disk state
(keyed by newer LastCheckedAt) before merging its own update.
Also stops re-logging identical repeat observations to History: the
ingest contract defines status_history as a transition log ("История
переходов статусов"), not a per-poll journal, but Record appended one
entry per call regardless — a continuously-polled PSU grew an unbounded
run of identical "still OK" entries. Now only appends when a source's
last recorded status for a key actually changes.
The PSU watchdog itself now backs off (60s -> doubling, capped at 30min)
while steady and resets to 60s the moment any PSU's status changes, cutting
ipmitool shellouts and file writes for a fleet that's been stable for a
while.
Also adds the missing "raid" case to the component-detail API (was
returning 404 for any RAID card's detail click on /topo).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- kmsg watcher now records kernel errors (GPU Xid, MCE, EDAC, storage I/O) at all times,
not only during SAT tasks; flushImmediate writes directly to ComponentStatusDB
- New health_poller: polls ipmitool sdr every 60s for PSU health (watchdog:psu source)
- Hardware Summary card auto-refreshes every 30s via htmx without page reload
- Component rows (CPU/Memory/Storage/GPU/PSU) are now clickable -- opens a modal
with per-component status, source, timestamp and last 20 history entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
