Commit Graph
5 Commits
Author SHA1 Message Date
Mikhail ChusavitinandClaude Sonnet 5 8cb250f3f4 fix(sat): fan check uses hottest GPU load + IPMI-hang-proof polling
- GPU load switches from bee-gpu-burn (compute burn, ~88% TDP) to
  dcgmproftester -t 1004 / targeted_power via
  resolveBenchmarkPowerLoadCommand — the same engine Power/Thermal Fit
  uses and the hottest sustained NVIDIA load we have, so fans are
  actually pushed toward their ceiling.
- Sample loop is now IPMI-hang-proof: every ipmitool read is time-boxed
  in an abandonable goroutine, and the poll interval backs off
  geometrically (1s→30s) when reads are slow, tightening again on
  recovery. A plateau is only trusted while telemetry is healthy;
  degraded runs ride out to MaxLoadSec. Summary gains fan_samples /
  telemetry_degraded. Drops the per-second nvidia-smi+power+cpu-temp
  sampling from the hot loop.
- Dead code removed: FanStressRow, GPUStressMetric, sampleFanStressRow,
  sampleGPUStressMetrics, WriteFanStressCSV/WriteFanSensorsCSV,
  analyzeMaxTemp, sampleSystemPowerResolved.

Topology fan tiles:
- size encodes the fan's ceiling RPM (its class), not current speed;
  coloured fill rising from the bottom encodes live duty cycle
  (current / ceiling), shown only when the ceiling was measured.
- glyph spin rate now maps absolute RPM into a human-perceptible band
  (fanSpinPeriodSec: 2.2s/turn at <=1000 RPM, 0.35s at >=13000).
- the "N fans · N OK · tile size ∝ …" caption line is gone.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
2026-09-04 11:02:31 +03:00
Mikhail ChusavitinandClaude Sonnet 5 bb2a501a28 feat(sat): fan ceiling check + topology fan tiles
Repurpose the previously-unwired RunFanStressTest into RunFanCheck, a
Load-tier SAT test that drives stressapptest (CPU+memory) and, when a GPU
is present, a GPU burn to 100% simultaneously, then watches every fan
until none has climbed for ~60s. The observed peak RPM per fan is the
"ceiling"; it is persisted through the existing fan-observation store.

MSI G4201 / AMI MegaRAC exposes no host-side fan force (every OEM IPMI
command returns 0xc1; Redfish Thermal is GET-only), so load-driven ramp
is the closest safe equivalent. See
bible-local/decisions/2026-09-04-fan-ceiling-check.md.

- platform.ResolveFanMaxRPM: per-fan max with fallback (persisted peak ->
  peer peak -> current RPM), resolved in platform, not the view.
- platform.ErrTestNotApplicable: no load source or no fan sensors ->
  task lands as cancelled ("not applicable"), never failed, so an
  engineer never sees a false red. executeTaskWithOptions maps the
  sentinel; finalizeTaskForResult honours a pre-set TaskCancelled.
- Verdict FAIL only for a fan at 0 RPM / IPMI cr-nr under load.
- /topo: one small spinning square per fan, sized by RPM / resolved max,
  clickable through to a new "fan" component-detail type; per-fan status
  recorded to the component-status DB from the fan SAT summary.
- Wiring: /api/sat/fan/run route, "fan" task target, Load-page card,
  stress-mode Run All.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
2026-09-04 10:35:34 +03:00
Mikhail ChusavitinandClaude Sonnet 5 319f08ac4b feat(topo): per-socket bar layout with disks under their controller
Rework the /topo diagram from side-by-side stacked cards into one tall
vertical bar per CPU socket with everything attached to it branching off
sideways (socket 0 left/branches right, socket 1 right/branches left).

Disks are now parented under the storage controller they physically hang
off (SATA/AHCI, SAS HBA, RAID) — itself a NUMA-affine PCIe device under one
socket — instead of a synthetic catch-all node. The disk->controller link
is read from a new storage-controllers.txt techdump
(platform.StorageControllerMapScript, a /sys/block walk); disks with no
resolvable controller fall back to an "Other" bar. No board/root node.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYjrG6bVmeG1Z2Wmc8kg3o
2026-09-03 12:15:49 +03:00
Mikhail Chusavitin 0a6ca8ba0f refactor: harden diagnostics and consolidate runtime code 2026-09-01 13:01:28 +03:00
Mikhail Chusavitin ac4bc0b2b7 refactor: modularize audit and harden build validation 2026-08-31 21:22:16 +03:00