Files

T

Mikhail ChusavitinandClaude Sonnet 5 8cb250f3f4 fix(sat): fan check uses hottest GPU load + IPMI-hang-proof polling

- GPU load switches from bee-gpu-burn (compute burn, ~88% TDP) to
  dcgmproftester -t 1004 / targeted_power via
  resolveBenchmarkPowerLoadCommand — the same engine Power/Thermal Fit
  uses and the hottest sustained NVIDIA load we have, so fans are
  actually pushed toward their ceiling.
- Sample loop is now IPMI-hang-proof: every ipmitool read is time-boxed
  in an abandonable goroutine, and the poll interval backs off
  geometrically (1s→30s) when reads are slow, tightening again on
  recovery. A plateau is only trusted while telemetry is healthy;
  degraded runs ride out to MaxLoadSec. Summary gains fan_samples /
  telemetry_degraded. Drops the per-second nvidia-smi+power+cpu-temp
  sampling from the hot loop.
- Dead code removed: FanStressRow, GPUStressMetric, sampleFanStressRow,
  sampleGPUStressMetrics, WriteFanStressCSV/WriteFanSensorsCSV,
  analyzeMaxTemp, sampleSystemPowerResolved.

Topology fan tiles:
- size encodes the fan's ceiling RPM (its class), not current speed;
  coloured fill rising from the bottom encodes live duty cycle
  (current / ceiling), shown only when the ceiling was measured.
- glyph spin rate now maps absolute RPM into a human-perceptible band
  (fanSpinPeriodSec: 2.2s/turn at <=1000 RPM, 0.35s at >=13000).
- the "N fans · N OK · tile size ∝ …" caption line is gone.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN

2026-09-04 11:02:31 +03:00

4.3 KiB

Raw Blame History

Decision: Fan check discovers the RPM ceiling by CPU+GPU load, not by forcing fans over the BMC

Date: 2026-09-04 Status: active

Context

We want a SAT test that pushes every system fan to its top speed, records that speed as the fan's ceiling, and flags a fan that will not spin. The recorded ceiling is what the Topology view sizes each fan tile against (platform.ResolveFanMaxRPM).

The obvious approach — force the fans to 100% PWM over IPMI/Redfish and read the resulting RPM — does not work on our current platform:

Stand: MSI G4201 / MS-S3831, AMI MegaRAC BMC (fw 1.08).
Every documented host-side fan-control OEM command returns rsp=0xc1 Invalid command: Supermicro 0x30 0x45, ASRock/AMI reference 0x3a 0x01 / 0xd0 0x12 / 0xd0 0x0f / 0xd6 / 0xd7 / 0xda, 0x30 0x30..0x32. BIOS KCS Access Control Policy = Allow All, so this is not KCS filtering — MSI simply does not implement them.
The MSI G4201 Redfish API guide documents Thermal as GET only; no fan-mode / fan-PWM PATCH endpoint exists.
The only fan knob in BIOS is Fan PWM Offset (0–100, additive to the auto curve, reboot-gated) — not a runtime 100% force.

Decision

The fan SAT test (platform.RunFanCheck, formerly the unwired RunFanStressTest) drives load, not the BMC:

Start stressapptest (CPU + memory) and, when a GPU is present, the hottest sustained GPU load — dcgmproftester -t 1004 / targeted_power, via resolveBenchmarkPowerLoadCommand, the same engine Power/Thermal Fit uses — simultaneously, each in its own goroutine. NOT bee-gpu-burn: that is a compute-throughput burn that tops out around 88% of TDP (measured ~525 W of 600 W on RTX PRO 6000 Blackwell) and never makes the fans demand their true ceiling. A first run on the MSI stand with bee-gpu-burn peaked F2U fans at 24 400 RPM vs a historical 26 000. A missing GPU is not an error — CPU/memory load alone exercises the cooling loop. Load sources reach full power at different times, so the plateau clock only starts once every launched source reports its process running (plus a fixed GPU ramp grace).
Sample every fan on an adaptive interval (floor 1 s). Each read is time-boxed (readFansBounded — a goroutine abandoned on timeout, so a KCS-wedged ipmitool can never block the loop); when reads are slow the interval backs off geometrically to 30 s and tightens again when they recover. Under 8-GPU + CPU load on the MSI stand, ipmitool sdr type Fan took ~14 s/call and at one point wedged for minutes — without this the "1 Hz" sampler silently degraded to one sample per 14 s and the plateau timer ran on stale data. A plateau is only declared while telemetry is healthy (interval near the floor, ≥5 recent samples); a degraded run just rides out to MaxLoadSec and records the peak it saw. Per fan, track the peak RPM and the last time it climbed by more than PlateauDeltaRPM (default 50).
When no fan has climbed for PlateauHoldSec (default 60 s) and at least one fan rose meaningfully above baseline, declare the ceiling found and stop — success. MaxLoadSec (default 900 s) is a hard cap; hitting it is also success (the highest RPM seen is still recorded).
Persist each peak through the existing updateFanObservation path (/var/log/bee-sat/fan-observation.json), which is what ResolveFanMaxRPM reads.

Verdict mapping

A fan reading 0 RPM, or IPMI status cr/nr, while under full load → FAILED (dead / stuck rotor).
No load source available (no stressapptest/stress-ng and no GPU burn tool), or no fan sensors at all → platform.ErrTestNotApplicable, which the task layer lands as cancelled ("not applicable"), never failed, with a detailed log. An engineer must not see a red for "this platform can't run the test".

Tier

Load only (/load, "3. Load"). It is a sustained full-load test, so it does not belong on the read-only Check page. Added to the stress-mode Run All.

Consequences

The ceiling is observed, not a spec value — only as high as the load drove the fans. Good enough for tile sizing and stuck-fan detection.
If a future platform does expose a safe host-side fan force, revisit: a real 100% force is a stronger test than load-driven ramp.

4.3 KiB Raw Blame History Unescape Escape