Files

T

Mikhail ChusavitinandClaude Sonnet 5 bb2a501a28 feat(sat): fan ceiling check + topology fan tiles

Repurpose the previously-unwired RunFanStressTest into RunFanCheck, a
Load-tier SAT test that drives stressapptest (CPU+memory) and, when a GPU
is present, a GPU burn to 100% simultaneously, then watches every fan
until none has climbed for ~60s. The observed peak RPM per fan is the
"ceiling"; it is persisted through the existing fan-observation store.

MSI G4201 / AMI MegaRAC exposes no host-side fan force (every OEM IPMI
command returns 0xc1; Redfish Thermal is GET-only), so load-driven ramp
is the closest safe equivalent. See
bible-local/decisions/2026-09-04-fan-ceiling-check.md.

- platform.ResolveFanMaxRPM: per-fan max with fallback (persisted peak ->
  peer peak -> current RPM), resolved in platform, not the view.
- platform.ErrTestNotApplicable: no load source or no fan sensors ->
  task lands as cancelled ("not applicable"), never failed, so an
  engineer never sees a false red. executeTaskWithOptions maps the
  sentinel; finalizeTaskForResult honours a pre-set TaskCancelled.
- Verdict FAIL only for a fan at 0 RPM / IPMI cr-nr under load.
- /topo: one small spinning square per fan, sized by RPM / resolved max,
  clickable through to a new "fan" component-detail type; per-fan status
  recorded to the component-status DB from the fan SAT summary.
- Wiring: /api/sat/fan/run route, "fan" task target, Load-page card,
  stress-mode Run All.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN

2026-09-04 10:35:34 +03:00

3.2 KiB

Raw Blame History

Decision: Fan check discovers the RPM ceiling by CPU+GPU load, not by forcing fans over the BMC

Date: 2026-09-04 Status: active

Context

We want a SAT test that pushes every system fan to its top speed, records that speed as the fan's ceiling, and flags a fan that will not spin. The recorded ceiling is what the Topology view sizes each fan tile against (platform.ResolveFanMaxRPM).

The obvious approach — force the fans to 100% PWM over IPMI/Redfish and read the resulting RPM — does not work on our current platform:

Stand: MSI G4201 / MS-S3831, AMI MegaRAC BMC (fw 1.08).
Every documented host-side fan-control OEM command returns rsp=0xc1 Invalid command: Supermicro 0x30 0x45, ASRock/AMI reference 0x3a 0x01 / 0xd0 0x12 / 0xd0 0x0f / 0xd6 / 0xd7 / 0xda, 0x30 0x30..0x32. BIOS KCS Access Control Policy = Allow All, so this is not KCS filtering — MSI simply does not implement them.
The MSI G4201 Redfish API guide documents Thermal as GET only; no fan-mode / fan-PWM PATCH endpoint exists.
The only fan knob in BIOS is Fan PWM Offset (0–100, additive to the auto curve, reboot-gated) — not a runtime 100% force.

Decision

The fan SAT test (platform.RunFanCheck, formerly the unwired RunFanStressTest) drives load, not the BMC:

Start stressapptest (CPU + memory) and, when a GPU is present, a GPU burn (bee-gpu-burn / rvs gst) simultaneously, each in its own goroutine. A missing GPU is not an error — CPU/memory load alone exercises the cooling loop. Load sources reach full power at different times, so the plateau clock only starts once every launched source reports its process running (plus a fixed GPU ramp grace).
Sample every fan once a second. Per fan, track the peak RPM and the last time it climbed by more than PlateauDeltaRPM (default 50).
When no fan has climbed for PlateauHoldSec (default 60 s) and at least one fan rose meaningfully above baseline, declare the ceiling found and stop — success. MaxLoadSec (default 900 s) is a hard cap; hitting it is also success (the highest RPM seen is still recorded).
Persist each peak through the existing updateFanObservation path (/var/log/bee-sat/fan-observation.json), which is what ResolveFanMaxRPM reads.

Verdict mapping

A fan reading 0 RPM, or IPMI status cr/nr, while under full load → FAILED (dead / stuck rotor).
No load source available (no stressapptest/stress-ng and no GPU burn tool), or no fan sensors at all → platform.ErrTestNotApplicable, which the task layer lands as cancelled ("not applicable"), never failed, with a detailed log. An engineer must not see a red for "this platform can't run the test".

Tier

Load only (/load, "3. Load"). It is a sustained full-load test, so it does not belong on the read-only Check page. Added to the stress-mode Run All.

Consequences

The ceiling is observed, not a spec value — only as high as the load drove the fans. Good enough for tile sizing and stuck-fan detection.
If a future platform does expose a safe host-side fan force, revisit: a real 100% force is a stronger test than load-driven ramp.

3.2 KiB Raw Blame History Unescape Escape