feat(sat): fan ceiling check + topology fan tiles

Repurpose the previously-unwired RunFanStressTest into RunFanCheck, a
Load-tier SAT test that drives stressapptest (CPU+memory) and, when a GPU
is present, a GPU burn to 100% simultaneously, then watches every fan
until none has climbed for ~60s. The observed peak RPM per fan is the
"ceiling"; it is persisted through the existing fan-observation store.

MSI G4201 / AMI MegaRAC exposes no host-side fan force (every OEM IPMI
command returns 0xc1; Redfish Thermal is GET-only), so load-driven ramp
is the closest safe equivalent. See
bible-local/decisions/2026-09-04-fan-ceiling-check.md.

- platform.ResolveFanMaxRPM: per-fan max with fallback (persisted peak ->
  peer peak -> current RPM), resolved in platform, not the view.
- platform.ErrTestNotApplicable: no load source or no fan sensors ->
  task lands as cancelled ("not applicable"), never failed, so an
  engineer never sees a false red. executeTaskWithOptions maps the
  sentinel; finalizeTaskForResult honours a pre-set TaskCancelled.
- Verdict FAIL only for a fan at 0 RPM / IPMI cr-nr under load.
- /topo: one small spinning square per fan, sized by RPM / resolved max,
  clickable through to a new "fan" component-detail type; per-fan status
  recorded to the component-status DB from the fan SAT summary.
- Wiring: /api/sat/fan/run route, "fan" task target, Load-page card,
  stress-mode Run All.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
This commit is contained in:
Mikhail Chusavitin
2026-09-04 10:35:34 +03:00
co-authored by Claude Sonnet 5
parent e4f7519ef3
commit bb2a501a28
22 changed files with 798 additions and 222 deletions
@@ -0,0 +1,68 @@
# Decision: Fan check discovers the RPM ceiling by CPU+GPU load, not by forcing fans over the BMC
**Date:** 2026-09-04
**Status:** active
## Context
We want a SAT test that pushes every system fan to its top speed, records that
speed as the fan's ceiling, and flags a fan that will not spin. The recorded
ceiling is what the Topology view sizes each fan tile against
(`platform.ResolveFanMaxRPM`).
The obvious approach — force the fans to 100% PWM over IPMI/Redfish and read the
resulting RPM — does not work on our current platform:
- Stand: MSI G4201 / MS-S3831, AMI MegaRAC BMC (fw 1.08).
- Every documented host-side fan-control OEM command returns
`rsp=0xc1 Invalid command`: Supermicro `0x30 0x45`, ASRock/AMI reference
`0x3a 0x01 / 0xd0 0x12 / 0xd0 0x0f / 0xd6 / 0xd7 / 0xda`, `0x30 0x30..0x32`.
BIOS `KCS Access Control Policy = Allow All`, so this is not KCS filtering —
MSI simply does not implement them.
- The MSI G4201 Redfish API guide documents Thermal as **GET only**; no
fan-mode / fan-PWM PATCH endpoint exists.
- The only fan knob in BIOS is `Fan PWM Offset` (0100, additive to the auto
curve, reboot-gated) — not a runtime 100% force.
## Decision
The `fan` SAT test (`platform.RunFanCheck`, formerly the unwired
`RunFanStressTest`) drives load, not the BMC:
1. Start `stressapptest` (CPU + memory) and, when a GPU is present, a GPU burn
(`bee-gpu-burn` / `rvs gst`) **simultaneously**, each in its own goroutine.
A missing GPU is not an error — CPU/memory load alone exercises the cooling
loop. Load sources reach full power at different times, so the plateau clock
only starts once every launched source reports its process running (plus a
fixed GPU ramp grace).
2. Sample every fan once a second. Per fan, track the peak RPM and the last time
it climbed by more than `PlateauDeltaRPM` (default 50).
3. When no fan has climbed for `PlateauHoldSec` (default 60 s) and at least one
fan rose meaningfully above baseline, declare the ceiling found and stop —
**success**. `MaxLoadSec` (default 900 s) is a hard cap; hitting it is also
success (the highest RPM seen is still recorded).
4. Persist each peak through the existing `updateFanObservation` path
(`/var/log/bee-sat/fan-observation.json`), which is what `ResolveFanMaxRPM`
reads.
## Verdict mapping
- A fan reading **0 RPM**, or IPMI status **cr/nr**, while under full load →
`FAILED` (dead / stuck rotor).
- No load source available (no `stressapptest`/`stress-ng` and no GPU burn
tool), or no fan sensors at all → `platform.ErrTestNotApplicable`, which the
task layer lands as **cancelled ("not applicable")**, never failed, with a
detailed log. An engineer must not see a red for "this platform can't run the
test".
## Tier
Load only (`/load`, "3. Load"). It is a sustained full-load test, so it does
not belong on the read-only Check page. Added to the stress-mode `Run All`.
## Consequences
- The ceiling is *observed*, not a spec value — only as high as the load drove
the fans. Good enough for tile sizing and stuck-fan detection.
- If a future platform does expose a safe host-side fan force, revisit: a real
100% force is a stronger test than load-driven ramp.