bee/bible-local/decisions/2026-09-04-fan-ceiling-check.md

Decision: Fan check discovers the RPM ceiling by CPU+GPU load, not by forcing fans over the BMC

Date: 2026-09-04 Status: active

Context

We want a SAT test that pushes every system fan to its top speed, records that speed as the fan's ceiling, and flags a fan that will not spin. The recorded ceiling is what the Topology view sizes each fan tile against (platform.ResolveFanMaxRPM).

The obvious approach — force the fans to 100% PWM over IPMI/Redfish and read the resulting RPM — does not work on our current platform:

• Stand: MSI G4201 / MS-S3831, AMI MegaRAC BMC (fw 1.08).
• Every documented host-side fan-control OEM command returns rsp=0xc1 Invalid command: Supermicro 0x30 0x45, ASRock/AMI reference 0x3a 0x01 / 0xd0 0x12 / 0xd0 0x0f / 0xd6 / 0xd7 / 0xda, 0x30 0x30..0x32. BIOS KCS Access Control Policy = Allow All, so this is not KCS filtering — MSI simply does not implement them.
• The MSI G4201 Redfish API guide documents Thermal as GET only; no fan-mode / fan-PWM PATCH endpoint exists.
• The only fan knob in BIOS is Fan PWM Offset (0–100, additive to the auto curve, reboot-gated) — not a runtime 100% force.

Decision

The fan SAT test (platform.RunFanCheck, formerly the unwired RunFanStressTest) drives load, not the BMC:

1. Start stressapptest (CPU + memory) and, when a GPU is present, the hottest sustained GPU load — dcgmproftester -t 1004 / targeted_power, via resolveBenchmarkPowerLoadCommand, the same engine Power/Thermal Fit uses — simultaneously, each in its own goroutine. NOT bee-gpu-burn: that is a compute-throughput burn that tops out around 88% of TDP (measured ~525 W of 600 W on RTX PRO 6000 Blackwell) and never makes the fans demand their true ceiling. A first run on the MSI stand with bee-gpu-burn peaked F2U fans at 24 400 RPM vs a historical 26 000. A missing GPU is not an error — CPU/memory load alone exercises the cooling loop. Load sources reach full power at different times, so the plateau clock only starts once every launched source reports its process running (plus a fixed GPU ramp grace).
2. Sample every fan on an adaptive interval (floor 1 s). Each read is time-boxed (readFansBounded — a goroutine abandoned on timeout, so a KCS-wedged ipmitool can never block the loop); when reads are slow the interval backs off geometrically to 30 s and tightens again when they recover. Under 8-GPU + CPU load on the MSI stand, ipmitool sdr type Fan took ~14 s/call and at one point wedged for minutes — without this the "1 Hz" sampler silently degraded to one sample per 14 s and the plateau timer ran on stale data. A plateau is only declared while telemetry is healthy (interval near the floor, ≥5 recent samples); a degraded run just rides out to MaxLoadSec and records the peak it saw. Per fan, track the peak RPM and the last time it climbed by more than PlateauDeltaRPM (default 50).
3. When no fan has climbed for PlateauHoldSec (default 60 s) and at least one fan rose meaningfully above baseline, declare the ceiling found and stop — success. MaxLoadSec (default 900 s) is a hard cap; hitting it is also success (the highest RPM seen is still recorded).
4. Persist each peak through updateFanObservation (/var/log/bee-sat/fan-observation.json), which is what ResolveFanMaxRPM reads.

The observed-peak store (autotune primitive)

platform.observedPeakStore (observed_peaks.go) is the shared mechanism: observe the max value per key while the box is under load, require a candidate to hold ≥ minHold before it sticks (rejects spikes), round up, persist to a {"<jsonKey>": {key: peak}} JSON file. Two instances:

• fanPeaks → fan-observation.json max_rpm, keyed by fan name, round-up 1000.
• psuPeaks → psu-observation.json max_w, keyed by PSU ordinal, round-up 50.

Both are fed from the ordinary telemetry paths — sampleFanSpeeds and samplePSUPower call update*Observation — so any full-load run refines them: the Fan Ceiling Check itself, a burn, thermal cycling, power calibration, and the 5 s web metrics collector while any of those run. There is no separate "PSU autotune" test: the fan check's max-CPU+GPU load is already the right moment to observe peak PSU draw, and it samples PSU power at a slow cadence off its own loop (psu_<i>_peak_w in the summary).

This exists because BMCs like the MSI stand's report only instantaneous PSU input power, no nameplate rating. /topo PSU cards scale the load fill by the real wattage_w when present, otherwise by ObservedPSUMaxW() — marked as an estimate (~N% load).

Verdict mapping

• A fan reading 0 RPM, or IPMI status cr/nr, while under full load → FAILED (dead / stuck rotor).
• No load source available (no stressapptest/stress-ng and no GPU burn tool), or no fan sensors at all → platform.ErrTestNotApplicable, which the task layer lands as cancelled ("not applicable"), never failed, with a detailed log. An engineer must not see a red for "this platform can't run the test".

Tier

Load only (/load, "3. Load"). It is a sustained full-load test, so it does not belong on the read-only Check page. Added to the stress-mode Run All.

Consequences

• The ceiling is observed, not a spec value — only as high as the load drove the fans. Good enough for tile sizing and stuck-fan detection.
• If a future platform does expose a safe host-side fan force, revisit: a real 100% force is a stronger test than load-driven ramp.