# Decision: Fan check discovers the RPM ceiling by CPU+GPU load, not by forcing fans over the BMC

**Date:** 2026-09-04
**Status:** active

## Context

We want a SAT test that pushes every system fan to its top speed, records that
speed as the fan's ceiling, and flags a fan that will not spin. The recorded
ceiling is what the Topology view sizes each fan tile against
(`platform.ResolveFanMaxRPM`).

The obvious approach — force the fans to 100% PWM over IPMI/Redfish and read the
resulting RPM — does not work on our current platform:

- Stand: MSI G4201 / MS-S3831, AMI MegaRAC BMC (fw 1.08).
- Every documented host-side fan-control OEM command returns
  `rsp=0xc1 Invalid command`: Supermicro `0x30 0x45`, ASRock/AMI reference
  `0x3a 0x01 / 0xd0 0x12 / 0xd0 0x0f / 0xd6 / 0xd7 / 0xda`, `0x30 0x30..0x32`.
  BIOS `KCS Access Control Policy = Allow All`, so this is not KCS filtering —
  MSI simply does not implement them.
- The MSI G4201 Redfish API guide documents Thermal as **GET only**; no
  fan-mode / fan-PWM PATCH endpoint exists.
- The only fan knob in BIOS is `Fan PWM Offset` (0–100, additive to the auto
  curve, reboot-gated) — not a runtime 100% force.

## Decision

The `fan` SAT test (`platform.RunFanCheck`, formerly the unwired
`RunFanStressTest`) drives load, not the BMC:

1. Start `stressapptest` (CPU + memory) and, when a GPU is present, the
   **hottest sustained GPU load** — `dcgmproftester -t 1004` /
   `targeted_power`, via `resolveBenchmarkPowerLoadCommand`, the same engine
   Power/Thermal Fit uses — **simultaneously**, each in its own goroutine.
   NOT `bee-gpu-burn`: that is a compute-throughput burn that tops out around
   88% of TDP (measured ~525 W of 600 W on RTX PRO 6000 Blackwell) and never
   makes the fans demand their true ceiling. A first run on the MSI stand with
   `bee-gpu-burn` peaked F2U fans at 24 400 RPM vs a historical 26 000.
   A missing GPU is not an error — CPU/memory load alone exercises the cooling
   loop. Load sources reach full power at different times, so the plateau clock
   only starts once every launched source reports its process running (plus a
   fixed GPU ramp grace).
2. Sample every fan on an adaptive interval (floor 1 s). Each read is
   time-boxed (`readFansBounded` — a goroutine abandoned on timeout, so a
   KCS-wedged `ipmitool` can never block the loop); when reads are slow the
   interval backs off geometrically to 30 s and tightens again when they
   recover. Under 8-GPU + CPU load on the MSI stand, `ipmitool sdr type Fan`
   took ~14 s/call and at one point wedged for minutes — without this the
   "1 Hz" sampler silently degraded to one sample per 14 s and the plateau
   timer ran on stale data. A plateau is only declared while telemetry is
   healthy (interval near the floor, ≥5 recent samples); a degraded run just
   rides out to `MaxLoadSec` and records the peak it saw. Per fan, track the
   peak RPM and the last time it climbed by more than `PlateauDeltaRPM`
   (default 50).
3. When no fan has climbed for `PlateauHoldSec` (default 60 s) and at least one
   fan rose meaningfully above baseline, declare the ceiling found and stop —
   **success**. `MaxLoadSec` (default 900 s) is a hard cap; hitting it is also
   success (the highest RPM seen is still recorded).
4. Persist each peak through `updateFanObservation`
   (`/var/log/bee-sat/fan-observation.json`), which is what `ResolveFanMaxRPM`
   reads.

## The observed-peak store (autotune primitive)

`platform.observedPeakStore` (`observed_peaks.go`) is the shared mechanism:
observe the max value per key while the box is under load, require a candidate
to hold ≥ `minHold` before it sticks (rejects spikes), round up, persist to a
`{"<jsonKey>": {key: peak}}` JSON file. Two instances:

- `fanPeaks` → `fan-observation.json` `max_rpm`, keyed by fan name, round-up 1000.
- `psuPeaks` → `psu-observation.json` `max_w`, keyed by **PSU ordinal**, round-up 50.

Both are fed from the ordinary telemetry paths — `sampleFanSpeeds` and
`samplePSUPower` call `update*Observation` — so **any** full-load run refines
them: the Fan Ceiling Check itself, a burn, thermal cycling, power
calibration, and the 5 s web metrics collector while any of those run. There
is no separate "PSU autotune" test: the fan check's max-CPU+GPU load is
already the right moment to observe peak PSU draw, and it samples PSU power at
a slow cadence off its own loop (`psu_<i>_peak_w` in the summary).

This exists because BMCs like the MSI stand's report only instantaneous PSU
input power, no nameplate rating. `/topo` PSU cards scale the load fill by the
real `wattage_w` when present, otherwise by `ObservedPSUMaxW()` — marked as an
estimate (`~N% load`).

## Verdict mapping

- A fan reading **0 RPM**, or IPMI status **cr/nr**, while under full load →
  `FAILED` (dead / stuck rotor).
- No load source available (no `stressapptest`/`stress-ng` and no GPU burn
  tool), or no fan sensors at all → `platform.ErrTestNotApplicable`, which the
  task layer lands as **cancelled ("not applicable")**, never failed, with a
  detailed log. An engineer must not see a red for "this platform can't run the
  test".

## Tier

Load only (`/load`, "3. Load"). It is a sustained full-load test, so it does
not belong on the read-only Check page. Added to the stress-mode `Run All`.

## Consequences

- The ceiling is *observed*, not a spec value — only as high as the load drove
  the fans. Good enough for tile sizing and stuck-fan detection.
- If a future platform does expose a safe host-side fan force, revisit: a real
  100% force is a stronger test than load-driven ramp.