Mikhail ChusavitinandClaude Sonnet 5
7ed652c5b1
Extract the fan peak-tracking into observedPeakStore (observe max under load, hold >= minHold to reject spikes, round, persist JSON) and add a second instance for PSU draw (psu-observation.json, keyed by PSU ordinal). Fed from samplePSUPower like fans are from sampleFanSpeeds, so any full-load run refines it — the Fan Ceiling Check (which also samples PSU power at a slow cadence off its loop and writes psu_<i>_peak_w), a burn, thermal cycling, and the 5s web metrics collector. /topo PSU cards now scale the load fill by wattage_w when the BMC reports it, else by the observed peak draw — marked "~N% load". This MSI stand's BMC gives only instantaneous input power, so the observed peak is the only capacity figure available. Fan behaviour is unchanged (tests exercise updateFanObservation / estimateFanDutyCyclePctFromObservation / ResolveFanMaxRPM through the new store). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
108 lines
5.5 KiB
Markdown
108 lines
5.5 KiB
Markdown
# Decision: Fan check discovers the RPM ceiling by CPU+GPU load, not by forcing fans over the BMC
|
||
|
||
**Date:** 2026-09-04
|
||
**Status:** active
|
||
|
||
## Context
|
||
|
||
We want a SAT test that pushes every system fan to its top speed, records that
|
||
speed as the fan's ceiling, and flags a fan that will not spin. The recorded
|
||
ceiling is what the Topology view sizes each fan tile against
|
||
(`platform.ResolveFanMaxRPM`).
|
||
|
||
The obvious approach — force the fans to 100% PWM over IPMI/Redfish and read the
|
||
resulting RPM — does not work on our current platform:
|
||
|
||
- Stand: MSI G4201 / MS-S3831, AMI MegaRAC BMC (fw 1.08).
|
||
- Every documented host-side fan-control OEM command returns
|
||
`rsp=0xc1 Invalid command`: Supermicro `0x30 0x45`, ASRock/AMI reference
|
||
`0x3a 0x01 / 0xd0 0x12 / 0xd0 0x0f / 0xd6 / 0xd7 / 0xda`, `0x30 0x30..0x32`.
|
||
BIOS `KCS Access Control Policy = Allow All`, so this is not KCS filtering —
|
||
MSI simply does not implement them.
|
||
- The MSI G4201 Redfish API guide documents Thermal as **GET only**; no
|
||
fan-mode / fan-PWM PATCH endpoint exists.
|
||
- The only fan knob in BIOS is `Fan PWM Offset` (0–100, additive to the auto
|
||
curve, reboot-gated) — not a runtime 100% force.
|
||
|
||
## Decision
|
||
|
||
The `fan` SAT test (`platform.RunFanCheck`, formerly the unwired
|
||
`RunFanStressTest`) drives load, not the BMC:
|
||
|
||
1. Start `stressapptest` (CPU + memory) and, when a GPU is present, the
|
||
**hottest sustained GPU load** — `dcgmproftester -t 1004` /
|
||
`targeted_power`, via `resolveBenchmarkPowerLoadCommand`, the same engine
|
||
Power/Thermal Fit uses — **simultaneously**, each in its own goroutine.
|
||
NOT `bee-gpu-burn`: that is a compute-throughput burn that tops out around
|
||
88% of TDP (measured ~525 W of 600 W on RTX PRO 6000 Blackwell) and never
|
||
makes the fans demand their true ceiling. A first run on the MSI stand with
|
||
`bee-gpu-burn` peaked F2U fans at 24 400 RPM vs a historical 26 000.
|
||
A missing GPU is not an error — CPU/memory load alone exercises the cooling
|
||
loop. Load sources reach full power at different times, so the plateau clock
|
||
only starts once every launched source reports its process running (plus a
|
||
fixed GPU ramp grace).
|
||
2. Sample every fan on an adaptive interval (floor 1 s). Each read is
|
||
time-boxed (`readFansBounded` — a goroutine abandoned on timeout, so a
|
||
KCS-wedged `ipmitool` can never block the loop); when reads are slow the
|
||
interval backs off geometrically to 30 s and tightens again when they
|
||
recover. Under 8-GPU + CPU load on the MSI stand, `ipmitool sdr type Fan`
|
||
took ~14 s/call and at one point wedged for minutes — without this the
|
||
"1 Hz" sampler silently degraded to one sample per 14 s and the plateau
|
||
timer ran on stale data. A plateau is only declared while telemetry is
|
||
healthy (interval near the floor, ≥5 recent samples); a degraded run just
|
||
rides out to `MaxLoadSec` and records the peak it saw. Per fan, track the
|
||
peak RPM and the last time it climbed by more than `PlateauDeltaRPM`
|
||
(default 50).
|
||
3. When no fan has climbed for `PlateauHoldSec` (default 60 s) and at least one
|
||
fan rose meaningfully above baseline, declare the ceiling found and stop —
|
||
**success**. `MaxLoadSec` (default 900 s) is a hard cap; hitting it is also
|
||
success (the highest RPM seen is still recorded).
|
||
4. Persist each peak through `updateFanObservation`
|
||
(`/var/log/bee-sat/fan-observation.json`), which is what `ResolveFanMaxRPM`
|
||
reads.
|
||
|
||
## The observed-peak store (autotune primitive)
|
||
|
||
`platform.observedPeakStore` (`observed_peaks.go`) is the shared mechanism:
|
||
observe the max value per key while the box is under load, require a candidate
|
||
to hold ≥ `minHold` before it sticks (rejects spikes), round up, persist to a
|
||
`{"<jsonKey>": {key: peak}}` JSON file. Two instances:
|
||
|
||
- `fanPeaks` → `fan-observation.json` `max_rpm`, keyed by fan name, round-up 1000.
|
||
- `psuPeaks` → `psu-observation.json` `max_w`, keyed by **PSU ordinal**, round-up 50.
|
||
|
||
Both are fed from the ordinary telemetry paths — `sampleFanSpeeds` and
|
||
`samplePSUPower` call `update*Observation` — so **any** full-load run refines
|
||
them: the Fan Ceiling Check itself, a burn, thermal cycling, power
|
||
calibration, and the 5 s web metrics collector while any of those run. There
|
||
is no separate "PSU autotune" test: the fan check's max-CPU+GPU load is
|
||
already the right moment to observe peak PSU draw, and it samples PSU power at
|
||
a slow cadence off its own loop (`psu_<i>_peak_w` in the summary).
|
||
|
||
This exists because BMCs like the MSI stand's report only instantaneous PSU
|
||
input power, no nameplate rating. `/topo` PSU cards scale the load fill by the
|
||
real `wattage_w` when present, otherwise by `ObservedPSUMaxW()` — marked as an
|
||
estimate (`~N% load`).
|
||
|
||
## Verdict mapping
|
||
|
||
- A fan reading **0 RPM**, or IPMI status **cr/nr**, while under full load →
|
||
`FAILED` (dead / stuck rotor).
|
||
- No load source available (no `stressapptest`/`stress-ng` and no GPU burn
|
||
tool), or no fan sensors at all → `platform.ErrTestNotApplicable`, which the
|
||
task layer lands as **cancelled ("not applicable")**, never failed, with a
|
||
detailed log. An engineer must not see a red for "this platform can't run the
|
||
test".
|
||
|
||
## Tier
|
||
|
||
Load only (`/load`, "3. Load"). It is a sustained full-load test, so it does
|
||
not belong on the read-only Check page. Added to the stress-mode `Run All`.
|
||
|
||
## Consequences
|
||
|
||
- The ceiling is *observed*, not a spec value — only as high as the load drove
|
||
the fans. Good enough for tile sizing and stuck-fan detection.
|
||
- If a future platform does expose a safe host-side fan force, revisit: a real
|
||
100% force is a stronger test than load-driven ramp.
|