Mikhail ChusavitinandClaude Sonnet 5
8cb250f3f4
- GPU load switches from bee-gpu-burn (compute burn, ~88% TDP) to dcgmproftester -t 1004 / targeted_power via resolveBenchmarkPowerLoadCommand — the same engine Power/Thermal Fit uses and the hottest sustained NVIDIA load we have, so fans are actually pushed toward their ceiling. - Sample loop is now IPMI-hang-proof: every ipmitool read is time-boxed in an abandonable goroutine, and the poll interval backs off geometrically (1s→30s) when reads are slow, tightening again on recovery. A plateau is only trusted while telemetry is healthy; degraded runs ride out to MaxLoadSec. Summary gains fan_samples / telemetry_degraded. Drops the per-second nvidia-smi+power+cpu-temp sampling from the hot loop. - Dead code removed: FanStressRow, GPUStressMetric, sampleFanStressRow, sampleGPUStressMetrics, WriteFanStressCSV/WriteFanSensorsCSV, analyzeMaxTemp, sampleSystemPowerResolved. Topology fan tiles: - size encodes the fan's ceiling RPM (its class), not current speed; coloured fill rising from the bottom encodes live duty cycle (current / ceiling), shown only when the ceiling was measured. - glyph spin rate now maps absolute RPM into a human-perceptible band (fanSpinPeriodSec: 2.2s/turn at <=1000 RPM, 0.35s at >=13000). - the "N fans · N OK · tile size ∝ …" caption line is gone. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
85 lines
4.3 KiB
Markdown
85 lines
4.3 KiB
Markdown
# Decision: Fan check discovers the RPM ceiling by CPU+GPU load, not by forcing fans over the BMC
|
||
|
||
**Date:** 2026-09-04
|
||
**Status:** active
|
||
|
||
## Context
|
||
|
||
We want a SAT test that pushes every system fan to its top speed, records that
|
||
speed as the fan's ceiling, and flags a fan that will not spin. The recorded
|
||
ceiling is what the Topology view sizes each fan tile against
|
||
(`platform.ResolveFanMaxRPM`).
|
||
|
||
The obvious approach — force the fans to 100% PWM over IPMI/Redfish and read the
|
||
resulting RPM — does not work on our current platform:
|
||
|
||
- Stand: MSI G4201 / MS-S3831, AMI MegaRAC BMC (fw 1.08).
|
||
- Every documented host-side fan-control OEM command returns
|
||
`rsp=0xc1 Invalid command`: Supermicro `0x30 0x45`, ASRock/AMI reference
|
||
`0x3a 0x01 / 0xd0 0x12 / 0xd0 0x0f / 0xd6 / 0xd7 / 0xda`, `0x30 0x30..0x32`.
|
||
BIOS `KCS Access Control Policy = Allow All`, so this is not KCS filtering —
|
||
MSI simply does not implement them.
|
||
- The MSI G4201 Redfish API guide documents Thermal as **GET only**; no
|
||
fan-mode / fan-PWM PATCH endpoint exists.
|
||
- The only fan knob in BIOS is `Fan PWM Offset` (0–100, additive to the auto
|
||
curve, reboot-gated) — not a runtime 100% force.
|
||
|
||
## Decision
|
||
|
||
The `fan` SAT test (`platform.RunFanCheck`, formerly the unwired
|
||
`RunFanStressTest`) drives load, not the BMC:
|
||
|
||
1. Start `stressapptest` (CPU + memory) and, when a GPU is present, the
|
||
**hottest sustained GPU load** — `dcgmproftester -t 1004` /
|
||
`targeted_power`, via `resolveBenchmarkPowerLoadCommand`, the same engine
|
||
Power/Thermal Fit uses — **simultaneously**, each in its own goroutine.
|
||
NOT `bee-gpu-burn`: that is a compute-throughput burn that tops out around
|
||
88% of TDP (measured ~525 W of 600 W on RTX PRO 6000 Blackwell) and never
|
||
makes the fans demand their true ceiling. A first run on the MSI stand with
|
||
`bee-gpu-burn` peaked F2U fans at 24 400 RPM vs a historical 26 000.
|
||
A missing GPU is not an error — CPU/memory load alone exercises the cooling
|
||
loop. Load sources reach full power at different times, so the plateau clock
|
||
only starts once every launched source reports its process running (plus a
|
||
fixed GPU ramp grace).
|
||
2. Sample every fan on an adaptive interval (floor 1 s). Each read is
|
||
time-boxed (`readFansBounded` — a goroutine abandoned on timeout, so a
|
||
KCS-wedged `ipmitool` can never block the loop); when reads are slow the
|
||
interval backs off geometrically to 30 s and tightens again when they
|
||
recover. Under 8-GPU + CPU load on the MSI stand, `ipmitool sdr type Fan`
|
||
took ~14 s/call and at one point wedged for minutes — without this the
|
||
"1 Hz" sampler silently degraded to one sample per 14 s and the plateau
|
||
timer ran on stale data. A plateau is only declared while telemetry is
|
||
healthy (interval near the floor, ≥5 recent samples); a degraded run just
|
||
rides out to `MaxLoadSec` and records the peak it saw. Per fan, track the
|
||
peak RPM and the last time it climbed by more than `PlateauDeltaRPM`
|
||
(default 50).
|
||
3. When no fan has climbed for `PlateauHoldSec` (default 60 s) and at least one
|
||
fan rose meaningfully above baseline, declare the ceiling found and stop —
|
||
**success**. `MaxLoadSec` (default 900 s) is a hard cap; hitting it is also
|
||
success (the highest RPM seen is still recorded).
|
||
4. Persist each peak through the existing `updateFanObservation` path
|
||
(`/var/log/bee-sat/fan-observation.json`), which is what `ResolveFanMaxRPM`
|
||
reads.
|
||
|
||
## Verdict mapping
|
||
|
||
- A fan reading **0 RPM**, or IPMI status **cr/nr**, while under full load →
|
||
`FAILED` (dead / stuck rotor).
|
||
- No load source available (no `stressapptest`/`stress-ng` and no GPU burn
|
||
tool), or no fan sensors at all → `platform.ErrTestNotApplicable`, which the
|
||
task layer lands as **cancelled ("not applicable")**, never failed, with a
|
||
detailed log. An engineer must not see a red for "this platform can't run the
|
||
test".
|
||
|
||
## Tier
|
||
|
||
Load only (`/load`, "3. Load"). It is a sustained full-load test, so it does
|
||
not belong on the read-only Check page. Added to the stress-mode `Run All`.
|
||
|
||
## Consequences
|
||
|
||
- The ceiling is *observed*, not a spec value — only as high as the load drove
|
||
the fans. Good enough for tile sizing and stuck-fan detection.
|
||
- If a future platform does expose a safe host-side fan force, revisit: a real
|
||
100% force is a stronger test than load-driven ramp.
|