feat(sat): fan ceiling check + topology fan tiles
Repurpose the previously-unwired RunFanStressTest into RunFanCheck, a
Load-tier SAT test that drives stressapptest (CPU+memory) and, when a GPU
is present, a GPU burn to 100% simultaneously, then watches every fan
until none has climbed for ~60s. The observed peak RPM per fan is the
"ceiling"; it is persisted through the existing fan-observation store.
MSI G4201 / AMI MegaRAC exposes no host-side fan force (every OEM IPMI
command returns 0xc1; Redfish Thermal is GET-only), so load-driven ramp
is the closest safe equivalent. See
bible-local/decisions/2026-09-04-fan-ceiling-check.md.
- platform.ResolveFanMaxRPM: per-fan max with fallback (persisted peak ->
peer peak -> current RPM), resolved in platform, not the view.
- platform.ErrTestNotApplicable: no load source or no fan sensors ->
task lands as cancelled ("not applicable"), never failed, so an
engineer never sees a false red. executeTaskWithOptions maps the
sentinel; finalizeTaskForResult honours a pre-set TaskCancelled.
- Verdict FAIL only for a fan at 0 RPM / IPMI cr-nr under load.
- /topo: one small spinning square per fan, sized by RPM / resolved max,
clickable through to a new "fan" component-detail type; per-fan status
recorded to the component-status DB from the fan SAT summary.
- Wiring: /api/sat/fan/run route, "fan" task target, Load-page card,
stress-mode Run All.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
This commit is contained in:
Mikhail Chusavitin
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
e4f7519ef3
commit
bb2a501a28
@@ -9,6 +9,7 @@ Generic engineering rules live in `bible/rules/patterns/`.
|
||||
|---|---|
|
||||
| `architecture/system-overview.md` | What bee does, scope, tech stack |
|
||||
| `architecture/runtime-flows.md` | Boot sequence, audit flow, service order |
|
||||
| `architecture/squashfs-layers.md` | Semantic SquashFS layer model, ownership, order, verification |
|
||||
| `docs/customer-gpu-test-methodology.md` | Customer-facing GPU PCIe Validate / Validate -> Stress test list |
|
||||
| `docs/hardware-ingest-contract.md` | Current Reanimator hardware ingest JSON contract |
|
||||
| `docs/validate-vs-burn.md` | Validate and Validate -> Stress hardware test policy |
|
||||
@@ -74,3 +75,8 @@ Generic engineering rules live in `bible/rules/patterns/`.
|
||||
- `all_reduce_perf`
|
||||
- GPU bandwidth check
|
||||
- `dcgmi diag -r nvbandwidth` (per CPU socket, then all selected GPUs, on multi-socket systems -- see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`)
|
||||
- Fan ceiling check (Load tier / `3. Load` only)
|
||||
- `stressapptest` (or `stress-ng`) + `bee-gpu-burn` / `rvs gst` at 100%, run simultaneously
|
||||
- `ipmitool sdr type Fan` sampled 1 Hz until every fan plateaus (~1 min flat) or the 15 min cap
|
||||
- records each fan's observed peak RPM to `/var/log/bee-sat/fan-observation.json` (used by the Topology fan tiles); FAIL only on a fan at 0 RPM / IPMI cr-nr under load; **cancelled ("not applicable")**, never failed, if the host cannot be loaded or has no fan sensors
|
||||
- see `decisions/2026-09-04-fan-ceiling-check.md`
|
||||
|
||||
@@ -53,6 +53,7 @@ All SAT run endpoints enqueue an async task. Response: `{"task_id": "..."}`.
|
||||
| POST | `/api/sat/memory-stress/run` | Memory stress |
|
||||
| POST | `/api/sat/sat-stress/run` | Combined storage+memory stress |
|
||||
| POST | `/api/sat/platform-stress/run` | Fan + thermal stress |
|
||||
| POST | `/api/sat/fan/run` | Fan ceiling check (CPU+GPU load until fans plateau; Load tier). Not-applicable → task cancelled, not failed. |
|
||||
| POST | `/api/sat/run-all` | Plan + enqueue the whole validate/check set server-side. Body: `{stress_mode, amd_targets[], nvidia_gpu_indices[]}` (operator intent only). Response: `{task_ids[], task_count, notes[]}`. Hardware presence/readiness and which tasks to run are decided by `handler.planSATRunAll`, not the page. |
|
||||
| GET | `/api/sat/stream` | SSE: live SAT log stream |
|
||||
| POST | `/api/sat/abort` | Abort the running SAT task |
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
# Decision: Fan check discovers the RPM ceiling by CPU+GPU load, not by forcing fans over the BMC
|
||||
|
||||
**Date:** 2026-09-04
|
||||
**Status:** active
|
||||
|
||||
## Context
|
||||
|
||||
We want a SAT test that pushes every system fan to its top speed, records that
|
||||
speed as the fan's ceiling, and flags a fan that will not spin. The recorded
|
||||
ceiling is what the Topology view sizes each fan tile against
|
||||
(`platform.ResolveFanMaxRPM`).
|
||||
|
||||
The obvious approach — force the fans to 100% PWM over IPMI/Redfish and read the
|
||||
resulting RPM — does not work on our current platform:
|
||||
|
||||
- Stand: MSI G4201 / MS-S3831, AMI MegaRAC BMC (fw 1.08).
|
||||
- Every documented host-side fan-control OEM command returns
|
||||
`rsp=0xc1 Invalid command`: Supermicro `0x30 0x45`, ASRock/AMI reference
|
||||
`0x3a 0x01 / 0xd0 0x12 / 0xd0 0x0f / 0xd6 / 0xd7 / 0xda`, `0x30 0x30..0x32`.
|
||||
BIOS `KCS Access Control Policy = Allow All`, so this is not KCS filtering —
|
||||
MSI simply does not implement them.
|
||||
- The MSI G4201 Redfish API guide documents Thermal as **GET only**; no
|
||||
fan-mode / fan-PWM PATCH endpoint exists.
|
||||
- The only fan knob in BIOS is `Fan PWM Offset` (0–100, additive to the auto
|
||||
curve, reboot-gated) — not a runtime 100% force.
|
||||
|
||||
## Decision
|
||||
|
||||
The `fan` SAT test (`platform.RunFanCheck`, formerly the unwired
|
||||
`RunFanStressTest`) drives load, not the BMC:
|
||||
|
||||
1. Start `stressapptest` (CPU + memory) and, when a GPU is present, a GPU burn
|
||||
(`bee-gpu-burn` / `rvs gst`) **simultaneously**, each in its own goroutine.
|
||||
A missing GPU is not an error — CPU/memory load alone exercises the cooling
|
||||
loop. Load sources reach full power at different times, so the plateau clock
|
||||
only starts once every launched source reports its process running (plus a
|
||||
fixed GPU ramp grace).
|
||||
2. Sample every fan once a second. Per fan, track the peak RPM and the last time
|
||||
it climbed by more than `PlateauDeltaRPM` (default 50).
|
||||
3. When no fan has climbed for `PlateauHoldSec` (default 60 s) and at least one
|
||||
fan rose meaningfully above baseline, declare the ceiling found and stop —
|
||||
**success**. `MaxLoadSec` (default 900 s) is a hard cap; hitting it is also
|
||||
success (the highest RPM seen is still recorded).
|
||||
4. Persist each peak through the existing `updateFanObservation` path
|
||||
(`/var/log/bee-sat/fan-observation.json`), which is what `ResolveFanMaxRPM`
|
||||
reads.
|
||||
|
||||
## Verdict mapping
|
||||
|
||||
- A fan reading **0 RPM**, or IPMI status **cr/nr**, while under full load →
|
||||
`FAILED` (dead / stuck rotor).
|
||||
- No load source available (no `stressapptest`/`stress-ng` and no GPU burn
|
||||
tool), or no fan sensors at all → `platform.ErrTestNotApplicable`, which the
|
||||
task layer lands as **cancelled ("not applicable")**, never failed, with a
|
||||
detailed log. An engineer must not see a red for "this platform can't run the
|
||||
test".
|
||||
|
||||
## Tier
|
||||
|
||||
Load only (`/load`, "3. Load"). It is a sustained full-load test, so it does
|
||||
not belong on the read-only Check page. Added to the stress-mode `Run All`.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The ceiling is *observed*, not a spec value — only as high as the load drove
|
||||
the fans. Good enough for tile sizing and stuck-fan detection.
|
||||
- If a future platform does expose a safe host-side fan force, revisit: a real
|
||||
100% force is a stronger test than load-driven ramp.
|
||||
Reference in New Issue
Block a user