reanimator/bee
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(sat): fan ceiling check + topology fan tiles

Browse Source 
Repurpose the previously-unwired RunFanStressTest into RunFanCheck, a
Load-tier SAT test that drives stressapptest (CPU+memory) and, when a GPU
is present, a GPU burn to 100% simultaneously, then watches every fan
until none has climbed for ~60s. The observed peak RPM per fan is the
"ceiling"; it is persisted through the existing fan-observation store.

MSI G4201 / AMI MegaRAC exposes no host-side fan force (every OEM IPMI
command returns 0xc1; Redfish Thermal is GET-only), so load-driven ramp
is the closest safe equivalent. See
bible-local/decisions/2026-09-04-fan-ceiling-check.md.

- platform.ResolveFanMaxRPM: per-fan max with fallback (persisted peak ->
  peer peak -> current RPM), resolved in platform, not the view.
- platform.ErrTestNotApplicable: no load source or no fan sensors ->
  task lands as cancelled ("not applicable"), never failed, so an
  engineer never sees a false red. executeTaskWithOptions maps the
  sentinel; finalizeTaskForResult honours a pre-set TaskCancelled.
- Verdict FAIL only for a fan at 0 RPM / IPMI cr-nr under load.
- /topo: one small spinning square per fan, sized by RPM / resolved max,
  clickable through to a new "fan" component-detail type; per-fan status
  recorded to the component-status DB from the fan SAT summary.
- Wiring: /api/sat/fan/run route, "fan" task target, Load-page card,
  stress-mode Run All.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
This commit is contained in:
Mikhail Chusavitin
2026-09-04 10:35:34 +03:00
co-authored by Claude Sonnet 5
parent e4f7519ef3
commit bb2a501a28
22 changed files with 798 additions and 222 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bible-local/README.md
+6
View File
@@ -9,6 +9,7 @@ Generic engineering rules live in `bible/rules/patterns/`.
|---|---|
| `architecture/system-overview.md` | What bee does, scope, tech stack |
| `architecture/runtime-flows.md` | Boot sequence, audit flow, service order |
| `architecture/squashfs-layers.md` | Semantic SquashFS layer model, ownership, order, verification |
| `docs/customer-gpu-test-methodology.md` | Customer-facing GPU PCIe Validate / Validate -> Stress test list |
| `docs/hardware-ingest-contract.md` | Current Reanimator hardware ingest JSON contract |
| `docs/validate-vs-burn.md` | Validate and Validate -> Stress hardware test policy |
@@ -74,3 +75,8 @@ Generic engineering rules live in `bible/rules/patterns/`.
- `all_reduce_perf`
- GPU bandwidth check
- `dcgmi diag -r nvbandwidth` (per CPU socket, then all selected GPUs, on multi-socket systems -- see `decisions/2026-07-27-nvbandwidth-per-socket-split.md`)
- Fan ceiling check (Load tier / `3. Load` only)
- `stressapptest` (or `stress-ng`) + `bee-gpu-burn` / `rvs gst` at 100%, run simultaneously
- `ipmitool sdr type Fan` sampled 1 Hz until every fan plateaus (~1 min flat) or the 15 min cap
- records each fan's observed peak RPM to `/var/log/bee-sat/fan-observation.json` (used by the Topology fan tiles); FAIL only on a fan at 0 RPM / IPMI cr-nr under load; **cancelled ("not applicable")**, never failed, if the host cannot be loaded or has no fan sensors
- see `decisions/2026-09-04-fan-ceiling-check.md`