reanimator/bee
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(sat): fan check uses hottest GPU load + IPMI-hang-proof polling

Browse Source 
- GPU load switches from bee-gpu-burn (compute burn, ~88% TDP) to
  dcgmproftester -t 1004 / targeted_power via
  resolveBenchmarkPowerLoadCommand — the same engine Power/Thermal Fit
  uses and the hottest sustained NVIDIA load we have, so fans are
  actually pushed toward their ceiling.
- Sample loop is now IPMI-hang-proof: every ipmitool read is time-boxed
  in an abandonable goroutine, and the poll interval backs off
  geometrically (1s→30s) when reads are slow, tightening again on
  recovery. A plateau is only trusted while telemetry is healthy;
  degraded runs ride out to MaxLoadSec. Summary gains fan_samples /
  telemetry_degraded. Drops the per-second nvidia-smi+power+cpu-temp
  sampling from the hot loop.
- Dead code removed: FanStressRow, GPUStressMetric, sampleFanStressRow,
  sampleGPUStressMetrics, WriteFanStressCSV/WriteFanSensorsCSV,
  analyzeMaxTemp, sampleSystemPowerResolved.

Topology fan tiles:
- size encodes the fan's ceiling RPM (its class), not current speed;
  coloured fill rising from the bottom encodes live duty cycle
  (current / ceiling), shown only when the ceiling was measured.
- glyph spin rate now maps absolute RPM into a human-perceptible band
  (fanSpinPeriodSec: 2.2s/turn at <=1000 RPM, 0.35s at >=13000).
- the "N fans · N OK · tile size ∝ …" caption line is gone.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
This commit is contained in:
Mikhail Chusavitin
2026-09-04 11:02:31 +03:00
co-authored by Claude Sonnet 5
parent bb2a501a28
commit 8cb250f3f4
6 changed files with 328 additions and 267 deletions
Download Patch File Download Diff File Expand all files Collapse all files
audit/internal/webui/page_validate.go
+3 -3
View File
@@ -117,9 +117,9 @@ func renderValidateMode(opts HandlerOptions, stressDefault bool) string {
)) +
renderSATCard("fan", "Fan Ceiling Check", "runSAT('fan')", "", renderValidateCardBody(
"All system fans reported over IPMI / lm-sensors.",
`Drives CPU (+memory) and, when a GPU is present, GPU load to 100% at the same time and watches every fan until none has climbed for ~1 min. The peak RPM reached is recorded as each fan's ceiling and is what the Topology view sizes the fan tiles against. Success once every fan plateaus (or the time cap is hit). A fan reading 0 RPM or an IPMI status of cr/nr under full load fails. If the host cannot be loaded at all, or exposes no fan sensors, the task is cancelled as "not applicable" rather than failed — the platform does not support forcing fans, so this is the closest safe equivalent.`,
`<code>stressapptest</code> / <code>stress-ng</code> + <code>bee-gpu-burn</code> / <code>rvs gst</code>; <code>ipmitool sdr type Fan</code>`,
`~2–8 min depending on how fast the fan curve settles (hard cap 15 min).`,
`Drives CPU (+memory) and, when a GPU is present, the hottest sustained GPU load (dcgmproftester targeted-power — the Power/Thermal Fit engine) at the same time, and watches every fan until none has climbed for ~1 min. The peak RPM reached is recorded as each fan's ceiling and is what the Topology view sizes the fan tiles and duty-cycle fill against. Success once every fan plateaus (or the time cap is hit). A fan reading 0 RPM or an IPMI status of cr/nr under full load fails. IPMI polling backs off automatically if the BMC gets slow under load. If the host cannot be loaded at all, or exposes no fan sensors, the task is cancelled as "not applicable" rather than failed — the platform does not support forcing fans, so this is the closest safe equivalent.`,
`<code>stressapptest</code> / <code>stress-ng</code> + <code>dcgmproftester -t 1004</code> / <code>rvs gst</code>; <code>ipmitool sdr type Fan</code>`,
`~3–10 min depending on how fast the fan curve settles (hard cap 15 min).`,
))
}