Mikhail ChusavitinandClaude Sonnet 5
7ed652c5b1
Extract the fan peak-tracking into observedPeakStore (observe max under load, hold >= minHold to reject spikes, round, persist JSON) and add a second instance for PSU draw (psu-observation.json, keyed by PSU ordinal). Fed from samplePSUPower like fans are from sampleFanSpeeds, so any full-load run refines it — the Fan Ceiling Check (which also samples PSU power at a slow cadence off its loop and writes psu_<i>_peak_w), a burn, thermal cycling, and the 5s web metrics collector. /topo PSU cards now scale the load fill by wattage_w when the BMC reports it, else by the observed peak draw — marked "~N% load". This MSI stand's BMC gives only instantaneous input power, so the observed peak is the only capacity figure available. Fan behaviour is unchanged (tests exercise updateFanObservation / estimateFanDutyCyclePctFromObservation / ResolveFanMaxRPM through the new store). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019VHG21rgTUiR1G3qFHTVmN
bee — Project Bible
Project-specific architecture, decisions, and runtime contracts.
Generic engineering rules live in bible/rules/patterns/.
Files
| File | Contents |
|---|---|
architecture/system-overview.md |
What bee does, scope, tech stack |
architecture/runtime-flows.md |
Boot sequence, audit flow, service order |
architecture/squashfs-layers.md |
Semantic SquashFS layer model, ownership, order, verification |
docs/customer-gpu-test-methodology.md |
Customer-facing GPU PCIe Validate / Validate -> Stress test list |
docs/hardware-ingest-contract.md |
Current Reanimator hardware ingest JSON contract |
docs/validate-vs-burn.md |
Validate and Validate -> Stress hardware test policy |
decisions/ |
Architectural decision log, including read-only submodule policy |
proposals/ |
RFCs and contract change proposals for Reanimator Core |
Validate Test Matrix
Validate
- CPU check
lscpusensorsstress-ng
- TPM check (read-only)
tpm2_getcap properties-fixedtpm2_getcap pcrstpm2_pcrreadtpm2_gettestresult(reads the existing result; does not startTPM2_SelfTest)
- Memory check
freetimeout <timeout_sec> memtesterfree
- NVMe storage check
nvme id-ctrlnvme smart-lognvme device-self-test
- SATA/SAS storage check
smartctl -H -Asmartctl -t short
- Basic NVIDIA GPU check
nvidia-smi -pm 1nvidia-smi -qdmidecode -t baseboarddmidecode -t systemdcgmi diag -r 2
- Inter-GPU communication check
all_reduce_perf
- GPU bandwidth check
dcgmi diag -r nvbandwidth(per CPU socket, then all selected GPUs, on multi-socket systems -- seedecisions/2026-07-27-nvbandwidth-per-socket-split.md)
Validate -> Stress
- Extended NVIDIA GPU check
nvidia-smi -pm 1nvidia-smi -qdmidecode -t baseboarddmidecode -t systemdcgmi diag -r 3
- NVIDIA targeted stress
nvidia-smi -pm 1nvidia-smi -qdcgmi diag -r targeted_stress
- NVIDIA targeted power
nvidia-smi -pm 1nvidia-smi -qdcgmi diag -r targeted_power
- NVIDIA pulse test
nvidia-smi -pm 1nvidia-smi -qdcgmi diag -r pulse_test
- Inter-GPU communication check
all_reduce_perf
- GPU bandwidth check
dcgmi diag -r nvbandwidth(per CPU socket, then all selected GPUs, on multi-socket systems -- seedecisions/2026-07-27-nvbandwidth-per-socket-split.md)
- Fan ceiling check (Load tier /
3. Loadonly)stressapptest(orstress-ng) + the hottest GPU load —dcgmproftester -t 1004/targeted_power(Power/Thermal Fit engine), notbee-gpu-burn— run simultaneouslyipmitool sdr type Fanon an adaptive interval (1 s floor, backs off to 30 s when the BMC gets slow, every read time-boxed) until every fan plateaus (~1 min flat, only trusted while telemetry is healthy) or the 15 min cap- records each fan's observed peak RPM to the fan-observation store (used by the Topology fan tiles: size ∝ ceiling, fill ∝ duty cycle); FAIL only on a fan at 0 RPM / IPMI cr-nr under load; cancelled ("not applicable"), never failed, if the host cannot be loaded or has no fan sensors
- see
decisions/2026-09-04-fan-ceiling-check.md