A CPU that had thermally throttled (sysfs thermal_throttle counter >
0) still reported status "OK" everywhere: dmidecode-derived CPU status
only distinguishes populated/enabled/disabled and never looked at the
throttle flag the collector already recorded next to it, and neither
SAT path meant to catch this actually could:
- The routine "cpu" SAT pack (RunCPUAcceptancePack) only checked
lscpu/sensors/stress-ng exit codes — stress-ng exits 0 whether or
not the CPU throttled while running it, so an 89°C/throttled CPU
right after a "successful" run still showed cpu:all as OK in
component-status.json.
- The more thorough platform-stress test already detected throttling
and fan-spindown correctly, but wrote its verdict as "Overall: FAIL
— ..." with no "=", which parseSATKV can't parse — so even a real
detected throttle event never reached the component-status DB.
Fixes:
- cpu_telemetry.go: escalate a CPU's status to Warning (only-escalate,
same severity ranking already used elsewhere) when Throttled is set.
- sat.go: add a before/after thermal-throttle-counter check job around
the "cpu" pack's stress-ng run, so a throttle event during the run
fails that job and (via the existing FAILED->Warning DB mapping)
flips cpu:all to Warning.
- platform_stress.go: emit a machine-readable overall_status= line
alongside the human-readable verdict so platform-stress results
actually reach ApplySATResultToDB.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers the sat.go fix from 2599d9c: a failing "dcgmi discovery -l"
preflight job must not flip the pack's overall status, and jobs marked
with retries should recover from a transient first-attempt failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dcgmi diag's failures only surface a terse wrapper message ("Detected
driver major version 0 is not between the required versions 346 and
2000"), which doesn't say why. Reading DCGM's own nvvs source
(NvidiaValidationSuite.cpp::CheckDriverVersion) shows this fires when
dcgmSystem.GetDeviceAttributes(0, ...) comes back empty — the version
string is never actually malformed. Diagnosing further requires info
bee never collected: whether dcgmi discovery even sees GPU 0/1
correctly, and nvvs's own internal debug log (normally
/var/log/nvidia-dcgm/nvvs.log, which bee never captured).
Add a "dcgmi discovery -l" job before every DCGM diag invocation
(check-gpu-dcgm-l2, nvbandwidth, targeted_power, pulse_test,
targeted_stress), and route dcgmi diag through
"-v -d DEBUG --debugLogFile {{run_dir}}/..." so the full nvvs debug
log lands in the SAT run dir and gets picked up by the support bundle
automatically, instead of needing a live SSH session on hardware we
usually can't get a second look at.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Check mode: read-only SMART/NVMe data collection, no self-test.
Load mode: same collection + short self-test (nvme device-self-test -s 1,
smartctl -t short). Card descriptions updated accordingly.
After each storage SAT run, a disk-N-devname-report.txt is written
per device into the runDir (auto-included in support bundles).
Web UI task page renders one card per disk directly below Task Report.
Also fixes pre-existing TestDashboardRendersRuntimeHealthTable failure:
test fixture used "inactive" status but code now treats inactive as OK
for completed oneshot services; updated to "failed" to match intent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace diag level 1-4 dropdown with Validate/Stress radio buttons
- Validate: dcgmi L2, 60s CPU, 256MB/1p memtester, SMART short
- Stress: dcgmi L3 + targeted_stress in Run All, 30min CPU, 1GB/3p memtester, SMART long/NVMe extended
- Parallel GPU mode: spawn single task for all GPUs instead of splitting per model
- Benchmark table: per-GPU columns for sequential runs, server-wide column for parallel
- Benchmark report converted to Markdown with server model, GPU model, version in header; only steady-state charts
- Fix IPMI power parsing in benchmark (was looking for 'Current Power', correct field is 'Instantaneous power reading')
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mikhail Chusavitin
mchusandClaude Sonnet 4.6