reanimator/bee
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

app/webui: replace generic "see summary.txt" SAT failure text with the real reason

Browse Source 
Every place that surfaced a FAILED SAT result — task error messages,
component-status.json detail, and the hardware snapshot's ErrorDescription/
StatusHistory — used to say only "SAT overall_status=FAILED (see
summary.txt)" or "<label> failed", forcing an engineer to go dig through the
run directory to find out what actually broke.

nvidia-config's summary.txt now carries a "warnings" field with the specific
GPU/NVLink finding. A new SATFailureDetail/satFailureDetailFromKV in
component_status_db.go reads that field, or falls back to naming whichever
generic SAT sub-job(s) reported non-OK/UNSUPPORTED status along with their
exit code. This feeds both the task-runner error message and the component
status DB. sat_overlay.go's satKeyStatus (which drives ErrorDescription on
the exported hardware snapshot) now does the same, with storage kept
per-device so one drive's rc doesn't get attributed to another's card.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-07-09 15:48:12 +03:00
co-authored by Claude Sonnet 5
parent 5aee146903
commit b30d34199a
8 changed files with 265 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
audit/internal/app/component_status_db_test.go
+58
View File
@@ -98,6 +98,64 @@ func TestApplyComponentStatusDBMatchesGPUByVendor(t *testing.T) {
func strPtr(s string) *string { return &s }
// TestSATFailureDetailPrefersWarningsField guards the real bug this
// exercises: an engineer looking at a failed nvidia-config task used to see
// only "SAT overall_status=FAILED (see summary.txt)" — no indication of
// which GPU or NVLink pair was the problem. nvidia-config's summary.txt
// (nvidia_config_check.go's renderNvidiaConfigCheckSummary) writes the
// specific reason into a "warnings" field; SATFailureDetail must surface it
// verbatim instead of falling through to the generic sub-job scan.
func TestSATFailureDetailPrefersWarningsField(t *testing.T) {
runDir := t.TempDir()
summary := "run_at_utc=2026-07-09T18:40:57Z\n" +
"nvlink_pairs_checked=1\n" +
"nvlink_pairs_with_issues=1\n" +
"overall_status=FAILED\n" +
"warnings=NVLink GPU0<->GPU1: 2/36 NVLinks inactive on a bonded pair\n"
if err := os.WriteFile(filepath.Join(runDir, "summary.txt"), []byte(summary), 0644); err != nil {
t.Fatal(err)
}
want := "NVLink GPU0<->GPU1: 2/36 NVLinks inactive on a bonded pair"
if got := SATFailureDetail(runDir); got != want {
t.Fatalf("SATFailureDetail() = %q, want %q", got, want)
}
}
// TestSATFailureDetailFallsBackToFailedSubJobs guards generic SAT acceptance
// packs (nvidia, memory, cpu, storage, ...), which have no "warnings" field
// — they record one "<job>_status"/"<job>_rc" pair per sub-job instead.
// SATFailureDetail must name the specific job(s) that failed rather than
// telling the reader nothing beyond "see summary.txt".
func TestSATFailureDetailFallsBackToFailedSubJobs(t *testing.T) {
runDir := t.TempDir()
summary := "run_at_utc=2026-07-09T18:40:57Z\n" +
"nvidia-smi-q_rc=0\n" +
"nvidia-smi-q_status=OK\n" +
"bee-gpu-burn_rc=1\n" +
"bee-gpu-burn_status=FAILED\n" +
"job_ok=1\n" +
"job_failed=1\n" +
"overall_status=FAILED\n"
if err := os.WriteFile(filepath.Join(runDir, "summary.txt"), []byte(summary), 0644); err != nil {
t.Fatal(err)
}
want := "failed sub-job(s): bee-gpu-burn=FAILED (rc=1)"
if got := SATFailureDetail(runDir); got != want {
t.Fatalf("SATFailureDetail() = %q, want %q", got, want)
}
}
// TestSATFailureDetailEmptyWhenNoReasonFound guards the ultimate fallback:
// when summary.txt carries no identifiable per-job or warnings detail (e.g.
// unreadable or from an older binary version), callers must get "" so they
// know to fall back to their own generic message rather than silently
// printing an empty explanation.
func TestSATFailureDetailEmptyWhenNoReasonFound(t *testing.T) {
if got := SATFailureDetail(filepath.Join(t.TempDir(), "does-not-exist")); got != "" {
t.Fatalf("SATFailureDetail() = %q, want empty for missing summary.txt", got)
}
}
// TestApplySATResultToDBCoversAllHealthCheckTargets guards against a target
// silently falling through ApplySATResultToDB's switch with no matching
// case — exactly what happened to the old "confidential-computing" target