NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power
saving, and applyPCIeLinkSpeedWarning fired on every idle collector
pass regardless - since component-status DB records never downgrade
(Record() only ever raises severity), one boot-time idle sample
permanently pinned pcie:gpu:nvidia to Warning for the rest of the
session even after every load-bearing GPU SAT test passed clean. Two
prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag)
didn't hold up against this hardware/driver combination - see
bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the
full history.
Rather than add a downgrade path, stop writing an unverified status in
the first place: parseLspciDevice no longer calls
applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay
populated as plain descriptive fields; only a verified-under-load
caller may now turn them into a status verdict.
Two new SAT targets provide that verified signal:
- pcie-link (platform/pcie_link_check.go): forces every enabled PCIe
device - not just GPUs - to retrain via the PCIe spec's Link Control
"Retrain Link" bit, then compares the negotiated speed against the
device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn
equivalent load tool. Classifies by PCI class code + vendor ID, not
name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into
their own component-status keys so a degraded NIC never reads as a
GPU fault.
- nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives
real host<->device traffic via dcgmi diag -r nvbandwidth and
resamples link speed immediately after, independent of nvbandwidth's
own pass/fail.
Both wired into the task queue/webui the same way as nvidia-config
(routes, dispatch, priority, Validate page cards, Run All Check SAT).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
statusSeverity was byte-for-byte duplicated in collector/contract.go and
app/sat_overlay.go (same switch, same comment). Export it as
collector.StatusSeverity and drop the app-package copy.
page_topo.go inlined the same GPU/NIC/RAID class switch as its own
pcieDeviceKind function in one spot — call the function instead.
page_validate.go had its own copy of pages.go's firstNonEmpty (all call
sites already pre-trim their inputs via validateTrimPtr, so the extra
TrimSpace was redundant) — drop the duplicate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
matchesGPUVendor (sat_overlay.go) matched any PCIe DeviceClass containing
the substring "Controller" instead of the actual GPU classes, so a
same-vendor non-GPU device (e.g. an NVIDIA-branded NIC/storage controller)
with a degraded PCIe link surfaced as a pcie:gpu:<vendor> alarm on the
Hardware Summary/webui.
isGPUDevice (app_format.go) had the same bug via a different path: an
"any NVIDIA-vendor device is a GPU" fallback that could inflate the
"GPU: N x <model>" audit summary line with non-GPU companion devices
(NVSwitch/NVLink bridges, etc).
Both now defer to collector.IsGPUClass (exported from the previously
unexported isGPUClass), the canonical exact-match classifier already used
by amdgpu.go — removing the duplicated/incorrect class checks instead of
reimplementing them. Also dedup two more locally-duplicated classifiers
within webui (which doesn't import collector by convention): NIC-class
matching (pages.go/page_topo.go) and GPU-class matching in
page_validate.go now share one copy per package instead of being
reimplemented per file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The standalone "confidential-computing" SAT target only ever checked CC
readiness, which most fleets never opt into (a NOT_READY verdict there
isn't a fault). Meanwhile DCGM diag never asserts GPU config compliance
(ECC/MIG/power-limit vs factory default) or NVLink topology (per NVIDIA's
own DGX BasePOD deployment guide, this needs a separate validation step)
— gaps confirmed against public DCGM docs and a real NV17-vs-expected-NV18
bonded pair found on a live bundle.
Repurposes the routine into "nvidia-config": reuses the existing
ListNvidiaGPUSettings() (already backing the GPU-settings page) to flag
ECC disabled, a MIG mode change stuck pending a reset/reboot, and a power
limit capped >5% below default; parses "nvidia-smi topo -m" bonded pairs
against "nvlink -s/-e" to flag any inactive lane or nonzero replay/
recovery/CRC counter on an otherwise-active bond. CC readiness is folded
in as one informational field (does not gate overall_status) rather than
a dedicated test. Reports under the same pcie:gpu:nvidia severity key as
every other nvidia-* SAT target instead of an isolated key, so a
config/NVLink FAILED result isn't invisible next to stress-test results.
Also fixes ApplySATResultToDB silently dropping any target with no
matching switch case (exactly what the old confidential-computing target
did) with a new coverage test enumerating every real SAT target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New read-only "Check" step reports whether this server can run NVIDIA
Confidential Computing: CPU TEE support (Intel TDX / AMD SEV-SNP, via
dmesg and kvm_amd sysfs params) and GPU firmware CC capability (via
`nvidia-smi conf-compute -q`). Also collect that command's output into
the techdump export bundle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Check mode: read-only SMART/NVMe data collection, no self-test.
Load mode: same collection + short self-test (nvme device-self-test -s 1,
smartctl -t short). Card descriptions updated accordingly.
After each storage SAT run, a disk-N-devname-report.txt is written
per device into the runDir (auto-included in support bundles).
Web UI task page renders one card per disk directly below Task Report.
Also fixes pre-existing TestDashboardRendersRuntimeHealthTable failure:
test fixture used "inactive" status but code now treats inactive as OK
for completed oneshot services; updated to "failed" to match intent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Check (2): validate mode only — no mode switcher, no stress-only cards
(nvidia-targeted-stress, nvidia-targeted-power, nvidia-pulse hidden)
- Load (3): stress mode only — no mode switcher, all cards shown
- satStressMode() hardcoded per page; satModeChanged() removed
- Profile card with radio buttons removed from both pages
- Replaced with simple Run All button + est. time
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the flat menu (Dashboard, Audit, Validate, Burn, Benchmark,
Tasks, Tools) with a numbered progression that guides engineers through
a logical acceptance workflow:
Dashboard (landing) → 1. Audit → 2. Check → 3. Load → 4. Speed
→ 5. Endurance → 6. Tools → 7. Settings
Key changes:
- layout.go: numbered nav labels, new hrefs, Tasks removed from nav
and replaced with a persistent sidebar badge (polls /api/tasks every
5 s, highlights amber when tasks are active)
- server.go: 301 redirects from /validate→/check, /burn→/load,
/benchmark→/speed for backward compatibility
- pages.go: dispatch cases for all new routes; old routes kept as
fallbacks
- page_validate.go: add renderCheck() — non-destructive check page
with validate-mode tests only (no stress toggle, no targeted-stress/
targeted-power/pulse cards)
- page_burn.go: add renderLoad() wrapper; update scope alert to
reference /check instead of /validate
- page_benchmark.go: add renderSpeed() (performance focus) and
renderEndurance() (stability/overnight focus) wrappers
- page_settings.go: new Settings page with blackbox logging toggle,
NVIDIA driver reset, and build info
- server_test.go: update five tests to use new route names and
content expectations
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A (hardware-ingest-json v2.8-2.9): remove sensor location fields from schema
and collector; tag HardwareMemory.Location as json:"-"; add PlatformConfig to
HardwareSnapshot.
B (no-hardcoded-vendors): consolidate PCI vendor IDs into collector/pci_vendors.go;
replace all vendor-name string checks in isGPUDevice, isNVIDIADevice, isMellanoxDevice,
isAMDGPUDevice, matchesGPUVendor (sat_overlay), and validateIsVendorGPU (page_validate)
with numeric vendor_id comparisons.
C (module-structure): split app/app.go (1413 lines) into app.go + app_format.go,
app_network.go, app_services.go, app_packs.go, app_install.go — no logic changes.
D (go-code-style): wrap bare return err in interfaceAdminState and
interfaceIPv4Addrs (platform/network.go) with fmt.Errorf context including
the interface name.
E (go-project-bible): add bible-local/architecture/data-model.md and
bible-local/architecture/api-surface.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nvme-cli emits smart-log counters as JSON strings and uses field names
avail_spare / percent_used instead of the prose names in the NVMe spec.
The nvmeSmartLog struct had int64 fields with wrong JSON tags — Unmarshal
returned an error and the whole health block was skipped, leaving every
NVMe drive with status=Unknown.
Fix: switch all numeric fields to jsonInt64 (already used for lsblk
block sizes) which accepts both bare numbers and quoted strings, and
correct the avail_spare / percent_used tag names.
Also fix validateIsVendorGPU for NVIDIA: previously counted any NVIDIA
PCIe device (including NVSwitch bridges) as a GPU, producing wrong
estimates (12 instead of 8 on an HGX H100 system). Now requires
device_class to be videocontroller or processingaccelerator, matching
the existing AMD filter logic.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
targeted_stress, targeted_power, and the Level 2/3 diag were dispatched
one GPU at a time from the UI, turning a single dcgmi command into 8
sequential ~350–450 s runs. DCGM supports -i with a comma-separated list
of GPU indices and runs the diagnostic on all of them in parallel.
Move nvidia, nvidia-targeted-stress, nvidia-targeted-power into
nvidiaAllGPUTargets so expandSATTarget passes all selected indices in one
API call. Simplify runNvidiaValidateSet to match runNvidiaFabricValidate.
Update sat.go constants and page_validate.go estimates to reflect all-GPU
simultaneous execution (remove n× multiplier from total time estimates).
Stress test on 8-GPU system: ~5.3 h → ~2.5 h.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
