Commit Graph
6 Commits
Author SHA1 Message Date
Mikhail ChusavitinandClaude Sonnet 5 b11018ac5e fix(collector): stop pinning GPU PCIe status on an unverified idle reading
NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power
saving, and applyPCIeLinkSpeedWarning fired on every idle collector
pass regardless - since component-status DB records never downgrade
(Record() only ever raises severity), one boot-time idle sample
permanently pinned pcie:gpu:nvidia to Warning for the rest of the
session even after every load-bearing GPU SAT test passed clean. Two
prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag)
didn't hold up against this hardware/driver combination - see
bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the
full history.

Rather than add a downgrade path, stop writing an unverified status in
the first place: parseLspciDevice no longer calls
applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay
populated as plain descriptive fields; only a verified-under-load
caller may now turn them into a status verdict.

Two new SAT targets provide that verified signal:

- pcie-link (platform/pcie_link_check.go): forces every enabled PCIe
  device - not just GPUs - to retrain via the PCIe spec's Link Control
  "Retrain Link" bit, then compares the negotiated speed against the
  device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn
  equivalent load tool. Classifies by PCI class code + vendor ID, not
  name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into
  their own component-status keys so a degraded NIC never reads as a
  GPU fault.
- nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives
  real host<->device traffic via dcgmi diag -r nvbandwidth and
  resamples link speed immediately after, independent of nvbandwidth's
  own pass/fail.

Both wired into the task queue/webui the same way as nvidia-config
(routes, dispatch, priority, Validate page cards, Run All Check SAT).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-24 18:39:19 +03:00
Mikhail ChusavitinandClaude Sonnet 5 bcf02e0515 webui: add "6. Scenario" page — run scriptable test scenarios from removable media through the normal task queue
Exposes the scenario engine (platform.System.RunScenario, added earlier)
in the web UI instead of only the `bee run` CLI: a new nav item lists every
scenarios/*.json found on mounted removable media (GET /api/scenario/list)
and runs one with a click (POST /api/scenario/run), enqueued as a normal
Task with target "scenario" — progress/logs live in Tasks like any other
SAT pack, no separate live-output UI needed.

- app.go: satRunner gains RunScenario, exportManager gains
  ListScenarioFilesOnRemovableMedia/ReadScenarioFromRemovableMedia — both
  already implemented on platform.System, just newly exposed through App.
- webui/tasks.go: taskParams.ScenarioName; runTask's "scenario" case reads
  the file from removable media, parses it, and runs it.
- webui/page_scenario.go: the page itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-28 18:58:11 +03:00
Mikhail ChusavitinandClaude Sonnet 5 cfaa15ec7c platform/webui: fold confidential-computing into a GPU config + NVLink check
The standalone "confidential-computing" SAT target only ever checked CC
readiness, which most fleets never opt into (a NOT_READY verdict there
isn't a fault). Meanwhile DCGM diag never asserts GPU config compliance
(ECC/MIG/power-limit vs factory default) or NVLink topology (per NVIDIA's
own DGX BasePOD deployment guide, this needs a separate validation step)
— gaps confirmed against public DCGM docs and a real NV17-vs-expected-NV18
bonded pair found on a live bundle.

Repurposes the routine into "nvidia-config": reuses the existing
ListNvidiaGPUSettings() (already backing the GPU-settings page) to flag
ECC disabled, a MIG mode change stuck pending a reset/reboot, and a power
limit capped >5% below default; parses "nvidia-smi topo -m" bonded pairs
against "nvlink -s/-e" to flag any inactive lane or nonzero replay/
recovery/CRC counter on an otherwise-active bond. CC readiness is folded
in as one informational field (does not gate overall_status) rather than
a dedicated test. Reports under the same pcie:gpu:nvidia severity key as
every other nvidia-* SAT target instead of an isolated key, so a
config/NVLink FAILED result isn't invisible next to stress-test results.

Also fixes ApplySATResultToDB silently dropping any target with no
matching switch case (exactly what the old confidential-computing target
did) with a new coverage test enumerating every real SAT target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 11:36:52 +03:00
Mikhail ChusavitinandClaude Sonnet 5 53c46465d2 webui: add persistent GPU settings management (ECC/MIG/CC/power limit)
Adds a GPU Settings card to /tools for the settings that actually
persist on NVIDIA data-center GPUs: ECC mode, MIG mode, and
Confidential Computing mode (all stored in the GPU's inforom/firmware,
take effect after a GPU reset or reboot) plus power limit (does not
persist — reapplied on demand). Includes a one-click "Reset All to
Defaults" that restores factory settings across every visible GPU,
touching only whatever has actually drifted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 15:25:25 +03:00
Mikhail ChusavitinandClaude Sonnet 5 abaeaea13f add Confidential Computing readiness check + collect nvidia-smi conf-compute -q
New read-only "Check" step reports whether this server can run NVIDIA
Confidential Computing: CPU TEE support (Intel TDX / AMD SEV-SNP, via
dmesg and kvm_amd sysfs params) and GPU firmware CC capability (via
`nvidia-smi conf-compute -q`). Also collect that command's output into
the techdump export bundle.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-02 19:18:41 +03:00
mchusandClaude Sonnet 4.6 7d2e904d14 Bring codebase into compliance with bible contracts (A–E)
A (hardware-ingest-json v2.8-2.9): remove sensor location fields from schema
and collector; tag HardwareMemory.Location as json:"-"; add PlatformConfig to
HardwareSnapshot.

B (no-hardcoded-vendors): consolidate PCI vendor IDs into collector/pci_vendors.go;
replace all vendor-name string checks in isGPUDevice, isNVIDIADevice, isMellanoxDevice,
isAMDGPUDevice, matchesGPUVendor (sat_overlay), and validateIsVendorGPU (page_validate)
with numeric vendor_id comparisons.

C (module-structure): split app/app.go (1413 lines) into app.go + app_format.go,
app_network.go, app_services.go, app_packs.go, app_install.go — no logic changes.

D (go-code-style): wrap bare return err in interfaceAdminState and
interfaceIPv4Addrs (platform/network.go) with fmt.Errorf context including
the interface name.

E (go-project-bible): add bible-local/architecture/data-model.md and
bible-local/architecture/api-surface.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 14:32:08 +03:00