Commit Graph
611 Commits
Author SHA1 Message Date
Mikhail ChusavitinandClaude Sonnet 5 2d84ddb577 webui/topo: read from techdump instead of live nvidia-smi, fix socket/NUMA mismatch, fix cross-tier SAT status merge
/topo blocked the HTTP request on live nvidia-smi calls with no timeout
(topo -m, nvlink -s/-e run on every page load), so a wedged driver hung
the page indefinitely. CaptureTechnicalDump now persists these dumps
once per audit cycle; the page reads them from techdump/ instead.

buildSocketIndex mapped NUMA node number to CPU by treating dmidecode's
Socket Designation (often 1-indexed, "CPU1"/"CPU2") as equal to the
NUMA node number (always 0-indexed) — GPUs/NICs on NUMA node 0 fell
into the "unknown" column, others attached to the wrong CPU box. Now
ranks CPUs by Socket value instead of assuming a shared numbering base.

Fixed a bug in ComponentStatusDB/applyComponentStatusDB where GPU SAT
results were keyed per-target ("pcie:gpu:nvidia-stress") instead of
per-vendor, which both broke cross-tier severity tracking (a later
clean "2. Check" run and an earlier failing "3. Load" run never
compared severities) and silently failed to match any real BDF, so the
DB overlay never reached the topology graph at all. GPU keys are now
normalized to vendor ("pcie:gpu:nvidia"/"pcie:gpu:amd"). Also skip
writing to the DB when a SAT task was aborted by the user (ctx
canceled), so a partial run can't stomp a previously recorded status.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 17:14:03 +03:00
Mikhail ChusavitinandClaude Sonnet 5 f46fc98110 sat: add tests for informational discovery jobs and retry logic
Covers the sat.go fix from 2599d9c: a failing "dcgmi discovery -l"
preflight job must not flip the pack's overall status, and jobs marked
with retries should recover from a transient first-attempt failure.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v12.57
2026-07-07 15:31:31 +03:00
Mikhail ChusavitinandClaude Sonnet 5 53c46465d2 webui: add persistent GPU settings management (ECC/MIG/CC/power limit)
Adds a GPU Settings card to /tools for the settings that actually
persist on NVIDIA data-center GPUs: ECC mode, MIG mode, and
Confidential Computing mode (all stored in the GPU's inforom/firmware,
take effect after a GPU reset or reboot) plus power limit (does not
persist — reapplied on demand). Includes a one-click "Reset All to
Defaults" that restores factory settings across every visible GPU,
touching only whatever has actually drifted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 15:25:25 +03:00
Mikhail ChusavitinandClaude Sonnet 5 2599d9c5e3 sat/nvidia-load: make dcgmi discovery non-fatal and bound systemctl restarts
dcgmi discovery -l is a preflight/metadata step ahead of the real DCGM
diag jobs; a transient failure racing nv-hostengine startup shouldn't
flip the whole pack's status, so it's now marked informational with a
couple of retries. Separately, bound the fabricmanager/nvidia-dcgm
systemctl restart/start calls in bee-nvidia-load with a timeout so a
wedged unit (e.g. fabric training stuck on a bad NVSwitch fabric)
can't hang bee-nvidia.service forever and block dcgm from ever
starting.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 15:25:14 +03:00
Mikhail ChusavitinandClaude Sonnet 5 b7f015c713 raid: add storcli2 support for Tri-Mode controllers (SAS3808-iMR/9500 series)
storcli64 enumerates these controllers but reports zero drives; storcli2
is the tool Broadcom ships for Tri-Mode/MegaRAID8 hardware and uses a
compatible JSON schema for drive listing. Wires storcli2 into both the
collector (structured drive data) and the webui RAID Management page
(dedup so a controller isn't double-listed if storcli64 already sees it
with zero drives), plus techdump raw collection and the ISO vendor-tool
build step.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 15:25:06 +03:00
Mikhail ChusavitinandClaude Sonnet 5 a3377083aa webui: add /topo server topology page
New read-only visualization page: CPU sockets as anchor nodes, PCIe
devices (GPU/NIC/RAID) linked to their NUMA-affine socket with edge
color derived strictly from link_speed vs max_link_speed (not from the
device's own Status, which can also be overwritten by SAT-test results
on the same field), PSU/BMC as standalone boxes with no connecting
line, and a separate NVLink Topology card (live nvidia-smi topo -m /
nvlink -s/-e queries, not persisted to any contract).

GPU-GPU edges are drawn strictly from the actual bonded-pair list
parsed out of "nvidia-smi topo -m" (parseGPUPairAdjacency), not from
adjacent box position in the layout — an earlier ASCII mockup drew a
"chain" through unrelated GPUs, which a dedicated regression test now
guards against. A bonded pair spanning two different NUMA nodes is
flagged Warning on the edge and on both GPU boxes, per project
decision that this is an anomaly worth surfacing, not a neutral fact.

Zero changes to the ingest contract: this reverts the HardwareNVLinkPort/
HardwarePCIeDevice.NVLinks field shipped in v11.55 (33d6eee) along with
its collector/nvidia.go enrichment — that field risked a 400 from
Reanimator Core's strict decoder without an RFC, and isn't needed since
the topo page queries nvidia-smi directly instead of reading it from
audit.json. The v11.55 systemd fix (bee-nvidia.service ordering,
nv-hostengine restart) and the nvlink-status/-errors/dcgmi dumps in the
support bundle are untouched.

Also reorganizes support bundle collection per project convention:
system/ is now LiveCD-operational logs only (Xorg, services, console,
network/FS of the host itself); all server-hardware dumps (lspci,
NVIDIA/NVLink/DCGM, fabric manager, PCIe AER, ethtool, mstflint) move
to techdump/, deduplicating two entries already produced by
platform/techdump.go. Adds nvidia-bug-report.sh (previously only
collected inside the NVIDIA SAT pack) and lscpu to the always-on dump.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.56
2026-07-07 11:50:33 +03:00
Mikhail Chusavitin 33d6eee9cf sat: collect NVLink status/errors, fix nv-hostengine restart via systemd
Add NVLink port status (nvidia-smi nvlink -s), error counters
(nvidia-smi nvlink -e), and dcgmi nvlink status to the support bundle,
and enrich HardwarePCIeDevice entries with per-link telemetry.

Replace the raw nv-hostengine pkill/restart dance in bee-nvidia-load
with systemctl restart/start of nvidia-dcgm.service, and order
bee-nvidia.service Before= nvidia-dcgm.service and
nvidia-fabricmanager.service so modules/device nodes exist before
those units start.
v11.55
2026-07-07 10:21:56 +03:00
Mikhail ChusavitinandClaude Sonnet 5 a98721b038 sat: collect dcgmi discovery + full nvvs debug log for all DCGM diags
dcgmi diag's failures only surface a terse wrapper message ("Detected
driver major version 0 is not between the required versions 346 and
2000"), which doesn't say why. Reading DCGM's own nvvs source
(NvidiaValidationSuite.cpp::CheckDriverVersion) shows this fires when
dcgmSystem.GetDeviceAttributes(0, ...) comes back empty — the version
string is never actually malformed. Diagnosing further requires info
bee never collected: whether dcgmi discovery even sees GPU 0/1
correctly, and nvvs's own internal debug log (normally
/var/log/nvidia-dcgm/nvvs.log, which bee never captured).

Add a "dcgmi discovery -l" job before every DCGM diag invocation
(check-gpu-dcgm-l2, nvbandwidth, targeted_power, pulse_test,
targeted_stress), and route dcgmi diag through
"-v -d DEBUG --debugLogFile {{run_dir}}/..." so the full nvvs debug
log lands in the SAT run dir and gets picked up by the support bundle
automatically, instead of needing a live SSH session on hardware we
usually can't get a second look at.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.54
2026-07-06 16:52:55 +03:00
Mikhail ChusavitinandClaude Sonnet 5 10b45bc267 iso: explicitly pin datacenter-gpu-manager-4-core
v11.52's ISO build failed at chroot_install-packages:

  datacenter-gpu-manager-4-cuda13 : Depends: datacenter-gpu-manager-4-core
    (= 1:4.5.2-1) but 1:4.6.0-1 is to be installed

cuda13/proprietary/proprietary-cuda13 all declare an exact-version
dependency on datacenter-gpu-manager-4-core, but core itself was never
listed, so apt's solver picked the newest core in the repo (4.6.0-1)
for the transaction and then reported the pinned variants as unmet
instead of resolving everything to 4.5.2-1. Pin core explicitly so
there's no ambiguity for the solver.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.53
2026-07-06 14:15:35 +03:00
Mikhail ChusavitinandClaude Sonnet 5 fb581c28d8 iso: fix driver pin — 580.126.20 404s on the installer CDN
v11.51 pinned 580.126.20 (DGX OS 7.5.0's validated combo), but
build-nvidia-module.sh downloads the .run installer from
download.nvidia.com/XFree86/Linux-x86_64/<version>/, a separate catalog
from the CUDA apt repo checked previously. 580.126.20 404s there —
same failure class as the original 550.54.15 CDN-404 incident.

Checked all 580.x point releases on that CDN; only some are mirrored
(580.126.09, 580.159.03/.04, 580.173.02 are 200; 580.126.16, 580.126.20,
580.167.08 are 404). Switched to 580.159.03: confirmed downloadable,
confirmed present as .deb in the apt repo, and it's the exact driver
version running on the real DGX Spark.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.52
2026-07-06 13:58:15 +03:00
Mikhail ChusavitinandClaude Sonnet 5 ffb85d55a2 iso: pin NVIDIA driver/DCGM to DGX OS 7.5.0-validated combo
590.48.01/DCGM 4.6.0-1 was never a deliberate compatibility choice — the
driver version traces back to a CDN 404 on the originally pinned
550.54.15 (commit 559fc29), and DCGM was bumped twice since purely
because NVIDIA pulled the previously-pinned core package from the repo
(6dee8f3, b2e177a). The 590 branch only has two point releases so far,
and its nvvs plugin fails dcgmi diag with "Detected driver major
version 0 is not between the required versions 346 and 2000" on H100 —
confirmed via a staging support bundle.

Switch to 580.126.20 / DCGM 4.5.2-1, the combo NVIDIA's own DGX OS 7.5.0
release notes validate for H100. Both packages confirmed present in the
NVIDIA CUDA debian12/x86_64 apt repo.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.51
2026-07-06 13:33:52 +03:00
Mikhail ChusavitinandClaude Sonnet 5 ee43a8e6ad fix: correctly extract archive path from ActionResult-wrapped SAT results
extractArchivePath only stripped the "Archive written to " prefix when
the string ended in ".tar.gz", but SAT packs write bare run directories,
never actual tar.gz archives. Any task routed through an
ActionResult-wrapping pack function (e.g. RunNvidiaAcceptancePackWithOptions,
used by the DCGM L1-L4 diag task) passed the whole prefixed string into
ReadSATOverallStatus, which then failed to find summary.txt and silently
returned "", masking real job failures as task-level "done".

Strip the prefix directly instead of gating on a suffix that SAT run
dirs never have.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-06 13:33:43 +03:00
Mikhail ChusavitinandClaude Sonnet 5 9db651c75a gpu topology: detect cross-NUMA-only paths between GPU pairs
Flags GPUs that reach one or more peers only via a SYS-class PCIe hop
(crossing the CPU/NUMA-node boundary) in nvidia-smi topo -m. On servers
where GPUs are only bridged pairwise via NVLink bridge (no switched
NVLink fabric), this is the exact path that traffic between different
bridge pairs has to cross, and can cut multi-GPU throughput by 2x+ for
workloads spanning more than one pair.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.50
2026-07-03 12:22:04 +03:00
Mikhail ChusavitinandClaude Sonnet 5 abaeaea13f add Confidential Computing readiness check + collect nvidia-smi conf-compute -q
New read-only "Check" step reports whether this server can run NVIDIA
Confidential Computing: CPU TEE support (Intel TDX / AMD SEV-SNP, via
dmesg and kvm_amd sysfs params) and GPU firmware CC capability (via
`nvidia-smi conf-compute -q`). Also collect that command's output into
the techdump export bundle.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.49
2026-07-02 19:18:41 +03:00
mchusandClaude Sonnet 5 5b98005d5d storage report: add new-vs-used disk verdict, human-readable data units, collect smartctl -i
Disk report now ends with a Conclusion section judging a drive NEW/USED
against loose thresholds (<110% capacity written, <210% read, <7d
uptime, <30 power cycles), listing which ones tripped. Data
Written/Read in the Usage section now scale to TB/PB via
formatBytesHuman instead of always printing raw GB. storageSATCommands
now runs smartctl with -i so SATA/SAS reports get Model/Serial/
Firmware/Capacity, which the Conclusion needs to evaluate the
write/read criteria (previously only -H -A was collected).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.48
2026-07-02 12:14:20 +03:00
mchusandClaude Sonnet 5 33bc275da2 storage SAT: fix NVMe SMART counters showing 0 for power-on hours/read/write
nvme-cli emits large 64-bit counters as JSON-quoted strings on some
versions; the disk-report text generator only handled bare numbers and
{lo,hi} objects, so power_on_hours/data_units_read/data_units_written
etc. silently parsed as 0 while the structured collector path already
handled this correctly. Unify both paths on a single exported
JSONInt64/NVMeSmartLog/NVMeIDCtrl type in collector/storage.go instead
of keeping two independent nvme-cli JSON parsers in sync.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.47
2026-07-02 11:58:37 +03:00
Mikhail Chusavitin 11ea640626 power ramp: fix missing step-1 GPU telemetry, add GPU/server power breakdown
Ramp Sequence table's Run 1 row showed "—" for GPU power because the
step-1 fast path (reusing single-card calibration) never populated
PerGPUTelemetry like steps 2+ do. Also add GPU total W / Server itself W
columns and an idle baseline row so server-vs-GPU consumption is visible
per ramp step.
v11.46
2026-07-01 17:39:58 +03:00
Mikhail ChusavitinandClaude Sonnet 5 796acdfec1 ipmi fru: add Asset Tag and vendor Extra field write support (in-band)
Product Asset Tag (p 5) and the repeated custom "Extra" fields (Product
Extra p 7, Board Extra b 5/6/7, Chassis Extra c 2/3) from the Inspur FRU
field doc weren't writable — ipmitool prints identically-named lines for
each custom field with no index of its own, so a plain name lookup
couldn't tell them apart. parseFRUOutput now counts occurrences per area
to recover the real index, and the existing area/index round-trip in the
FRU editor write path picks it up automatically. Out-of-band (-H/-U/-P)
writing remains out of scope.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 17:21:26 +03:00
Mikhail ChusavitinandClaude Sonnet 5 2a7d366e50 bump bible and chart submodules
bible: add chart-based web visual baseline contract
chart: support hardware contract 2.10

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
v11.45
2026-07-01 14:56:07 +03:00
Mikhail ChusavitinandClaude Sonnet 5 5bfaecd417 storage SAT: wait for smartctl self-test completion, add human-readable resource summary
smartctl -t short only launches the self-test and returns immediately
("Testing has begun"); unlike nvme device-self-test --wait, it has no
blocking mode. Validate/Load runs closed the task and produced reports
before the drive actually finished the test. Now poll smartctl -a until
the test completes (or times out) and report the real result.

Also add a per-disk "Resource" section with pseudographic progress bars
for uptime (vs 5y design life), bytes written (vs 1 DWPD x 5y budget),
and bytes read (percent from SMART attribute 242), all rendered in
human-scaled units (days/years, TB/PB) instead of raw hour/byte counts.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 14:54:48 +03:00
Mikhail ChusavitinandClaude Sonnet 5 8575cf06f8 webui: show all RAID drives per controller and add drive-prepare action
RAID Controller Management previously hid any LSI drive that wasn't
already Frgn/UGood/JBOD, and scoped VROC "free drives" from all system
disks instead of the ones actually wired to the VROC controller's
ports - drives attached directly to the CPU or another HBA could leak
in. Now every drive is listed per its own controller, and LSI drives
not already ready for array creation get a "Prepare" button that
forces them to Unconfigured Good via storcli.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 13:32:03 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 d1d5f63257 storage SAT: split collect/self-test modes, add per-disk text reports
Check mode: read-only SMART/NVMe data collection, no self-test.
Load mode: same collection + short self-test (nvme device-self-test -s 1,
smartctl -t short). Card descriptions updated accordingly.

After each storage SAT run, a disk-N-devname-report.txt is written
per device into the runDir (auto-included in support bundles).
Web UI task page renders one card per disk directly below Task Report.

Also fixes pre-existing TestDashboardRendersRuntimeHealthTable failure:
test fixture used "inactive" status but code now treats inactive as OK
for completed oneshot services; updated to "failed" to match intent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.44
2026-06-30 19:08:15 +03:00
mchusandClaude Opus 4.8 fc9b446d2e webui: per-source status bar in FRU/Elabel card + fix stale runtime-health test
Show an explicit per-source status line after "Read All" instead of hiding
failed/blocked sources in a "(skipped: …)" tail. Sources blocked by a missing
Supermicro license (SFT-OOB-LIC / SFT-DCMS-SINGLE) are flagged in red with an
actionable message, so engineers see that SAA DMI is gated rather than silently
falling back to the futile ipmitool FRU path (BIOS re-syncs FRU from DMI on boot).

Also fix TestDashboardRendersRuntimeHealthTable, stale since 4f6579e moved
"inactive" to the OK service states: the fixture now uses a failed service and
the assertion matches the current contract (failed flagged, inactive not).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 17:30:50 +03:00
Mikhail Chusavitin ea68318744 contract: bump to v2.11 — add sfp_modules[], deprecate scalar sfp_* fields 2026-06-19 18:26:29 +03:00
Mikhail Chusavitin 518082c2e2 proposals: RFC for sfp_modules[] contract extension (v2.10 → v2.11) 2026-06-19 18:14:46 +03:00
Mikhail Chusavitin 056dce0b98 backlog: add SFP module collection with contract gap analysis 2026-06-19 16:25:57 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 24f2e65b6e Add unified FRU/Elabel card with Huawei iBMC OEM IPMI support
Replaces separate IPMI FRU and SAA DMI cards with a single FRU / Elabel
card that reads all available sources in parallel and shows each field
with a color-coded source chip (IPMI FRU / Huawei iBMC / SAA DMI).

Huawei elabel fields are read/written via OEM IPMI raw commands
(NetFn 0x30, cmd 0x90) with 19-byte chunking protocol, matching
the FusionServer ElabelTool V511 wire format. Covers DeviceName,
DeviceSerialNumber, ProductName, ProductSerialNumber, ProductAssetTag,
ProductManufacturer, MainboardManufacturer, BoardProductName,
ChassisPartnumber, ChassisType (read-only), IOChassisSerial,
IOChassisAssetTag, and GUID (read-only via standard 0x06 0x08).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.43
2026-06-19 15:29:07 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 7f27b9aa38 Fix AMD GPU false detection, blackbox deadlock, and NOGPU build bloat
- sat.go: DetectGPUVendor lspci fallback now checks GPU device classes
  ([0300]/[0302]/[0380]) per line instead of scanning the whole output for
  vendor name; AMD EPYC servers have dozens of AMD-branded PCIe entries
  (Root Complex, IOMMU, Host Bridge) that were triggering the old check
- blackbox.go: fix deadlock in finishCycle — it held w.mu while calling
  persistState(), which acquires rt.mu then re-acquires w.mu inside
  persistStateLocked(); now w.mu is released before persistState()
- build.sh: remove NVIDIA-specific overlay files (bee-gpu-burn,
  bee-john-gpu-stress, bee-nccl-gpu-stress, bee-nvidia-recover,
  bee-dcgmproftester-staggered, bee-check-nvswitch,
  nvidia-fabricmanager.service.d/) for non-nvidia build variants
- bee-selfheal: gate NVIDIA recovery on BEE_GPU_VENDOR=nvidia so the
  script does not attempt to restart bee-nvidia.service on NOGPU builds

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.42
2026-06-19 09:37:26 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 cf29131116 Rework FRU and DMI editors: per-row inline save, all fields editable
- Replace global Save button with per-row ✓ (save) / ✗ (cancel) buttons
  that appear only when a field is changed
- All fields shown as editable inputs; server rejects unknown fields
  with a clear error message instead of hiding them in the UI
- Monospace font and 1.5px border for all value inputs
- Server-side name→area/index lookup for fields sent without area
- SAA DMI card: same per-row UX, confirm dialog kept (requires reboot)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.41
2026-06-19 09:30:39 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 13e6324853 Fix IPMI FRU editable field detection for abbreviated ipmitool names
ipmitool fru print on some BMC implementations returns short names
("Chassis Serial", "Board Mfg", "Board Product", "Board Serial",
"Product Serial") instead of the full names in the vendor doc.
Add both variants to fruEditableFields so all fields are editable
regardless of which naming convention the BMC uses.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.40
2026-06-19 09:24:15 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 892ef6fb7d Add Reboot and Shutdown buttons to Settings page
POST /api/system/reboot → systemctl reboot
POST /api/system/shutdown → systemctl poweroff
Both require confirm() before executing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.39
2026-06-19 09:18:30 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 ce46a97975 Remove duplicate Blackbox Logging card from Settings page
The USB Black-Box card already provides enable/disable per device.
The standalone Blackbox Logging card was non-functional and redundant.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.38
2026-06-19 09:15:31 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 258ecb3453 Add RAID Controller Management to Tools page
Unified card for LSI/Broadcom and Intel VROC controllers: auto-detects
foreign configurations and warns the operator with Import/Clear actions;
allows creating RAID 1 mirrors from unconfigured drives regardless of
controller type. Live output streams via SSE into an inline terminal.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 08:58:19 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 cbb0d1e522 Collect IPMI sensors, SEL and dmesg errors into audit JSON and support bundle
- audit JSON: IPMI sensor readings (ipmitool sensor) merged into hardware.sensors alongside lm-sensors data
- audit JSON: IPMI SEL entries (ipmitool sel list) in hardware.event_logs with source "ipmi-sel"
- audit JSON: dmesg error/warning lines in hardware.event_logs with source "dmesg" (filtered by error/warn/AER/Xid/NVRM/ECC/panic patterns)
- support bundle: added ipmitool-sensor.txt, ipmitool-sel.txt, ipmitool-sel-time.txt to techdump
- saa_dmi.go: fix dmiItemRE to accept SHN with parentheses (e.g. PS(4)LC for PSU fields)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.37
2026-06-19 08:41:37 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 bab941ccf1 Fix SAA: set CWD=/usr/local/bin; include all SAA package binaries
- saa_dmi.go: set cmd.Dir=/usr/local/bin on all saa exec calls so
  acpica_bin/acpidump is found relative to correct working directory
- build.sh: copy all saa companion dirs (acpica_bin, ExternalData,
  tool, stunnel, GO_SNMP) to /usr/local/bin/ preserving structure
- iso/vendor: add acpica_bin/acpiexec, ExternalData/, tool/gpu/nVidia/x64/,
  tool/USBController/, stunnel/, GO_SNMP/ from SAA 1.5.0 release package

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.36
2026-06-19 08:24:50 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 b49c71a980 Add IPMI FRU editor to Tools page
- New card "IPMI — FRU" on Tools page (device 0, in-band)
- Read: GET /api/tools/ipmi-fru → ipmitool fru print 0 → editable table
- Editable fields: chassis (part#, serial, extra), board (mfr, product, serial, part#),
  product (mfr, name, part#, version, serial); read-only fields displayed as text
- Write: POST /api/tools/ipmi-fru/write → task → backup to fru-backups/ → ipmitool fru edit per field
- Dirty tracking + Save (N changed) button, same UX as Supermicro DMI card

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.35
2026-06-19 08:13:35 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 85d1acdaa3 Split validate/stress into separate fixed-mode pages
- Check (2): validate mode only — no mode switcher, no stress-only cards
  (nvidia-targeted-stress, nvidia-targeted-power, nvidia-pulse hidden)
- Load (3): stress mode only — no mode switcher, all cards shown
- satStressMode() hardcoded per page; satModeChanged() removed
- Profile card with radio buttons removed from both pages
- Replaced with simple Run All button + est. time

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.34
2026-06-18 19:12:17 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 a2d7513153 Restructure nav to Load/Burn/Benchmark; fix SAA acpidump dependency
- Nav steps 3-5: Load (validate), Burn (burn-in), Benchmark (speed+endurance merged)
- /load now renders validate mode; /burn renders burn-in; /benchmark replaces /speed+/endurance
- Legacy redirects updated: /validate→/load, /burn-in→/burn, /speed+/endurance→/benchmark
- Add acpica_bin/acpidump from SAA 1.5.0 package; required by saa GetDmiInfo (ExitCode 8)
- build.sh copies acpica_bin/acpidump to /usr/local/bin/acpica_bin/ alongside saa

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.33
2026-06-18 19:07:51 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 5b5d8609d3 Refactor nav: remove numbers from Tools/Settings, add separator and Tasks item
- Remove "6." / "7." prefixes from Tools and Settings nav labels and page titles
- Add a horizontal separator (nav-sep) before the Tools/Settings group
- Move Tasks into the nav as a regular nav-item after the separator,
  replacing the separate tasks-nav-btn at the sidebar bottom
- Tasks item retains the active-count badge (tasks-nav-count)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.32
2026-06-18 17:54:54 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 e7442972d1 Move session-scoped LiveCD tools from Tools to Settings
Tools page now contains only NVMe Block Format and Supermicro - DMI.

Moved to Settings (7):
- System Install (Install to RAM + Install to Disk)
- Support Bundle + USB Black-Box
- Tool Check
- NVIDIA Self Heal (replaces simple NVIDIA Recovery card)
- Network
- Services

Update TestToolsPageRendersNvidiaSelfHealSection to assert the moved
cards on /settings instead of /tools.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v10.31
2026-06-18 17:52:19 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 4c6daa1c5e Add SAA binary to ISO vendor, rename card to Supermicro - DMI
- Extract saa 1.5.0 (Linux x86_64) into iso/vendor/saa — baked into ISO
  at /usr/local/bin/saa via the existing vendor loop in build.sh
- Add saa to the vendor tool loop in iso/builder/build.sh
- Rename the web UI card from "SAA - DMI" to "Supermicro - DMI"
- Remove the redundant description hint about saa on PATH

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v10.30
2026-06-18 17:49:12 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 e420888d71 Add DMI test fixtures from linuxhw/DMI and expand placeholder detection
Adds board and memory parser test fixtures based on real dmidecode output
from Dell PowerEdge R740xd, HPE ProLiant DL380 Gen10, and Supermicro
SYS-6028R-WTR sourced from the linuxhw/DMI dataset. Extends cleanDMIValue
with four additional vendor placeholder strings found in the dataset:
"0123456789", "1234567890", "NOT AVAILABLE", and "TO BE FILLED BY O.E.M"
(without trailing dot). Adds memory_test.go covering mixed populated/empty
DIMM slots and both GB and MB size formats.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v11.29
2026-06-18 16:15:20 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 8149360410 Fix SAA DMI parser to match real DMI.txt format
Replace the guessed pipe/key=value parser with the correct format
documented in SAA User Guide 4.8.1:

  [Section]
  Item Name   {SHN}  = "value"   // comment

Handles string values (strips surrounding quotes), non-string values
(UUID, hex), section headers for display names, version line, and
// comments. Verified against the SAA 1.5.0 User Guide sample.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v10.29
2026-06-18 15:58:02 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 4262c5b798 Add SAA DMI editor to Tools page
Adds a new card to the web UI Tools page for reading and editing DMI
fields via SAA (In-Band). Reads current DMI configuration with GetDmiInfo,
displays all fields as an editable table, and applies only the changed
fields via EditDmiInfo + ChangeDmiInfo. Backs up the original DMI file to
dmi-backups/ before any write, making it available in the support bundle
for rollback. Also adds "saa" to the standard tool check list.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v10.28
2026-06-18 15:50:42 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 b2e177af31 Bump DCGM to 4.6.0-1 to fix broken repo dependency
NVIDIA removed datacenter-gpu-manager-4-core 1:4.5.3-1 from the
repository and published 1:4.6.0-1. The cuda13 and proprietary
packages still declared an exact-version dependency on 4.5.3-1 core,
making the old pin unresolvable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v10.27
2026-06-18 14:21:24 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 271dadda03 Restructure web UI navigation into 7 numbered workflow stages
Replace the flat menu (Dashboard, Audit, Validate, Burn, Benchmark,
Tasks, Tools) with a numbered progression that guides engineers through
a logical acceptance workflow:

  Dashboard (landing) → 1. Audit → 2. Check → 3. Load → 4. Speed
  → 5. Endurance → 6. Tools → 7. Settings

Key changes:
- layout.go: numbered nav labels, new hrefs, Tasks removed from nav
  and replaced with a persistent sidebar badge (polls /api/tasks every
  5 s, highlights amber when tasks are active)
- server.go: 301 redirects from /validate→/check, /burn→/load,
  /benchmark→/speed for backward compatibility
- pages.go: dispatch cases for all new routes; old routes kept as
  fallbacks
- page_validate.go: add renderCheck() — non-destructive check page
  with validate-mode tests only (no stress toggle, no targeted-stress/
  targeted-power/pulse cards)
- page_burn.go: add renderLoad() wrapper; update scope alert to
  reference /check instead of /validate
- page_benchmark.go: add renderSpeed() (performance focus) and
  renderEndurance() (stability/overnight focus) wrappers
- page_settings.go: new Settings page with blackbox logging toggle,
  NVIDIA driver reset, and build info
- server_test.go: update five tests to use new route names and
  content expectations

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v10.26
2026-06-18 11:00:02 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 20766ccc76 Order nvidia-fabricmanager after bee-nvidia to fix boot race
bee-nvidia.service loads NVIDIA kernel modules; without After=bee-nvidia.service
fabricmanager starts before /dev/nvidiactl is ready, fails, and relies on
systemd restart to recover (~38s delay on affected systems).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-18 10:11:52 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 966944d6d8 Fix audit hanging on smartpqi SAS HBA scan file write
smartpqi uses scsi_transport_sas but does not register a sas_host
object, so /sys/class/sas_host/host14 does not exist and the existing
SAS detection check passes right through. Writing to host14/scan then
calls sas_user_scan which blocks indefinitely on scsi_scan_target's
mutex (confirmed by kernel hung-task traces in the field).

Add a second detection path via /sys/class/scsi_host/hostX/proc_name:
skip hosts whose driver is "smartpqi" or "hpsa" (HPE Smart Array
predecessors that exhibit the same behaviour).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v10.25
2026-06-15 16:07:54 +03:00
mchusandClaude Sonnet 4.6 ce6b1e0eb7 Update internal/chart submodule pointer to 8105c7e
Tracks origin/main after rebase: adds per-column header filters for
severity in the viewer (feat(viewer): replace severity dropdown).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v10.24
2026-06-13 14:48:04 +03:00
mchusandClaude Sonnet 4.6 4066e842a9 Update bible submodule to v0.2.0-13-g1977730
Picks up new contracts: hardware-ingest-json, submodule-integration,
go-database cursor safety, and several contract deduplication passes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 14:44:52 +03:00