Commit Graph
170 Commits
Author SHA1 Message Date
Mikhail ChusavitinandClaude Sonnet 5 cc3997f7b1 app/webui: fix component-status.json multi-process clobber and unbounded growth
The long-lived bee-web process (writing PSU/kmsg watchdog records ~every
60s) and each short-lived "bee bee-worker" SAT-task subprocess each held
an independent in-memory copy of component-status.json. Whichever saved
last won outright, silently erasing whatever the other had just written —
e.g. a GPU SAT task's pcie:gpu:nvidia result vanishing the next time the
PSU watchdog ticked. ComponentStatusDB.Record now reloads on-disk state
(keyed by newer LastCheckedAt) before merging its own update.

Also stops re-logging identical repeat observations to History: the
ingest contract defines status_history as a transition log ("История
переходов статусов"), not a per-poll journal, but Record appended one
entry per call regardless — a continuously-polled PSU grew an unbounded
run of identical "still OK" entries. Now only appends when a source's
last recorded status for a key actually changes.

The PSU watchdog itself now backs off (60s -> doubling, capped at 30min)
while steady and resets to 60s the moment any PSU's status changes, cutting
ipmitool shellouts and file writes for a fleet that's been stable for a
while.

Also adds the missing "raid" case to the component-detail API (was
returning 404 for any RAID card's detail click on /topo).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 11:24:18 +03:00
Mikhail Chusavitin 1d5c02ebaa webui/topo: responsive PSU/firmware layout, stacked group cards, fix zero-CPU crash
PSU and firmware (BMC/BIOS) boxes were absolutely-positioned SVG rects in a
single fixed-width row with no wrap, so an arbitrary/larger count piled up
and overlapped once a board had more of either than fit in that row. Move
them to plain flex-wrap HTML below the diagram (Firmware row first, then
Power Supplies), which reflows naturally for any count. The main CPU/PCIe/
GPU diagram now scrolls horizontally (overflow-x:auto) instead of being
squashed to fit narrow viewports, matching the wide-table convention used
elsewhere in webui.

Also fixes a crash: renderTopoMainDiagram forced numCols to 1 for layout
purposes when a snapshot has zero CPUs, then unconditionally indexed
hw.CPUs[0], panicking the whole /topo page on any audit without CPU data.

Same-kind/same-column components (e.g. 4 GPUs in one NUMA node) now render
as one stacked card summarizing worst-case status plus a tally line ("3 OK,
1 Warning") instead of one box per component, and card severity coloring
uses real fill/stroke vars instead of HTML badge classes that don't apply
any style to SVG shapes.
2026-07-09 10:52:38 +03:00
mchusandClaude Fable 5 10557ec0f6 webui/collector: strip ANSI escapes in nvidia-smi topo parsing, auto-prepare drives for RAID mirror creation
- nvidia-smi underlines the topo -m header with ANSI CSI codes even when
  writing to a file; both NVLink matrix parsers failed to find the header
  and /topo showed "No NVLink-bonded GPU pairs found" on bonded systems.
- raid-lsi-create-mirror failed with "resources already in use" (exit 11)
  on JBOD drives; the task now reads drive states and auto-converts
  JBOD/UBad (set good force), releases hotspares, refuses Frgn/Onln with
  actionable messages, and dumps preservedcache info when add vd fails.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 23:16:04 +03:00
Mikhail ChusavitinandClaude Sonnet 5 1175e6ccd8 gpu: collect reset-required/row-remap status and classify Xid codes by severity
nvidia-smi exposes reset_status.reset_required and remapped_rows.* only on
newer drivers for Ampere+ GPUs; queried via a separate exec call since an
unrecognized field name fails the whole --query-gpu command and would have
wiped out unrelated telemetry (temp/ECC/power) on older drivers otherwise.

Also refines Xid severity in both the ingest dmesg collector and the
always-on kmsg watcher: Xid 64 (row-remap InfoROM write failure) now
escalates to Critical instead of the generic warning every other Xid got,
and fixes the SAT-window flush path which previously hardcoded "Warning"
and ignored pattern severity entirely.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 19:28:59 +03:00
Mikhail ChusavitinandClaude Sonnet 5 4e306ff78c raid: parse storcli64/storcli2's actual per-slot drive JSON shape
RAID Controller Management showed "No drives detected" for a live
SAS3808-iMR controller even though storcli64 clearly enumerates its
drives. Root cause: the "eall/sall show all J" drive-listing parser
(collector/raid.go and its webui/raid_mgmt.go duplicate) assumed
drives are reported as a single "Drive Information" array, but real
storcli64/storcli2 output nests each drive under its own dynamically
named key ("Drive /c0/e69/s0", paired with a "... - Detailed
Information" key) — confirmed against a live techdump/storcli64-drives.json
capture. The assumed shape was never actually produced by the tool, so
this affected every storcli64/storcli2 controller, not just Tri-Mode
ones (the storcli2 fallback added in b7f015c never got a chance to
mask it in practice, since storcli2 itself reports zero controllers on
this hardware — a separate, unrelated tool-side gap).

Both parsers now read "Response Data" as a raw key map and pull drives
from either shape, so older storcli output using the array form still
works alongside the real per-slot form.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 17:21:05 +03:00
Mikhail ChusavitinandClaude Sonnet 5 2d84ddb577 webui/topo: read from techdump instead of live nvidia-smi, fix socket/NUMA mismatch, fix cross-tier SAT status merge
/topo blocked the HTTP request on live nvidia-smi calls with no timeout
(topo -m, nvlink -s/-e run on every page load), so a wedged driver hung
the page indefinitely. CaptureTechnicalDump now persists these dumps
once per audit cycle; the page reads them from techdump/ instead.

buildSocketIndex mapped NUMA node number to CPU by treating dmidecode's
Socket Designation (often 1-indexed, "CPU1"/"CPU2") as equal to the
NUMA node number (always 0-indexed) — GPUs/NICs on NUMA node 0 fell
into the "unknown" column, others attached to the wrong CPU box. Now
ranks CPUs by Socket value instead of assuming a shared numbering base.

Fixed a bug in ComponentStatusDB/applyComponentStatusDB where GPU SAT
results were keyed per-target ("pcie:gpu:nvidia-stress") instead of
per-vendor, which both broke cross-tier severity tracking (a later
clean "2. Check" run and an earlier failing "3. Load" run never
compared severities) and silently failed to match any real BDF, so the
DB overlay never reached the topology graph at all. GPU keys are now
normalized to vendor ("pcie:gpu:nvidia"/"pcie:gpu:amd"). Also skip
writing to the DB when a SAT task was aborted by the user (ctx
canceled), so a partial run can't stomp a previously recorded status.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 17:14:03 +03:00
Mikhail ChusavitinandClaude Sonnet 5 53c46465d2 webui: add persistent GPU settings management (ECC/MIG/CC/power limit)
Adds a GPU Settings card to /tools for the settings that actually
persist on NVIDIA data-center GPUs: ECC mode, MIG mode, and
Confidential Computing mode (all stored in the GPU's inforom/firmware,
take effect after a GPU reset or reboot) plus power limit (does not
persist — reapplied on demand). Includes a one-click "Reset All to
Defaults" that restores factory settings across every visible GPU,
touching only whatever has actually drifted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 15:25:25 +03:00
Mikhail ChusavitinandClaude Sonnet 5 b7f015c713 raid: add storcli2 support for Tri-Mode controllers (SAS3808-iMR/9500 series)
storcli64 enumerates these controllers but reports zero drives; storcli2
is the tool Broadcom ships for Tri-Mode/MegaRAID8 hardware and uses a
compatible JSON schema for drive listing. Wires storcli2 into both the
collector (structured drive data) and the webui RAID Management page
(dedup so a controller isn't double-listed if storcli64 already sees it
with zero drives), plus techdump raw collection and the ISO vendor-tool
build step.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 15:25:06 +03:00
Mikhail ChusavitinandClaude Sonnet 5 a3377083aa webui: add /topo server topology page
New read-only visualization page: CPU sockets as anchor nodes, PCIe
devices (GPU/NIC/RAID) linked to their NUMA-affine socket with edge
color derived strictly from link_speed vs max_link_speed (not from the
device's own Status, which can also be overwritten by SAT-test results
on the same field), PSU/BMC as standalone boxes with no connecting
line, and a separate NVLink Topology card (live nvidia-smi topo -m /
nvlink -s/-e queries, not persisted to any contract).

GPU-GPU edges are drawn strictly from the actual bonded-pair list
parsed out of "nvidia-smi topo -m" (parseGPUPairAdjacency), not from
adjacent box position in the layout — an earlier ASCII mockup drew a
"chain" through unrelated GPUs, which a dedicated regression test now
guards against. A bonded pair spanning two different NUMA nodes is
flagged Warning on the edge and on both GPU boxes, per project
decision that this is an anomaly worth surfacing, not a neutral fact.

Zero changes to the ingest contract: this reverts the HardwareNVLinkPort/
HardwarePCIeDevice.NVLinks field shipped in v11.55 (33d6eee) along with
its collector/nvidia.go enrichment — that field risked a 400 from
Reanimator Core's strict decoder without an RFC, and isn't needed since
the topo page queries nvidia-smi directly instead of reading it from
audit.json. The v11.55 systemd fix (bee-nvidia.service ordering,
nv-hostengine restart) and the nvlink-status/-errors/dcgmi dumps in the
support bundle are untouched.

Also reorganizes support bundle collection per project convention:
system/ is now LiveCD-operational logs only (Xorg, services, console,
network/FS of the host itself); all server-hardware dumps (lspci,
NVIDIA/NVLink/DCGM, fabric manager, PCIe AER, ethtool, mstflint) move
to techdump/, deduplicating two entries already produced by
platform/techdump.go. Adds nvidia-bug-report.sh (previously only
collected inside the NVIDIA SAT pack) and lscpu to the always-on dump.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 11:50:33 +03:00
Mikhail ChusavitinandClaude Sonnet 5 abaeaea13f add Confidential Computing readiness check + collect nvidia-smi conf-compute -q
New read-only "Check" step reports whether this server can run NVIDIA
Confidential Computing: CPU TEE support (Intel TDX / AMD SEV-SNP, via
dmesg and kvm_amd sysfs params) and GPU firmware CC capability (via
`nvidia-smi conf-compute -q`). Also collect that command's output into
the techdump export bundle.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-02 19:18:41 +03:00
Mikhail ChusavitinandClaude Sonnet 5 796acdfec1 ipmi fru: add Asset Tag and vendor Extra field write support (in-band)
Product Asset Tag (p 5) and the repeated custom "Extra" fields (Product
Extra p 7, Board Extra b 5/6/7, Chassis Extra c 2/3) from the Inspur FRU
field doc weren't writable — ipmitool prints identically-named lines for
each custom field with no index of its own, so a plain name lookup
couldn't tell them apart. parseFRUOutput now counts occurrences per area
to recover the real index, and the existing area/index round-trip in the
FRU editor write path picks it up automatically. Out-of-band (-H/-U/-P)
writing remains out of scope.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 17:21:26 +03:00
Mikhail ChusavitinandClaude Sonnet 5 8575cf06f8 webui: show all RAID drives per controller and add drive-prepare action
RAID Controller Management previously hid any LSI drive that wasn't
already Frgn/UGood/JBOD, and scoped VROC "free drives" from all system
disks instead of the ones actually wired to the VROC controller's
ports - drives attached directly to the CPU or another HBA could leak
in. Now every drive is listed per its own controller, and LSI drives
not already ready for array creation get a "Prepare" button that
forces them to Unconfigured Good via storcli.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 13:32:03 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 d1d5f63257 storage SAT: split collect/self-test modes, add per-disk text reports
Check mode: read-only SMART/NVMe data collection, no self-test.
Load mode: same collection + short self-test (nvme device-self-test -s 1,
smartctl -t short). Card descriptions updated accordingly.

After each storage SAT run, a disk-N-devname-report.txt is written
per device into the runDir (auto-included in support bundles).
Web UI task page renders one card per disk directly below Task Report.

Also fixes pre-existing TestDashboardRendersRuntimeHealthTable failure:
test fixture used "inactive" status but code now treats inactive as OK
for completed oneshot services; updated to "failed" to match intent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-30 19:08:15 +03:00
mchusandClaude Opus 4.8 fc9b446d2e webui: per-source status bar in FRU/Elabel card + fix stale runtime-health test
Show an explicit per-source status line after "Read All" instead of hiding
failed/blocked sources in a "(skipped: …)" tail. Sources blocked by a missing
Supermicro license (SFT-OOB-LIC / SFT-DCMS-SINGLE) are flagged in red with an
actionable message, so engineers see that SAA DMI is gated rather than silently
falling back to the futile ipmitool FRU path (BIOS re-syncs FRU from DMI on boot).

Also fix TestDashboardRendersRuntimeHealthTable, stale since 4f6579e moved
"inactive" to the OK service states: the fixture now uses a failed service and
the assertion matches the current contract (failed flagged, inactive not).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 17:30:50 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 24f2e65b6e Add unified FRU/Elabel card with Huawei iBMC OEM IPMI support
Replaces separate IPMI FRU and SAA DMI cards with a single FRU / Elabel
card that reads all available sources in parallel and shows each field
with a color-coded source chip (IPMI FRU / Huawei iBMC / SAA DMI).

Huawei elabel fields are read/written via OEM IPMI raw commands
(NetFn 0x30, cmd 0x90) with 19-byte chunking protocol, matching
the FusionServer ElabelTool V511 wire format. Covers DeviceName,
DeviceSerialNumber, ProductName, ProductSerialNumber, ProductAssetTag,
ProductManufacturer, MainboardManufacturer, BoardProductName,
ChassisPartnumber, ChassisType (read-only), IOChassisSerial,
IOChassisAssetTag, and GUID (read-only via standard 0x06 0x08).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 15:29:07 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 cf29131116 Rework FRU and DMI editors: per-row inline save, all fields editable
- Replace global Save button with per-row ✓ (save) / ✗ (cancel) buttons
  that appear only when a field is changed
- All fields shown as editable inputs; server rejects unknown fields
  with a clear error message instead of hiding them in the UI
- Monospace font and 1.5px border for all value inputs
- Server-side name→area/index lookup for fields sent without area
- SAA DMI card: same per-row UX, confirm dialog kept (requires reboot)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 09:30:39 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 13e6324853 Fix IPMI FRU editable field detection for abbreviated ipmitool names
ipmitool fru print on some BMC implementations returns short names
("Chassis Serial", "Board Mfg", "Board Product", "Board Serial",
"Product Serial") instead of the full names in the vendor doc.
Add both variants to fruEditableFields so all fields are editable
regardless of which naming convention the BMC uses.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 09:24:15 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 892ef6fb7d Add Reboot and Shutdown buttons to Settings page
POST /api/system/reboot → systemctl reboot
POST /api/system/shutdown → systemctl poweroff
Both require confirm() before executing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 09:18:30 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 ce46a97975 Remove duplicate Blackbox Logging card from Settings page
The USB Black-Box card already provides enable/disable per device.
The standalone Blackbox Logging card was non-functional and redundant.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 09:15:31 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 258ecb3453 Add RAID Controller Management to Tools page
Unified card for LSI/Broadcom and Intel VROC controllers: auto-detects
foreign configurations and warns the operator with Import/Clear actions;
allows creating RAID 1 mirrors from unconfigured drives regardless of
controller type. Live output streams via SSE into an inline terminal.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 08:58:19 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 cbb0d1e522 Collect IPMI sensors, SEL and dmesg errors into audit JSON and support bundle
- audit JSON: IPMI sensor readings (ipmitool sensor) merged into hardware.sensors alongside lm-sensors data
- audit JSON: IPMI SEL entries (ipmitool sel list) in hardware.event_logs with source "ipmi-sel"
- audit JSON: dmesg error/warning lines in hardware.event_logs with source "dmesg" (filtered by error/warn/AER/Xid/NVRM/ECC/panic patterns)
- support bundle: added ipmitool-sensor.txt, ipmitool-sel.txt, ipmitool-sel-time.txt to techdump
- saa_dmi.go: fix dmiItemRE to accept SHN with parentheses (e.g. PS(4)LC for PSU fields)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 08:41:37 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 bab941ccf1 Fix SAA: set CWD=/usr/local/bin; include all SAA package binaries
- saa_dmi.go: set cmd.Dir=/usr/local/bin on all saa exec calls so
  acpica_bin/acpidump is found relative to correct working directory
- build.sh: copy all saa companion dirs (acpica_bin, ExternalData,
  tool, stunnel, GO_SNMP) to /usr/local/bin/ preserving structure
- iso/vendor: add acpica_bin/acpiexec, ExternalData/, tool/gpu/nVidia/x64/,
  tool/USBController/, stunnel/, GO_SNMP/ from SAA 1.5.0 release package

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 08:24:50 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 b49c71a980 Add IPMI FRU editor to Tools page
- New card "IPMI — FRU" on Tools page (device 0, in-band)
- Read: GET /api/tools/ipmi-fru → ipmitool fru print 0 → editable table
- Editable fields: chassis (part#, serial, extra), board (mfr, product, serial, part#),
  product (mfr, name, part#, version, serial); read-only fields displayed as text
- Write: POST /api/tools/ipmi-fru/write → task → backup to fru-backups/ → ipmitool fru edit per field
- Dirty tracking + Save (N changed) button, same UX as Supermicro DMI card

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 08:13:35 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 85d1acdaa3 Split validate/stress into separate fixed-mode pages
- Check (2): validate mode only — no mode switcher, no stress-only cards
  (nvidia-targeted-stress, nvidia-targeted-power, nvidia-pulse hidden)
- Load (3): stress mode only — no mode switcher, all cards shown
- satStressMode() hardcoded per page; satModeChanged() removed
- Profile card with radio buttons removed from both pages
- Replaced with simple Run All button + est. time

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-18 19:12:17 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 a2d7513153 Restructure nav to Load/Burn/Benchmark; fix SAA acpidump dependency
- Nav steps 3-5: Load (validate), Burn (burn-in), Benchmark (speed+endurance merged)
- /load now renders validate mode; /burn renders burn-in; /benchmark replaces /speed+/endurance
- Legacy redirects updated: /validate→/load, /burn-in→/burn, /speed+/endurance→/benchmark
- Add acpica_bin/acpidump from SAA 1.5.0 package; required by saa GetDmiInfo (ExitCode 8)
- build.sh copies acpica_bin/acpidump to /usr/local/bin/acpica_bin/ alongside saa

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-18 19:07:51 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 5b5d8609d3 Refactor nav: remove numbers from Tools/Settings, add separator and Tasks item
- Remove "6." / "7." prefixes from Tools and Settings nav labels and page titles
- Add a horizontal separator (nav-sep) before the Tools/Settings group
- Move Tasks into the nav as a regular nav-item after the separator,
  replacing the separate tasks-nav-btn at the sidebar bottom
- Tasks item retains the active-count badge (tasks-nav-count)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-18 17:54:54 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 e7442972d1 Move session-scoped LiveCD tools from Tools to Settings
Tools page now contains only NVMe Block Format and Supermicro - DMI.

Moved to Settings (7):
- System Install (Install to RAM + Install to Disk)
- Support Bundle + USB Black-Box
- Tool Check
- NVIDIA Self Heal (replaces simple NVIDIA Recovery card)
- Network
- Services

Update TestToolsPageRendersNvidiaSelfHealSection to assert the moved
cards on /settings instead of /tools.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-18 17:52:19 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 4c6daa1c5e Add SAA binary to ISO vendor, rename card to Supermicro - DMI
- Extract saa 1.5.0 (Linux x86_64) into iso/vendor/saa — baked into ISO
  at /usr/local/bin/saa via the existing vendor loop in build.sh
- Add saa to the vendor tool loop in iso/builder/build.sh
- Rename the web UI card from "SAA - DMI" to "Supermicro - DMI"
- Remove the redundant description hint about saa on PATH

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-18 17:49:12 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 8149360410 Fix SAA DMI parser to match real DMI.txt format
Replace the guessed pipe/key=value parser with the correct format
documented in SAA User Guide 4.8.1:

  [Section]
  Item Name   {SHN}  = "value"   // comment

Handles string values (strips surrounding quotes), non-string values
(UUID, hex), section headers for display names, version line, and
// comments. Verified against the SAA 1.5.0 User Guide sample.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-18 15:58:02 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 4262c5b798 Add SAA DMI editor to Tools page
Adds a new card to the web UI Tools page for reading and editing DMI
fields via SAA (In-Band). Reads current DMI configuration with GetDmiInfo,
displays all fields as an editable table, and applies only the changed
fields via EditDmiInfo + ChangeDmiInfo. Backs up the original DMI file to
dmi-backups/ before any write, making it available in the support bundle
for rollback. Also adds "saa" to the standard tool check list.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-18 15:50:42 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 271dadda03 Restructure web UI navigation into 7 numbered workflow stages
Replace the flat menu (Dashboard, Audit, Validate, Burn, Benchmark,
Tasks, Tools) with a numbered progression that guides engineers through
a logical acceptance workflow:

  Dashboard (landing) → 1. Audit → 2. Check → 3. Load → 4. Speed
  → 5. Endurance → 6. Tools → 7. Settings

Key changes:
- layout.go: numbered nav labels, new hrefs, Tasks removed from nav
  and replaced with a persistent sidebar badge (polls /api/tasks every
  5 s, highlights amber when tasks are active)
- server.go: 301 redirects from /validate→/check, /burn→/load,
  /benchmark→/speed for backward compatibility
- pages.go: dispatch cases for all new routes; old routes kept as
  fallbacks
- page_validate.go: add renderCheck() — non-destructive check page
  with validate-mode tests only (no stress toggle, no targeted-stress/
  targeted-power/pulse cards)
- page_burn.go: add renderLoad() wrapper; update scope alert to
  reference /check instead of /validate
- page_benchmark.go: add renderSpeed() (performance focus) and
  renderEndurance() (stability/overnight focus) wrappers
- page_settings.go: new Settings page with blackbox logging toggle,
  NVIDIA driver reset, and build info
- server_test.go: update five tests to use new route names and
  content expectations

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-18 11:00:02 +03:00
mchusandClaude Sonnet 4.6 7d2e904d14 Bring codebase into compliance with bible contracts (A–E)
A (hardware-ingest-json v2.8-2.9): remove sensor location fields from schema
and collector; tag HardwareMemory.Location as json:"-"; add PlatformConfig to
HardwareSnapshot.

B (no-hardcoded-vendors): consolidate PCI vendor IDs into collector/pci_vendors.go;
replace all vendor-name string checks in isGPUDevice, isNVIDIADevice, isMellanoxDevice,
isAMDGPUDevice, matchesGPUVendor (sat_overlay), and validateIsVendorGPU (page_validate)
with numeric vendor_id comparisons.

C (module-structure): split app/app.go (1413 lines) into app.go + app_format.go,
app_network.go, app_services.go, app_packs.go, app_install.go — no logic changes.

D (go-code-style): wrap bare return err in interfaceAdminState and
interfaceIPv4Addrs (platform/network.go) with fmt.Errorf context including
the interface name.

E (go-project-bible): add bible-local/architecture/data-model.md and
bible-local/architecture/api-surface.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 14:32:08 +03:00
mchusandClaude Sonnet 4.6 e169a7722c Fix NVMe SMART status always Unknown; fix GPU count including NVSwitches
nvme-cli emits smart-log counters as JSON strings and uses field names
avail_spare / percent_used instead of the prose names in the NVMe spec.
The nvmeSmartLog struct had int64 fields with wrong JSON tags — Unmarshal
returned an error and the whole health block was skipped, leaving every
NVMe drive with status=Unknown.

Fix: switch all numeric fields to jsonInt64 (already used for lsblk
block sizes) which accepts both bare numbers and quoted strings, and
correct the avail_spare / percent_used tag names.

Also fix validateIsVendorGPU for NVIDIA: previously counted any NVIDIA
PCIe device (including NVSwitch bridges) as a GPU, producing wrong
estimates (12 instead of 8 on an HGX H100 system). Now requires
device_class to be videocontroller or processingaccelerator, matching
the existing AMD filter logic.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 18:06:32 +03:00
mchusandClaude Sonnet 4.6 4f6579e040 Fix Runtime Health criteria: network, services, nvidia-fabricmanager
Network: green if at least one interface has IPv4 (drop PARTIAL state).

Bee Services: treat inactive as OK — oneshot services (bee-sshsetup,
bee-preflight, bee-network, bee-audit, etc.) complete successfully and
exit to inactive; only failed is a real problem.

nvidia-fabricmanager: add ExecCondition=bee-check-nvswitch drop-in so
the service is silently skipped (inactive, not failed) on systems
without NVSwitch hardware (e.g. H200 NVL with direct NVLink, no
NVSwitch chips). bee-check-nvswitch detects NVSwitch via lspci
(vendor 10de, class 0680).

bee-nvidia.service: add ConditionPathExists=/usr/local/bin/bee-nvidia-load
so the unit is a no-op if somehow present in a non-nvidia build.

bee-boot-status: read /etc/bee-gpu-vendor and exclude bee-nvidia from
CRITICAL/ALL on non-nvidia builds, preventing boot hang if the unit
is unexpectedly present.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-14 05:20:25 +03:00
mchusandClaude Sonnet 4.6 dc07580adc Add AER decode, event counter, and sparkline to component detail modal
- decodeAERStatus: parses aer_status hex from kernel error strings and
  maps PCIe AER register bits to human-readable names with correctable/
  uncorrectable classification (e.g. "Receiver Error, Replay Timer Timeout (correctable)")
- renderSparkline: 100px inline SVG showing non-OK events over time,
  bars positioned proportionally to timestamp; evenly spaced when timestamps coincide
- renderComponentDetail: shows event count badge and sparkline in the
  component header row; decoded AER line appears below the raw error summary

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 23:54:54 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 805a3b277d Track PCIe AER correctable errors; fix GPU status key routing
Add nvidia-aer-correctable and pcie-aer-correctable patterns to catch
"bus correctable error" events seen in SEL (Critical Interrupt / offset 7).
Both patterns carry severity "warning" — correctable errors are
hardware-recovered and should not flag a card as failed.

Fix kmsg_watcher routing: GPU-category events were keyed as pcie:<BDF>
but the UI queries for pcie:gpu: prefix. Split the switch so "gpu" →
pcie:gpu:<BDF> and "pcie" → pcie:<BDF>. This applies to both
flushWindow (SAT-window path) and flushImmediate (always-on path).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 12:50:14 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 0939a647ea Fix component detail modal: replace dead hx-* with fetch-based JS
HTMX was never loaded on the page, so hx-get on the component label
spans was dead code — the dialog opened empty. Replace with a plain
openComponentDetail() fetch call. Also fix dialog positioning broken
by the CSS reset (*{margin:0} overrode the UA margin:auto that centers
<dialog>). Replace card hx-trigger polling with a setInterval.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 10:53:20 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 ae80d7711e Add continuous hardware health monitoring and component detail view
- kmsg watcher now records kernel errors (GPU Xid, MCE, EDAC, storage I/O) at all times,
  not only during SAT tasks; flushImmediate writes directly to ComponentStatusDB
- New health_poller: polls ipmitool sdr every 60s for PSU health (watchdog:psu source)
- Hardware Summary card auto-refreshes every 30s via htmx without page reload
- Component rows (CPU/Memory/Storage/GPU/PSU) are now clickable -- opens a modal
  with per-component status, source, timestamp and last 20 history entries

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 09:56:39 +03:00
mchus 6c2b188ec9 Add no-GUI boot mode and quieter boot diagnostics 2026-05-03 21:14:45 +03:00
mchus 14505ef24a Remove easy bee ASCII logo banners 2026-05-03 21:07:13 +03:00
mchus 0e39e7d960 Make toram default and add install-to-ram CLI 2026-05-03 14:07:47 +03:00
Mikhail Chusavitin 58d6da0e4f Fix live task logs and SAT windows 2026-04-30 17:26:45 +03:00
Mikhail Chusavitin 7ce73e34a4 Add NVMe block format tool 2026-04-30 16:27:25 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 2c22b01fe3 Fix IPMI hangs, add VROC license, fix blackbox service, drop qrencode
IPMI hang fix (Lenovo XCC SR650 V3):
- Add pluggable ipmi_profile system with per-vendor timeouts and fruEarlyExit flag
- Lenovo profile: 90s FRU timeout, streaming early-exit stops after PSU blocks found
- collectFRUEarlyExit streams ipmitool fru print and kills process once PSU blocks
  are followed by a non-PSU header (~6s instead of ~108s on 54-device FRU list)
- collectBMCFirmware and collectPSUs accept manufacturer and apply profile timeouts

VROC license detection:
- Detect VMD/VROC controller in PCIe list, run mdadm --detail-platform
- Parse "License:" line; store as snap.VROCLicense in HardwareSnapshot

Blackbox service fix:
- bee-blackbox.service was missing from systemctl enable list in ISO build hook
- Service never started on boot; state file never written; UI button stayed "Enable"

Drop qrencode:
- Remove from package list, standardTools API check, and runtime-flows doc

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-30 10:46:59 +03:00
Mikhail Chusavitin ec89616585 Add storage block geometry to audit and viewer 2026-04-29 17:39:11 +03:00
mchus 29179917c3 Add USB blackbox log mirroring service 2026-04-24 10:20:12 +03:00
mchus be4b439804 Commit remaining workspace changes 2026-04-23 20:32:26 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 679aeb9947 Run NVIDIA DCGM diag tests on all selected GPUs simultaneously
targeted_stress, targeted_power, and the Level 2/3 diag were dispatched
one GPU at a time from the UI, turning a single dcgmi command into 8
sequential ~350–450 s runs. DCGM supports -i with a comma-separated list
of GPU indices and runs the diagnostic on all of them in parallel.

Move nvidia, nvidia-targeted-stress, nvidia-targeted-power into
nvidiaAllGPUTargets so expandSATTarget passes all selected indices in one
API call. Simplify runNvidiaValidateSet to match runNvidiaFabricValidate.
Update sat.go constants and page_validate.go estimates to reflect all-GPU
simultaneous execution (remove n× multiplier from total time estimates).

Stress test on 8-GPU system: ~5.3 h → ~2.5 h.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-20 11:53:25 +03:00
mchus b3cf8e3893 Globalize autotuned system power source 2026-04-20 07:02:12 +03:00
mchus 65bcc9ce81 refactor(webui): split pages into task modules 2026-04-20 06:56:52 +03:00