Commit Graph
55 Commits
Author SHA1 Message Date
mchus ccc781856e fix: complete hardware collection diagnostics 2026-08-25 17:04:33 +03:00
Mikhail ChusavitinandClaude Sonnet 5 b11018ac5e fix(collector): stop pinning GPU PCIe status on an unverified idle reading
NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power
saving, and applyPCIeLinkSpeedWarning fired on every idle collector
pass regardless - since component-status DB records never downgrade
(Record() only ever raises severity), one boot-time idle sample
permanently pinned pcie:gpu:nvidia to Warning for the rest of the
session even after every load-bearing GPU SAT test passed clean. Two
prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag)
didn't hold up against this hardware/driver combination - see
bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the
full history.

Rather than add a downgrade path, stop writing an unverified status in
the first place: parseLspciDevice no longer calls
applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay
populated as plain descriptive fields; only a verified-under-load
caller may now turn them into a status verdict.

Two new SAT targets provide that verified signal:

- pcie-link (platform/pcie_link_check.go): forces every enabled PCIe
  device - not just GPUs - to retrain via the PCIe spec's Link Control
  "Retrain Link" bit, then compares the negotiated speed against the
  device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn
  equivalent load tool. Classifies by PCI class code + vendor ID, not
  name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into
  their own component-status keys so a degraded NIC never reads as a
  GPU fault.
- nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives
  real host<->device traffic via dcgmi diag -r nvbandwidth and
  resamples link speed immediately after, independent of nvbandwidth's
  own pass/fail.

Both wired into the task queue/webui the same way as nvidia-config
(routes, dispatch, priority, Validate page cards, Run All Check SAT).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-24 18:39:19 +03:00
Mikhail ChusavitinandClaude Sonnet 5 198567dffe fix: surface CPU thermal throttling in status and SAT results
A CPU that had thermally throttled (sysfs thermal_throttle counter >
0) still reported status "OK" everywhere: dmidecode-derived CPU status
only distinguishes populated/enabled/disabled and never looked at the
throttle flag the collector already recorded next to it, and neither
SAT path meant to catch this actually could:

- The routine "cpu" SAT pack (RunCPUAcceptancePack) only checked
  lscpu/sensors/stress-ng exit codes — stress-ng exits 0 whether or
  not the CPU throttled while running it, so an 89°C/throttled CPU
  right after a "successful" run still showed cpu:all as OK in
  component-status.json.
- The more thorough platform-stress test already detected throttling
  and fan-spindown correctly, but wrote its verdict as "Overall: FAIL
  — ..." with no "=", which parseSATKV can't parse — so even a real
  detected throttle event never reached the component-status DB.

Fixes:
- cpu_telemetry.go: escalate a CPU's status to Warning (only-escalate,
  same severity ranking already used elsewhere) when Throttled is set.
- sat.go: add a before/after thermal-throttle-counter check job around
  the "cpu" pack's stress-ng run, so a throttle event during the run
  fails that job and (via the existing FAILED->Warning DB mapping)
  flips cpu:all to Warning.
- platform_stress.go: emit a machine-readable overall_status= line
  alongside the human-readable verdict so platform-stress results
  actually reach ApplySATResultToDB.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-24 17:41:40 +03:00
Mikhail ChusavitinandClaude Sonnet 5 56d12b1f3c fix(collector): use word-boundary matching for dmesg severity
dmesgSeverity classified a kernel log line as Critical via plain
strings.Contains, without word boundaries, so common words that merely
contain a keyword as a substring false-positive — "disabled by
default" was flagged Critical because "default" contains "fault"
(de-fault). Found in a support bundle where 140/238 event_logs entries
came back Critical, most of them harmless boot messages (module load
notices, "... is initialized", "disabled by default"), drowning out
genuinely critical entries (Xid, AER, ECC) in the same list.

Switch to the same \b-bounded regexes already used for the capture
patterns above it in this file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-24 17:41:28 +03:00
Mikhail ChusavitinandClaude Sonnet 5 a09201c7ab fix(collector): recover storage model for NVMe drives exposed as SCSI
An NVMe SSD passed through a RAID/HBA as /dev/sdX (not /dev/nvmeX) gets
smartctl'd over the SCSI protocol, which reports its model in
scsi_model_name/scsi_vendor/scsi_product instead of the ATA/NVMe
model_name field. smartctlInfo only mapped model_name, so the drive's
model was silently dropped from reanimator.json even though smartctl
ran successfully and the serial number came through fine — caught via
a support bundle where lsblk showed "SSSTC CA6-8D1024" for a drive but
reanimator.json's storage entry had no model at all.

Add scsi_model_name/scsi_vendor/scsi_product to smartctlInfo and fall
back to them, in that order, when model_name is empty.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-24 17:41:23 +03:00
Mikhail ChusavitinandClaude Sonnet 5 134bed3eae refactor: dedupe status-severity ranking and small webui helpers
statusSeverity was byte-for-byte duplicated in collector/contract.go and
app/sat_overlay.go (same switch, same comment). Export it as
collector.StatusSeverity and drop the app-package copy.

page_topo.go inlined the same GPU/NIC/RAID class switch as its own
pcieDeviceKind function in one spot — call the function instead.

page_validate.go had its own copy of pages.go's firstNonEmpty (all call
sites already pre-trim their inputs via validateTrimPtr, so the extra
TrimSpace was redundant) — drop the duplicate.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-18 14:44:52 +03:00
Mikhail ChusavitinandClaude Sonnet 5 e3697c0a11 fix(pcie): stop misclassifying same-vendor non-GPU devices as GPUs
matchesGPUVendor (sat_overlay.go) matched any PCIe DeviceClass containing
the substring "Controller" instead of the actual GPU classes, so a
same-vendor non-GPU device (e.g. an NVIDIA-branded NIC/storage controller)
with a degraded PCIe link surfaced as a pcie:gpu:<vendor> alarm on the
Hardware Summary/webui.

isGPUDevice (app_format.go) had the same bug via a different path: an
"any NVIDIA-vendor device is a GPU" fallback that could inflate the
"GPU: N x <model>" audit summary line with non-GPU companion devices
(NVSwitch/NVLink bridges, etc).

Both now defer to collector.IsGPUClass (exported from the previously
unexported isGPUClass), the canonical exact-match classifier already used
by amdgpu.go — removing the duplicated/incorrect class checks instead of
reimplementing them. Also dedup two more locally-duplicated classifiers
within webui (which doesn't import collector by convention): NIC-class
matching (pages.go/page_topo.go) and GPU-class matching in
page_validate.go now share one copy per package instead of being
reimplemented per file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-17 12:50:50 +03:00
Mikhail ChusavitinandClaude Sonnet 5 b2b3f86c8d fix(nvidia): surface Xid 79/154 GPU bus-fall-off as a physical-reboot-required signal
nvidia-bug-report.sh appends .gz to --output-file when gzip is available,
which silently produced empty nvidia-bug-report.txt in support bundles
(cat looked for the uncompressed name that never existed).

Also: a GPU that falls off the PCIe/NVLink bus (Xid 79) or gets flagged
for Node Reboot Required (Xid 154) mid-SAT-run left every downstream test
failing with generic, unrelated-looking errors (CUDA "unknown error",
"unable to determine device handle") with no indication the GPU needed a
physical power-cycle to recover. Detect these codes from SAT run logs and
surface a plain-English "physical reboot required" message in the task's
failure detail, the persisted component-status DB, a dashboard banner on
the Hardware Summary card, and topology diagram GPU-node severity.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-06 12:28:34 +03:00
Mikhail ChusavitinandClaude Sonnet 5 a34e823f82 fix(collector): stop NVIDIA enrichment from clobbering PCIe status
enrichPCIeWithNVIDIAData unconditionally overwrote dev.Status after
collectPCIe() had already flagged a Warning/Critical (e.g. PCIe link
speed degraded), so a clean ECC/remap/reset readout silently downgraded
that finding back to OK while leaving the stale ErrorDescription behind.

Add a severity-ordered merge (OK/Unknown < Warning < Critical) shared
via mergeDeviceStatus in contract.go, and route both the NVIDIA status
calculation and the driver-unavailable fallback through it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-04 17:28:47 +03:00
mchusandClaude Fable 5 10557ec0f6 webui/collector: strip ANSI escapes in nvidia-smi topo parsing, auto-prepare drives for RAID mirror creation
- nvidia-smi underlines the topo -m header with ANSI CSI codes even when
  writing to a file; both NVLink matrix parsers failed to find the header
  and /topo showed "No NVLink-bonded GPU pairs found" on bonded systems.
- raid-lsi-create-mirror failed with "resources already in use" (exit 11)
  on JBOD drives; the task now reads drive states and auto-converts
  JBOD/UBad (set good force), releases hotspares, refuses Frgn/Onln with
  actionable messages, and dumps preservedcache info when add vd fails.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 23:16:04 +03:00
Mikhail ChusavitinandClaude Sonnet 5 1175e6ccd8 gpu: collect reset-required/row-remap status and classify Xid codes by severity
nvidia-smi exposes reset_status.reset_required and remapped_rows.* only on
newer drivers for Ampere+ GPUs; queried via a separate exec call since an
unrecognized field name fails the whole --query-gpu command and would have
wiped out unrelated telemetry (temp/ECC/power) on older drivers otherwise.

Also refines Xid severity in both the ingest dmesg collector and the
always-on kmsg watcher: Xid 64 (row-remap InfoROM write failure) now
escalates to Critical instead of the generic warning every other Xid got,
and fixes the SAT-window flush path which previously hardcoded "Warning"
and ignored pattern severity entirely.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 19:28:59 +03:00
Mikhail ChusavitinandClaude Sonnet 5 4e306ff78c raid: parse storcli64/storcli2's actual per-slot drive JSON shape
RAID Controller Management showed "No drives detected" for a live
SAS3808-iMR controller even though storcli64 clearly enumerates its
drives. Root cause: the "eall/sall show all J" drive-listing parser
(collector/raid.go and its webui/raid_mgmt.go duplicate) assumed
drives are reported as a single "Drive Information" array, but real
storcli64/storcli2 output nests each drive under its own dynamically
named key ("Drive /c0/e69/s0", paired with a "... - Detailed
Information" key) — confirmed against a live techdump/storcli64-drives.json
capture. The assumed shape was never actually produced by the tool, so
this affected every storcli64/storcli2 controller, not just Tri-Mode
ones (the storcli2 fallback added in b7f015c never got a chance to
mask it in practice, since storcli2 itself reports zero controllers on
this hardware — a separate, unrelated tool-side gap).

Both parsers now read "Response Data" as a raw key map and pull drives
from either shape, so older storcli output using the array form still
works alongside the real per-slot form.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 17:21:05 +03:00
Mikhail ChusavitinandClaude Sonnet 5 b7f015c713 raid: add storcli2 support for Tri-Mode controllers (SAS3808-iMR/9500 series)
storcli64 enumerates these controllers but reports zero drives; storcli2
is the tool Broadcom ships for Tri-Mode/MegaRAID8 hardware and uses a
compatible JSON schema for drive listing. Wires storcli2 into both the
collector (structured drive data) and the webui RAID Management page
(dedup so a controller isn't double-listed if storcli64 already sees it
with zero drives), plus techdump raw collection and the ISO vendor-tool
build step.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 15:25:06 +03:00
Mikhail ChusavitinandClaude Sonnet 5 a3377083aa webui: add /topo server topology page
New read-only visualization page: CPU sockets as anchor nodes, PCIe
devices (GPU/NIC/RAID) linked to their NUMA-affine socket with edge
color derived strictly from link_speed vs max_link_speed (not from the
device's own Status, which can also be overwritten by SAT-test results
on the same field), PSU/BMC as standalone boxes with no connecting
line, and a separate NVLink Topology card (live nvidia-smi topo -m /
nvlink -s/-e queries, not persisted to any contract).

GPU-GPU edges are drawn strictly from the actual bonded-pair list
parsed out of "nvidia-smi topo -m" (parseGPUPairAdjacency), not from
adjacent box position in the layout — an earlier ASCII mockup drew a
"chain" through unrelated GPUs, which a dedicated regression test now
guards against. A bonded pair spanning two different NUMA nodes is
flagged Warning on the edge and on both GPU boxes, per project
decision that this is an anomaly worth surfacing, not a neutral fact.

Zero changes to the ingest contract: this reverts the HardwareNVLinkPort/
HardwarePCIeDevice.NVLinks field shipped in v11.55 (33d6eee) along with
its collector/nvidia.go enrichment — that field risked a 400 from
Reanimator Core's strict decoder without an RFC, and isn't needed since
the topo page queries nvidia-smi directly instead of reading it from
audit.json. The v11.55 systemd fix (bee-nvidia.service ordering,
nv-hostengine restart) and the nvlink-status/-errors/dcgmi dumps in the
support bundle are untouched.

Also reorganizes support bundle collection per project convention:
system/ is now LiveCD-operational logs only (Xorg, services, console,
network/FS of the host itself); all server-hardware dumps (lspci,
NVIDIA/NVLink/DCGM, fabric manager, PCIe AER, ethtool, mstflint) move
to techdump/, deduplicating two entries already produced by
platform/techdump.go. Adds nvidia-bug-report.sh (previously only
collected inside the NVIDIA SAT pack) and lscpu to the always-on dump.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 11:50:33 +03:00
Mikhail Chusavitin 33d6eee9cf sat: collect NVLink status/errors, fix nv-hostengine restart via systemd
Add NVLink port status (nvidia-smi nvlink -s), error counters
(nvidia-smi nvlink -e), and dcgmi nvlink status to the support bundle,
and enrich HardwarePCIeDevice entries with per-link telemetry.

Replace the raw nv-hostengine pkill/restart dance in bee-nvidia-load
with systemctl restart/start of nvidia-dcgm.service, and order
bee-nvidia.service Before= nvidia-dcgm.service and
nvidia-fabricmanager.service so modules/device nodes exist before
those units start.
2026-07-07 10:21:56 +03:00
Mikhail ChusavitinandClaude Sonnet 5 9db651c75a gpu topology: detect cross-NUMA-only paths between GPU pairs
Flags GPUs that reach one or more peers only via a SYS-class PCIe hop
(crossing the CPU/NUMA-node boundary) in nvidia-smi topo -m. On servers
where GPUs are only bridged pairwise via NVLink bridge (no switched
NVLink fabric), this is the exact path that traffic between different
bridge pairs has to cross, and can cut multi-GPU throughput by 2x+ for
workloads spanning more than one pair.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-03 12:22:04 +03:00
mchusandClaude Sonnet 5 33bc275da2 storage SAT: fix NVMe SMART counters showing 0 for power-on hours/read/write
nvme-cli emits large 64-bit counters as JSON-quoted strings on some
versions; the disk-report text generator only handled bare numbers and
{lo,hi} objects, so power_on_hours/data_units_read/data_units_written
etc. silently parsed as 0 while the structured collector path already
handled this correctly. Unify both paths on a single exported
JSONInt64/NVMeSmartLog/NVMeIDCtrl type in collector/storage.go instead
of keeping two independent nvme-cli JSON parsers in sync.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-02 11:58:37 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 cbb0d1e522 Collect IPMI sensors, SEL and dmesg errors into audit JSON and support bundle
- audit JSON: IPMI sensor readings (ipmitool sensor) merged into hardware.sensors alongside lm-sensors data
- audit JSON: IPMI SEL entries (ipmitool sel list) in hardware.event_logs with source "ipmi-sel"
- audit JSON: dmesg error/warning lines in hardware.event_logs with source "dmesg" (filtered by error/warn/AER/Xid/NVRM/ECC/panic patterns)
- support bundle: added ipmitool-sensor.txt, ipmitool-sel.txt, ipmitool-sel-time.txt to techdump
- saa_dmi.go: fix dmiItemRE to accept SHN with parentheses (e.g. PS(4)LC for PSU fields)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 08:41:37 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 e420888d71 Add DMI test fixtures from linuxhw/DMI and expand placeholder detection
Adds board and memory parser test fixtures based on real dmidecode output
from Dell PowerEdge R740xd, HPE ProLiant DL380 Gen10, and Supermicro
SYS-6028R-WTR sourced from the linuxhw/DMI dataset. Extends cleanDMIValue
with four additional vendor placeholder strings found in the dataset:
"0123456789", "1234567890", "NOT AVAILABLE", and "TO BE FILLED BY O.E.M"
(without trailing dot). Adds memory_test.go covering mixed populated/empty
DIMM slots and both GB and MB size formats.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-18 16:15:20 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 966944d6d8 Fix audit hanging on smartpqi SAS HBA scan file write
smartpqi uses scsi_transport_sas but does not register a sas_host
object, so /sys/class/sas_host/host14 does not exist and the existing
SAS detection check passes right through. Writing to host14/scan then
calls sas_user_scan which blocks indefinitely on scsi_scan_target's
mutex (confirmed by kernel hung-task traces in the field).

Add a second detection path via /sys/class/scsi_host/hostX/proc_name:
skip hosts whose driver is "smartpqi" or "hpsa" (HPE Smart Array
predecessors that exhibit the same behaviour).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 16:07:54 +03:00
mchusandClaude Sonnet 4.6 7d2e904d14 Bring codebase into compliance with bible contracts (A–E)
A (hardware-ingest-json v2.8-2.9): remove sensor location fields from schema
and collector; tag HardwareMemory.Location as json:"-"; add PlatformConfig to
HardwareSnapshot.

B (no-hardcoded-vendors): consolidate PCI vendor IDs into collector/pci_vendors.go;
replace all vendor-name string checks in isGPUDevice, isNVIDIADevice, isMellanoxDevice,
isAMDGPUDevice, matchesGPUVendor (sat_overlay), and validateIsVendorGPU (page_validate)
with numeric vendor_id comparisons.

C (module-structure): split app/app.go (1413 lines) into app.go + app_format.go,
app_network.go, app_services.go, app_packs.go, app_install.go — no logic changes.

D (go-code-style): wrap bare return err in interfaceAdminState and
interfaceIPv4Addrs (platform/network.go) with fmt.Errorf context including
the interface name.

E (go-project-bible): add bible-local/architecture/data-model.md and
bible-local/architecture/api-surface.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 14:32:08 +03:00
mchusandClaude Sonnet 4.6 2320925433 Skip PCIe link-speed warnings for disabled devices
Disabled PCIe devices (sysfs enable==0) carry no data traffic; their
link state has no operational impact. Switchtec PCIe switch management
endpoints on NVIDIA HGX H100 baseboards (and similar fabric controllers)
train at reduced speed intentionally and were producing spurious warnings.

Check is vendor-agnostic: reads enable attribute via existing helper,
no vendor/device ID hardcoding.

Documented in bible-local/decisions/2026-06-12-pcie-disabled-device-link-warning.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-12 03:42:19 +03:00
mchusandClaude Sonnet 4.6 e169a7722c Fix NVMe SMART status always Unknown; fix GPU count including NVSwitches
nvme-cli emits smart-log counters as JSON strings and uses field names
avail_spare / percent_used instead of the prose names in the NVMe spec.
The nvmeSmartLog struct had int64 fields with wrong JSON tags — Unmarshal
returned an error and the whole health block was skipped, leaving every
NVMe drive with status=Unknown.

Fix: switch all numeric fields to jsonInt64 (already used for lsblk
block sizes) which accepts both bare numbers and quoted strings, and
correct the avail_spare / percent_used tag names.

Also fix validateIsVendorGPU for NVIDIA: previously counted any NVIDIA
PCIe device (including NVSwitch bridges) as a GPU, producing wrong
estimates (12 instead of 8 on an HGX H100 system). Now requires
device_class to be videocontroller or processingaccelerator, matching
the existing AMD filter logic.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 18:06:32 +03:00
mchusandClaude Sonnet 4.6 884988cb2a Fix audit hang on SAS HBAs: skip scsi host scan for SAS hosts
Writing to /sys/class/scsi_host/hostX/scan on SAS controllers (e.g.
Adaptec smartpqi/PM8222-SHBA) triggers sas_user_scan which blocks
indefinitely, causing the audit to hang forever. Skip hosts that appear
under /sys/class/sas_host/ — SAS topology is discovered by the driver.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-01 18:50:20 +03:00
mchusandClaude Sonnet 4.6 963bc960ca Fix SATA discovery, add NVLink bridge detection, add infiniband-diags
- storage: add jsonInt64 dual-format unmarshaler to handle lsblk output
  change in util-linux 2.38 (LOG-SEC/PHY-SEC now emitted as JSON
  integers, not quoted strings); fixes SATA disks invisible on Debian 12
- pcie: detect NVLink bridge mezzanine CX-7 cards (Mellanox x2, no host
  net ifaces, DeviceName contains "NVLINK" in lspci -v) and mark them
  with device_class="NVLinkBridge"; escalate PCIe link speed downgrade to
  Critical for these cards (Gen3 on a fixed internal connector = hardware
  fault, not a transient warning)
- pcie: cross-reference nvidia-smi topo to capture NVLink bond counts and
  active status for all NVLink bridge cards
- packages: add infiniband-diags to ISO; provides ibstat required by
  nvidia-fabricmanager-start.sh to enumerate IB devices before FM launch
  (absence causes CUDA_ERROR_SYSTEM_NOT_READY)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-28 20:57:04 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 2c22b01fe3 Fix IPMI hangs, add VROC license, fix blackbox service, drop qrencode
IPMI hang fix (Lenovo XCC SR650 V3):
- Add pluggable ipmi_profile system with per-vendor timeouts and fruEarlyExit flag
- Lenovo profile: 90s FRU timeout, streaming early-exit stops after PSU blocks found
- collectFRUEarlyExit streams ipmitool fru print and kills process once PSU blocks
  are followed by a non-PSU header (~6s instead of ~108s on 54-device FRU list)
- collectBMCFirmware and collectPSUs accept manufacturer and apply profile timeouts

VROC license detection:
- Detect VMD/VROC controller in PCIe list, run mdadm --detail-platform
- Parse "License:" line; store as snap.VROCLicense in HardwareSnapshot

Blackbox service fix:
- bee-blackbox.service was missing from systemctl enable list in ISO build hook
- Service never started on boot; state file never written; UI button stayed "Enable"

Drop qrencode:
- Remove from package list, standardTools API check, and runtime-flows doc

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-30 10:46:59 +03:00
Mikhail Chusavitin ec89616585 Add storage block geometry to audit and viewer 2026-04-29 17:39:11 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 7c504e5056 Collect IOMMU group per PCIe device from sysfs
Reads the iommu_group symlink for each BDF and exposes the group number
as iommu_group in the hardware snapshot JSON.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-29 12:34:54 +03:00
Mikhail Chusavitin 2163017a98 Collect and report storage telemetry 2026-04-29 09:40:58 +03:00
mchusandClaude Sonnet 4.6 3053cb0710 Fix PSU slot regex: match MSI underscore format PSU1_POWER_IN
\b does not fire between a digit and '_' because '_' is \w in RE2.
The pattern \bpsu?\s*([0-9]+)\b never matched PSU1_POWER_IN style
sensors, so parsePSUSDR (and PSUSlotsFromSDR / samplePSUPower) returned
empty results for MSI servers — causing all power graphs to fall back
to DCMI which reports ~half actual draw.

Added an explicit underscore-terminated pattern first in the list and
tests covering the MSI format.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 19:03:02 +03:00
mchusandClaude Sonnet 4.6 61c7abaa80 Add multi-source PSU power triangulation and per-slot distribution table
- collector/psu.go: export PSUSlotsFromSDR() reusing slot regex patterns;
  add isPSUInputPower/isPSUOutputPower helpers covering MSI/MLT/xFusion/HPE
  naming; add xFusion Power<N> slot pattern; parseBoundedFloat for self-healing
  (rejects zero/negative/out-of-range sensor readings); default fallback treats
  unclassified PSU sensors as AC input
- benchmark_types.go: BenchmarkPSUSlotPower struct; BenchmarkServerPower gains
  PSUInputIdle/Loaded, PSUOutputIdle/Loaded, PSUSlotReadingsIdle/Loaded,
  GPUSlotTotalW, DCMICoverageRatio fields
- benchmark.go: sampleIPMISDRPowerSensors uses collector.PSUSlotsFromSDR instead
  of custom classifier; detectDCMIPartialCoverage replaces ramp heuristic —
  compares DCMI idle vs SDR PSU sum, flags <0.70 ratio as partial coverage;
  detectIPMISaturationFallback kept for servers without SDR PSU sensors;
  report gains PSU Load Distribution table (per-slot AC/DC idle vs loaded, Δ)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 13:07:48 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 05c1fde233 Warn on PCIe link speed degradation and collect lspci -vvv in techdump
- collector/pcie: add applyPCIeLinkSpeedWarning that sets status=Warning
  and ErrorDescription when current link speed is below maximum negotiated
  speed (e.g. Gen1 running on a Gen5 slot)
- collector/pcie: add pcieLinkSpeedRank helper for Gen string comparison
- collector/pcie_filter_test: cover degraded and healthy link speed cases
- platform/techdump: collect lspci -vvv → lspci-vvv.txt for LnkCap/LnkSta

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-12 12:42:17 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 bb1218ddd4 Fix GPU inventory: exclude BMC virtual VGA, show real NVIDIA model names
Two issues:
1. BMC/management VGA chips (e.g. Huawei iBMC Hi171x, ASPEED) were included
   in GPU inventory because shouldIncludePCIeDevice only checked the PCI class,
   not the device name. Added a name-based filter for known BMC/management
   patterns when the class is VGA/display/3d.

2. New NVIDIA GPUs (e.g. RTX PRO 6000 Blackwell, device ID 2bb5) showed as
   "Device 2bb5" because lspci's database lags behind. Added "name" to the
   nvidia-smi query and use it to override dev.Model during enrichment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 13:57:26 +03:00
mchusandClaude Sonnet 4.6 05241f2e0e Redesign dashboard: split Runtime Health and Hardware Summary
- Runtime Health now shows only LiveCD system status (services, tools,
  drivers, network, CUDA/ROCm) — hardware component rows removed
- Hardware Summary now shows server components with readable descriptions
  (model, count×size) and component-status.json health badges
- Add Network Adapters row to Hardware Summary
- SFP module static info (vendor, PN, SN, connector, type, wavelength)
  now collected via ethtool -m regardless of carrier state
- PSU statuses from IPMI audit written to component-status.json so PSU
  badge shows actual status after first audit instead of UNKNOWN

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 23:41:23 +03:00
Mikhail Chusavitin 531d1ca366 Add NVIDIA self-heal tools and per-GPU SAT status 2026-04-07 20:20:05 +03:00
Mikhail Chusavitin f3c14cd893 Harden NIC probing for empty SFP ports 2026-04-04 15:23:15 +03:00
Mikhail Chusavitin b2b0444131 audit: ignore virtual hdisk and coprocessor noise 2026-04-02 09:56:17 +03:00
mchusandClaude Sonnet 4.6 964ab39656 fix: run john stress in parallel per GPU, fix chromium fullscreen, filter BMC virtual disks
- bee-john-gpu-stress: spawn one john process per OpenCL device in parallel
  so all GPUs are stressed simultaneously instead of only device 1
- bee-openbox-session: --start-fullscreen → --start-maximized to fix blank
  white page on first render in fbdev environment
- storage collector: skip Virtual HDisk* devices reported by BMC/iDRAC

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 23:14:21 +03:00
mchusandClaude Sonnet 4.6 eb60100297 fix: pcie gen, nccl binary, netconf sudo, boot noise, firmware cleanup
- nvidia collector: read pcie.link.gen.current/max from nvidia-smi instead
  of sysfs to avoid false Gen1 readings when GPU is in ASPM idle state
- build: remove bee-nccl-gpu-stress from rm -f list so shell script from
  overlay is not silently dropped from the ISO
- smoketest: add explicit checks for bee-gpu-burn, bee-john-gpu-stress,
  bee-nccl-gpu-stress, all_reduce_perf
- netconf: re-exec via sudo when not root to fix RTNETLINK/resolv.conf errors
- auto/config: reduce loglevel 7→3 to show clean systemd output on boot
- auto/config: blacklist snd_hda_intel and related audio modules (unused on servers)
- package-lists: remove firmware-intel-sound and firmware-amd-graphics from
  base list; move firmware-amd-graphics to bee-amd variant only
- bible-local: mark memtest ADR resolved, document working solution

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 21:25:23 +03:00
Mikhail Chusavitin 8b9d3447d7 Overlay SAT results into audit JSON 2026-03-25 20:11:03 +03:00
Mikhail Chusavitin 614b7cad61 Improve PCIe inventory and hardware identity collection 2026-03-25 20:00:38 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 36dff6e584 feat: CPU SAT via stress-ng + BMC version via ipmitool
BMC:
- collector/board.go: collectBMCFirmware() via ipmitool mc info, graceful skip if /dev/ipmi0 absent
- collector/collector.go: append BMC firmware record to snap.Firmware
- app/panel.go: show BMC version in TUI right-panel header alongside BIOS

CPU SAT:
- platform/sat.go: RunCPUAcceptancePack(baseDir, durationSec) — lscpu + sensors before/after + stress-ng
- app/app.go: RunCPUAcceptancePack + RunCPUAcceptancePackResult methods, satRunner interface updated
- app/panel.go: CPU row now reads real PASS/FAIL from cpu-*/summary.txt via satStatuses(); cpuDetailResult shows last SAT summary + audit data
- tui/types.go: actionRunCPUSAT, confirmBody for CPU test with mode label
- tui/screen_health_check.go: hcCPUDurations [60,300,900]s; hcRunSingle(CPU)→confirm screen; executeRunAll uses RunCPUAcceptancePackResult
- tui/forms.go: actionRunCPUSAT → RunCPUAcceptancePackResult with mode duration
- cmd/bee/main.go: bee sat cpu [--duration N] subcommand

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-25 11:06:12 +03:00
Mikhail Chusavitin b25a2f6d30 feat: add support bundle and raw audit export 2026-03-16 18:20:26 +03:00
Mikhail Chusavitin 78c6dfc0ef Sync hardware ingest contract v2.7 2026-03-15 23:03:38 +03:00
Mikhail Chusavitin a6023372b1 Use microcode as CPU firmware 2026-03-15 21:16:17 +03:00
Mikhail Chusavitin ab5a4be7ac Align hardware export with ingest contract 2026-03-15 21:04:53 +03:00
Mikhail Chusavitin b483e2ce35 Add health verdicts and acceptance tests 2026-03-14 17:53:58 +03:00
Mikhail Chusavitin 6aca1682b9 Refactor bee CLI and LiveCD integration 2026-03-13 16:52:16 +03:00
Mikhail Chusavitin 18b8c69bc5 Implement audit enrichments, TUI workflows, and production ISO scaffold 2026-03-06 11:56:26 +03:00
mchus ab22e3ad74 add: NVMe wear telemetry via nvme smart-log (1.8b) 2026-03-05 14:55:53 +03:00