Commit Graph
75 Commits
Author SHA1 Message Date
mchus 983f41a1f0 fix(nvidia): detect NVSwitch fabric via driver, not host-PCIe
On HGX B200 (Kaytus KR9288-X3) NVSwitch is managed via HMC/ERoT and never
appears on host lspci as 10de:0680, so the old check always returned
"not present" and silently skipped fabricmanager — leaving the fabric
untrained and CUDA/NCCL/dcgmi failing with "system not yet initialized"
on otherwise-healthy hardware. bee-check-nvswitch now checks, in order,
nvidia-smi's GPU Fabric GUID, /proc/driver/nvidia-nvswitch/devices, and
finally the legacy host-PCIe scan for older HGX generations.
2026-07-29 18:58:19 +03:00
mchusandClaude Sonnet 5 8a91f0f783 fix(webui): repair broken scenario Run button onclick, dedupe build.sh overlay staging
- page_scenario.go: onclick built via JSON.stringify() embedded raw double
  quotes inside a double-quoted HTML attribute, truncating the attribute so
  the click handler never compiled; pass the name through an escaped
  data-scenario-name attribute instead.
- build.sh: overlay staging rsyncs (OVERLAY_DIR->stage, stage->includes.chroot)
  ran without --delete, so a scenario removed from the repo (a9924b0) stayed
  baked into every ISO built from the persistent stage cache since — the
  "second script" in the Scenario page's list.
- blackbox: rewritten around a deterministic local zip + incremental
  patch-the-changed-suffix onto removable media, instead of walking/copying
  ~90 files through a synchronous ntfs-3g FUSE mount every cycle. journalctl
  captures are now "--since last sync" (were "--since boot", growing with
  uptime) and metrics.db is excluded (was copied whole every cycle).
- scenario: nvbandwidth-acs-ab now escalates GPU count (same-socket pair,
  other socket's pair, one cross-socket pair, all GPUs) under each ACS state
  instead of always running all 6 GPUs at once, using a new `bee
  gpu-bandwidth-groups` subcommand that discovers socket layout from
  `nvidia-smi topo -m` at runtime — gpu_indices is host-specific, so this
  can't be baked into the scenario file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-29 18:11:24 +03:00
Mikhail ChusavitinandClaude Sonnet 5 5a780e96b4 platform/app/webui: ship the power-watch scenario baked into the image
platform.LocalScenariosDir (/usr/share/bee/scenarios, populated from
iso/overlay/usr/share/bee/scenarios/ by build.sh's overlay rsync) is now
checked before removable media for both `bee run <name>` and the "6.
Scenario" page — a scenario shipped with the image works with no USB
stick required. ReadScenario/ListAvailableScenarios merge local + USB;
the removable-media-only functions from the previous commit are kept
as-is (still used directly where that's actually what's wanted) rather
than renamed out from under existing callers/tests.

iso/overlay/usr/share/bee/scenarios/nvbandwidth-all-gpu-power-watch.json
is a copy of scenarios/nvbandwidth-all-gpu-power-watch.json — the two
aren't auto-synced (documented in scenarios/README.md), so shipping a
scenario baked-in means checking it into both places.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-28 19:10:35 +03:00
Mikhail ChusavitinandClaude Sonnet 5 bcf02e0515 webui: add "6. Scenario" page — run scriptable test scenarios from removable media through the normal task queue
Exposes the scenario engine (platform.System.RunScenario, added earlier)
in the web UI instead of only the `bee run` CLI: a new nav item lists every
scenarios/*.json found on mounted removable media (GET /api/scenario/list)
and runs one with a click (POST /api/scenario/run), enqueued as a normal
Task with target "scenario" — progress/logs live in Tasks like any other
SAT pack, no separate live-output UI needed.

- app.go: satRunner gains RunScenario, exportManager gains
  ListScenarioFilesOnRemovableMedia/ReadScenarioFromRemovableMedia — both
  already implemented on platform.System, just newly exposed through App.
- webui/tasks.go: taskParams.ScenarioName; runTask's "scenario" case reads
  the file from removable media, parses it, and runs it.
- webui/page_scenario.go: the page itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-28 18:58:11 +03:00
Mikhail ChusavitinandClaude Sonnet 5 49979c4da4 app: don't let a renamed-away source path abort the whole blackbox/support-bundle copy
copyPath/copyPathFiltered treated any os.Stat/os.ReadDir/os.Open error on a
source entry as fatal, aborting the entire tree copy. A source path is read
from the live export dir while bee's own task runner concurrently renames
task directories (e.g. "_pending" -> "_done") — an entry present in the
parent's os.ReadDir a moment ago disappearing by the time it's individually
Stat'd/Open'd is an expected race, not a real failure.

Seen on a real crash bundle: blackbox got stuck in status "degraded" from
early in the run (first hit during the CPU pack, well before the GPU tests)
after exactly this race, and every syncBracket wait then timed out for the
rest of the run — the discovery/wait plumbing from the previous fix works,
but had nothing working under it to wait on. The target also accumulated
stale "_pending" copies alongside "_done" ones with no cleanup, though
fixing that dedup is left for a follow-up.

os.IsNotExist(err) now skips the vanished entry instead of propagating.
Added regression tests simulating the race directly (copy_path_race_test.go).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-28 16:28:11 +03:00
Mikhail ChusavitinandClaude Sonnet 5 fa618d5abb platform/app: restructure support bundle/blackbox mirror into device-type export/, status/, tasks/, livecd/
Replaces the flat export/ + techdump/ + system/ + systemd/ layout (two
same-named-but-different techdump/ dirs, disk reports duplicated between a
SAT run dir and techdump/) with one shared categorizeExportTree used by both
BuildSupportBundle and the blackbox USB mirror:

- export/{cpu,memory,storage,gpu,network,platform}/ — raw vendor-tool
  output grouped by device type, plus export/reanimator.json ready to POST
  to Reanimator's /ingest/hardware endpoint.
- status/ — computed diagnosis (component-status.json, runtime-health.*,
  and metrics.db, previously missing from every bundle entirely).
- tasks/ — bee's own task-run bookkeeping (bee-sat/, bee-bench/, task
  reports, service logs) where duplication with export/ is expected.
- livecd/{gui,host}/ — live-boot/kiosk-session-only diagnostics, kept out
  of the hardware-facing tree.

Drops the redundant disk-report mirror write in RunStorageAcceptancePack
and the now-dead syncDirectoryTree/removeMissingPaths; adds a size+mtime
skip in copyPath so the blackbox mirror doesn't rewrite unchanged files
every cycle. README.md rewritten to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-28 14:33:26 +03:00
Mikhail ChusavitinandClaude Sonnet 5 e03267a72f platform/app: fix nvbandwidth split fallback, blackbox discovery churn, and add blocking sync brackets around load steps
- gpuBandwidthSocketGroups: a single GPU whose NUMA node fails to resolve
  no longer collapses the whole per-socket nvbandwidth split into one
  fallback pass — it now folds into the last resolved group instead,
  preserving isolation for the sockets that did resolve.
- blackbox discoverMarkedTargets: skip mounting/unmounting devices that
  already have a running worker on every 2s discovery tick. This was
  observed hammering the same USB target continuously (mount+unmount
  every ~2s for the whole session) and contending with the worker's own
  sync cycle, plausibly explaining multi-minute sync cycles seen on a
  real crash bundle.
- syncFilesystem now calls syscall.Sync() directly instead of spawning
  /bin/sync per copied file; blackbox mounts removable targets with
  -o sync so writes are durable without relying on the app-level sync as
  the primary mechanism.
- New platform.SetSyncBracketHook / satJob.syncBracket: blocks (with a
  bounded timeout) on blackbox actually reaching removable media right
  before and right after a diagnostic's real load step (nvbandwidth,
  memtester, stress-ng, dcgmi diag, nccl, smartctl/nvme self-test...),
  instead of only firing a fire-and-forget kick after the job's own log
  file is written. A crash mid-load now has durable evidence the load
  started, not just whatever streamed to the RAM-backed export dir before
  blackbox's next scheduled cycle.

Found investigating a real support bundle where blackbox's last
successful sync (19:55:25) predated both the previous job finishing and
the crashing nvbandwidth job starting (19:56:57) — none of the crash
window ever reached durable media.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-28 14:22:05 +03:00
Mikhail Chusavitin 781cf5dcbf platform/app: stream SAT job output live and kick blackbox on job completion
A crash mid-command (e.g. the nvbandwidth reboot) previously lost that
job's entire output: streamExecOutput only buffered stdout/stderr in
memory and the job's log file was written once, after the process
exited. It now also streams each line straight to that file as it
arrives, so whatever printed before a crash survives.

Root filesystem here is a tmpfs overlay (toram boot), so the only real
persistence boundary is blackbox's mirror to removable media, not the
local write itself. platform.SetJobBoundaryHook lets app wire a touch
of a small kick-file after each job's output is written; blackboxWorker
now polls that file's mtime alongside its normal adaptive timer and
syncs immediately on a kick instead of waiting out the current flush
period (up to 30s).
2026-07-27 17:40:22 +03:00
Mikhail Chusavitin ced2175fb0 platform: split nvbandwidth SAT into per-socket + all-GPU passes
On multi-socket systems, run the NVIDIA bandwidth diagnostic once per
CPU socket before the all-GPU pass, so a crash confined to the all-GPU
run (with clean per-socket passes preceding it) isolates a cross-socket
peer-to-peer fault instead of leaving it conflated with a general
GPU/PCIe issue. Single-socket systems keep the original one-pass shape.

Also expand the support-bundle README with reference notes distilled
from a real analysis pass (BMC clock drift, "0/empty" tool output
meaning absent hardware rather than a fault, timestamp-matching before
assigning causality, and a normal-power-cycle SEL signature), plus a
step-by-step recipe for diagnosing an unexpected reboot/crash during a
specific test.
2026-07-27 17:01:39 +03:00
Mikhail ChusavitinandClaude Sonnet 5 b30d34199a app/webui: replace generic "see summary.txt" SAT failure text with the real reason
Every place that surfaced a FAILED SAT result — task error messages,
component-status.json detail, and the hardware snapshot's ErrorDescription/
StatusHistory — used to say only "SAT overall_status=FAILED (see
summary.txt)" or "<label> failed", forcing an engineer to go dig through the
run directory to find out what actually broke.

nvidia-config's summary.txt now carries a "warnings" field with the specific
GPU/NVLink finding. A new SATFailureDetail/satFailureDetailFromKV in
component_status_db.go reads that field, or falls back to naming whichever
generic SAT sub-job(s) reported non-OK/UNSUPPORTED status along with their
exit code. This feeds both the task-runner error message and the component
status DB. sat_overlay.go's satKeyStatus (which drives ErrorDescription on
the exported hardware snapshot) now does the same, with storage kept
per-device so one drive's rc doesn't get attributed to another's card.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 15:48:12 +03:00
Mikhail ChusavitinandClaude Sonnet 5 5aee146903 app: ship a README.md in every support bundle and blackbox capture
An agent handed a bundle path had no way to know its layout without
grepping through random logs first — confirmed by watching a separate
session read bee-nvidia.log before anything else on a real bundle. Embeds
a single README.md (bee-embed, internal/app/assets/) explaining what bee
is and giving direct answers to the questions someone analyzing a bundle
is most likely to ask (did the tests pass, what hardware is this, is a
service healthy, RAID/GPU/NVLink state, etc), written at the bundle root
by both BuildSupportBundle (support-bundle archive root, sibling of
manifest.txt) and blackboxWorker.syncCycle (removable-media boot-folder
root, which otherwise has no manifest.txt-equivalent pointing anywhere).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 11:37:44 +03:00
Mikhail ChusavitinandClaude Sonnet 5 cfaa15ec7c platform/webui: fold confidential-computing into a GPU config + NVLink check
The standalone "confidential-computing" SAT target only ever checked CC
readiness, which most fleets never opt into (a NOT_READY verdict there
isn't a fault). Meanwhile DCGM diag never asserts GPU config compliance
(ECC/MIG/power-limit vs factory default) or NVLink topology (per NVIDIA's
own DGX BasePOD deployment guide, this needs a separate validation step)
— gaps confirmed against public DCGM docs and a real NV17-vs-expected-NV18
bonded pair found on a live bundle.

Repurposes the routine into "nvidia-config": reuses the existing
ListNvidiaGPUSettings() (already backing the GPU-settings page) to flag
ECC disabled, a MIG mode change stuck pending a reset/reboot, and a power
limit capped >5% below default; parses "nvidia-smi topo -m" bonded pairs
against "nvlink -s/-e" to flag any inactive lane or nonzero replay/
recovery/CRC counter on an otherwise-active bond. CC readiness is folded
in as one informational field (does not gate overall_status) rather than
a dedicated test. Reports under the same pcie:gpu:nvidia severity key as
every other nvidia-* SAT target instead of an isolated key, so a
config/NVLink FAILED result isn't invisible next to stress-test results.

Also fixes ApplySATResultToDB silently dropping any target with no
matching switch case (exactly what the old confidential-computing target
did) with a new coverage test enumerating every real SAT target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 11:36:52 +03:00
Mikhail ChusavitinandClaude Sonnet 5 cc3997f7b1 app/webui: fix component-status.json multi-process clobber and unbounded growth
The long-lived bee-web process (writing PSU/kmsg watchdog records ~every
60s) and each short-lived "bee bee-worker" SAT-task subprocess each held
an independent in-memory copy of component-status.json. Whichever saved
last won outright, silently erasing whatever the other had just written —
e.g. a GPU SAT task's pcie:gpu:nvidia result vanishing the next time the
PSU watchdog ticked. ComponentStatusDB.Record now reloads on-disk state
(keyed by newer LastCheckedAt) before merging its own update.

Also stops re-logging identical repeat observations to History: the
ingest contract defines status_history as a transition log ("История
переходов статусов"), not a per-poll journal, but Record appended one
entry per call regardless — a continuously-polled PSU grew an unbounded
run of identical "still OK" entries. Now only appends when a source's
last recorded status for a key actually changes.

The PSU watchdog itself now backs off (60s -> doubling, capped at 30min)
while steady and resets to 60s the moment any PSU's status changes, cutting
ipmitool shellouts and file writes for a fleet that's been stable for a
while.

Also adds the missing "raid" case to the component-detail API (was
returning 404 for any RAID card's detail click on /topo).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 11:24:18 +03:00
Mikhail ChusavitinandClaude Sonnet 5 2d84ddb577 webui/topo: read from techdump instead of live nvidia-smi, fix socket/NUMA mismatch, fix cross-tier SAT status merge
/topo blocked the HTTP request on live nvidia-smi calls with no timeout
(topo -m, nvlink -s/-e run on every page load), so a wedged driver hung
the page indefinitely. CaptureTechnicalDump now persists these dumps
once per audit cycle; the page reads them from techdump/ instead.

buildSocketIndex mapped NUMA node number to CPU by treating dmidecode's
Socket Designation (often 1-indexed, "CPU1"/"CPU2") as equal to the
NUMA node number (always 0-indexed) — GPUs/NICs on NUMA node 0 fell
into the "unknown" column, others attached to the wrong CPU box. Now
ranks CPUs by Socket value instead of assuming a shared numbering base.

Fixed a bug in ComponentStatusDB/applyComponentStatusDB where GPU SAT
results were keyed per-target ("pcie:gpu:nvidia-stress") instead of
per-vendor, which both broke cross-tier severity tracking (a later
clean "2. Check" run and an earlier failing "3. Load" run never
compared severities) and silently failed to match any real BDF, so the
DB overlay never reached the topology graph at all. GPU keys are now
normalized to vendor ("pcie:gpu:nvidia"/"pcie:gpu:amd"). Also skip
writing to the DB when a SAT task was aborted by the user (ctx
canceled), so a partial run can't stomp a previously recorded status.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 17:14:03 +03:00
Mikhail ChusavitinandClaude Sonnet 5 53c46465d2 webui: add persistent GPU settings management (ECC/MIG/CC/power limit)
Adds a GPU Settings card to /tools for the settings that actually
persist on NVIDIA data-center GPUs: ECC mode, MIG mode, and
Confidential Computing mode (all stored in the GPU's inforom/firmware,
take effect after a GPU reset or reboot) plus power limit (does not
persist — reapplied on demand). Includes a one-click "Reset All to
Defaults" that restores factory settings across every visible GPU,
touching only whatever has actually drifted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 15:25:25 +03:00
Mikhail ChusavitinandClaude Sonnet 5 a3377083aa webui: add /topo server topology page
New read-only visualization page: CPU sockets as anchor nodes, PCIe
devices (GPU/NIC/RAID) linked to their NUMA-affine socket with edge
color derived strictly from link_speed vs max_link_speed (not from the
device's own Status, which can also be overwritten by SAT-test results
on the same field), PSU/BMC as standalone boxes with no connecting
line, and a separate NVLink Topology card (live nvidia-smi topo -m /
nvlink -s/-e queries, not persisted to any contract).

GPU-GPU edges are drawn strictly from the actual bonded-pair list
parsed out of "nvidia-smi topo -m" (parseGPUPairAdjacency), not from
adjacent box position in the layout — an earlier ASCII mockup drew a
"chain" through unrelated GPUs, which a dedicated regression test now
guards against. A bonded pair spanning two different NUMA nodes is
flagged Warning on the edge and on both GPU boxes, per project
decision that this is an anomaly worth surfacing, not a neutral fact.

Zero changes to the ingest contract: this reverts the HardwareNVLinkPort/
HardwarePCIeDevice.NVLinks field shipped in v11.55 (33d6eee) along with
its collector/nvidia.go enrichment — that field risked a 400 from
Reanimator Core's strict decoder without an RFC, and isn't needed since
the topo page queries nvidia-smi directly instead of reading it from
audit.json. The v11.55 systemd fix (bee-nvidia.service ordering,
nv-hostengine restart) and the nvlink-status/-errors/dcgmi dumps in the
support bundle are untouched.

Also reorganizes support bundle collection per project convention:
system/ is now LiveCD-operational logs only (Xorg, services, console,
network/FS of the host itself); all server-hardware dumps (lspci,
NVIDIA/NVLink/DCGM, fabric manager, PCIe AER, ethtool, mstflint) move
to techdump/, deduplicating two entries already produced by
platform/techdump.go. Adds nvidia-bug-report.sh (previously only
collected inside the NVIDIA SAT pack) and lscpu to the always-on dump.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 11:50:33 +03:00
Mikhail Chusavitin 33d6eee9cf sat: collect NVLink status/errors, fix nv-hostengine restart via systemd
Add NVLink port status (nvidia-smi nvlink -s), error counters
(nvidia-smi nvlink -e), and dcgmi nvlink status to the support bundle,
and enrich HardwarePCIeDevice entries with per-link telemetry.

Replace the raw nv-hostengine pkill/restart dance in bee-nvidia-load
with systemctl restart/start of nvidia-dcgm.service, and order
bee-nvidia.service Before= nvidia-dcgm.service and
nvidia-fabricmanager.service so modules/device nodes exist before
those units start.
2026-07-07 10:21:56 +03:00
Mikhail ChusavitinandClaude Sonnet 5 ee43a8e6ad fix: correctly extract archive path from ActionResult-wrapped SAT results
extractArchivePath only stripped the "Archive written to " prefix when
the string ended in ".tar.gz", but SAT packs write bare run directories,
never actual tar.gz archives. Any task routed through an
ActionResult-wrapping pack function (e.g. RunNvidiaAcceptancePackWithOptions,
used by the DCGM L1-L4 diag task) passed the whole prefixed string into
ReadSATOverallStatus, which then failed to find summary.txt and silently
returned "", masking real job failures as task-level "done".

Strip the prefix directly instead of gating on a suffix that SAT run
dirs never have.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-06 13:33:43 +03:00
Mikhail ChusavitinandClaude Sonnet 5 abaeaea13f add Confidential Computing readiness check + collect nvidia-smi conf-compute -q
New read-only "Check" step reports whether this server can run NVIDIA
Confidential Computing: CPU TEE support (Intel TDX / AMD SEV-SNP, via
dmesg and kvm_amd sysfs params) and GPU firmware CC capability (via
`nvidia-smi conf-compute -q`). Also collect that command's output into
the techdump export bundle.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-02 19:18:41 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 7f27b9aa38 Fix AMD GPU false detection, blackbox deadlock, and NOGPU build bloat
- sat.go: DetectGPUVendor lspci fallback now checks GPU device classes
  ([0300]/[0302]/[0380]) per line instead of scanning the whole output for
  vendor name; AMD EPYC servers have dozens of AMD-branded PCIe entries
  (Root Complex, IOMMU, Host Bridge) that were triggering the old check
- blackbox.go: fix deadlock in finishCycle — it held w.mu while calling
  persistState(), which acquires rt.mu then re-acquires w.mu inside
  persistStateLocked(); now w.mu is released before persistState()
- build.sh: remove NVIDIA-specific overlay files (bee-gpu-burn,
  bee-john-gpu-stress, bee-nccl-gpu-stress, bee-nvidia-recover,
  bee-dcgmproftester-staggered, bee-check-nvswitch,
  nvidia-fabricmanager.service.d/) for non-nvidia build variants
- bee-selfheal: gate NVIDIA recovery on BEE_GPU_VENDOR=nvidia so the
  script does not attempt to restart bee-nvidia.service on NOGPU builds

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 09:37:26 +03:00
mchusandClaude Sonnet 4.6 7d2e904d14 Bring codebase into compliance with bible contracts (A–E)
A (hardware-ingest-json v2.8-2.9): remove sensor location fields from schema
and collector; tag HardwareMemory.Location as json:"-"; add PlatformConfig to
HardwareSnapshot.

B (no-hardcoded-vendors): consolidate PCI vendor IDs into collector/pci_vendors.go;
replace all vendor-name string checks in isGPUDevice, isNVIDIADevice, isMellanoxDevice,
isAMDGPUDevice, matchesGPUVendor (sat_overlay), and validateIsVendorGPU (page_validate)
with numeric vendor_id comparisons.

C (module-structure): split app/app.go (1413 lines) into app.go + app_format.go,
app_network.go, app_services.go, app_packs.go, app_install.go — no logic changes.

D (go-code-style): wrap bare return err in interfaceAdminState and
interfaceIPv4Addrs (platform/network.go) with fmt.Errorf context including
the interface name.

E (go-project-bible): add bible-local/architecture/data-model.md and
bible-local/architecture/api-surface.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 14:32:08 +03:00
mchus 6c2b188ec9 Add no-GUI boot mode and quieter boot diagnostics 2026-05-03 21:14:45 +03:00
mchus 29179917c3 Add USB blackbox log mirroring service 2026-04-24 10:20:12 +03:00
mchus b3cf8e3893 Globalize autotuned system power source 2026-04-20 07:02:12 +03:00
mchusandClaude Sonnet 4.6 d52ec67f8f Stability hardening, build script fixes, GRUB bee logo
Stability hardening (webui/app):
- readFileLimited(): защита от OOM при чтении audit JSON (100 MB),
  component-status DB (10 MB) и лога задачи (50 MB)
- jobs.go: буферизованный лог задачи — один открытый fd на задачу
  вместо open/write/close на каждую строку (устраняет тысячи syscall/сек
  при GPU стресс-тестах)
- stability.go: экспоненциальный backoff в goRecoverLoop (2s→4s→…→60s),
  сброс при успешном прогоне >30s, счётчик перезапусков в slog
- kill_workers.go: таймаут 5s на скан /proc, warn при срабатывании
- bee-web.service: MemoryMax=3G — OOM killer защищён

Build script:
- build.sh: удалён блок генерации grub-pc/grub.cfg + live.cfg.in —
  мёртвый код с v8.25; grub-pc игнорируется live-build, а генерируемый
  live.cfg.in перезаписывал правильный статический файл устаревшей
  версией без tuning-параметров ядра и пунктов gsp-off/kms+gsp-off
- build.sh: dump_memtest_debug теперь логирует grub-efi/grub.cfg
  вместо grub-pc/grub.cfg (было всегда "missing")

GRUB:
- live-theme/bee-logo.png: логотип пчелы 400×400px на чёрном фоне
- live-theme/theme.txt: + image компонент по центру в верхней трети
  экрана; меню сдвинуто с 62% до 65%

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 13:08:31 +03:00
Mikhail Chusavitin b4280941f5 Move NCCL and NVBandwidth into validate mode 2026-04-16 11:02:30 +03:00
Mikhail Chusavitin 7237e4d3e4 Add fabric manager boot and support diagnostics 2026-04-15 16:14:26 +03:00
Mikhail Chusavitin 303de2df04 Add slot-aware ramp sequence to bee-bench power 2026-04-14 17:47:40 +03:00
Mikhail Chusavitin 95124d228f Split bee-bench into perf and power workflows 2026-04-14 17:33:13 +03:00
Mikhail Chusavitin 54338dbae5 Unify live RAM runtime state 2026-04-14 16:18:33 +03:00
mchusandClaude Sonnet 4.6 ba16021cdb Fix GPU model propagation, export filenames, PSU/service status, and chart perf
- nvidia.go: add Name field to nvidiaGPUInfo, include model name in
  nvidia-smi query, set dev.Model in enrichPCIeWithNVIDIAData
- pages.go: fix duplicate GPU count in validate card summary (4 GPU: 4 x …
  → 4 x … GPU); fix PSU UNKNOWN fallback from hw.PowerSupplies; treat
  activating/deactivating/reloading service states as OK in Runtime Health
- support_bundle.go: use "150405" time format (no colons) for exFAT compat
- sat.go / benchmark.go / platform_stress.go / sat_fan_stress.go: remove
  .tar.gz archive creation from export dirs — export packs everything itself
- charts_svg.go: add min-max downsampling (1400 pt cap) for SVG chart perf
- benchmark_report.go / sat.go: normalize GPU fallback to "Unknown GPU"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 10:05:27 +03:00
mchusandClaude Sonnet 4.6 05241f2e0e Redesign dashboard: split Runtime Health and Hardware Summary
- Runtime Health now shows only LiveCD system status (services, tools,
  drivers, network, CUDA/ROCm) — hardware component rows removed
- Hardware Summary now shows server components with readable descriptions
  (model, count×size) and component-status.json health badges
- Add Network Adapters row to Hardware Summary
- SFP module static info (vendor, PN, SN, connector, type, wavelength)
  now collected via ethtool -m regardless of carrier state
- PSU statuses from IPMI audit written to component-status.json so PSU
  badge shows actual status after first audit instead of UNKNOWN

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 23:41:23 +03:00
Mikhail Chusavitin c1690a084b Fix app tests that mutate global defaults 2026-04-09 15:28:25 +03:00
Mikhail Chusavitin 9481ca2805 Add staged NVIDIA burn ramp-up mode 2026-04-09 15:21:14 +03:00
Mikhail Chusavitin 4ef403898f Tighten NVIDIA GPU PCI detection 2026-04-09 15:14:48 +03:00
mchusandClaude Sonnet 4.6 025548ab3c UI: amber accents, smaller wallpaper logo, new support bundle name, drop display resolution
- Bootloader: GRUB fallback text colors → yellow/brown (amber tone)
- CLI charts: all GPU metric series use single amber color (xterm-256 #214)
- Wallpaper: logo width scaled to 400 px dynamically, shadow scales with font size
- Support bundle: renamed to YYYY-MM-DD (BEE-SP vX.X) SRV_MODEL SRV_SN ToD.tar.gz
  using dmidecode for server model (spaces→underscores) and serial number
- Remove display resolution feature (UI card, API routes, handlers, tests)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 21:37:01 +03:00
Mikhail Chusavitin e0d94d7f47 Remove HPL from build and audit flows 2026-04-08 10:00:23 +03:00
mchusandClaude Sonnet 4.6 16e7ae00e7 Add HPL (LINPACK) benchmark as validate/stress task
HPL 2.3 from netlib compiled against OpenBLAS with a minimal
single-process MPI stub — no MPI package required in the ISO.
Matrix size is auto-sized to 80% of total RAM at runtime.

Build:
- VERSIONS: HPL_VERSION=2.3, HPL_SHA256=32c5c17d…
- build-hpl.sh: downloads HPL + OpenBLAS from Debian 12 repo,
  compiles xhpl with a self-contained mpi_stub.c
- build.sh: step 80-hpl, injects xhpl + libopenblas into overlay

Runtime:
- bee-hpl: generates HPL.dat (N auto from /proc/meminfo, NB=256,
  P=1 Q=1), runs xhpl, prints standard WR... Gflops output
- platform/hpl.go: RunHPL(), parses WR line → GFlops + PASSED/FAILED
- tasks.go: target "hpl"
- pages.go: LINPACK (HPL) card in validate/stress grid (stress-only)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 07:08:18 +03:00
mchusandClaude Sonnet 4.6 b2f8626fee Refactor validate modes, fix benchmark report and IPMI power
- Replace diag level 1-4 dropdown with Validate/Stress radio buttons
- Validate: dcgmi L2, 60s CPU, 256MB/1p memtester, SMART short
- Stress: dcgmi L3 + targeted_stress in Run All, 30min CPU, 1GB/3p memtester, SMART long/NVMe extended
- Parallel GPU mode: spawn single task for all GPUs instead of splitting per model
- Benchmark table: per-GPU columns for sequential runs, server-wide column for parallel
- Benchmark report converted to Markdown with server model, GPU model, version in header; only steady-state charts
- Fix IPMI power parsing in benchmark (was looking for 'Current Power', correct field is 'Instantaneous power reading')

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 00:42:12 +03:00
Mikhail Chusavitin 531d1ca366 Add NVIDIA self-heal tools and per-GPU SAT status 2026-04-07 20:20:05 +03:00
mchus 38e79143eb Refine burn UI and NVIDIA stress flows 2026-04-05 13:43:43 +03:00
mchus bf47c8dbd2 Add NVIDIA benchmark reporting flow 2026-04-05 10:30:56 +03:00
mchus 143b7dca5d Add stability hardening and self-heal recovery 2026-04-05 10:29:37 +03:00
Mikhail Chusavitin 728270dc8e Unblock bee-web startup and expand support bundle diagnostics 2026-04-04 15:18:43 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 fd722692a4 feat(watchdog): hardware error monitor + unified component status store
- Add platform/error_patterns.go: pluggable table of kernel log patterns
  (NVIDIA/GPU, PCIe AER, storage I/O, MCE, EDAC) — extend by adding one struct
- Add app/component_status_db.go: persistent JSON store (component-status.json)
  keyed by "pcie:BDF", "storage:dev", "cpu:all", "memory:all"; OK never
  downgrades Warning or Critical
- Add webui/kmsg_watcher.go: goroutine reads /dev/kmsg during SAT tasks,
  writes Warning to DB for matched hardware errors
- Fix task status: overall_status=FAILED in summary.txt now marks task failed
- Audit routine overlays component DB statuses into bee-audit.json on every read

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-02 19:20:59 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 99cece524c feat(support-bundle): add PCIe link diagnostics and system logs
- Add full dmesg (was tail -200), kern.log, syslog
- Add /proc/cmdline, lspci -vvv, nvidia-smi -q
- Add per-GPU PCIe link speed/width from sysfs (NVIDIA devices only)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-02 15:42:28 +03:00
Mikhail Chusavitin c27449c60e feat(webui): show current boot source 2026-04-02 15:36:32 +03:00
Mikhail Chusavitin e7df63bae1 fix(app): include extra system logs in support bundle 2026-04-02 13:44:58 +03:00
Mikhail Chusavitin b2b0444131 audit: ignore virtual hdisk and coprocessor noise 2026-04-02 09:56:17 +03:00
Mikhail Chusavitin c394845b34 refactor(webui): queue install and bundle tasks - v3.18 2026-04-01 08:46:46 +03:00