Commit Graph
21 Commits
Author SHA1 Message Date
Mikhail ChusavitinandClaude Sonnet 5 49979c4da4 app: don't let a renamed-away source path abort the whole blackbox/support-bundle copy
copyPath/copyPathFiltered treated any os.Stat/os.ReadDir/os.Open error on a
source entry as fatal, aborting the entire tree copy. A source path is read
from the live export dir while bee's own task runner concurrently renames
task directories (e.g. "_pending" -> "_done") — an entry present in the
parent's os.ReadDir a moment ago disappearing by the time it's individually
Stat'd/Open'd is an expected race, not a real failure.

Seen on a real crash bundle: blackbox got stuck in status "degraded" from
early in the run (first hit during the CPU pack, well before the GPU tests)
after exactly this race, and every syncBracket wait then timed out for the
rest of the run — the discovery/wait plumbing from the previous fix works,
but had nothing working under it to wait on. The target also accumulated
stale "_pending" copies alongside "_done" ones with no cleanup, though
fixing that dedup is left for a follow-up.

os.IsNotExist(err) now skips the vanished entry instead of propagating.
Added regression tests simulating the race directly (copy_path_race_test.go).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-28 16:28:11 +03:00
Mikhail ChusavitinandClaude Sonnet 5 fa618d5abb platform/app: restructure support bundle/blackbox mirror into device-type export/, status/, tasks/, livecd/
Replaces the flat export/ + techdump/ + system/ + systemd/ layout (two
same-named-but-different techdump/ dirs, disk reports duplicated between a
SAT run dir and techdump/) with one shared categorizeExportTree used by both
BuildSupportBundle and the blackbox USB mirror:

- export/{cpu,memory,storage,gpu,network,platform}/ — raw vendor-tool
  output grouped by device type, plus export/reanimator.json ready to POST
  to Reanimator's /ingest/hardware endpoint.
- status/ — computed diagnosis (component-status.json, runtime-health.*,
  and metrics.db, previously missing from every bundle entirely).
- tasks/ — bee's own task-run bookkeeping (bee-sat/, bee-bench/, task
  reports, service logs) where duplication with export/ is expected.
- livecd/{gui,host}/ — live-boot/kiosk-session-only diagnostics, kept out
  of the hardware-facing tree.

Drops the redundant disk-report mirror write in RunStorageAcceptancePack
and the now-dead syncDirectoryTree/removeMissingPaths; adds a size+mtime
skip in copyPath so the blackbox mirror doesn't rewrite unchanged files
every cycle. README.md rewritten to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-28 14:33:26 +03:00
Mikhail ChusavitinandClaude Sonnet 5 5aee146903 app: ship a README.md in every support bundle and blackbox capture
An agent handed a bundle path had no way to know its layout without
grepping through random logs first — confirmed by watching a separate
session read bee-nvidia.log before anything else on a real bundle. Embeds
a single README.md (bee-embed, internal/app/assets/) explaining what bee
is and giving direct answers to the questions someone analyzing a bundle
is most likely to ask (did the tests pass, what hardware is this, is a
service healthy, RAID/GPU/NVLink state, etc), written at the bundle root
by both BuildSupportBundle (support-bundle archive root, sibling of
manifest.txt) and blackboxWorker.syncCycle (removable-media boot-folder
root, which otherwise has no manifest.txt-equivalent pointing anywhere).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 11:37:44 +03:00
Mikhail ChusavitinandClaude Sonnet 5 a3377083aa webui: add /topo server topology page
New read-only visualization page: CPU sockets as anchor nodes, PCIe
devices (GPU/NIC/RAID) linked to their NUMA-affine socket with edge
color derived strictly from link_speed vs max_link_speed (not from the
device's own Status, which can also be overwritten by SAT-test results
on the same field), PSU/BMC as standalone boxes with no connecting
line, and a separate NVLink Topology card (live nvidia-smi topo -m /
nvlink -s/-e queries, not persisted to any contract).

GPU-GPU edges are drawn strictly from the actual bonded-pair list
parsed out of "nvidia-smi topo -m" (parseGPUPairAdjacency), not from
adjacent box position in the layout — an earlier ASCII mockup drew a
"chain" through unrelated GPUs, which a dedicated regression test now
guards against. A bonded pair spanning two different NUMA nodes is
flagged Warning on the edge and on both GPU boxes, per project
decision that this is an anomaly worth surfacing, not a neutral fact.

Zero changes to the ingest contract: this reverts the HardwareNVLinkPort/
HardwarePCIeDevice.NVLinks field shipped in v11.55 (33d6eee) along with
its collector/nvidia.go enrichment — that field risked a 400 from
Reanimator Core's strict decoder without an RFC, and isn't needed since
the topo page queries nvidia-smi directly instead of reading it from
audit.json. The v11.55 systemd fix (bee-nvidia.service ordering,
nv-hostengine restart) and the nvlink-status/-errors/dcgmi dumps in the
support bundle are untouched.

Also reorganizes support bundle collection per project convention:
system/ is now LiveCD-operational logs only (Xorg, services, console,
network/FS of the host itself); all server-hardware dumps (lspci,
NVIDIA/NVLink/DCGM, fabric manager, PCIe AER, ethtool, mstflint) move
to techdump/, deduplicating two entries already produced by
platform/techdump.go. Adds nvidia-bug-report.sh (previously only
collected inside the NVIDIA SAT pack) and lscpu to the always-on dump.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 11:50:33 +03:00
Mikhail Chusavitin 33d6eee9cf sat: collect NVLink status/errors, fix nv-hostengine restart via systemd
Add NVLink port status (nvidia-smi nvlink -s), error counters
(nvidia-smi nvlink -e), and dcgmi nvlink status to the support bundle,
and enrich HardwarePCIeDevice entries with per-link telemetry.

Replace the raw nv-hostengine pkill/restart dance in bee-nvidia-load
with systemctl restart/start of nvidia-dcgm.service, and order
bee-nvidia.service Before= nvidia-dcgm.service and
nvidia-fabricmanager.service so modules/device nodes exist before
those units start.
2026-07-07 10:21:56 +03:00
mchus 6c2b188ec9 Add no-GUI boot mode and quieter boot diagnostics 2026-05-03 21:14:45 +03:00
mchus 29179917c3 Add USB blackbox log mirroring service 2026-04-24 10:20:12 +03:00
mchus b3cf8e3893 Globalize autotuned system power source 2026-04-20 07:02:12 +03:00
Mikhail Chusavitin 7237e4d3e4 Add fabric manager boot and support diagnostics 2026-04-15 16:14:26 +03:00
mchusandClaude Sonnet 4.6 ba16021cdb Fix GPU model propagation, export filenames, PSU/service status, and chart perf
- nvidia.go: add Name field to nvidiaGPUInfo, include model name in
  nvidia-smi query, set dev.Model in enrichPCIeWithNVIDIAData
- pages.go: fix duplicate GPU count in validate card summary (4 GPU: 4 x …
  → 4 x … GPU); fix PSU UNKNOWN fallback from hw.PowerSupplies; treat
  activating/deactivating/reloading service states as OK in Runtime Health
- support_bundle.go: use "150405" time format (no colons) for exFAT compat
- sat.go / benchmark.go / platform_stress.go / sat_fan_stress.go: remove
  .tar.gz archive creation from export dirs — export packs everything itself
- charts_svg.go: add min-max downsampling (1400 pt cap) for SVG chart perf
- benchmark_report.go / sat.go: normalize GPU fallback to "Unknown GPU"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 10:05:27 +03:00
Mikhail Chusavitin 4ef403898f Tighten NVIDIA GPU PCI detection 2026-04-09 15:14:48 +03:00
mchusandClaude Sonnet 4.6 025548ab3c UI: amber accents, smaller wallpaper logo, new support bundle name, drop display resolution
- Bootloader: GRUB fallback text colors → yellow/brown (amber tone)
- CLI charts: all GPU metric series use single amber color (xterm-256 #214)
- Wallpaper: logo width scaled to 400 px dynamically, shadow scales with font size
- Support bundle: renamed to YYYY-MM-DD (BEE-SP vX.X) SRV_MODEL SRV_SN ToD.tar.gz
  using dmidecode for server model (spaces→underscores) and serial number
- Remove display resolution feature (UI card, API routes, handlers, tests)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 21:37:01 +03:00
Mikhail Chusavitin 531d1ca366 Add NVIDIA self-heal tools and per-GPU SAT status 2026-04-07 20:20:05 +03:00
mchus 143b7dca5d Add stability hardening and self-heal recovery 2026-04-05 10:29:37 +03:00
Mikhail Chusavitin 728270dc8e Unblock bee-web startup and expand support bundle diagnostics 2026-04-04 15:18:43 +03:00
Mikhail ChusavitinandClaude Sonnet 4.6 99cece524c feat(support-bundle): add PCIe link diagnostics and system logs
- Add full dmesg (was tail -200), kern.log, syslog
- Add /proc/cmdline, lspci -vvv, nvidia-smi -q
- Add per-GPU PCIe link speed/width from sysfs (NVIDIA devices only)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-02 15:42:28 +03:00
Mikhail Chusavitin e7df63bae1 fix(app): include extra system logs in support bundle 2026-04-02 13:44:58 +03:00
Mikhail Chusavitin b2b0444131 audit: ignore virtual hdisk and coprocessor noise 2026-04-02 09:56:17 +03:00
Mikhail Chusavitin c394845b34 refactor(webui): queue install and bundle tasks - v3.18 2026-04-01 08:46:46 +03:00
Mikhail Chusavitin 9a1df9b1ba Tighten support bundles and fix AMD runtime checks 2026-03-25 19:35:25 +03:00
Mikhail Chusavitin b25a2f6d30 feat: add support bundle and raw audit export 2026-03-16 18:20:26 +03:00