refactor: harden diagnostics and consolidate runtime code

This commit is contained in:
Mikhail Chusavitin
2026-09-01 13:01:28 +03:00
parent ac4bc0b2b7
commit 0a6ca8ba0f
49 changed files with 1441 additions and 837 deletions
+5 -1
View File
@@ -125,7 +125,7 @@ All SAT run endpoints enqueue an async task. Response: `{"task_id": "..."}`.
| Method | Path | Description |
|--------|----------------------------|----------------------------------------------------|
| GET | `/api/gpu/presence` | `{"nvidia": bool, "amd": bool}` |
| GET | `/api/gpu/presence` | `{"nvidia", "amd", "nvidia_initializing", "amd_initializing": bool}`. `*_initializing` = PCI GPU present but driver not up yet (distinct from absent). Backed by `app.DetectGPUPresence`. |
| GET | `/api/gpu/nvidia` | List NVIDIA GPUs from nvidia-smi |
| GET | `/api/gpu/nvidia-status` | Per-GPU status (ECC, power, throttle) |
| POST | `/api/gpu/nvidia-reset` | GPU reset by index |
@@ -139,6 +139,10 @@ All SAT run endpoints enqueue an async task. Response: `{"task_id": "..."}`.
|--------|------------------------------|---------------------------------------------------|
| GET | `/api/system/ram-status` | toram boot state and ISO copy status |
| POST | `/api/system/install-to-ram` | Copy ISO to RAM (background task) |
| GET | `/api/system/time` | Current host epoch, local wall-clock text, and timezone |
| POST | `/api/system/time-sync` | Set host time and timezone from the browser |
| POST | `/api/system/reboot` | Reboot the host |
| POST | `/api/system/shutdown` | Power off the host |
| GET | `/api/install/disks` | List block devices suitable for disk installation |
| POST | `/api/install/run` | Install bee to disk (background task) |
+17
View File
@@ -263,6 +263,11 @@ bee-blackbox.service (separate process from bee-web/bee-audit)
a time-derived staging path: two builds in the same wall-clock second (two
operators, or an on-demand build racing the blackbox worker) then share one
tree and one's deferred `os.RemoveAll` truncates the other's archive.
- The resulting archive is also unique (`os.CreateTemp` with the readable
support-bundle name as its prefix) and remains a hidden `.partial` from
retention cleanup until an atomic rename publishes the closed `.tar.gz`.
A timestamp-only output path lets same-second writers share one inode and
can corrupt both gzip streams.
## NVIDIA SAT Web UI flow
@@ -306,3 +311,15 @@ Web UI: "Run All" button -> POST /api/sat/run-all
- `RunNvidiaBandwidthPack` runs one all-GPU `nvbandwidth` pass in Validate; the
per-NUMA-node matrix is Stress-tier only (`fullMatrix` arg). See
`decisions/2026-08-31-nvbandwidth-validate-single-deep-matrix.md`.
- `nvidia-config` validates the active-link count independently at both ends of
every `NVx` pair reported by `nvidia-smi topo -m`; aggregate counts must not
be compared with a per-endpoint `NVx` value. It also persists matching
NVIDIA degraded-mode kernel lines in `06-dmesg-nvlink-degraded.log`.
- Generic SAT jobs may attach an output validator. A validator runs only after
exit status 0 and returns `OK`, `FAILED`, or `UNSUPPORTED`; its detail is
persisted in both the job log and `<job>_detail` in `summary.txt`.
- `nccl-tests` explicitly validates the pinned `all_reduce_perf` correctness
summary and both `#wrong` columns, and cross-checks selected NVLink endpoints
against the captured topology/status. `Avg bus bandwidth` is retained as a
diagnostic value only: no model-independent bandwidth threshold is valid.
NCCL `P2P` alone is not proof of NVLink because P2P can also use PCIe.