refactor: harden diagnostics and consolidate runtime code

This commit is contained in:
Mikhail Chusavitin
2026-09-01 13:01:28 +03:00
parent ac4bc0b2b7
commit 0a6ca8ba0f
49 changed files with 1441 additions and 837 deletions
+17
View File
@@ -263,6 +263,11 @@ bee-blackbox.service (separate process from bee-web/bee-audit)
a time-derived staging path: two builds in the same wall-clock second (two
operators, or an on-demand build racing the blackbox worker) then share one
tree and one's deferred `os.RemoveAll` truncates the other's archive.
- The resulting archive is also unique (`os.CreateTemp` with the readable
support-bundle name as its prefix) and remains a hidden `.partial` from
retention cleanup until an atomic rename publishes the closed `.tar.gz`.
A timestamp-only output path lets same-second writers share one inode and
can corrupt both gzip streams.
## NVIDIA SAT Web UI flow
@@ -306,3 +311,15 @@ Web UI: "Run All" button -> POST /api/sat/run-all
- `RunNvidiaBandwidthPack` runs one all-GPU `nvbandwidth` pass in Validate; the
per-NUMA-node matrix is Stress-tier only (`fullMatrix` arg). See
`decisions/2026-08-31-nvbandwidth-validate-single-deep-matrix.md`.
- `nvidia-config` validates the active-link count independently at both ends of
every `NVx` pair reported by `nvidia-smi topo -m`; aggregate counts must not
be compared with a per-endpoint `NVx` value. It also persists matching
NVIDIA degraded-mode kernel lines in `06-dmesg-nvlink-degraded.log`.
- Generic SAT jobs may attach an output validator. A validator runs only after
exit status 0 and returns `OK`, `FAILED`, or `UNSUPPORTED`; its detail is
persisted in both the job log and `<job>_detail` in `summary.txt`.
- `nccl-tests` explicitly validates the pinned `all_reduce_perf` correctness
summary and both `#wrong` columns, and cross-checks selected NVLink endpoints
against the captured topology/status. `Avg bus bandwidth` is retained as a
diagnostic value only: no model-independent bandwidth threshold is valid.
NCCL `P2P` alone is not proof of NVLink because P2P can also use PCIe.