platform/app: fix nvbandwidth split fallback, blackbox discovery churn, and add blocking sync brackets around load steps

- gpuBandwidthSocketGroups: a single GPU whose NUMA node fails to resolve
  no longer collapses the whole per-socket nvbandwidth split into one
  fallback pass — it now folds into the last resolved group instead,
  preserving isolation for the sockets that did resolve.
- blackbox discoverMarkedTargets: skip mounting/unmounting devices that
  already have a running worker on every 2s discovery tick. This was
  observed hammering the same USB target continuously (mount+unmount
  every ~2s for the whole session) and contending with the worker's own
  sync cycle, plausibly explaining multi-minute sync cycles seen on a
  real crash bundle.
- syncFilesystem now calls syscall.Sync() directly instead of spawning
  /bin/sync per copied file; blackbox mounts removable targets with
  -o sync so writes are durable without relying on the app-level sync as
  the primary mechanism.
- New platform.SetSyncBracketHook / satJob.syncBracket: blocks (with a
  bounded timeout) on blackbox actually reaching removable media right
  before and right after a diagnostic's real load step (nvbandwidth,
  memtester, stress-ng, dcgmi diag, nccl, smartctl/nvme self-test...),
  instead of only firing a fire-and-forget kick after the job's own log
  file is written. A crash mid-load now has durable evidence the load
  started, not just whatever streamed to the RAM-backed export dir before
  blackbox's next scheduled cycle.

Found investigating a real support bundle where blackbox's last
successful sync (19:55:25) predated both the previous job finishing and
the crashing nvbandwidth job starting (19:56:57) — none of the crash
window ever reached durable media.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-07-28 14:22:05 +03:00
co-authored by Claude Sonnet 5
parent 6d13c17d36
commit e03267a72f
8 changed files with 438 additions and 51 deletions
@@ -31,16 +31,22 @@ func gpuBandwidthSocketGroups(gpuIndices []int, logFunc func(string)) [][]int {
}
byNode := map[int][]int{}
var unresolved []int
for _, idx := range gpuIndices {
node, ok := nodes[idx]
if !ok {
if logFunc != nil {
logFunc(fmt.Sprintf("nvbandwidth: no NUMA node resolved for GPU %d; running all GPUs as one group", idx))
logFunc(fmt.Sprintf("nvbandwidth: no NUMA node resolved for GPU %d; will fold it into a resolved socket group instead of dropping the split", idx))
}
return [][]int{gpuIndices}
unresolved = append(unresolved, idx)
continue
}
byNode[node] = append(byNode[node], idx)
}
// Fewer than two resolved sockets means there's nothing to split either
// way: every GPU's node is unknown, or every resolved GPU shares one
// socket. A single unresolved GPU among an otherwise clean multi-socket
// system shouldn't cost us the split, so only bail out here.
if len(byNode) < 2 {
return [][]int{gpuIndices}
}
@@ -55,6 +61,14 @@ func gpuBandwidthSocketGroups(gpuIndices []int, logFunc func(string)) [][]int {
for _, node := range sortedNodes {
groups = append(groups, dedupeSortedIndices(byNode[node]))
}
if len(unresolved) > 0 {
// Fold into the last group rather than running unresolved GPUs in a
// group of their own — a lone GPU can't run a GPU-to-GPU bandwidth
// test by itself, and the point of the split is to isolate the
// sockets we *do* know about, not to also isolate the unknown one.
last := len(groups) - 1
groups[last] = dedupeSortedIndices(append(groups[last], unresolved...))
}
return groups
}