refactor: modularize audit and harden build validation

2026-08-31 21:22:16 +03:00
parent bb22ccfafe
commit ac4bc0b2b7
78 changed files with 13598 additions and 13130 deletions
@@ -0,0 +1,66 @@
+# bee-nvidia.service: never call blocking `systemctl restart` on units ordered After= itself
+
+**Date:** 2026-08-31
+**Status:** active
+
+## Symptom
+
+Every affected NVIDIA boot reaches two 60-second wrapper timeouts in
+`bee-nvidia.service`. `bee-nvidia.log`:
+
+```
+[bee-nvidia] restarting nvidia-fabricmanager.service (timeout 60s)
+[bee-nvidia] WARN: systemctl restart nvidia-fabricmanager.service timed out after 60s
+[bee-nvidia] restarting nvidia-dcgm.service (timeout 60s)
+[bee-nvidia] WARN: systemctl restart nvidia-dcgm.service timed out after 60s
+[bee-nvidia] done
+```
+
+Seen on both `210619KUGGXGS2000008` and `...017` (8x H200 NVL). That 120 s
+window overlaps exactly with when an operator opens the web UI and clicks
+"Run All" - during it `nvidia-smi` may not yet enumerate all GPUs, which is
+how bundle `...017` ended up running the check set with **no GPU tests**
+(see [2026-08-31-backend-driven-sat-planning.md](2026-08-31-backend-driven-sat-planning.md)).
+
+## Root cause
+
+`bee-nvidia.service` is `Type=oneshot` and `Before=nvidia-fabricmanager.service
+nvidia-dcgm.service`. Its `ExecStart` (`bee-nvidia-load`) then ran, synchronously:
+
+```
+timeout 60 systemctl restart nvidia-fabricmanager.service
+timeout 60 systemctl restart nvidia-dcgm.service
+```
+
+A oneshot unit is not "active" until `ExecStart` returns. Both target units
+are ordered `After=bee-nvidia.service`, so systemd queues them behind
+bee-nvidia and will not run them while `bee-nvidia-load` is still executing.
+`bee-nvidia-load` blocks on `systemctl restart` waiting for exactly that job
+to complete -> deadlock -> broken only when `timeout 60` fires. Twice.
+
+`nvidia-smi -q` inside `bee-check-nvswitch` (the fabricmanager ExecCondition)
+is not part of this ordering cycle; the deadlock is structural.
+
+## Decision
+
+`bee-nvidia-load` no longer blocks on those units. It calls
+`systemctl --no-block try-restart` for each:
+
+- `--no-block` returns immediately; systemd runs the job after bee-nvidia
+  exits, via the existing `Before=` ordering.
+- `try-restart` only acts if the unit is already running (the "stale instance
+  from a reload / re-run" case the old code worried about). If it is inactive,
+  this command does nothing; an enabled unit already queued by the normal boot
+  transaction starts after `bee-nvidia.service` via the declared ordering.
+
+The `SYSTEMCTL_TIMEOUT` / `timeout_systemctl` wrapper and the fallback
+`systemctl start` / `systemctl status` branches are gone. `--no-block` means
+systemctl does not wait for the queued unit job to finish.
+
+## Consequences
+
+- NVIDIA boot no longer waits for those two 60-second wrapper deadlines.
+- DO NOT reintroduce a synchronous `systemctl {start,restart}` of any unit
+  that is `After=bee-nvidia.service` from inside `bee-nvidia-load`. If a unit
+  genuinely must be up before the script returns, invert the ordering
+  instead.