mchusandClaude Opus 4.8
a9924b08b0
Replaces the old nvbandwidth-all-gpu-power-watch scenario (dropped: the power/IPMI-sampler framing was a dead-end — the blackbox shows every other load test, incl. NCCL all-reduce across all 6 GPUs, passes with power fully connected; only nvbandwidth on cross-socket GPU P2P reboots the box). nvbandwidth-acs-ab is a two-phase reproduction for the MSI CG480-S6053 case. Phase 1 disables PCIe ACS redirect at runtime via setpci (saving each bridge's original ACS control word to /run/bee-acs-orig) then runs nvbandwidth across all 6 GPUs — expected to survive. Phase 2 restores the saved ACS values and repeats — expected to reproduce the reboot. Safe phase first so its full blackbox reaches the USB stick before the risky phase can reset the box (sync-bracket before the final job flushes phase-1 evidence). ACS mutation runs via `bash -c` since scenario cmd is exec'd directly with no shell. Both shipped copies (repo scenarios/ and image overlay) identical; verified both parse via ParseScenarioJSON (4 jobs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
30 lines
1.8 KiB
JSON
30 lines
1.8 KiB
JSON
{
|
|
"name": "nvbandwidth-acs-ab",
|
|
"description": "Two-phase A/B for the MSI CG480-S6053 cross-socket GPU P2P hard-reboot. Phase 1 disables PCIe ACS redirect at runtime via setpci (P2P goes device-direct instead of up to the root complex) then runs nvbandwidth across all 6 GPUs — the run expected to SURVIVE. Phase 2 restores ACS to the BIOS default and repeats — the run expected to REPRODUCE the reboot. Safe phase runs first so its full blackbox reaches the USB stick before the risky phase can reset the machine.",
|
|
"timeout_sec": 1800,
|
|
"jobs": [
|
|
{
|
|
"name": "acs-disable",
|
|
"type": "command",
|
|
"cmd": ["bash", "-c", ": > /run/bee-acs-orig; for bdf in $(lspci -D | awk '{print $1}'); do setpci -s $bdf ECAP_ACS.w >/dev/null 2>&1 || continue; v=$(setpci -s $bdf ECAP_ACS+0x6.w 2>/dev/null) || continue; echo $bdf $v >> /run/bee-acs-orig; setpci -s $bdf ECAP_ACS+0x6.w=0000 >/dev/null 2>&1 || true; done; echo ACS disabled on $(wc -l < /run/bee-acs-orig) bridges; echo -n 'bridges still ReqRedir+: '; lspci -vvv 2>/dev/null | grep -c 'ReqRedir+' || true"]
|
|
},
|
|
{
|
|
"name": "nvbandwidth-acs-off",
|
|
"type": "command",
|
|
"cmd": ["dcgmi", "diag", "-r", "nvbandwidth", "-i", "{{gpus}}"],
|
|
"gpu_indices": [0, 1, 2, 3, 4, 5]
|
|
},
|
|
{
|
|
"name": "acs-restore",
|
|
"type": "command",
|
|
"cmd": ["bash", "-c", "test -s /run/bee-acs-orig || { echo no saved ACS state; exit 0; }; while read -r bdf v; do setpci -s $bdf ECAP_ACS+0x6.w=$v >/dev/null 2>&1 || true; done < /run/bee-acs-orig; echo ACS restored on $(wc -l < /run/bee-acs-orig) bridges; echo -n 'bridges ReqRedir+ now: '; lspci -vvv 2>/dev/null | grep -c 'ReqRedir+' || true"]
|
|
},
|
|
{
|
|
"name": "nvbandwidth-acs-on",
|
|
"type": "command",
|
|
"cmd": ["dcgmi", "diag", "-r", "nvbandwidth", "-i", "{{gpus}}"],
|
|
"gpu_indices": [0, 1, 2, 3, 4, 5]
|
|
}
|
|
]
|
|
}
|