bee/bible-local/docs/customer-gpu-test-methodology.md

GPU PCIe Test Methodology

Which of the checks below run is decided by the backend (POST /api/sat/run-all -> handler.planSATRunAll), not the web page: it enumerates the hardware, waits for the NVIDIA driver to finish enumerating GPUs, and enqueues only the applicable checks. The TPM check is planned only when sysfs reports a TPM 2.x device (tpm_version_major=2). See bible-local/decisions/2026-08-31-backend-driven-sat-planning.md.

Validate

• CPU check
• lscpu
• sensors
• stress-ng
• Memory check
• free
• timeout <timeout_sec> memtester
• free
• NVMe storage check
• nvme id-ctrl
• nvme smart-log
• nvme device-self-test
• SATA/SAS storage check
• smartctl -H -A
• smartctl -t short
• Basic NVIDIA GPU check
• nvidia-smi -pm 1
• nvidia-smi -q
• dmidecode -t baseboard
• dmidecode -t system
• dcgmi diag -r 2
• Inter-GPU communication check
• all_reduce_perf
• GPU bandwidth check
• dcgmi diag -r nvbandwidth - one pass across all GPUs (-i <all>). No NUMA-locality split in Validate.

Validate -> Stress

• Extended NVIDIA GPU check
• nvidia-smi -pm 1
• nvidia-smi -q
• dmidecode -t baseboard
• dmidecode -t system
• dcgmi diag -r 3
• NVIDIA targeted stress
• nvidia-smi -pm 1
• nvidia-smi -q
• dcgmi diag -r targeted_stress
• NVIDIA targeted power
• nvidia-smi -pm 1
• nvidia-smi -q
• dcgmi diag -r targeted_power
• NVIDIA pulse test
• nvidia-smi -pm 1
• nvidia-smi -q
• dcgmi diag -r pulse_test
• Inter-GPU communication check
• all_reduce_perf
• GPU bandwidth check
• dcgmi diag -r nvbandwidth - per-NUMA-node matrix: one pass per completely resolved Linux PCI NUMA group, then one all-GPU pass. If any selected GPU has no resolved numa_node, no group is guessed and the test falls back to one all-GPU pass. See bible-local/decisions/2026-07-27-nvbandwidth-per-socket-split.md and bible-local/decisions/2026-08-31-nvbandwidth-validate-single-deep-matrix.md.