fix(collector): stop pinning GPU PCIe status on an unverified idle reading
NVIDIA GPUs deliberately downclock PCIe to Gen1 at idle for power saving, and applyPCIeLinkSpeedWarning fired on every idle collector pass regardless - since component-status DB records never downgrade (Record() only ever raises severity), one boot-time idle sample permanently pinned pcie:gpu:nvidia to Warning for the rest of the session even after every load-bearing GPU SAT test passed clean. Two prior fixes (nvidia-smi-sourced link speed, pcie_aspm=off boot flag) didn't hold up against this hardware/driver combination - see bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning.md for the full history. Rather than add a downgrade path, stop writing an unverified status in the first place: parseLspciDevice no longer calls applyPCIeLinkSpeedWarning on the idle path. LinkSpeed/MaxLinkSpeed stay populated as plain descriptive fields; only a verified-under-load caller may now turn them into a status verdict. Two new SAT targets provide that verified signal: - pcie-link (platform/pcie_link_check.go): forces every enabled PCIe device - not just GPUs - to retrain via the PCIe spec's Link Control "Retrain Link" bit, then compares the negotiated speed against the device's max. Covers NICs/HBAs/switches that have no bee-gpu-burn equivalent load tool. Classifies by PCI class code + vendor ID, not name substrings. Routes gpu_nvidia/gpu_amd/other sub-verdicts into their own component-status keys so a degraded NIC never reads as a GPU fault. - nvidia-pcie-bandwidth (platform/nvidia_pcie_bandwidth.go): drives real host<->device traffic via dcgmi diag -r nvbandwidth and resamples link speed immediately after, independent of nvbandwidth's own pass/fail. Both wired into the task queue/webui the same way as nvidia-config (routes, dispatch, priority, Validate page cards, Run All Check SAT). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
7aa276320b
commit
b11018ac5e
@@ -139,6 +139,8 @@ type satRunner interface {
|
||||
RunMemoryAcceptancePack(ctx context.Context, baseDir string, sizeMB, passes int, logFunc func(string)) (string, error)
|
||||
RunStorageAcceptancePack(ctx context.Context, baseDir string, extended bool, logFunc func(string)) (string, error)
|
||||
RunNvidiaConfigCheckPack(ctx context.Context, baseDir string, logFunc func(string)) (string, error)
|
||||
RunPCIeLinkCheckPack(ctx context.Context, baseDir string, logFunc func(string)) (string, error)
|
||||
RunNvidiaPCIeBandwidthPack(ctx context.Context, baseDir string, gpuIndices []int, logFunc func(string)) (string, error)
|
||||
RunCPUAcceptancePack(ctx context.Context, baseDir string, durationSec int, logFunc func(string)) (string, error)
|
||||
ListNvidiaGPUs() ([]platform.NvidiaGPU, error)
|
||||
ListNvidiaGPUSettings() ([]platform.NvidiaGPUSetting, error)
|
||||
|
||||
@@ -274,6 +274,38 @@ func (a *App) RunNvidiaConfigCheckPackResult(baseDir string) (ActionResult, erro
|
||||
return ActionResult{Title: "GPU Config & NVLink Check", Body: satResultBody(path)}, err
|
||||
}
|
||||
|
||||
func (a *App) RunPCIeLinkCheckPackCtx(ctx context.Context, baseDir string, logFunc func(string)) (string, error) {
|
||||
if strings.TrimSpace(baseDir) == "" {
|
||||
baseDir = DefaultSATBaseDir
|
||||
}
|
||||
return a.sat.RunPCIeLinkCheckPack(ctx, baseDir, logFunc)
|
||||
}
|
||||
|
||||
func (a *App) RunPCIeLinkCheckPack(baseDir string, logFunc func(string)) (string, error) {
|
||||
return a.RunPCIeLinkCheckPackCtx(context.Background(), baseDir, logFunc)
|
||||
}
|
||||
|
||||
func (a *App) RunPCIeLinkCheckPackResult(baseDir string) (ActionResult, error) {
|
||||
path, err := a.RunPCIeLinkCheckPack(baseDir, nil)
|
||||
return ActionResult{Title: "PCIe Link Check", Body: satResultBody(path)}, err
|
||||
}
|
||||
|
||||
func (a *App) RunNvidiaPCIeBandwidthPackCtx(ctx context.Context, baseDir string, gpuIndices []int, logFunc func(string)) (string, error) {
|
||||
if strings.TrimSpace(baseDir) == "" {
|
||||
baseDir = DefaultSATBaseDir
|
||||
}
|
||||
return a.sat.RunNvidiaPCIeBandwidthPack(ctx, baseDir, gpuIndices, logFunc)
|
||||
}
|
||||
|
||||
func (a *App) RunNvidiaPCIeBandwidthPack(baseDir string, gpuIndices []int, logFunc func(string)) (string, error) {
|
||||
return a.RunNvidiaPCIeBandwidthPackCtx(context.Background(), baseDir, gpuIndices, logFunc)
|
||||
}
|
||||
|
||||
func (a *App) RunNvidiaPCIeBandwidthPackResult(baseDir string) (ActionResult, error) {
|
||||
path, err := a.RunNvidiaPCIeBandwidthPack(baseDir, nil, nil)
|
||||
return ActionResult{Title: "NVIDIA GPU PCIe Bandwidth Check", Body: satResultBody(path)}, err
|
||||
}
|
||||
|
||||
func (a *App) DetectGPUVendor() string {
|
||||
return a.sat.DetectGPUVendor()
|
||||
}
|
||||
|
||||
@@ -287,6 +287,14 @@ func (f fakeSAT) RunNvidiaConfigCheckPack(_ context.Context, baseDir string, _ f
|
||||
return "", nil
|
||||
}
|
||||
|
||||
func (f fakeSAT) RunPCIeLinkCheckPack(_ context.Context, baseDir string, _ func(string)) (string, error) {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
func (f fakeSAT) RunNvidiaPCIeBandwidthPack(_ context.Context, baseDir string, _ []int, _ func(string)) (string, error) {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
func (f fakeSAT) RunCPUAcceptancePack(_ context.Context, baseDir string, durationSec int, _ func(string)) (string, error) {
|
||||
if f.runCPUFn != nil {
|
||||
return f.runCPUFn(baseDir, durationSec)
|
||||
|
||||
@@ -253,10 +253,32 @@ func ApplySATResultToDB(db *ComponentStatusDB, target, archivePath string) {
|
||||
// otherwise fails to match any real BDF.
|
||||
switch target {
|
||||
case "nvidia", "nvidia-targeted-stress", "nvidia-compute", "nvidia-targeted-power", "nvidia-pulse",
|
||||
"nvidia-interconnect", "nvidia-bandwidth", "nvidia-stress", "nvidia-config":
|
||||
"nvidia-interconnect", "nvidia-bandwidth", "nvidia-stress", "nvidia-config", "nvidia-pcie-bandwidth":
|
||||
db.Record("pcie:gpu:nvidia", source, dbStatus, detail)
|
||||
case "amd", "amd-stress", "amd-mem", "amd-bandwidth":
|
||||
db.Record("pcie:gpu:amd", source, dbStatus, detail)
|
||||
case "pcie-link":
|
||||
// Forced-retrain PCIe link check (audit/internal/platform/pcie_link_check.go):
|
||||
// the only verified (non-idle-sampled) source for PCIe link-speed
|
||||
// status. summary.txt carries up to three independent sub-verdicts
|
||||
// — record each into its own component key rather than collapsing
|
||||
// them into one, since a degraded NIC/HBA shouldn't be reported as
|
||||
// a GPU fault or vice versa.
|
||||
recordPCIeLinkSubStatus := func(key, kvKey string) {
|
||||
v, ok := kv[kvKey]
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
st := strings.ToUpper(strings.TrimSpace(v))
|
||||
d := "pcie-link SAT: " + st
|
||||
if st != "OK" && kv["warnings"] != "" {
|
||||
d += " — " + kv["warnings"]
|
||||
}
|
||||
db.Record(key, source, satStatusToDBStatus(st), d)
|
||||
}
|
||||
recordPCIeLinkSubStatus("pcie:gpu:nvidia", "gpu_nvidia_status")
|
||||
recordPCIeLinkSubStatus("pcie:gpu:amd", "gpu_amd_status")
|
||||
recordPCIeLinkSubStatus("pcie:link:other", "other_status")
|
||||
case "memory", "memory-stress", "sat-stress":
|
||||
db.Record("memory:all", source, dbStatus, detail)
|
||||
case "cpu", "platform-stress":
|
||||
|
||||
@@ -187,8 +187,20 @@ func parseLspciDevice(fields map[string]string) schema.HardwarePCIeDevice {
|
||||
markNVLinkBridge(&dev)
|
||||
}
|
||||
|
||||
// Warn (or Critical for NVLink bridges) if PCIe link is running below max.
|
||||
applyPCIeLinkSpeedWarning(&dev)
|
||||
// Deliberately not calling applyPCIeLinkSpeedWarning here: this function
|
||||
// runs on every idle-time collector pass, and NVIDIA GPUs legitimately
|
||||
// downclock their PCIe link at idle to save power, making "current <
|
||||
// max" a false positive most of the time it fires. There's no reliable
|
||||
// way to downgrade a status once recorded (component-status DB history
|
||||
// is a one-way transition log, by design — see component_status_db.go),
|
||||
// so writing an unverified idle reading as Warning meant it stuck for
|
||||
// the rest of the session even after every load-bearing SAT test passed
|
||||
// clean. See bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning-unresolved.md.
|
||||
// applyPCIeLinkSpeedWarning is reserved for a future caller that samples
|
||||
// link speed while the device is under verified load (a dedicated SAT,
|
||||
// or a forced link retrain) — only that caller should turn this data
|
||||
// into a status verdict. LinkSpeed/MaxLinkSpeed above stay populated as
|
||||
// plain descriptive fields regardless.
|
||||
|
||||
return dev
|
||||
}
|
||||
@@ -283,6 +295,12 @@ func readPCIStringAttribute(bdf, attribute string) (string, bool) {
|
||||
// their link state has no operational impact. This covers management endpoints
|
||||
// (e.g. PCIe switch fabric controllers on HGX baseboards) that the kernel never
|
||||
// activates but that lspci still reports with link stats.
|
||||
//
|
||||
// Not called from the idle collector path (parseLspciDevice) — see the
|
||||
// comment there. Call this only from a context that just put the device
|
||||
// under real traffic (a load-bearing SAT, a forced link retrain), so a
|
||||
// genuine Warning/Critical is never based on an ASPM/power-managed idle
|
||||
// downclock that a human or SAT would immediately clear anyway.
|
||||
func applyPCIeLinkSpeedWarning(dev *schema.HardwarePCIeDevice) {
|
||||
if dev.LinkSpeed == nil || dev.MaxLinkSpeed == nil {
|
||||
return
|
||||
@@ -330,6 +348,14 @@ func pcieLinkSpeedRank(gen string) int {
|
||||
}
|
||||
|
||||
func normalizePCILinkSpeed(raw string) string {
|
||||
return NormalizePCILinkSpeed(raw)
|
||||
}
|
||||
|
||||
// NormalizePCILinkSpeed maps a raw sysfs current_link_speed/max_link_speed
|
||||
// string (e.g. "16.0 GT/s PCIe") to a "GenN" label. Exported for reuse by
|
||||
// the platform package's PCIe link-retrain SAT, which needs the same
|
||||
// mapping outside the idle collector path.
|
||||
func NormalizePCILinkSpeed(raw string) string {
|
||||
raw = strings.TrimSpace(strings.ToLower(raw))
|
||||
switch {
|
||||
case strings.Contains(raw, "2.5"):
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
package platform
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// nvidiaPCIeBandwidthFinding is one GPU's post-load PCIe link-speed result.
|
||||
type nvidiaPCIeBandwidthFinding struct {
|
||||
Index int
|
||||
BDF string
|
||||
BeforeSpeed string
|
||||
AfterSpeed string
|
||||
MaxSpeed string
|
||||
Width int
|
||||
MaxWidth int
|
||||
Degraded bool
|
||||
}
|
||||
|
||||
// RunNvidiaPCIeBandwidthPack drives real host<->device traffic across the
|
||||
// GPUs' PCIe links (via `dcgmi diag -r nvbandwidth`, the same tool the
|
||||
// existing nvidia-bandwidth SAT uses for GPU-to-GPU throughput) and then
|
||||
// resamples each GPU's PCIe link speed from sysfs immediately afterward.
|
||||
//
|
||||
// This is deliberately a separate, narrower check from "nvidia-bandwidth":
|
||||
// that SAT's overall_status reflects nvbandwidth's own pass/fail (did the
|
||||
// measured GB/s clear its internal threshold), never PCIe link speed
|
||||
// itself. This pack exists purely to answer "did sustained real traffic
|
||||
// make the link train up to its negotiated maximum" — the load-bearing
|
||||
// counterpart to the idle-time collector reading that used to (falsely)
|
||||
// drive pcie:gpu:nvidia's status. See
|
||||
// bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning-unresolved.md.
|
||||
func (s *System) RunNvidiaPCIeBandwidthPack(ctx context.Context, baseDir string, gpuIndices []int, logFunc func(string)) (string, error) {
|
||||
if ctx == nil {
|
||||
ctx = context.Background()
|
||||
}
|
||||
if baseDir == "" {
|
||||
baseDir = "/var/log/bee-sat"
|
||||
}
|
||||
ts := time.Now().UTC().Format("20060102-150405")
|
||||
runDir := filepath.Join(baseDir, "nvidia-pcie-bandwidth-"+ts)
|
||||
if err := os.MkdirAll(runDir, 0755); err != nil {
|
||||
return "", err
|
||||
}
|
||||
verboseLog := filepath.Join(runDir, "verbose.log")
|
||||
|
||||
bdfByIndex, err := gpuIndexToBDF(gpuIndices)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("resolve GPU BDFs: %w", err)
|
||||
}
|
||||
indices := make([]int, 0, len(bdfByIndex))
|
||||
for idx := range bdfByIndex {
|
||||
indices = append(indices, idx)
|
||||
}
|
||||
sort.Ints(indices)
|
||||
|
||||
findings := make([]nvidiaPCIeBandwidthFinding, 0, len(indices))
|
||||
for _, idx := range indices {
|
||||
bdf := bdfByIndex[idx]
|
||||
before, _ := readPCIeSysfsString(bdf, "current_link_speed")
|
||||
max, _ := readPCIeSysfsString(bdf, "max_link_speed")
|
||||
maxWidth, _ := readPCIeSysfsInt(bdf, "max_link_width")
|
||||
findings = append(findings, nvidiaPCIeBandwidthFinding{
|
||||
Index: idx, BDF: bdf, BeforeSpeed: before, MaxSpeed: max, MaxWidth: maxWidth,
|
||||
})
|
||||
}
|
||||
|
||||
cmd := []string{"dcgmi", "diag", "-r", "nvbandwidth"}
|
||||
if len(indices) > 0 {
|
||||
cmd = append(cmd, "-i", joinIndexList(indices))
|
||||
}
|
||||
out, runErr := runSATCommandCtx(ctx, verboseLog, "dcgmi-nvbandwidth", cmd, nil, logFunc)
|
||||
_ = os.WriteFile(filepath.Join(runDir, "01-dcgmi-nvbandwidth.log"), out, 0644)
|
||||
|
||||
for i := range findings {
|
||||
bdf := findings[i].BDF
|
||||
after, _ := readPCIeSysfsString(bdf, "current_link_speed")
|
||||
width, _ := readPCIeSysfsInt(bdf, "current_link_width")
|
||||
findings[i].AfterSpeed = after
|
||||
findings[i].Width = width
|
||||
findings[i].Degraded = after != findings[i].MaxSpeed
|
||||
}
|
||||
|
||||
summary := renderNvidiaPCIeBandwidthSummary(findings, runErr)
|
||||
if err := os.WriteFile(filepath.Join(runDir, "summary.txt"), []byte(summary), 0644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
report := renderNvidiaPCIeBandwidthReport(findings, runErr)
|
||||
if err := os.WriteFile(filepath.Join(runDir, "nvidia-pcie-bandwidth-report.txt"), []byte(report), 0644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return runDir, nil
|
||||
}
|
||||
|
||||
// gpuIndexToBDF resolves each of gpuIndices to its PCI BDF via nvidia-smi.
|
||||
// An empty gpuIndices resolves every GPU nvidia-smi reports.
|
||||
func gpuIndexToBDF(gpuIndices []int) (map[int]string, error) {
|
||||
out, err := satExecCommand("nvidia-smi", "--query-gpu=index,pci.bus_id", "--format=csv,noheader,nounits").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("nvidia-smi: %w", err)
|
||||
}
|
||||
want := make(map[int]struct{}, len(gpuIndices))
|
||||
for _, idx := range gpuIndices {
|
||||
want[idx] = struct{}{}
|
||||
}
|
||||
filterByIndex := len(want) > 0
|
||||
|
||||
result := make(map[int]string)
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
fields := strings.SplitN(line, ",", 2)
|
||||
if len(fields) != 2 {
|
||||
continue
|
||||
}
|
||||
idx, err := strconv.Atoi(strings.TrimSpace(fields[0]))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if filterByIndex {
|
||||
if _, ok := want[idx]; !ok {
|
||||
continue
|
||||
}
|
||||
}
|
||||
result[idx] = normalizeNvidiaBDF(strings.TrimSpace(fields[1]))
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func renderNvidiaPCIeBandwidthSummary(findings []nvidiaPCIeBandwidthFinding, runErr error) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "run_at_utc=%s\n", time.Now().UTC().Format(time.RFC3339))
|
||||
fmt.Fprintf(&b, "gpu_count=%d\n", len(findings))
|
||||
degraded := 0
|
||||
for _, f := range findings {
|
||||
fmt.Fprintf(&b, "gpu%d_status=%s\n", f.Index, statusLabel(!f.Degraded))
|
||||
if f.Degraded {
|
||||
degraded++
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(&b, "degraded=%d\n", degraded)
|
||||
if runErr != nil {
|
||||
fmt.Fprintf(&b, "dcgmi_nvbandwidth_error=%s\n", runErr.Error())
|
||||
}
|
||||
if degraded > 0 {
|
||||
fmt.Fprintln(&b, "overall_status=FAILED")
|
||||
var reasons []string
|
||||
for _, f := range findings {
|
||||
if f.Degraded {
|
||||
reasons = append(reasons, fmt.Sprintf("GPU%d (%s): still at %s under load, capable of %s",
|
||||
f.Index, f.BDF, f.AfterSpeed, f.MaxSpeed))
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(&b, "warnings=%s\n", strings.Join(reasons, "; "))
|
||||
} else {
|
||||
fmt.Fprintln(&b, "overall_status=OK")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func statusLabel(ok bool) string {
|
||||
if ok {
|
||||
return "OK"
|
||||
}
|
||||
return "FAILED"
|
||||
}
|
||||
|
||||
func renderNvidiaPCIeBandwidthReport(findings []nvidiaPCIeBandwidthFinding, runErr error) string {
|
||||
var b strings.Builder
|
||||
line := strings.Repeat("=", 80)
|
||||
b.WriteString(line + "\n")
|
||||
b.WriteString("NVIDIA GPU PCIe Bandwidth / Link-Under-Load Check\n")
|
||||
b.WriteString(line + "\n\n")
|
||||
if runErr != nil {
|
||||
fmt.Fprintf(&b, "dcgmi diag -r nvbandwidth: %s\n\n", runErr.Error())
|
||||
}
|
||||
for _, f := range findings {
|
||||
verdict := "OK"
|
||||
if f.Degraded {
|
||||
verdict = "DEGRADED"
|
||||
}
|
||||
fmt.Fprintf(&b, "GPU%d (%s): %s\n", f.Index, f.BDF, verdict)
|
||||
fmt.Fprintf(&b, " before=%s after=%s max=%s width=%d/%d\n",
|
||||
f.BeforeSpeed, f.AfterSpeed, f.MaxSpeed, f.Width, f.MaxWidth)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -0,0 +1,353 @@
|
||||
package platform
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"bee/audit/internal/collector"
|
||||
)
|
||||
|
||||
// pcieLinkRetrainTimeout bounds how long we wait for a device to finish
|
||||
// retraining (clear the Link Status "Link Training" bit) before giving up
|
||||
// and reading whatever speed it settled on anyway. The PCIe spec allows up
|
||||
// to 100ms for Gen1-3 and longer for higher generations with equalization;
|
||||
// this is generous headroom above that.
|
||||
const pcieLinkRetrainTimeout = 2 * time.Second
|
||||
|
||||
// pcieLinkFinding is one device's before/after link-speed retrain result.
|
||||
type pcieLinkFinding struct {
|
||||
BDF string
|
||||
Description string
|
||||
VendorID string
|
||||
ClassCode string
|
||||
IsGPU bool
|
||||
GPUVendor string // "nvidia" or "amd", only set when IsGPU
|
||||
Skipped string // non-empty reason this device wasn't retrained
|
||||
BeforeSpeed string
|
||||
AfterSpeed string
|
||||
MaxSpeed string
|
||||
Width int
|
||||
MaxWidth int
|
||||
Degraded bool
|
||||
}
|
||||
|
||||
// RunPCIeLinkCheckPack forces every enabled PCIe device to retrain its link
|
||||
// (via the PCIe Link Control register's spec-defined Retrain Link bit — see
|
||||
// PCIe base spec, Link Control Register, bit 5) and compares the
|
||||
// post-retrain negotiated speed against the device's own reported maximum.
|
||||
//
|
||||
// This exists because a plain idle-time sysfs read of current_link_speed is
|
||||
// not a reliable fault signal: NVIDIA GPUs (and other devices with runtime
|
||||
// power management) deliberately downclock their PCIe link to save power
|
||||
// while idle, which is indistinguishable from a real degraded slot/riser/
|
||||
// cable without either sustained traffic or a forced retrain. Forcing a
|
||||
// retrain sidesteps needing a device-specific load generator (bee-gpu-burn
|
||||
// exists for GPUs; nothing plays that role for NICs, HBAs, or PCIe
|
||||
// switches) — retraining is a PCIe-spec mechanism every endpoint supports,
|
||||
// so this one check covers every PCIe device in the machine, not just
|
||||
// GPUs. See bible-local/decisions/2026-08-24-pcie-gpu-gen1-idle-warning-unresolved.md
|
||||
// for the history of narrower attempts that didn't generalize.
|
||||
//
|
||||
// Disabled devices (sysfs enable==0 — e.g. PCIe fabric-management endpoints
|
||||
// the kernel never activates, per the 2026-06-12 decision) are left alone:
|
||||
// they carry no data traffic, so there is nothing to verify and no reason
|
||||
// to poke them.
|
||||
func (s *System) RunPCIeLinkCheckPack(ctx context.Context, baseDir string, logFunc func(string)) (string, error) {
|
||||
if ctx == nil {
|
||||
ctx = context.Background()
|
||||
}
|
||||
if baseDir == "" {
|
||||
baseDir = "/var/log/bee-sat"
|
||||
}
|
||||
ts := time.Now().UTC().Format("20060102-150405")
|
||||
runDir := filepath.Join(baseDir, "pcie-link-"+ts)
|
||||
if err := os.MkdirAll(runDir, 0755); err != nil {
|
||||
return "", err
|
||||
}
|
||||
verboseLog := filepath.Join(runDir, "verbose.log")
|
||||
|
||||
bdfs, err := listPCIDeviceBDFs()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("list PCI devices: %w", err)
|
||||
}
|
||||
|
||||
var findings []pcieLinkFinding
|
||||
for _, bdf := range bdfs {
|
||||
if logFunc != nil {
|
||||
logFunc(fmt.Sprintf("=== %s ===", bdf))
|
||||
}
|
||||
f := retrainAndSamplePCIeDevice(ctx, verboseLog, bdf, logFunc)
|
||||
findings = append(findings, f)
|
||||
}
|
||||
|
||||
summary := renderPCIeLinkCheckSummary(findings)
|
||||
if err := os.WriteFile(filepath.Join(runDir, "summary.txt"), []byte(summary), 0644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
report := renderPCIeLinkCheckReport(findings)
|
||||
if err := os.WriteFile(filepath.Join(runDir, "pcie-link-report.txt"), []byte(report), 0644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return runDir, nil
|
||||
}
|
||||
|
||||
// listPCIDeviceBDFs returns every BDF under /sys/bus/pci/devices, sorted for
|
||||
// deterministic report ordering.
|
||||
func listPCIDeviceBDFs() ([]string, error) {
|
||||
entries, err := os.ReadDir("/sys/bus/pci/devices")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
bdfs := make([]string, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
bdfs = append(bdfs, e.Name())
|
||||
}
|
||||
sort.Strings(bdfs)
|
||||
return bdfs, nil
|
||||
}
|
||||
|
||||
func retrainAndSamplePCIeDevice(ctx context.Context, verboseLog, bdf string, logFunc func(string)) pcieLinkFinding {
|
||||
f := pcieLinkFinding{BDF: bdf}
|
||||
|
||||
vendor, _ := readPCIeSysfsHex(bdf, "vendor")
|
||||
class, _ := readPCIeSysfsHex(bdf, "class")
|
||||
f.VendorID = vendor
|
||||
f.ClassCode = class
|
||||
f.IsGPU, f.GPUVendor = classifyGPUFromVendorClass(vendor, class)
|
||||
f.Description = pcieDeviceDescription(ctx, verboseLog, bdf, logFunc)
|
||||
|
||||
if enabled, ok := readPCIeSysfsInt(bdf, "enable"); ok && enabled == 0 {
|
||||
f.Skipped = "device disabled (no data traffic; link state has no operational impact)"
|
||||
return f
|
||||
}
|
||||
|
||||
before, beforeOK := readPCIeSysfsString(bdf, "current_link_speed")
|
||||
maxSpeed, maxOK := readPCIeSysfsString(bdf, "max_link_speed")
|
||||
width, _ := readPCIeSysfsInt(bdf, "current_link_width")
|
||||
maxWidth, _ := readPCIeSysfsInt(bdf, "max_link_width")
|
||||
f.BeforeSpeed = before
|
||||
f.MaxSpeed = maxSpeed
|
||||
f.MaxWidth = maxWidth
|
||||
if !beforeOK || !maxOK {
|
||||
f.Skipped = "no PCIe link-speed attributes in sysfs (not a link-trained endpoint)"
|
||||
return f
|
||||
}
|
||||
|
||||
if err := retrainPCIeLink(ctx, verboseLog, bdf, logFunc); err != nil {
|
||||
f.Skipped = "retrain failed: " + err.Error()
|
||||
f.AfterSpeed = before
|
||||
f.Width = width
|
||||
// before/maxSpeed are already normalized "GenN" labels (see
|
||||
// readPCIeSysfsString) — compare directly, don't re-normalize.
|
||||
f.Degraded = before != maxSpeed
|
||||
return f
|
||||
}
|
||||
|
||||
after, _ := readPCIeSysfsString(bdf, "current_link_speed")
|
||||
widthAfter, _ := readPCIeSysfsInt(bdf, "current_link_width")
|
||||
f.AfterSpeed = after
|
||||
f.Width = widthAfter
|
||||
f.Degraded = after != maxSpeed
|
||||
return f
|
||||
}
|
||||
|
||||
// retrainPCIeLink sets the Retrain Link bit (bit 5) of the PCI Express
|
||||
// Capability's Link Control register via setpci, then polls the Link
|
||||
// Status register's Link Training bit (bit 11) until it clears or
|
||||
// pcieLinkRetrainTimeout elapses.
|
||||
func retrainPCIeLink(ctx context.Context, verboseLog, bdf string, logFunc func(string)) error {
|
||||
linkCtrlOut, err := runSATCommandCtx(ctx, verboseLog, "setpci-read-"+bdf,
|
||||
[]string{"setpci", "-s", bdf, "CAP_EXP+0x10.w"}, nil, logFunc)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read Link Control: %w", err)
|
||||
}
|
||||
cur, err := strconv.ParseUint(strings.TrimSpace(string(linkCtrlOut)), 16, 16)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse Link Control %q: %w", linkCtrlOut, err)
|
||||
}
|
||||
const retrainLinkBit = 0x0020
|
||||
newVal := uint16(cur) | retrainLinkBit
|
||||
|
||||
if _, err := runSATCommandCtx(ctx, verboseLog, "setpci-retrain-"+bdf,
|
||||
[]string{"setpci", "-s", bdf, fmt.Sprintf("CAP_EXP+0x10.w=%04x", newVal)}, nil, logFunc); err != nil {
|
||||
return fmt.Errorf("write Retrain Link bit: %w", err)
|
||||
}
|
||||
|
||||
const linkTrainingBit = 0x0800
|
||||
deadline := time.Now().Add(pcieLinkRetrainTimeout)
|
||||
for time.Now().Before(deadline) {
|
||||
statusOut, err := runSATCommandCtx(ctx, verboseLog, "setpci-status-"+bdf,
|
||||
[]string{"setpci", "-s", bdf, "CAP_EXP+0x12.w"}, nil, nil)
|
||||
if err == nil {
|
||||
if status, perr := strconv.ParseUint(strings.TrimSpace(string(statusOut)), 16, 16); perr == nil {
|
||||
if status&linkTrainingBit == 0 {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
// Timed out waiting for training to clear; the caller still samples
|
||||
// whatever speed sysfs reports, which is the honest answer either way.
|
||||
return nil
|
||||
}
|
||||
|
||||
func pcieDeviceDescription(ctx context.Context, verboseLog, bdf string, logFunc func(string)) string {
|
||||
out, err := runSATCommandCtx(ctx, verboseLog, "lspci-"+bdf, []string{"lspci", "-s", bdf}, nil, logFunc)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
line := strings.TrimSpace(string(out))
|
||||
if idx := strings.Index(line, "\n"); idx >= 0 {
|
||||
line = line[:idx]
|
||||
}
|
||||
if idx := strings.Index(line, " "); idx >= 0 {
|
||||
return strings.TrimSpace(line[idx+1:])
|
||||
}
|
||||
return line
|
||||
}
|
||||
|
||||
func readPCIeSysfsString(bdf, attr string) (string, bool) {
|
||||
raw, err := os.ReadFile(filepath.Join("/sys/bus/pci/devices", bdf, attr))
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
v := strings.TrimSpace(string(raw))
|
||||
if v == "" {
|
||||
return "", false
|
||||
}
|
||||
return collector.NormalizePCILinkSpeed(v), true
|
||||
}
|
||||
|
||||
func readPCIeSysfsInt(bdf, attr string) (int, bool) {
|
||||
raw, err := os.ReadFile(filepath.Join("/sys/bus/pci/devices", bdf, attr))
|
||||
if err != nil {
|
||||
return 0, false
|
||||
}
|
||||
v, err := strconv.Atoi(strings.TrimSpace(string(raw)))
|
||||
if err != nil || v < 0 {
|
||||
return 0, false
|
||||
}
|
||||
return v, true
|
||||
}
|
||||
|
||||
func readPCIeSysfsHex(bdf, attr string) (string, bool) {
|
||||
raw, err := os.ReadFile(filepath.Join("/sys/bus/pci/devices", bdf, attr))
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
return strings.TrimSpace(string(raw)), true
|
||||
}
|
||||
|
||||
// classifyGPUFromVendorClass reports whether a device is a GPU die itself
|
||||
// (PCI base class 0x03 — Display Controller — under NVIDIA/AMD's vendor
|
||||
// ID), as opposed to a same-vendor companion device (NIC, storage
|
||||
// controller, NVLink bridge) that shares the GPU's PCI vendor ID. Class-code
|
||||
// based, not name-substring based, per the same reasoning as
|
||||
// collector.IsGPUClass.
|
||||
func classifyGPUFromVendorClass(vendorHex, classHex string) (isGPU bool, vendor string) {
|
||||
v := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(vendorHex)), "0x")
|
||||
c := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(classHex)), "0x")
|
||||
if len(c) < 2 || c[:2] != "03" {
|
||||
return false, ""
|
||||
}
|
||||
switch v {
|
||||
case "10de":
|
||||
return true, "nvidia"
|
||||
case "1002":
|
||||
return true, "amd"
|
||||
default:
|
||||
return false, ""
|
||||
}
|
||||
}
|
||||
|
||||
func renderPCIeLinkCheckSummary(findings []pcieLinkFinding) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "run_at_utc=%s\n", time.Now().UTC().Format(time.RFC3339))
|
||||
fmt.Fprintf(&b, "devices_tested=%d\n", len(findings))
|
||||
|
||||
gpuStatus := map[string]string{} // vendor -> OK/FAILED
|
||||
otherDegraded := 0
|
||||
otherTested := 0
|
||||
anyDegraded := false
|
||||
for _, f := range findings {
|
||||
if f.Skipped != "" && f.AfterSpeed == "" {
|
||||
continue
|
||||
}
|
||||
if f.IsGPU {
|
||||
if _, ok := gpuStatus[f.GPUVendor]; !ok {
|
||||
gpuStatus[f.GPUVendor] = "OK"
|
||||
}
|
||||
if f.Degraded {
|
||||
gpuStatus[f.GPUVendor] = "FAILED"
|
||||
anyDegraded = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
otherTested++
|
||||
if f.Degraded {
|
||||
otherDegraded++
|
||||
anyDegraded = true
|
||||
}
|
||||
}
|
||||
for _, vendor := range []string{"nvidia", "amd"} {
|
||||
if status, ok := gpuStatus[vendor]; ok {
|
||||
fmt.Fprintf(&b, "gpu_%s_status=%s\n", vendor, status)
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(&b, "other_devices_tested=%d\n", otherTested)
|
||||
fmt.Fprintf(&b, "other_devices_degraded=%d\n", otherDegraded)
|
||||
if otherTested > 0 {
|
||||
if otherDegraded > 0 {
|
||||
fmt.Fprintln(&b, "other_status=FAILED")
|
||||
} else {
|
||||
fmt.Fprintln(&b, "other_status=OK")
|
||||
}
|
||||
}
|
||||
|
||||
if anyDegraded {
|
||||
fmt.Fprintln(&b, "overall_status=FAILED")
|
||||
var reasons []string
|
||||
for _, f := range findings {
|
||||
if f.Degraded {
|
||||
reasons = append(reasons, fmt.Sprintf("%s (%s): retrained to %s, capable of %s",
|
||||
f.BDF, nonEmptyOr(f.Description, "unknown device"), f.AfterSpeed, f.MaxSpeed))
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(&b, "warnings=%s\n", strings.Join(reasons, "; "))
|
||||
} else {
|
||||
fmt.Fprintln(&b, "overall_status=OK")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func renderPCIeLinkCheckReport(findings []pcieLinkFinding) string {
|
||||
var b strings.Builder
|
||||
line := strings.Repeat("=", 80)
|
||||
b.WriteString(line + "\n")
|
||||
b.WriteString("PCIe Link Retrain Check\n")
|
||||
b.WriteString(line + "\n\n")
|
||||
for _, f := range findings {
|
||||
fmt.Fprintf(&b, "%s %s\n", f.BDF, nonEmptyOr(f.Description, "(unknown device)"))
|
||||
if f.Skipped != "" && f.AfterSpeed == "" {
|
||||
fmt.Fprintf(&b, " skipped: %s\n", f.Skipped)
|
||||
continue
|
||||
}
|
||||
verdict := "OK"
|
||||
if f.Degraded {
|
||||
verdict = "DEGRADED"
|
||||
}
|
||||
fmt.Fprintf(&b, " %s: before=%s after=%s max=%s width=%d/%d\n",
|
||||
verdict, f.BeforeSpeed, f.AfterSpeed, f.MaxSpeed, f.Width, f.MaxWidth)
|
||||
if f.Skipped != "" {
|
||||
fmt.Fprintf(&b, " note: %s\n", f.Skipped)
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
package platform
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestClassifyGPUFromVendorClass(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
vendor string
|
||||
class string
|
||||
wantIsGPU bool
|
||||
wantVendor string
|
||||
}{
|
||||
{"nvidia GPU", "0x10de", "0x030200", true, "nvidia"},
|
||||
{"amd GPU", "0x1002", "0x030000", true, "amd"},
|
||||
{"nvidia NIC (same vendor, not display class)", "0x10de", "0x020000", false, ""},
|
||||
{"intel NIC", "0x8086", "0x020000", false, ""},
|
||||
{"unrelated display-class vendor", "0x1234", "0x030000", false, ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
gotIsGPU, gotVendor := classifyGPUFromVendorClass(tc.vendor, tc.class)
|
||||
if gotIsGPU != tc.wantIsGPU || gotVendor != tc.wantVendor {
|
||||
t.Fatalf("classifyGPUFromVendorClass(%q,%q) = (%v,%q), want (%v,%q)",
|
||||
tc.vendor, tc.class, gotIsGPU, gotVendor, tc.wantIsGPU, tc.wantVendor)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderPCIeLinkCheckSummaryDegradedGPU(t *testing.T) {
|
||||
findings := []pcieLinkFinding{
|
||||
{BDF: "0000:0d:00.0", Description: "NVIDIA GPU", IsGPU: true, GPUVendor: "nvidia",
|
||||
BeforeSpeed: "Gen1", AfterSpeed: "Gen1", MaxSpeed: "Gen5", Degraded: true},
|
||||
{BDF: "0000:37:00.0", Description: "NVIDIA GPU", IsGPU: true, GPUVendor: "nvidia",
|
||||
BeforeSpeed: "Gen1", AfterSpeed: "Gen5", MaxSpeed: "Gen5", Degraded: false},
|
||||
{BDF: "0000:01:00.0", Description: "Mellanox NIC", IsGPU: false,
|
||||
BeforeSpeed: "Gen3", AfterSpeed: "Gen4", MaxSpeed: "Gen4", Degraded: false},
|
||||
}
|
||||
summary := renderPCIeLinkCheckSummary(findings)
|
||||
if !strings.Contains(summary, "gpu_nvidia_status=FAILED") {
|
||||
t.Fatalf("expected gpu_nvidia_status=FAILED (one degraded GPU should fail the vendor group), got:\n%s", summary)
|
||||
}
|
||||
if !strings.Contains(summary, "other_status=OK") {
|
||||
t.Fatalf("expected other_status=OK, got:\n%s", summary)
|
||||
}
|
||||
if !strings.Contains(summary, "overall_status=FAILED") {
|
||||
t.Fatalf("expected overall_status=FAILED, got:\n%s", summary)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderPCIeLinkCheckSummaryAllClean(t *testing.T) {
|
||||
findings := []pcieLinkFinding{
|
||||
{BDF: "0000:0d:00.0", IsGPU: true, GPUVendor: "nvidia", BeforeSpeed: "Gen1", AfterSpeed: "Gen5", MaxSpeed: "Gen5"},
|
||||
{BDF: "0000:2c:00.0", Skipped: "device disabled (no data traffic; link state has no operational impact)"},
|
||||
}
|
||||
summary := renderPCIeLinkCheckSummary(findings)
|
||||
if !strings.Contains(summary, "overall_status=OK") {
|
||||
t.Fatalf("expected overall_status=OK, got:\n%s", summary)
|
||||
}
|
||||
if strings.Contains(summary, "other_status=") {
|
||||
t.Fatalf("disabled/skipped-only device should not produce an other_status line, got:\n%s", summary)
|
||||
}
|
||||
}
|
||||
@@ -135,7 +135,7 @@ func defaultTaskPriority(target string, params taskParams) int {
|
||||
return taskPriorityBurn
|
||||
case "nvidia", "nvidia-targeted-stress", "nvidia-targeted-power", "nvidia-pulse",
|
||||
"nvidia-interconnect", "nvidia-bandwidth", "memory", "storage", "cpu",
|
||||
"amd", "amd-mem", "amd-bandwidth", "nvidia-config":
|
||||
"amd", "amd-mem", "amd-bandwidth", "nvidia-config", "pcie-link", "nvidia-pcie-bandwidth":
|
||||
if params.StressMode {
|
||||
return taskPriorityValidateStress
|
||||
}
|
||||
|
||||
@@ -671,6 +671,18 @@ func renderCheck(opts HandlerOptions) string {
|
||||
`<code>nvidia-smi --query-gpu=...</code>, <code>nvidia-smi topo -m</code>, <code>nvidia-smi nvlink -s/-e</code>, <code>nvidia-smi conf-compute -q</code>, <code>dmesg</code>`,
|
||||
`Seconds — read-only query only.`,
|
||||
)) +
|
||||
renderSATCard("pcie-link", "PCIe Link Check", "runSAT('pcie-link')", "", renderValidateCardBody(
|
||||
`Every enabled PCIe device in the machine (GPUs, NICs, RAID/HBA controllers, PCIe switches).`,
|
||||
`Forces every enabled PCIe device to retrain its link (PCIe spec Link Control "Retrain Link" bit) and compares the negotiated speed against the device's own maximum. An idle sysfs reading alone can't tell a real degraded slot/riser from a device that's simply power-managed down at idle (GPUs do this routinely); a forced retrain settles that without needing a device-specific load generator, so this one check covers non-GPU PCIe hardware too, not just GPUs.`,
|
||||
`<code>setpci</code> (Link Control/Link Status registers), <code>lspci</code>`,
|
||||
`Seconds per device — brief link retrain, no sustained traffic.`,
|
||||
)) +
|
||||
renderSATCard("nvidia-pcie-bandwidth", "GPU PCIe Bandwidth", "runSAT('nvidia-pcie-bandwidth')", "", renderValidateCardBody(
|
||||
inv.NVIDIA,
|
||||
`Drives real host<->device traffic across each GPU's PCIe link and resamples link speed immediately after, to confirm the link actually trains to its negotiated maximum under real load — the load-bearing counterpart to PCIe Link Check for GPUs specifically.`,
|
||||
`<code>dcgmi diag -r nvbandwidth</code>, sysfs link-speed resample`,
|
||||
`Depends on nvbandwidth's built-in test duration.`,
|
||||
)) +
|
||||
`</div>
|
||||
<div style="height:1px;background:var(--border);margin:16px 0"></div>
|
||||
<div class="card" style="margin-bottom:16px">
|
||||
@@ -732,7 +744,7 @@ func renderCheck(opts HandlerOptions) string {
|
||||
<script>
|
||||
let satES = null;
|
||||
function satLabels() {
|
||||
return {nvidia:'Check GPU (DCGM L2)', 'nvidia-interconnect':'NVIDIA Interconnect (NCCL)', 'nvidia-bandwidth':'NVIDIA Bandwidth (NVBandwidth)', memory:'Check Memory', storage:'Check Storage', cpu:'Check CPU', amd:'Check AMD GPU', 'amd-mem':'AMD GPU MEM Integrity', 'amd-bandwidth':'AMD GPU MEM Bandwidth', 'nvidia-config':'Check GPU Config & NVLink'};
|
||||
return {nvidia:'Check GPU (DCGM L2)', 'nvidia-interconnect':'NVIDIA Interconnect (NCCL)', 'nvidia-bandwidth':'NVIDIA Bandwidth (NVBandwidth)', memory:'Check Memory', storage:'Check Storage', cpu:'Check CPU', amd:'Check AMD GPU', 'amd-mem':'AMD GPU MEM Integrity', 'amd-bandwidth':'AMD GPU MEM Bandwidth', 'nvidia-config':'Check GPU Config & NVLink', 'pcie-link':'PCIe Link Check', 'nvidia-pcie-bandwidth':'GPU PCIe Bandwidth Check'};
|
||||
}
|
||||
let satNvidiaGPUsPromise = null;
|
||||
function loadSatNvidiaGPUs() {
|
||||
@@ -867,8 +879,8 @@ function runAllCheckSAT() {
|
||||
const status = document.getElementById('sat-all-status');
|
||||
status.textContent = 'Enqueuing...';
|
||||
const nvidiaIndices = satSelectedGPUIndices();
|
||||
const nvidiaAllTargets = ['nvidia', 'nvidia-interconnect', 'nvidia-bandwidth'];
|
||||
const baseTargets = ['cpu', 'memory', 'storage', 'nvidia-config'];
|
||||
const nvidiaAllTargets = ['nvidia', 'nvidia-interconnect', 'nvidia-bandwidth', 'nvidia-pcie-bandwidth'];
|
||||
const baseTargets = ['cpu', 'memory', 'storage', 'nvidia-config', 'pcie-link'];
|
||||
const amdTargets = selectedAMDValidateTargets();
|
||||
const expanded = [];
|
||||
baseTargets.forEach(t => expanded.push({target: t}));
|
||||
|
||||
@@ -265,6 +265,8 @@ func NewHandler(opts HandlerOptions) http.Handler {
|
||||
mux.HandleFunc("POST /api/sat/memory/run", h.handleAPISATRun("memory"))
|
||||
mux.HandleFunc("POST /api/sat/storage/run", h.handleAPISATRun("storage"))
|
||||
mux.HandleFunc("POST /api/sat/nvidia-config/run", h.handleAPISATRun("nvidia-config"))
|
||||
mux.HandleFunc("POST /api/sat/pcie-link/run", h.handleAPISATRun("pcie-link"))
|
||||
mux.HandleFunc("POST /api/sat/nvidia-pcie-bandwidth/run", h.handleAPISATRun("nvidia-pcie-bandwidth"))
|
||||
mux.HandleFunc("POST /api/sat/cpu/run", h.handleAPISATRun("cpu"))
|
||||
mux.HandleFunc("POST /api/sat/amd/run", h.handleAPISATRun("amd"))
|
||||
mux.HandleFunc("POST /api/sat/amd-mem/run", h.handleAPISATRun("amd-mem"))
|
||||
|
||||
@@ -294,6 +294,18 @@ func executeTaskWithOptions(opts *HandlerOptions, t *Task, j *jobState, ctx cont
|
||||
break
|
||||
}
|
||||
archive, err = runNvidiaConfigCheckPackCtx(a, ctx, "", j.append)
|
||||
case "pcie-link":
|
||||
if a == nil {
|
||||
err = fmt.Errorf("app not configured")
|
||||
break
|
||||
}
|
||||
archive, err = runPCIeLinkCheckPackCtx(a, ctx, "", j.append)
|
||||
case "nvidia-pcie-bandwidth":
|
||||
if a == nil {
|
||||
err = fmt.Errorf("app not configured")
|
||||
break
|
||||
}
|
||||
archive, err = runNvidiaPCIeBandwidthPackCtx(a, ctx, "", t.params.GPUIndices, j.append)
|
||||
case "cpu":
|
||||
if a == nil {
|
||||
err = fmt.Errorf("app not configured")
|
||||
|
||||
@@ -46,6 +46,8 @@ var taskNames = map[string]string{
|
||||
"memory": "Memory SAT",
|
||||
"storage": "Storage SAT",
|
||||
"nvidia-config": "GPU Config & NVLink Check",
|
||||
"pcie-link": "PCIe Link Check (forced retrain)",
|
||||
"nvidia-pcie-bandwidth": "NVIDIA GPU PCIe Bandwidth Check",
|
||||
"cpu": "CPU SAT",
|
||||
"amd": "AMD GPU SAT",
|
||||
"amd-mem": "AMD GPU MEM Integrity",
|
||||
@@ -317,6 +319,12 @@ var (
|
||||
runNvidiaConfigCheckPackCtx = func(a *app.App, ctx context.Context, baseDir string, logFunc func(string)) (string, error) {
|
||||
return a.RunNvidiaConfigCheckPackCtx(ctx, baseDir, logFunc)
|
||||
}
|
||||
runPCIeLinkCheckPackCtx = func(a *app.App, ctx context.Context, baseDir string, logFunc func(string)) (string, error) {
|
||||
return a.RunPCIeLinkCheckPackCtx(ctx, baseDir, logFunc)
|
||||
}
|
||||
runNvidiaPCIeBandwidthPackCtx = func(a *app.App, ctx context.Context, baseDir string, gpuIndices []int, logFunc func(string)) (string, error) {
|
||||
return a.RunNvidiaPCIeBandwidthPackCtx(ctx, baseDir, gpuIndices, logFunc)
|
||||
}
|
||||
runCPUAcceptancePackCtx = func(a *app.App, ctx context.Context, baseDir string, durationSec int, logFunc func(string)) (string, error) {
|
||||
return a.RunCPUAcceptancePackCtx(ctx, baseDir, durationSec, logFunc)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
# PCIe Gen1-at-idle GPU warning: history of attempts, and the fix
|
||||
|
||||
**Date:** 2026-08-24
|
||||
**Status:** active — implemented (see "Resolution" at the end)
|
||||
|
||||
## Symptom
|
||||
|
||||
On NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, `reanimator.json` /
|
||||
`status/component-status.json` (`pcie:gpu:nvidia`) report `Warning`:
|
||||
"PCIe link speed degraded: running at Gen1, capable of Gen5" for every GPU in
|
||||
the system, for the entire diagnostic session — even though every functional
|
||||
test that actually exercises the GPUs (dcgmi diag targeted-stress/targeted-power/
|
||||
pulse_test, nvbandwidth, NCCL all_reduce_perf, GPU config check) passes clean,
|
||||
link width stays x16/x16 throughout, and there is no AER/Xid activity in
|
||||
dmesg. Concretely observed in the blackbox `2026-08-13 (BEE-SP v12.84)
|
||||
NF5468-M7-A0-R0-00 28CC05483`.
|
||||
|
||||
This is not a one-off — five separate commits over five months have targeted
|
||||
this exact false-positive-vs-real-fault ambiguity, and it is still not fully
|
||||
solved. This doc is the record of what was tried and why each attempt only
|
||||
closed part of the gap, so we stop re-discovering the same dead ends.
|
||||
|
||||
## Timeline
|
||||
|
||||
| Date | Commit | What it did | Gap it left |
|
||||
|---|---|---|---|
|
||||
| 2026-04-01 | `eb60100` | NVIDIA collector switched `LinkSpeed`/`MaxLinkSpeed` from raw sysfs (`current_link_speed`) to `nvidia-smi --query-gpu=pcie.link.gen.current,...`, on the theory that "the driver knows the negotiated speed regardless of current power state" and sysfs reflects only instantaneous physical state. | **False premise, confirmed wrong by this very bundle.** `export/gpu/nvidia-smi-q.txt` (the actual `nvidia-smi -q` dump, not sysfs) shows `PCIe Generation / Device Current: 1` for every GPU. `nvidia-smi`'s own query reflects the same ASPM/power-managed downshift as sysfs on driver 580.159.03 + Blackwell — switching the data source changed nothing for this failure mode. |
|
||||
| 2026-04-02 | `99cece5` | Added `lspci -vvv` and per-GPU sysfs link files to the **on-demand support bundle** (`export/gpu/`) so a human could manually cross-check `LnkCap`/`LnkSta`. | Diagnostic aid only, no automated verdict. Also — see below — this file only ships when someone clicks "Download Support Bundle" in the web UI; it never reaches a blackbox capture. |
|
||||
| 2026-04-12 | `05c1fde` | Added `applyPCIeLinkSpeedWarning`: sets `Warning` (or `Critical` for NVLink bridges) whenever `LinkSpeed < MaxLinkSpeed`, using whatever `LinkSpeed` was populated by (at that point) `eb60100`'s nvidia-smi values. This is the commit that actually introduced the warning we're chasing. | Compares "current" to "max" as an absolute rule with no allowance for idle power states — this is the root of the false-positive on any idle GPU. |
|
||||
| 2026-04-12 | `4f94ebc` | Same day: added `pcie_aspm=off`, `intel_idle.max_cstate=1`, `processor.max_cstate=1` to the boot kernel command line, as an OS-level attempt to stop links from ever downshifting in the first place. | **Confirmed present but ineffective for this failure mode.** This exact bundle's `dmesg.txt` shows `pcie_aspm=off ... intel_idle.max_cstate=1 processor.max_cstate=1` active in the kernel command line — and the GPUs still trained down to Gen1 at idle anyway. `pcie_aspm=off` disables the platform's standard PCIe ASPM (L0s/L1) link states; it does **not** touch the GPU driver's own independent runtime power management, which downclocks the PCIe link as part of the GPU's P-state machine. Two different mechanisms, only one of which this flag controls. |
|
||||
| 2026-06-12 | `2320925` | Excluded permanently-disabled PCIe devices (e.g. Switchtec fabric-management endpoints on HGX H100 baseboards) from the warning entirely, via `sysfs enable==0`. Documented in `2026-06-12-pcie-disabled-device-link-warning.md`. | Fixed a different false-positive (management-plane chips, not GPUs at idle). Doesn't touch this bug. |
|
||||
| 2026-08-04 | `a34e823` | Fixed `enrichPCIeWithNVIDIAData` unconditionally clobbering `dev.Status` after `applyPCIeLinkSpeedWarning` had already set `Warning` — a later NVIDIA-data enrichment pass was silently downgrading it back to `OK` while leaving the stale `ErrorDescription` behind. Added severity-ordered merge (`OK/Unknown < Warning < Critical`). | Made the `Warning` **stick** reliably in the hardware snapshot (`reanimator.json`) for the first time — before this, it could randomly vanish depending on collector pass ordering. Necessary fix, but it also means the false-positive from `05c1fde` now survives *more* reliably than before. |
|
||||
| 2026-08-06 | `b1f165e` | Two things: (1) `writePCIeGPUStatusesToDB` — pushes the collector's PCIe status into `component-status.json` every audit cycle, because none of the SAT jobs (`nvidia`, `nvidia-config`, `nvidia-interconnect`, `nvidia-bandwidth`) ever checked link speed, so the DB-backed status (what the web UI "Hardware Summary" chip and `status/component-status.json` in every bundle read) had been silently showing `OK` regardless of the collector's own `Warning`. (2) Added `export/gpu/pcie-nvidia-link-under-load.txt` to the **support bundle**: runs `bee-gpu-burn --seconds 8` and resamples the same sysfs link attributes mid-load, so a human/agent can tell idle power-saving from a real degraded slot/riser by comparing it against the idle `pcie-nvidia-link.txt`. | (1) closed the "SAT says OK" mismatch — this is why `status/component-status.json` in the 08-13 bundle correctly shows `Warning` and doesn't get silently overwritten by the string of `sat:nvidia*: OK` entries in its history. (2) is the one piece of tooling actually designed to answer "is this real or just idle" — **and it is absent from this bundle** (see next section). Also: `component_status_db.go`'s `Record()` merge (`newSev > curSev`) has **no downgrade path** — once `Warning` is recorded, nothing (not even a later `audit:pcie` poll reporting the link back at Gen5) can lower it back to `OK` within that DB file's lifetime. Not new in this commit, but this is the mechanism that pins the warning for the rest of the session even if the link genuinely retrains under load later. |
|
||||
| 2026-08-17 | `e3697c0` | Fixed `matchesGPUVendor`/`isGPUDevice` matching any PCIe device with "Controller" in its class string or same-vendor ID as a GPU — same-vendor NICs/NVSwitch bridges could trip the `pcie:gpu:<vendor>` alarm. | Correctness fix for *which* devices get judged, not for the idle-vs-real ambiguity itself. |
|
||||
| 2026-08-24 (today) | `9add561` | Bounded every `support_bundle.go` subprocess (including the `bee-gpu-burn`-driven under-load capture from `b1f165e`) with a timeout, because a genuinely wedged GPU could hang bundle generation forever. | Unrelated to the false-positive itself, but relevant context: confirms the under-load capture is a live, still-evolving code path, not dead code. |
|
||||
|
||||
## Why this specific bundle still shows the confusing state
|
||||
|
||||
Two independent things line up to explain exactly what you're looking at:
|
||||
|
||||
1. **The blackbox (USB auto-sync) capture path never runs `b1f165e`'s
|
||||
disambiguation tooling at all**, regardless of build freshness.
|
||||
`pcie-nvidia-link.txt` / `pcie-nvidia-link-under-load.txt` live in
|
||||
`support_bundle.go`'s `supportBundleCommands`, which only executes inside
|
||||
`BuildSupportBundle` — triggered by a human clicking "Download Support
|
||||
Bundle" in the web UI. The blackbox worker (`blackbox.go`
|
||||
`syncCycle`/`captureSnapshots`) instead calls `categorizeExportTree` on
|
||||
whatever the periodic collector already staged in the live export
|
||||
directory, plus journalctl/dmesg/status snapshots — it does not invoke
|
||||
`supportBundleCommands`. That's also why this bundle has no
|
||||
`manifest.txt`: `blackbox.go:604` notes the blackbox path intentionally
|
||||
skips it, shipping only `README.md` as the "how to read this" doc. Net
|
||||
effect: **the one artifact designed to answer "idle or real fault" for a
|
||||
GPU PCIe warning is structurally unreachable from a blackbox pull.** You
|
||||
would only get it by opening the web UI and downloading a support bundle
|
||||
from that same host while the warning is still active — not useful for a
|
||||
server that already shipped or is already offline.
|
||||
|
||||
2. **Even where the fixes did land, they don't address the actual
|
||||
mechanism.** This bundle's own `dmesg.txt` proves `pcie_aspm=off` and
|
||||
both max_cstate=1 flags from `4f94ebc` were active at boot, and the GPU
|
||||
still trained to Gen1 at idle — and `nvidia-smi-q.txt` proves `eb60100`'s
|
||||
switch to nvidia-smi's own query (instead of sysfs) reports the exact
|
||||
same Gen1 reading. Both mitigations were built on an assumption (ASPM is
|
||||
the mechanism / nvidia-smi reports negotiated capability, not current
|
||||
power state) that this hardware+driver combination (RTX PRO 6000
|
||||
Blackwell Server Edition, driver 580.159.03) disproves. The actual
|
||||
mechanism is the NVIDIA driver's own runtime power management
|
||||
downclocking the link independent of platform ASPM — nothing in the
|
||||
current codebase distinguishes "GPU driver decided to save power" from
|
||||
"riser/slot is actually degraded."
|
||||
|
||||
3. Separately, `component_status_db.go`'s `Record()` has no downgrade path
|
||||
(see `b1f165e` row above): once any source writes `Warning` for
|
||||
`pcie:gpu:nvidia`, that status cannot go back to `OK` within the same
|
||||
session even if a later poll of the *same* source reports the link back
|
||||
at full speed. Combined with #2, a single idle-time sample taken at
|
||||
14:20:05 (before any load test ran) permanently pins the whole GPU
|
||||
subsystem's status for the rest of that ~9-day diagnostic session
|
||||
(14:21 → 08-17 10:03), regardless of how many load tests pass clean in
|
||||
between.
|
||||
|
||||
## Options going forward
|
||||
|
||||
Pick one (or combine):
|
||||
|
||||
1. **Make the under-load resample part of the periodic collector /
|
||||
blackbox path**, not support-bundle-only — e.g. run it once per boot
|
||||
from `bee-audit`/`bee-nvidia` right after the SAT GPU load tests
|
||||
(`nvidia`, `nvidia-bandwidth`, `nvidia-targeted-stress` already load the
|
||||
GPUs for minutes at a time — piggyback the sysfs resample on one of
|
||||
those instead of a dedicated 8s `bee-gpu-burn` run) and write it into
|
||||
the live export tree so blackbox mirrors it automatically.
|
||||
2. **Let a real load test clear the Warning.** Have one of the GPU SAT
|
||||
jobs that's already running a sustained load (`nvidia-bandwidth`,
|
||||
`nvidia-targeted-stress`) resample link speed at the end of its run and
|
||||
call `writePCIeGPUStatusesToDB` with the fresh reading, allowing an
|
||||
explicit downgrade path for this one key when the *same* mechanism that
|
||||
raised the warning reports it resolved under controlled load — rather
|
||||
than opening up `Record()`'s merge logic in general (which is
|
||||
deliberately sticky for a reason: don't want a real intermittent PSU/ECC
|
||||
fault to be silently forgotten because one later poll came back clean).
|
||||
3. **Stop comparing against idle sysfs/nvidia-smi state at all** for
|
||||
NVIDIA GPUs specifically, and instead rely on the SAT bandwidth numbers
|
||||
themselves (nvbandwidth's measured GB/s vs. expected-for-Gen5-x16
|
||||
threshold) as the actual link-health signal — this sidesteps the
|
||||
idle-vs-load ambiguity entirely since it measures the thing you actually
|
||||
care about (does the link deliver Gen5 throughput when asked), not a
|
||||
point-in-time speed field.
|
||||
4. **Leave detection as-is, fix only the messaging**: keep flagging Gen1 at
|
||||
idle as `Warning`, but make the `error_description` explicit that this
|
||||
is unconfirmed/idle-sampled ("Warning: idle PCIe link at Gen1 (max Gen5,
|
||||
unconfirmed under load)") so a human reading `reanimator.json` cold
|
||||
isn't misled into thinking it's a proven hardware fault — closest to a
|
||||
documentation-only fix, cheapest, but keeps the ambiguity forever.
|
||||
|
||||
## Resolution (2026-08-24)
|
||||
|
||||
Landed a variant of options 1–3 that turned out simpler than any of them
|
||||
individually once we stopped trying to make the idle reading recoverable:
|
||||
|
||||
**Stop writing a status from the idle reading at all.** `parseLspciDevice`
|
||||
(`collector/pcie.go`) no longer calls `applyPCIeLinkSpeedWarning` on every
|
||||
idle collector pass — `LinkSpeed`/`MaxLinkSpeed` stay populated as plain
|
||||
descriptive fields in `reanimator.json`, but nothing sets `Status` to
|
||||
`Warning` from them anymore. This makes `Record()`'s one-way severity merge
|
||||
(no downgrade path, see the `b1f165e` row above) a non-issue by
|
||||
construction: if nothing ever writes an unverified `Warning`, there is
|
||||
nothing that later needs downgrading. `applyPCIeLinkSpeedWarning` itself is
|
||||
kept, now documented as reserved for a caller that already put the device
|
||||
under real traffic.
|
||||
|
||||
**Two new verified-load SAT targets replace the idle signal:**
|
||||
|
||||
- `pcie-link` (`platform/pcie_link_check.go`) — forces every *enabled*
|
||||
PCIe device (not just GPUs) to retrain via the PCIe spec's Link Control
|
||||
"Retrain Link" bit (`setpci … CAP_EXP+0x10.w`, poll Link Status bit 11
|
||||
until training clears), then compares the post-retrain negotiated speed
|
||||
against the device's own max. This is the generic answer to "what about
|
||||
non-GPU PCIe cards" from the prior discussion: retraining is a mechanism
|
||||
every PCIe endpoint supports, so one check now covers NICs/HBAs/switches
|
||||
that have no `bee-gpu-burn`-equivalent load tool. Classifies devices by
|
||||
PCI class code (`0x03` = Display) + vendor ID (`0x10de`/`0x1002`), not
|
||||
name substrings, per the existing `no-hardcoded-vendors` contract.
|
||||
Disabled devices (`enable==0`) are left alone, same carve-out as the
|
||||
2026-06-12 decision. Writes `gpu_nvidia_status` / `gpu_amd_status` /
|
||||
`other_status` into `summary.txt`; `ApplySATResultToDB` routes each into
|
||||
its own component key (`pcie:gpu:nvidia`, `pcie:gpu:amd`,
|
||||
`pcie:link:other`) so a degraded NIC is never reported as a GPU fault.
|
||||
- `nvidia-pcie-bandwidth` (`platform/nvidia_pcie_bandwidth.go`) — GPU-only,
|
||||
drives `dcgmi diag -r nvbandwidth` (the same tool `nvidia-bandwidth`
|
||||
already uses for P2P throughput) and resamples each involved GPU's link
|
||||
speed via sysfs immediately after, deliberately independent of
|
||||
nvbandwidth's own GB/s pass/fail — this SAT's verdict is purely "did the
|
||||
link train up to max under real traffic." Feeds `pcie:gpu:nvidia`
|
||||
alongside `pcie-link`.
|
||||
|
||||
Both are wired into the existing task queue/webui exactly like
|
||||
`nvidia-config` (routes in `server.go`, dispatch case in `task_runner.go`,
|
||||
priority in `api.go`'s `defaultTaskPriority`, cards on the Validate "Check"
|
||||
page, included in "Run All Check SAT").
|
||||
|
||||
**Open follow-up, not yet built:** neither target runs automatically today
|
||||
— a Warning only clears/confirms when an operator runs one of these two
|
||||
SATs (or "Run All Check SAT", which now includes `pcie-link`). If that
|
||||
turns out to matter in practice, wire `pcie-link` (it's fast — a retrain is
|
||||
milliseconds per device, not a sustained burn) into `bee-audit`'s periodic
|
||||
cycle.
|
||||
|
||||
## Follow-up (2026-08-24, same day): the original artifact never reached blackbox at all
|
||||
|
||||
Separately from the SAT work above, re-examined why `pcie-nvidia-link.txt` /
|
||||
`pcie-nvidia-link-under-load.txt` (the `b1f165e` diagnostic pair) were
|
||||
missing from the blackbox analyzed earlier in this doc, even on a build new
|
||||
enough to have that code. Root cause, confirmed against the actual pipeline
|
||||
(not build/version drift as first guessed): those two files only ever lived
|
||||
in `supportBundleCommands` (`app/support_bundle.go`), which exclusively
|
||||
runs inside `BuildSupportBundle` — the on-demand "Download Support Bundle"
|
||||
web UI action. The blackbox USB auto-sync worker (`blackbox.go`
|
||||
`syncCycle`) never calls that function; it only mirrors whatever
|
||||
`platform.CaptureTechnicalDump` already wrote into the live export tree at
|
||||
boot (`bee-audit.service`, `Type=oneshot`) via `categorizeExportTree`. So
|
||||
the artifact was structurally unreachable from a blackbox pull no matter
|
||||
how fresh the ISO was — this was the real explanation for the earlier
|
||||
"missing file" mystery, not the stale-build hypothesis floated above.
|
||||
|
||||
**Fix:** moved both scripts (as shared exported constants,
|
||||
`platform.PCIeNvidiaLinkScript` / `PCIeNvidiaLinkUnderLoadScript`, so
|
||||
support-bundle and techdump run byte-identical scripts instead of two
|
||||
copies that can drift) into `platform.techDumpNvidiaCommands`, so
|
||||
`CaptureTechnicalDump` captures them once at boot alongside the existing
|
||||
`nvidia-smi-*` dumps. `bee-audit` is oneshot, so the one-time ~8s
|
||||
`bee-gpu-burn` cost for the under-load sample is a boot-time cost, not a
|
||||
per-blackbox-sync-cycle one. Added both filenames to `techdumpBucketFor`
|
||||
(→ `export/gpu/`) so `categorizeExportTree` buckets them correctly.
|
||||
`supportBundleCommands` still re-runs both on-demand for a fresher sample
|
||||
than the boot-time one — that's intentional, not a duplicate to clean up.
|
||||
|
||||
**Same-pattern audit of the rest of `supportBundleCommands`:** while in
|
||||
there, checked every other `export/*` entry (the ones the README
|
||||
tags as hardware-facing, i.e. everything except `livecd/*`) for the same
|
||||
"only reachable via on-demand support bundle, never blackbox" gap. Also
|
||||
fixed **`export/gpu/kernel-aer-nvidia.txt`** (dmesg filtered for
|
||||
AER/NVRM/Xid lines) the same way — same file, same fix, now
|
||||
`platform.KernelAERNvidiaScript` shared between both paths. This one stood
|
||||
out because it's the exact file I reached for by hand (grepping raw
|
||||
`dmesg.txt` myself) when first triaging the blackbox that started this
|
||||
whole investigation — its absence wasn't hypothetical.
|
||||
|
||||
Found but **not** changed — flagging for a decision, not fixed unilaterally,
|
||||
since promoting more of these turns boot-time audit into a slower fixed
|
||||
cost for every boot, not just support-bundle downloads:
|
||||
- `export/platform/lspci-nn.txt` (`lspci -nn`) — cheap, static, would be a
|
||||
trivial add.
|
||||
- `export/gpu/lspci-video-vv.txt`, `export/gpu/lspci-nvidia-bridges-vv.txt`
|
||||
— cheap-ish, somewhat redundant with `lspci-vvv.txt` (already in
|
||||
techdump) but with NVIDIA-specific bridge-chain framing that's genuinely
|
||||
easier to read.
|
||||
- `export/gpu/systemctl-nvidia-units.txt`, `export/gpu/dcgmi-nvlink-status.txt`,
|
||||
`export/gpu/fabric-manager-paths.txt` — cheap, static.
|
||||
- `export/gpu/nvidia-smi-topo-fresh.txt` / `-nvlink-status-fresh.txt` /
|
||||
`-nvlink-errors-fresh.txt` — deliberately **not** a gap: their entire
|
||||
purpose is being a fresher resample than the boot-time
|
||||
`nvidia-smi-topo.txt` etc. that techdump already captures and blackbox
|
||||
already mirrors; only useful as an on-demand recapture.
|
||||
- `export/gpu/nvidia-bug-report.txt` (30–120s via `nvidia-bug-report.sh`)
|
||||
and the network `export/network/ethtool-*`/`mstflint-query.txt` entries
|
||||
— left alone; meaningfully heavier or more device-count-dependent than
|
||||
the others, worth a deliberate cost/benefit call rather than folding in
|
||||
by default.
|
||||
Reference in New Issue
Block a user