fix(parser): support Inspur HGX dump_<serial>_<timestamp>/ onekeylog layout
Fixes #20. This onekeylog variant has no devicefrusdr.log at all: FRU/sensors come from raw ipmitool text output, PCIe/GPU presence has a dedicated structural snapshot, SEL lives at a different path, and BMC component failures are logged separately from SEL/IDL. - Fall back to component/fru.txt (same FRU block format as devicefrusdr.log) and component/sensor.txt / sdr.txt (ipmitool sensor list / sdr elist) when devicefrusdr.log is absent. - Parse log/bmc/diagnose/OtrdDiagnoseComponent.json's PCIe Device Info array for GPU/PCIe presence and link state, independent of SEL/IDL alarm history; flag devices running below their negotiated max link speed/width as degraded with a Warning event. - Fall back to log/sel.csv (same format as selelist.csv) when selelist.csv is absent. - Parse log/bmc/commer-comp/{commerslot,commerhmc,commerswvr,commerswcpld} logs (including rotated *.tar.gz.N parts) into failure events, filtering known-noisy lines. - Collapse SEL events duplicated across sources by (timestamp, event_type, description). - Surface a CollectionError when FRU/sensors are still empty after all fallbacks, instead of silently returning an empty inventory. - Fix ParseFRU: a later placeholder "Product Serial : 0" / "Product Part Number : NULL" line in the same FRU block (e.g. SCM_FRU) was overwriting an already-parsed real Board Serial/Part Number. Verified against dump_23DB01633_20260727-1359.tar.gz (HGX B200, KR9288-X3): fru 0→21, sensors 0→303, 8 GPUs present at Gen5 x16 in slots 100-107. Deferred (not covered by this change): BIOS-change-settings context and BIOS POST codes from the same layout — see bible-local/10-decisions.md ADL-048. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1229,3 +1229,58 @@ later content-free duplicate cannot blank out an earlier real reading.
|
||||
Revisit only if a real archive needs that specific data and asset.json does not cover it.
|
||||
- Any future component/*.txt GETALL consumer should reuse `parseDBusGetAllObjects` rather than
|
||||
re-implementing block splitting or field extraction.
|
||||
|
||||
---
|
||||
|
||||
## ADL-048 — Inspur HGX `dump_<serial>_<timestamp>/` layout: ipmitool-output FRU/sensors, diagnose-json PCIe, commer-comp events
|
||||
|
||||
**Date:** 2026-07-29
|
||||
**Context:** A field dump from an HGX B200 / KR9288-X3 (`dump_23DB01633_20260727-1359.tar.gz`)
|
||||
uses yet another onekeylog variant that has no `devicefrusdr.log` at all (unlike ADL-047's
|
||||
per-file D-Bus layout, which is missing only `component.log`). FRU and sensors come from raw
|
||||
`ipmitool fru print` / `ipmitool sensor list` / `ipmitool sdr elist` output under
|
||||
`component/{fru,sensor,sdr}.txt`. Without a fallback the parser returned `fru: 0, sensors: 0` on
|
||||
an otherwise well-detected archive (`Detect()` still scored via `onekeylog_dreport.log`), silently
|
||||
dropping inventory a diagnostic case (intermittent GPU PCIe dropout) depended on.
|
||||
**Decision:**
|
||||
- `component/fru.txt` uses the exact same `FRU Device Description :` block format as the FRU
|
||||
section of `devicefrusdr.log`, so it's parsed with the existing `ParseFRU` unchanged.
|
||||
- `component/sensor.txt` (`ipmitool sensor list`, has value+unit+status per line) is the primary
|
||||
sensor fallback; `component/sdr.txt` (`ipmitool sdr elist`, no value on disabled sensors) is used
|
||||
only if `sensor.txt` is absent. New parsers in `component_fallback.go`
|
||||
(`ParseSensorList`, `ParseSDRElist`).
|
||||
- `log/bmc/diagnose/OtrdDiagnoseComponent.json`'s `"Pcie Device Info"` array is a structural,
|
||||
point-in-time PCIe snapshot (independent of SEL/IDL alarm history) — parsed by
|
||||
`ParseOtrdDiagnosePCIe` in `diagnose.go` into `models.PCIeDevice` (using the existing
|
||||
`Present *bool` field) and merged via the existing `MergePCIeDevices`. A device present but
|
||||
running below its negotiated max link speed/width gets `Status = "Link Degraded"` and a
|
||||
matching Warning event (`BuildPCIeLinkDegradationEvents`), independent of GPU-fault SEL/IDL
|
||||
parsing in `gpu_status.go`.
|
||||
- `log/sel.csv` is textually identical to `selelist.csv` (`ipmitool sel elist -c -Z` output with
|
||||
the same 6-column CSV body) and is now a fallback source when `selelist.csv` is absent, reusing
|
||||
`ParseSELListWithLocation` unchanged.
|
||||
- `log/bmc/commer-comp/<commerslot|commerhmc|commerswvr|commerswcpld>/` component logs (plus
|
||||
rotated `*.tar.gz.N` parts, decoded in-memory) share one line format
|
||||
(`[timestamp][file, line][level] message`); `ParseCommerCompEvents` in `commercomp.go` emits an
|
||||
event per line that looks like a real failure, after dropping a fixed noise list (`Invalid Comp
|
||||
Data`, `Error in update redis`, `MutexTimeout = 0`).
|
||||
- Fixed two long-standing `ParseFRU` bugs surfaced by this archive: `Board Serial`/`Board Part
|
||||
Number` values were overwritten by a later, placeholder `Product Serial : 0` / `Product Part
|
||||
Number : NULL` line within the same FRU block (e.g. `SCM_FRU`, which has no product-level FRU
|
||||
fields). Placeholder values (`"0"`, `"NULL"`) no longer overwrite an already-set real value.
|
||||
- Since duplicate SEL entries can now surface through more than one source file (`sel.csv` +
|
||||
IDL/BMC logs) for the same moment, `Parse()` runs a final `dedupSELEvents` pass collapsing
|
||||
`Source == "SEL"` events with identical `(timestamp, event_type, description)`. This is
|
||||
deliberately narrower than IDL's own timestamp-inclusive dedup in `idl.go`, which must keep
|
||||
distinct occurrences of a recurring alarm.
|
||||
- When `Parse()` still ends up with `len(FRU) == 0 || len(Sensors) == 0` after all fallbacks, it
|
||||
appends a `CollectionError{Section: "inventory"}` so the gap is visible in the API response
|
||||
instead of silently returning an empty inventory.
|
||||
**Consequences:**
|
||||
- `stats.fru`/`stats.sensors` are non-zero on this dump class; GPU presence/link state is
|
||||
available from a source independent of SEL/IDL alarm parsing.
|
||||
- BIOS-change-settings context (`biosChangedSettings/Bios_change_settings*.json`) and BIOS POST
|
||||
codes (`log/server/progress_log/host0/*`) from the same issue are not yet parsed — deferred,
|
||||
no acceptance case depended on them yet.
|
||||
- Any future ipmitool-text-output fallback (FRU/sensor/sdr) should extend
|
||||
`component_fallback.go` rather than adding another one-off parser.
|
||||
|
||||
+169
@@ -0,0 +1,169 @@
|
||||
// Parses log/bmc/commer-comp/*, per-component BMC transcripts (slot presence,
|
||||
// HGX management controller, switch VR/power, switch CPLD). Each rotated log
|
||||
// (component.log plus component.tar.gz.0..9) uses the same
|
||||
// "[timestamp][file, line][level] message" line format.
|
||||
package inspur
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bufio"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"fmt"
|
||||
"io"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.mchus.pro/mchus/logpile/internal/models"
|
||||
"git.mchus.pro/mchus/logpile/internal/parser"
|
||||
)
|
||||
|
||||
var commerCompComponents = []string{"commerslot", "commerhmc", "commerswvr", "commerswcpld"}
|
||||
|
||||
var commerCompLineRegex = regexp.MustCompile(`^\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\]\[[^\]]*\]\[(\w+)\]\s*(.+)$`)
|
||||
|
||||
var commerCompNoisePatterns = []*regexp.Regexp{
|
||||
regexp.MustCompile(`(?i)invalid comp data`),
|
||||
regexp.MustCompile(`(?i)error in update redis`),
|
||||
regexp.MustCompile(`(?i)mutextimeout\s*=\s*0`),
|
||||
}
|
||||
|
||||
var commerCompFailureRegex = regexp.MustCompile(`(?i)\b(fail|failed|failure|error)\b`)
|
||||
|
||||
// ParseCommerCompEvents extracts BMC component failure events from
|
||||
// log/bmc/commer-comp/<component>/ logs, including rotated *.tar.gz.N parts.
|
||||
func ParseCommerCompEvents(files []parser.ExtractedFile, location *time.Location) []models.Event {
|
||||
var events []models.Event
|
||||
for _, f := range files {
|
||||
path := strings.ToLower(f.Path)
|
||||
if !strings.Contains(path, "/commer-comp/") {
|
||||
continue
|
||||
}
|
||||
component := commerCompComponentFromPath(path)
|
||||
if component == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
if strings.Contains(path, ".tar.gz") {
|
||||
for _, part := range decodeNestedTarGz(f.Content) {
|
||||
events = append(events, parseCommerCompLog(part, component, location)...)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if !strings.HasSuffix(path, ".log") {
|
||||
continue
|
||||
}
|
||||
events = append(events, parseCommerCompLog(f.Content, component, location)...)
|
||||
}
|
||||
return events
|
||||
}
|
||||
|
||||
func commerCompComponentFromPath(path string) string {
|
||||
for _, c := range commerCompComponents {
|
||||
if strings.Contains(path, "/"+c+"/") {
|
||||
return c
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// decodeNestedTarGz unpacks a rotated log part (a full tar.gz archive stored
|
||||
// as e.g. commerhmc.tar.gz.0) and returns the contents of every regular file
|
||||
// inside it.
|
||||
func decodeNestedTarGz(content []byte) [][]byte {
|
||||
gz, err := gzip.NewReader(bytes.NewReader(content))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
defer gz.Close()
|
||||
|
||||
tr := tar.NewReader(gz)
|
||||
var out [][]byte
|
||||
for {
|
||||
hdr, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
if hdr.Typeflag != tar.TypeReg {
|
||||
continue
|
||||
}
|
||||
buf, err := io.ReadAll(tr)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, buf)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func parseCommerCompLog(content []byte, component string, location *time.Location) []models.Event {
|
||||
var events []models.Event
|
||||
scanner := bufio.NewScanner(bytes.NewReader(content))
|
||||
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
||||
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
m := commerCompLineRegex.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
msg := strings.TrimSpace(m[3])
|
||||
if isCommerCompNoise(msg) || !commerCompFailureRegex.MatchString(msg) {
|
||||
continue
|
||||
}
|
||||
|
||||
ts, err := time.ParseInLocation("2006-01-02 15:04:05", m[1], location)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
events = append(events, models.Event{
|
||||
ID: fmt.Sprintf("commer_comp_%s_%d_%s", component, ts.Unix(), commerCompIDSuffix(msg)),
|
||||
Timestamp: ts,
|
||||
Source: fmt.Sprintf("BMC/%s", component),
|
||||
SensorType: "bmc_component",
|
||||
EventType: m[2],
|
||||
Severity: commerCompSeverity(m[2]),
|
||||
Description: msg,
|
||||
RawData: line,
|
||||
})
|
||||
}
|
||||
return events
|
||||
}
|
||||
|
||||
func isCommerCompNoise(msg string) bool {
|
||||
for _, re := range commerCompNoisePatterns {
|
||||
if re.MatchString(msg) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
var commerCompNonAlnumRegex = regexp.MustCompile(`[^a-zA-Z0-9]+`)
|
||||
|
||||
// commerCompIDSuffix derives a short, stable suffix from the message so that
|
||||
// two different failures logged in the same second get distinct event IDs.
|
||||
func commerCompIDSuffix(msg string) string {
|
||||
s := commerCompNonAlnumRegex.ReplaceAllString(strings.ToLower(msg), "_")
|
||||
s = strings.Trim(s, "_")
|
||||
if len(s) > 40 {
|
||||
s = s[:40]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func commerCompSeverity(level string) models.Severity {
|
||||
switch strings.ToLower(level) {
|
||||
case "critial", "critical", "error":
|
||||
return models.SeverityCritical
|
||||
case "warning", "warn":
|
||||
return models.SeverityWarning
|
||||
default:
|
||||
return models.SeverityInfo
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package inspur
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.mchus.pro/mchus/logpile/internal/parser"
|
||||
)
|
||||
|
||||
func TestParseCommerCompEvents(t *testing.T) {
|
||||
content := []byte(`[2026-07-08 16:46:19][commer_comp_anal.c, 635][Critial] [COMP]Invalid Comp Data, Continue... ParamList: Index-3, Valid-0x1, MediaType-0(1-PCIE,2-I2C)
|
||||
[2026-07-27 04:52:55][commer_comp_slot.c, 319][Critial] Data Update fail for dev serial-6!
|
||||
[2026-07-27 04:52:55][commer_comp_slot.c, 319][Critial] Data Update fail for dev serial-7!
|
||||
[2026-07-27 05:00:00][commer_comp_slot.c, 100][Info] Error in update redis, retry
|
||||
`)
|
||||
|
||||
files := []parser.ExtractedFile{
|
||||
{Path: "log/bmc/commer-comp/commerslot/commerslot.log", Content: content},
|
||||
}
|
||||
|
||||
events := ParseCommerCompEvents(files, time.UTC)
|
||||
if len(events) != 2 {
|
||||
t.Fatalf("expected 2 events (noise filtered), got %d: %+v", len(events), events)
|
||||
}
|
||||
for _, e := range events {
|
||||
if e.Source != "BMC/commerslot" {
|
||||
t.Fatalf("unexpected source %q", e.Source)
|
||||
}
|
||||
}
|
||||
if events[0].Timestamp.Format("2006-01-02 15:04:05") != "2026-07-27 04:52:55" {
|
||||
t.Fatalf("unexpected timestamp: %v", events[0].Timestamp)
|
||||
}
|
||||
if events[0].ID == events[1].ID {
|
||||
t.Fatalf("expected distinct event IDs for serial-6/serial-7, got same: %q", events[0].ID)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
// Fallback inventory parsing for the per-file dump_<serial>_<timestamp>/ onekeylog
|
||||
// layout, which has no combined devicefrusdr.log. Instead FRU and sensors live in
|
||||
// separate ipmitool-output text files under component/.
|
||||
package inspur
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"git.mchus.pro/mchus/logpile/internal/models"
|
||||
)
|
||||
|
||||
// ParseSensorList parses component/sensor.txt ("ipmitool sensor list" output).
|
||||
// Line format: Name | Value | Unit | Status | lnr | lcr | lnc | unc | ucr | unr
|
||||
func ParseSensorList(content []byte) []models.SensorReading {
|
||||
var readings []models.SensorReading
|
||||
|
||||
scanner := bufio.NewScanner(strings.NewReader(string(content)))
|
||||
for scanner.Scan() {
|
||||
line := strings.TrimSpace(scanner.Text())
|
||||
if line == "" || !strings.Contains(line, "|") {
|
||||
continue
|
||||
}
|
||||
fields := strings.Split(line, "|")
|
||||
if len(fields) < 4 {
|
||||
continue
|
||||
}
|
||||
name := strings.TrimSpace(fields[0])
|
||||
valueStr := strings.TrimSpace(fields[1])
|
||||
unit := strings.TrimSpace(fields[2])
|
||||
status := strings.TrimSpace(fields[3])
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
reading := models.SensorReading{
|
||||
Name: name,
|
||||
Status: status,
|
||||
Unit: unit,
|
||||
Type: determineSensorType(name),
|
||||
}
|
||||
|
||||
if valueStr != "" && valueStr != "na" {
|
||||
if v, err := strconv.ParseFloat(valueStr, 64); err == nil {
|
||||
reading.Value = v
|
||||
reading.RawValue = valueStr + " " + unit
|
||||
} else {
|
||||
reading.RawValue = valueStr
|
||||
}
|
||||
}
|
||||
|
||||
readings = append(readings, reading)
|
||||
}
|
||||
|
||||
return readings
|
||||
}
|
||||
|
||||
// ParseSDRElist parses component/sdr.txt ("ipmitool sdr elist" output), used as a
|
||||
// fallback when component/sensor.txt is unavailable.
|
||||
// Line format: Name | ID | Status | Entity | Reading (e.g. "21 degrees C" or "no reading")
|
||||
func ParseSDRElist(content []byte) []models.SensorReading {
|
||||
var readings []models.SensorReading
|
||||
|
||||
scanner := bufio.NewScanner(strings.NewReader(string(content)))
|
||||
for scanner.Scan() {
|
||||
line := strings.TrimSpace(scanner.Text())
|
||||
if line == "" || !strings.Contains(line, "|") {
|
||||
continue
|
||||
}
|
||||
fields := strings.Split(line, "|")
|
||||
if len(fields) < 3 {
|
||||
continue
|
||||
}
|
||||
name := strings.TrimSpace(fields[0])
|
||||
status := strings.TrimSpace(fields[2])
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
reading := models.SensorReading{
|
||||
Name: name,
|
||||
Status: status,
|
||||
Type: determineSensorType(name),
|
||||
}
|
||||
|
||||
if len(fields) >= 5 {
|
||||
readingStr := strings.TrimSpace(fields[4])
|
||||
if readingStr != "" && readingStr != "no reading" {
|
||||
reading.RawValue = readingStr
|
||||
if m := valueRegex.FindStringSubmatch(readingStr); m != nil {
|
||||
if v, err := strconv.ParseFloat(m[1], 64); err == nil {
|
||||
reading.Value = v
|
||||
reading.Unit = strings.TrimSpace(m[2])
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
readings = append(readings, reading)
|
||||
}
|
||||
|
||||
return readings
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package inspur
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestParseSensorList(t *testing.T) {
|
||||
content := []byte(`============
|
||||
Description:sensor list
|
||||
Command: ipmitool sensor list
|
||||
Response:
|
||||
Inlet_Temp | 21.000 | degrees C | ok | na | na | na | na | 40.000 | na
|
||||
GPU0_Temp | 29.000 | degrees C | ok | na | na | na | na | 86.000 | na
|
||||
PSU0_Status | 0x0 | discrete | 0x0180| na | na | na | na | na | na
|
||||
`)
|
||||
|
||||
readings := ParseSensorList(content)
|
||||
if len(readings) != 3 {
|
||||
t.Fatalf("expected 3 readings, got %d", len(readings))
|
||||
}
|
||||
if readings[0].Name != "Inlet_Temp" || readings[0].Value != 21.0 || readings[0].Unit != "degrees C" {
|
||||
t.Fatalf("unexpected Inlet_Temp reading: %+v", readings[0])
|
||||
}
|
||||
if readings[1].Type != "temperature" {
|
||||
t.Fatalf("expected temperature type, got %q", readings[1].Type)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseSDRElist(t *testing.T) {
|
||||
content := []byte(`============
|
||||
Description:sdr elist
|
||||
Command: ipmitool sdr elist
|
||||
Response:
|
||||
Inlet_Temp | 00h | ok | 3.0 | 21 degrees C
|
||||
GPU0_Temp | 2Ah | ok | 11.0 | 29 degrees C
|
||||
Event Logging Disabled SEL_Status | 15h | ok | 7.0 | no reading
|
||||
`)
|
||||
|
||||
readings := ParseSDRElist(content)
|
||||
if len(readings) != 3 {
|
||||
t.Fatalf("expected 3 readings, got %d", len(readings))
|
||||
}
|
||||
if readings[0].Name != "Inlet_Temp" || readings[0].Value != 21 || readings[0].Unit != "degrees C" {
|
||||
t.Fatalf("unexpected Inlet_Temp reading: %+v", readings[0])
|
||||
}
|
||||
if readings[2].RawValue != "" {
|
||||
t.Fatalf("expected no reading to leave RawValue empty, got %q", readings[2].RawValue)
|
||||
}
|
||||
}
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
// Parses log/bmc/diagnose/OtrdDiagnoseComponent.json, an HGX BMC structural
|
||||
// snapshot of PCIe topology (present GPUs/NICs/NVMe, negotiated link state).
|
||||
// This is a direct point-in-time source for GPU presence and PCIe link
|
||||
// degradation, independent of SEL/IDL alarm history.
|
||||
package inspur
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"git.mchus.pro/mchus/logpile/internal/models"
|
||||
"git.mchus.pro/mchus/logpile/internal/parser/vendors/pciids"
|
||||
)
|
||||
|
||||
// otrdDiagnosePCIeEntry mirrors the relevant fields of "Pcie Device Info"
|
||||
// entries in OtrdDiagnoseComponent.json.
|
||||
type otrdDiagnosePCIeEntry struct {
|
||||
LocString string `json:"LocString"`
|
||||
PcieSlot int `json:"PcieSlot"`
|
||||
PresentStatus int `json:"PresentStatus"`
|
||||
VendorId int `json:"VendorId"`
|
||||
DeviceId int `json:"DeviceId"`
|
||||
BusNumber int `json:"BusNumber"`
|
||||
DeviceNumber int `json:"DeviceNumber"`
|
||||
FunctionNumber int `json:"FunctionNumber"`
|
||||
CurrentLinkSpeed int `json:"CurrentLinkSpeed"`
|
||||
MaxLinkSpeed int `json:"MaxLinkSpeed"`
|
||||
NegotiatedLinkWidth int `json:"NegotiatedLinkWidth"`
|
||||
MaxLinkWidth int `json:"MaxLinkWidth"`
|
||||
SerialNumber *string `json:"SerialNumber"`
|
||||
PartNumber *string `json:"PartNumber"`
|
||||
}
|
||||
|
||||
type otrdDiagnoseComponent struct {
|
||||
PcieDeviceInfo []otrdDiagnosePCIeEntry `json:"Pcie Device Info"`
|
||||
}
|
||||
|
||||
// ParseOtrdDiagnosePCIe parses the "Pcie Device Info" array from
|
||||
// OtrdDiagnoseComponent.json into PCIe device inventory entries.
|
||||
func ParseOtrdDiagnosePCIe(content []byte) []models.PCIeDevice {
|
||||
var doc otrdDiagnoseComponent
|
||||
if err := json.Unmarshal(content, &doc); err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
devices := make([]models.PCIeDevice, 0, len(doc.PcieDeviceInfo))
|
||||
for _, e := range doc.PcieDeviceInfo {
|
||||
present := e.PresentStatus == 1
|
||||
_, deviceName := pciids.DeviceInfo(e.VendorId, e.DeviceId)
|
||||
deviceClass := normalizeModelLabel(deviceName)
|
||||
if strings.Contains(strings.ToUpper(deviceName), "NVIDIA") {
|
||||
deviceClass = "GPU (" + deviceName + ")"
|
||||
}
|
||||
|
||||
serial := ""
|
||||
if e.SerialNumber != nil {
|
||||
serial = strings.TrimSpace(*e.SerialNumber)
|
||||
}
|
||||
partNum := ""
|
||||
if e.PartNumber != nil {
|
||||
partNum = strings.TrimSpace(*e.PartNumber)
|
||||
}
|
||||
|
||||
status := ""
|
||||
if present && isLinkDegraded(e) {
|
||||
status = "Link Degraded"
|
||||
}
|
||||
|
||||
devices = append(devices, models.PCIeDevice{
|
||||
Slot: e.LocString,
|
||||
VendorID: e.VendorId,
|
||||
DeviceID: e.DeviceId,
|
||||
BDF: formatBDF(e.BusNumber, e.DeviceNumber, e.FunctionNumber),
|
||||
DeviceClass: deviceClass,
|
||||
Manufacturer: normalizeModelLabel(pciids.VendorName(e.VendorId)),
|
||||
LinkWidth: e.NegotiatedLinkWidth,
|
||||
LinkSpeed: fmt.Sprintf("GEN%d", e.CurrentLinkSpeed),
|
||||
MaxLinkWidth: e.MaxLinkWidth,
|
||||
MaxLinkSpeed: fmt.Sprintf("GEN%d", e.MaxLinkSpeed),
|
||||
PartNumber: partNum,
|
||||
SerialNumber: serial,
|
||||
Present: &present,
|
||||
Status: status,
|
||||
})
|
||||
}
|
||||
|
||||
return devices
|
||||
}
|
||||
|
||||
func isLinkDegraded(e otrdDiagnosePCIeEntry) bool {
|
||||
if e.MaxLinkSpeed > 0 && e.CurrentLinkSpeed < e.MaxLinkSpeed {
|
||||
return true
|
||||
}
|
||||
if e.MaxLinkWidth > 0 && e.NegotiatedLinkWidth < e.MaxLinkWidth {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// BuildPCIeLinkDegradationEvents emits a Warning event for every PCIe device
|
||||
// present in OtrdDiagnoseComponent.json but running below its negotiated max
|
||||
// link speed/width (e.g. a GPU baseboard degrading from Gen5 x16 to a lower
|
||||
// state), so the log viewer surfaces the same signal the IDL alarms would.
|
||||
func BuildPCIeLinkDegradationEvents(devices []models.PCIeDevice) []models.Event {
|
||||
var events []models.Event
|
||||
for _, d := range devices {
|
||||
if d.Present == nil || !*d.Present || d.Status != "Link Degraded" {
|
||||
continue
|
||||
}
|
||||
events = append(events, models.Event{
|
||||
ID: fmt.Sprintf("pcie_link_degraded_%s", strings.TrimPrefix(d.Slot, "#")),
|
||||
Source: "PCIe Diagnose",
|
||||
SensorType: "pcie_link",
|
||||
EventType: "Link Degraded",
|
||||
Severity: models.SeverityWarning,
|
||||
Description: fmt.Sprintf("%s: link degraded to %s x%d (max %s x%d)", d.Slot, d.LinkSpeed, d.LinkWidth, d.MaxLinkSpeed, d.MaxLinkWidth),
|
||||
})
|
||||
}
|
||||
return events
|
||||
}
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
package inspur
|
||||
|
||||
import "testing"
|
||||
|
||||
const diagnoseJSONFixture = `{
|
||||
"Pcie Device Info": [
|
||||
{
|
||||
"LocString": "#GPU0",
|
||||
"PcieSlot": 100,
|
||||
"PresentStatus": 1,
|
||||
"VendorId": 4318,
|
||||
"DeviceId": 10497,
|
||||
"BusNumber": 23,
|
||||
"DeviceNumber": 0,
|
||||
"FunctionNumber": 0,
|
||||
"CurrentLinkSpeed": 5,
|
||||
"MaxLinkSpeed": 5,
|
||||
"NegotiatedLinkWidth": 16,
|
||||
"MaxLinkWidth": 16
|
||||
},
|
||||
{
|
||||
"LocString": "#GPU6",
|
||||
"PcieSlot": 106,
|
||||
"PresentStatus": 1,
|
||||
"VendorId": 4318,
|
||||
"DeviceId": 10497,
|
||||
"BusNumber": 220,
|
||||
"DeviceNumber": 0,
|
||||
"FunctionNumber": 0,
|
||||
"CurrentLinkSpeed": 2,
|
||||
"MaxLinkSpeed": 5,
|
||||
"NegotiatedLinkWidth": 8,
|
||||
"MaxLinkWidth": 16
|
||||
}
|
||||
]
|
||||
}`
|
||||
|
||||
func TestParseOtrdDiagnosePCIe(t *testing.T) {
|
||||
devices := ParseOtrdDiagnosePCIe([]byte(diagnoseJSONFixture))
|
||||
if len(devices) != 2 {
|
||||
t.Fatalf("expected 2 devices, got %d", len(devices))
|
||||
}
|
||||
|
||||
gpu0 := devices[0]
|
||||
if gpu0.Slot != "#GPU0" || gpu0.Present == nil || !*gpu0.Present {
|
||||
t.Fatalf("expected GPU0 present, got %+v", gpu0)
|
||||
}
|
||||
if gpu0.Status == "Link Degraded" {
|
||||
t.Fatalf("GPU0 should not be flagged as degraded: %+v", gpu0)
|
||||
}
|
||||
if gpu0.LinkSpeed != "GEN5" || gpu0.MaxLinkSpeed != "GEN5" || gpu0.LinkWidth != 16 {
|
||||
t.Fatalf("unexpected GPU0 link info: %+v", gpu0)
|
||||
}
|
||||
|
||||
gpu6 := devices[1]
|
||||
if gpu6.Status != "Link Degraded" {
|
||||
t.Fatalf("expected GPU6 to be flagged as degraded, got %+v", gpu6)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPCIeLinkDegradationEvents(t *testing.T) {
|
||||
devices := ParseOtrdDiagnosePCIe([]byte(diagnoseJSONFixture))
|
||||
events := BuildPCIeLinkDegradationEvents(devices)
|
||||
if len(events) != 1 {
|
||||
t.Fatalf("expected 1 degradation event, got %d", len(events))
|
||||
}
|
||||
if events[0].Severity != "warning" {
|
||||
t.Fatalf("expected warning severity, got %q", events[0].Severity)
|
||||
}
|
||||
}
|
||||
+9
-2
@@ -72,9 +72,16 @@ func ParseFRU(content []byte) []models.FRUInfo {
|
||||
current.ProductName = fieldValue
|
||||
}
|
||||
case "Board Serial", "Product Serial":
|
||||
current.SerialNumber = fieldValue
|
||||
// Modules with no product-level FRU (e.g. SCM_FRU) report
|
||||
// "Product Serial : 0" after a real "Board Serial" value;
|
||||
// don't let the placeholder overwrite the real serial.
|
||||
if fieldValue != "0" && fieldValue != "NULL" {
|
||||
current.SerialNumber = fieldValue
|
||||
}
|
||||
case "Board Part Number", "Product Part Number":
|
||||
if fieldValue != "0" {
|
||||
// See "Board Serial" above: placeholder "NULL"/"0" product-level
|
||||
// fields must not overwrite a real board-level part number.
|
||||
if fieldValue != "0" && fieldValue != "NULL" {
|
||||
current.PartNumber = fieldValue
|
||||
}
|
||||
case "Product Version":
|
||||
|
||||
+48
-2
@@ -16,7 +16,7 @@ import (
|
||||
|
||||
// parserVersion - version of this parser module
|
||||
// IMPORTANT: Increment this version when making changes to parser logic!
|
||||
const parserVersion = "2.2"
|
||||
const parserVersion = "2.3"
|
||||
|
||||
func init() {
|
||||
parser.Register(&Parser{})
|
||||
@@ -203,6 +203,26 @@ func (p *Parser) Parse(files []parser.ExtractedFile) (*models.AnalysisResult, er
|
||||
}
|
||||
}
|
||||
|
||||
// New dump_<serial>_<timestamp>/ layout has no devicefrusdr.log: FRU and
|
||||
// sensors live in separate ipmitool-output text files under component/.
|
||||
if parser.FindFileByName(files, "devicefrusdr.log") == nil {
|
||||
if f := parser.FindFileByName(files, "fru.txt"); f != nil && len(result.FRU) == 0 {
|
||||
result.FRU = ParseFRU(f.Content)
|
||||
extractBoardInfo(result.FRU, result.Hardware)
|
||||
}
|
||||
if f := parser.FindFileByName(files, "sensor.txt"); f != nil && len(result.Sensors) == 0 {
|
||||
result.Sensors = mergeSensorReadings(result.Sensors, ParseSensorList(f.Content))
|
||||
} else if f := parser.FindFileByName(files, "sdr.txt"); f != nil && len(result.Sensors) == 0 {
|
||||
result.Sensors = mergeSensorReadings(result.Sensors, ParseSDRElist(f.Content))
|
||||
}
|
||||
// Structural PCIe snapshot (GPU presence/link state) from the HGX diagnose dump.
|
||||
if f := parser.FindFileByName(files, "OtrdDiagnoseComponent.json"); f != nil && result.Hardware != nil {
|
||||
diagPCIe := ParseOtrdDiagnosePCIe(f.Content)
|
||||
result.Hardware.PCIeDevices = MergePCIeDevices(result.Hardware.PCIeDevices, diagPCIe)
|
||||
result.Events = append(result.Events, BuildPCIeLinkDegradationEvents(diagPCIe)...)
|
||||
}
|
||||
}
|
||||
|
||||
// Enrich runtime component data from Redis snapshot (serials, FW, telemetry),
|
||||
// when text logs miss these fields.
|
||||
if f := parser.FindFileByName(files, "redis-dump.rdb"); f != nil && result.Hardware != nil {
|
||||
@@ -216,10 +236,14 @@ func (p *Parser) Parse(files []parser.ExtractedFile) (*models.AnalysisResult, er
|
||||
result.Events = append(result.Events, idlEvents...)
|
||||
}
|
||||
|
||||
// Parse SEL list (selelist.csv)
|
||||
// Parse SEL list (selelist.csv), falling back to log/sel.csv on the
|
||||
// dump_<serial>_<timestamp>/ layout that has no selelist.csv.
|
||||
if f := parser.FindFileByName(files, "selelist.csv"); f != nil {
|
||||
selEvents := ParseSELListWithLocation(f.Content, selLocation)
|
||||
result.Events = append(result.Events, selEvents...)
|
||||
} else if f := parser.FindFileByName(files, "sel.csv"); f != nil {
|
||||
selEvents := ParseSELListWithLocation(f.Content, selLocation)
|
||||
result.Events = append(result.Events, selEvents...)
|
||||
}
|
||||
|
||||
// Parse syslog files
|
||||
@@ -229,6 +253,14 @@ func (p *Parser) Parse(files []parser.ExtractedFile) (*models.AnalysisResult, er
|
||||
result.Events = append(result.Events, events...)
|
||||
}
|
||||
|
||||
// Parse BMC component failure logs (log/bmc/commer-comp/*): HGX management
|
||||
// controller, switch VR/power, switch CPLD and slot presence failures.
|
||||
result.Events = append(result.Events, ParseCommerCompEvents(files, selLocation)...)
|
||||
|
||||
// Same SEL event can be reported twice by the BMC (e.g. once via sel.csv,
|
||||
// once via idl.log); collapse exact duplicates so they don't double-count.
|
||||
result.Events = dedupSELEvents(result.Events)
|
||||
|
||||
// Fallback for archives where board serial is missing in parsed FRU/asset data:
|
||||
// recover it from log content, never from archive filename.
|
||||
if strings.TrimSpace(result.Hardware.BoardInfo.SerialNumber) == "" {
|
||||
@@ -261,6 +293,20 @@ func (p *Parser) Parse(files []parser.ExtractedFile) (*models.AnalysisResult, er
|
||||
parser.ApplyManufacturedYearWeekFromFRU(result.FRU, result.Hardware)
|
||||
}
|
||||
|
||||
if len(result.FRU) == 0 || len(result.Sensors) == 0 {
|
||||
var missing []string
|
||||
if len(result.FRU) == 0 {
|
||||
missing = append(missing, "FRU")
|
||||
}
|
||||
if len(result.Sensors) == 0 {
|
||||
missing = append(missing, "sensors")
|
||||
}
|
||||
result.CollectionErrors = append(result.CollectionErrors, models.CollectionError{
|
||||
Section: "inventory",
|
||||
Message: fmt.Sprintf("inventory sources not found: devicefrusdr.log absent, fell back to component/fru.txt+sensor.txt/sdr.txt, still missing: %s", strings.Join(missing, ", ")),
|
||||
})
|
||||
}
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
|
||||
+6
@@ -314,6 +314,12 @@ func enrichPCIeDevice(dst *models.PCIeDevice, src models.PCIeDevice) {
|
||||
if isGenericPCIeClass(dst.DeviceClass) && !isGenericPCIeClass(src.DeviceClass) {
|
||||
dst.DeviceClass = src.DeviceClass
|
||||
}
|
||||
if dst.Present == nil {
|
||||
dst.Present = src.Present
|
||||
}
|
||||
if strings.TrimSpace(dst.Status) == "" {
|
||||
dst.Status = src.Status
|
||||
}
|
||||
}
|
||||
|
||||
func normalizePCIeBDF(bdf string) string {
|
||||
|
||||
+26
@@ -176,6 +176,32 @@ func determineSELSeverity(sensorStr, eventDesc, status string) models.Severity {
|
||||
return models.SeverityInfo
|
||||
}
|
||||
|
||||
// dedupSELEvents collapses events reported more than once with the same
|
||||
// (timestamp, event_type, description) triple. Unlike ParseIDLLog's
|
||||
// dedup (which must keep recurring alarms with distinct timestamps), this
|
||||
// only removes true duplicates: the same SEL entry surfacing through more
|
||||
// than one source file for the exact same moment.
|
||||
func dedupSELEvents(events []models.Event) []models.Event {
|
||||
if len(events) == 0 {
|
||||
return events
|
||||
}
|
||||
seen := make(map[string]struct{}, len(events))
|
||||
out := make([]models.Event, 0, len(events))
|
||||
for _, e := range events {
|
||||
if e.Source != "SEL" {
|
||||
out = append(out, e)
|
||||
continue
|
||||
}
|
||||
key := e.Timestamp.String() + "|" + e.EventType + "|" + e.Description
|
||||
if _, ok := seen[key]; ok {
|
||||
continue
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
out = append(out, e)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// buildSELDescription builds human-readable description
|
||||
func buildSELDescription(eventDesc, status string) string {
|
||||
if status == "Asserted" || status == "Deasserted" {
|
||||
|
||||
+17
@@ -3,8 +3,25 @@ package inspur
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.mchus.pro/mchus/logpile/internal/models"
|
||||
)
|
||||
|
||||
func TestDedupSELEvents(t *testing.T) {
|
||||
ts := time.Date(2026, 7, 27, 4, 52, 55, 0, time.UTC)
|
||||
events := []models.Event{
|
||||
{Source: "SEL", Timestamp: ts, EventType: "Asserted", Description: "PSU0 fail"},
|
||||
{Source: "SEL", Timestamp: ts, EventType: "Asserted", Description: "PSU0 fail"},
|
||||
{Source: "BMC", Timestamp: ts, EventType: "Assert", Description: "recurring alarm"},
|
||||
{Source: "BMC", Timestamp: ts, EventType: "Assert", Description: "recurring alarm"},
|
||||
}
|
||||
|
||||
out := dedupSELEvents(events)
|
||||
if len(out) != 3 {
|
||||
t.Fatalf("expected 3 events (1 SEL dup removed, BMC untouched), got %d: %+v", len(out), out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseSELListWithLocation_UsesProvidedTimezone(t *testing.T) {
|
||||
content := []byte("sel elist:\n1,02/28/2026,04:18:18,Sensor X,Event,Asserted\n")
|
||||
shanghai, err := time.LoadLocation("Asia/Shanghai")
|
||||
|
||||
Reference in New Issue
Block a user