fix(parser): support Inspur HGX dump_<serial>_<timestamp>/ onekeylog layout
Fixes #20. This onekeylog variant has no devicefrusdr.log at all: FRU/sensors come from raw ipmitool text output, PCIe/GPU presence has a dedicated structural snapshot, SEL lives at a different path, and BMC component failures are logged separately from SEL/IDL. - Fall back to component/fru.txt (same FRU block format as devicefrusdr.log) and component/sensor.txt / sdr.txt (ipmitool sensor list / sdr elist) when devicefrusdr.log is absent. - Parse log/bmc/diagnose/OtrdDiagnoseComponent.json's PCIe Device Info array for GPU/PCIe presence and link state, independent of SEL/IDL alarm history; flag devices running below their negotiated max link speed/width as degraded with a Warning event. - Fall back to log/sel.csv (same format as selelist.csv) when selelist.csv is absent. - Parse log/bmc/commer-comp/{commerslot,commerhmc,commerswvr,commerswcpld} logs (including rotated *.tar.gz.N parts) into failure events, filtering known-noisy lines. - Collapse SEL events duplicated across sources by (timestamp, event_type, description). - Surface a CollectionError when FRU/sensors are still empty after all fallbacks, instead of silently returning an empty inventory. - Fix ParseFRU: a later placeholder "Product Serial : 0" / "Product Part Number : NULL" line in the same FRU block (e.g. SCM_FRU) was overwriting an already-parsed real Board Serial/Part Number. Verified against dump_23DB01633_20260727-1359.tar.gz (HGX B200, KR9288-X3): fru 0→21, sensors 0→303, 8 GPUs present at Gen5 x16 in slots 100-107. Deferred (not covered by this change): BIOS-change-settings context and BIOS POST codes from the same layout — see bible-local/10-decisions.md ADL-048. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1229,3 +1229,58 @@ later content-free duplicate cannot blank out an earlier real reading.
|
|||||||
Revisit only if a real archive needs that specific data and asset.json does not cover it.
|
Revisit only if a real archive needs that specific data and asset.json does not cover it.
|
||||||
- Any future component/*.txt GETALL consumer should reuse `parseDBusGetAllObjects` rather than
|
- Any future component/*.txt GETALL consumer should reuse `parseDBusGetAllObjects` rather than
|
||||||
re-implementing block splitting or field extraction.
|
re-implementing block splitting or field extraction.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ADL-048 — Inspur HGX `dump_<serial>_<timestamp>/` layout: ipmitool-output FRU/sensors, diagnose-json PCIe, commer-comp events
|
||||||
|
|
||||||
|
**Date:** 2026-07-29
|
||||||
|
**Context:** A field dump from an HGX B200 / KR9288-X3 (`dump_23DB01633_20260727-1359.tar.gz`)
|
||||||
|
uses yet another onekeylog variant that has no `devicefrusdr.log` at all (unlike ADL-047's
|
||||||
|
per-file D-Bus layout, which is missing only `component.log`). FRU and sensors come from raw
|
||||||
|
`ipmitool fru print` / `ipmitool sensor list` / `ipmitool sdr elist` output under
|
||||||
|
`component/{fru,sensor,sdr}.txt`. Without a fallback the parser returned `fru: 0, sensors: 0` on
|
||||||
|
an otherwise well-detected archive (`Detect()` still scored via `onekeylog_dreport.log`), silently
|
||||||
|
dropping inventory a diagnostic case (intermittent GPU PCIe dropout) depended on.
|
||||||
|
**Decision:**
|
||||||
|
- `component/fru.txt` uses the exact same `FRU Device Description :` block format as the FRU
|
||||||
|
section of `devicefrusdr.log`, so it's parsed with the existing `ParseFRU` unchanged.
|
||||||
|
- `component/sensor.txt` (`ipmitool sensor list`, has value+unit+status per line) is the primary
|
||||||
|
sensor fallback; `component/sdr.txt` (`ipmitool sdr elist`, no value on disabled sensors) is used
|
||||||
|
only if `sensor.txt` is absent. New parsers in `component_fallback.go`
|
||||||
|
(`ParseSensorList`, `ParseSDRElist`).
|
||||||
|
- `log/bmc/diagnose/OtrdDiagnoseComponent.json`'s `"Pcie Device Info"` array is a structural,
|
||||||
|
point-in-time PCIe snapshot (independent of SEL/IDL alarm history) — parsed by
|
||||||
|
`ParseOtrdDiagnosePCIe` in `diagnose.go` into `models.PCIeDevice` (using the existing
|
||||||
|
`Present *bool` field) and merged via the existing `MergePCIeDevices`. A device present but
|
||||||
|
running below its negotiated max link speed/width gets `Status = "Link Degraded"` and a
|
||||||
|
matching Warning event (`BuildPCIeLinkDegradationEvents`), independent of GPU-fault SEL/IDL
|
||||||
|
parsing in `gpu_status.go`.
|
||||||
|
- `log/sel.csv` is textually identical to `selelist.csv` (`ipmitool sel elist -c -Z` output with
|
||||||
|
the same 6-column CSV body) and is now a fallback source when `selelist.csv` is absent, reusing
|
||||||
|
`ParseSELListWithLocation` unchanged.
|
||||||
|
- `log/bmc/commer-comp/<commerslot|commerhmc|commerswvr|commerswcpld>/` component logs (plus
|
||||||
|
rotated `*.tar.gz.N` parts, decoded in-memory) share one line format
|
||||||
|
(`[timestamp][file, line][level] message`); `ParseCommerCompEvents` in `commercomp.go` emits an
|
||||||
|
event per line that looks like a real failure, after dropping a fixed noise list (`Invalid Comp
|
||||||
|
Data`, `Error in update redis`, `MutexTimeout = 0`).
|
||||||
|
- Fixed two long-standing `ParseFRU` bugs surfaced by this archive: `Board Serial`/`Board Part
|
||||||
|
Number` values were overwritten by a later, placeholder `Product Serial : 0` / `Product Part
|
||||||
|
Number : NULL` line within the same FRU block (e.g. `SCM_FRU`, which has no product-level FRU
|
||||||
|
fields). Placeholder values (`"0"`, `"NULL"`) no longer overwrite an already-set real value.
|
||||||
|
- Since duplicate SEL entries can now surface through more than one source file (`sel.csv` +
|
||||||
|
IDL/BMC logs) for the same moment, `Parse()` runs a final `dedupSELEvents` pass collapsing
|
||||||
|
`Source == "SEL"` events with identical `(timestamp, event_type, description)`. This is
|
||||||
|
deliberately narrower than IDL's own timestamp-inclusive dedup in `idl.go`, which must keep
|
||||||
|
distinct occurrences of a recurring alarm.
|
||||||
|
- When `Parse()` still ends up with `len(FRU) == 0 || len(Sensors) == 0` after all fallbacks, it
|
||||||
|
appends a `CollectionError{Section: "inventory"}` so the gap is visible in the API response
|
||||||
|
instead of silently returning an empty inventory.
|
||||||
|
**Consequences:**
|
||||||
|
- `stats.fru`/`stats.sensors` are non-zero on this dump class; GPU presence/link state is
|
||||||
|
available from a source independent of SEL/IDL alarm parsing.
|
||||||
|
- BIOS-change-settings context (`biosChangedSettings/Bios_change_settings*.json`) and BIOS POST
|
||||||
|
codes (`log/server/progress_log/host0/*`) from the same issue are not yet parsed — deferred,
|
||||||
|
no acceptance case depended on them yet.
|
||||||
|
- Any future ipmitool-text-output fallback (FRU/sensor/sdr) should extend
|
||||||
|
`component_fallback.go` rather than adding another one-off parser.
|
||||||
|
|||||||
+169
@@ -0,0 +1,169 @@
|
|||||||
|
// Parses log/bmc/commer-comp/*, per-component BMC transcripts (slot presence,
|
||||||
|
// HGX management controller, switch VR/power, switch CPLD). Each rotated log
|
||||||
|
// (component.log plus component.tar.gz.0..9) uses the same
|
||||||
|
// "[timestamp][file, line][level] message" line format.
|
||||||
|
package inspur
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/tar"
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"compress/gzip"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.mchus.pro/mchus/logpile/internal/models"
|
||||||
|
"git.mchus.pro/mchus/logpile/internal/parser"
|
||||||
|
)
|
||||||
|
|
||||||
|
var commerCompComponents = []string{"commerslot", "commerhmc", "commerswvr", "commerswcpld"}
|
||||||
|
|
||||||
|
var commerCompLineRegex = regexp.MustCompile(`^\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\]\[[^\]]*\]\[(\w+)\]\s*(.+)$`)
|
||||||
|
|
||||||
|
var commerCompNoisePatterns = []*regexp.Regexp{
|
||||||
|
regexp.MustCompile(`(?i)invalid comp data`),
|
||||||
|
regexp.MustCompile(`(?i)error in update redis`),
|
||||||
|
regexp.MustCompile(`(?i)mutextimeout\s*=\s*0`),
|
||||||
|
}
|
||||||
|
|
||||||
|
var commerCompFailureRegex = regexp.MustCompile(`(?i)\b(fail|failed|failure|error)\b`)
|
||||||
|
|
||||||
|
// ParseCommerCompEvents extracts BMC component failure events from
|
||||||
|
// log/bmc/commer-comp/<component>/ logs, including rotated *.tar.gz.N parts.
|
||||||
|
func ParseCommerCompEvents(files []parser.ExtractedFile, location *time.Location) []models.Event {
|
||||||
|
var events []models.Event
|
||||||
|
for _, f := range files {
|
||||||
|
path := strings.ToLower(f.Path)
|
||||||
|
if !strings.Contains(path, "/commer-comp/") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
component := commerCompComponentFromPath(path)
|
||||||
|
if component == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.Contains(path, ".tar.gz") {
|
||||||
|
for _, part := range decodeNestedTarGz(f.Content) {
|
||||||
|
events = append(events, parseCommerCompLog(part, component, location)...)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !strings.HasSuffix(path, ".log") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
events = append(events, parseCommerCompLog(f.Content, component, location)...)
|
||||||
|
}
|
||||||
|
return events
|
||||||
|
}
|
||||||
|
|
||||||
|
func commerCompComponentFromPath(path string) string {
|
||||||
|
for _, c := range commerCompComponents {
|
||||||
|
if strings.Contains(path, "/"+c+"/") {
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// decodeNestedTarGz unpacks a rotated log part (a full tar.gz archive stored
|
||||||
|
// as e.g. commerhmc.tar.gz.0) and returns the contents of every regular file
|
||||||
|
// inside it.
|
||||||
|
func decodeNestedTarGz(content []byte) [][]byte {
|
||||||
|
gz, err := gzip.NewReader(bytes.NewReader(content))
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
defer gz.Close()
|
||||||
|
|
||||||
|
tr := tar.NewReader(gz)
|
||||||
|
var out [][]byte
|
||||||
|
for {
|
||||||
|
hdr, err := tr.Next()
|
||||||
|
if err == io.EOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if hdr.Typeflag != tar.TypeReg {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
buf, err := io.ReadAll(tr)
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, buf)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseCommerCompLog(content []byte, component string, location *time.Location) []models.Event {
|
||||||
|
var events []models.Event
|
||||||
|
scanner := bufio.NewScanner(bytes.NewReader(content))
|
||||||
|
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
||||||
|
|
||||||
|
for scanner.Scan() {
|
||||||
|
line := scanner.Text()
|
||||||
|
m := commerCompLineRegex.FindStringSubmatch(line)
|
||||||
|
if m == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
msg := strings.TrimSpace(m[3])
|
||||||
|
if isCommerCompNoise(msg) || !commerCompFailureRegex.MatchString(msg) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
ts, err := time.ParseInLocation("2006-01-02 15:04:05", m[1], location)
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
events = append(events, models.Event{
|
||||||
|
ID: fmt.Sprintf("commer_comp_%s_%d_%s", component, ts.Unix(), commerCompIDSuffix(msg)),
|
||||||
|
Timestamp: ts,
|
||||||
|
Source: fmt.Sprintf("BMC/%s", component),
|
||||||
|
SensorType: "bmc_component",
|
||||||
|
EventType: m[2],
|
||||||
|
Severity: commerCompSeverity(m[2]),
|
||||||
|
Description: msg,
|
||||||
|
RawData: line,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return events
|
||||||
|
}
|
||||||
|
|
||||||
|
func isCommerCompNoise(msg string) bool {
|
||||||
|
for _, re := range commerCompNoisePatterns {
|
||||||
|
if re.MatchString(msg) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
var commerCompNonAlnumRegex = regexp.MustCompile(`[^a-zA-Z0-9]+`)
|
||||||
|
|
||||||
|
// commerCompIDSuffix derives a short, stable suffix from the message so that
|
||||||
|
// two different failures logged in the same second get distinct event IDs.
|
||||||
|
func commerCompIDSuffix(msg string) string {
|
||||||
|
s := commerCompNonAlnumRegex.ReplaceAllString(strings.ToLower(msg), "_")
|
||||||
|
s = strings.Trim(s, "_")
|
||||||
|
if len(s) > 40 {
|
||||||
|
s = s[:40]
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
func commerCompSeverity(level string) models.Severity {
|
||||||
|
switch strings.ToLower(level) {
|
||||||
|
case "critial", "critical", "error":
|
||||||
|
return models.SeverityCritical
|
||||||
|
case "warning", "warn":
|
||||||
|
return models.SeverityWarning
|
||||||
|
default:
|
||||||
|
return models.SeverityInfo
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
package inspur
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.mchus.pro/mchus/logpile/internal/parser"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestParseCommerCompEvents(t *testing.T) {
|
||||||
|
content := []byte(`[2026-07-08 16:46:19][commer_comp_anal.c, 635][Critial] [COMP]Invalid Comp Data, Continue... ParamList: Index-3, Valid-0x1, MediaType-0(1-PCIE,2-I2C)
|
||||||
|
[2026-07-27 04:52:55][commer_comp_slot.c, 319][Critial] Data Update fail for dev serial-6!
|
||||||
|
[2026-07-27 04:52:55][commer_comp_slot.c, 319][Critial] Data Update fail for dev serial-7!
|
||||||
|
[2026-07-27 05:00:00][commer_comp_slot.c, 100][Info] Error in update redis, retry
|
||||||
|
`)
|
||||||
|
|
||||||
|
files := []parser.ExtractedFile{
|
||||||
|
{Path: "log/bmc/commer-comp/commerslot/commerslot.log", Content: content},
|
||||||
|
}
|
||||||
|
|
||||||
|
events := ParseCommerCompEvents(files, time.UTC)
|
||||||
|
if len(events) != 2 {
|
||||||
|
t.Fatalf("expected 2 events (noise filtered), got %d: %+v", len(events), events)
|
||||||
|
}
|
||||||
|
for _, e := range events {
|
||||||
|
if e.Source != "BMC/commerslot" {
|
||||||
|
t.Fatalf("unexpected source %q", e.Source)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if events[0].Timestamp.Format("2006-01-02 15:04:05") != "2026-07-27 04:52:55" {
|
||||||
|
t.Fatalf("unexpected timestamp: %v", events[0].Timestamp)
|
||||||
|
}
|
||||||
|
if events[0].ID == events[1].ID {
|
||||||
|
t.Fatalf("expected distinct event IDs for serial-6/serial-7, got same: %q", events[0].ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
// Fallback inventory parsing for the per-file dump_<serial>_<timestamp>/ onekeylog
|
||||||
|
// layout, which has no combined devicefrusdr.log. Instead FRU and sensors live in
|
||||||
|
// separate ipmitool-output text files under component/.
|
||||||
|
package inspur
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"git.mchus.pro/mchus/logpile/internal/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ParseSensorList parses component/sensor.txt ("ipmitool sensor list" output).
|
||||||
|
// Line format: Name | Value | Unit | Status | lnr | lcr | lnc | unc | ucr | unr
|
||||||
|
func ParseSensorList(content []byte) []models.SensorReading {
|
||||||
|
var readings []models.SensorReading
|
||||||
|
|
||||||
|
scanner := bufio.NewScanner(strings.NewReader(string(content)))
|
||||||
|
for scanner.Scan() {
|
||||||
|
line := strings.TrimSpace(scanner.Text())
|
||||||
|
if line == "" || !strings.Contains(line, "|") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fields := strings.Split(line, "|")
|
||||||
|
if len(fields) < 4 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
name := strings.TrimSpace(fields[0])
|
||||||
|
valueStr := strings.TrimSpace(fields[1])
|
||||||
|
unit := strings.TrimSpace(fields[2])
|
||||||
|
status := strings.TrimSpace(fields[3])
|
||||||
|
if name == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
reading := models.SensorReading{
|
||||||
|
Name: name,
|
||||||
|
Status: status,
|
||||||
|
Unit: unit,
|
||||||
|
Type: determineSensorType(name),
|
||||||
|
}
|
||||||
|
|
||||||
|
if valueStr != "" && valueStr != "na" {
|
||||||
|
if v, err := strconv.ParseFloat(valueStr, 64); err == nil {
|
||||||
|
reading.Value = v
|
||||||
|
reading.RawValue = valueStr + " " + unit
|
||||||
|
} else {
|
||||||
|
reading.RawValue = valueStr
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
readings = append(readings, reading)
|
||||||
|
}
|
||||||
|
|
||||||
|
return readings
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseSDRElist parses component/sdr.txt ("ipmitool sdr elist" output), used as a
|
||||||
|
// fallback when component/sensor.txt is unavailable.
|
||||||
|
// Line format: Name | ID | Status | Entity | Reading (e.g. "21 degrees C" or "no reading")
|
||||||
|
func ParseSDRElist(content []byte) []models.SensorReading {
|
||||||
|
var readings []models.SensorReading
|
||||||
|
|
||||||
|
scanner := bufio.NewScanner(strings.NewReader(string(content)))
|
||||||
|
for scanner.Scan() {
|
||||||
|
line := strings.TrimSpace(scanner.Text())
|
||||||
|
if line == "" || !strings.Contains(line, "|") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fields := strings.Split(line, "|")
|
||||||
|
if len(fields) < 3 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
name := strings.TrimSpace(fields[0])
|
||||||
|
status := strings.TrimSpace(fields[2])
|
||||||
|
if name == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
reading := models.SensorReading{
|
||||||
|
Name: name,
|
||||||
|
Status: status,
|
||||||
|
Type: determineSensorType(name),
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(fields) >= 5 {
|
||||||
|
readingStr := strings.TrimSpace(fields[4])
|
||||||
|
if readingStr != "" && readingStr != "no reading" {
|
||||||
|
reading.RawValue = readingStr
|
||||||
|
if m := valueRegex.FindStringSubmatch(readingStr); m != nil {
|
||||||
|
if v, err := strconv.ParseFloat(m[1], 64); err == nil {
|
||||||
|
reading.Value = v
|
||||||
|
reading.Unit = strings.TrimSpace(m[2])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
readings = append(readings, reading)
|
||||||
|
}
|
||||||
|
|
||||||
|
return readings
|
||||||
|
}
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
package inspur
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestParseSensorList(t *testing.T) {
|
||||||
|
content := []byte(`============
|
||||||
|
Description:sensor list
|
||||||
|
Command: ipmitool sensor list
|
||||||
|
Response:
|
||||||
|
Inlet_Temp | 21.000 | degrees C | ok | na | na | na | na | 40.000 | na
|
||||||
|
GPU0_Temp | 29.000 | degrees C | ok | na | na | na | na | 86.000 | na
|
||||||
|
PSU0_Status | 0x0 | discrete | 0x0180| na | na | na | na | na | na
|
||||||
|
`)
|
||||||
|
|
||||||
|
readings := ParseSensorList(content)
|
||||||
|
if len(readings) != 3 {
|
||||||
|
t.Fatalf("expected 3 readings, got %d", len(readings))
|
||||||
|
}
|
||||||
|
if readings[0].Name != "Inlet_Temp" || readings[0].Value != 21.0 || readings[0].Unit != "degrees C" {
|
||||||
|
t.Fatalf("unexpected Inlet_Temp reading: %+v", readings[0])
|
||||||
|
}
|
||||||
|
if readings[1].Type != "temperature" {
|
||||||
|
t.Fatalf("expected temperature type, got %q", readings[1].Type)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseSDRElist(t *testing.T) {
|
||||||
|
content := []byte(`============
|
||||||
|
Description:sdr elist
|
||||||
|
Command: ipmitool sdr elist
|
||||||
|
Response:
|
||||||
|
Inlet_Temp | 00h | ok | 3.0 | 21 degrees C
|
||||||
|
GPU0_Temp | 2Ah | ok | 11.0 | 29 degrees C
|
||||||
|
Event Logging Disabled SEL_Status | 15h | ok | 7.0 | no reading
|
||||||
|
`)
|
||||||
|
|
||||||
|
readings := ParseSDRElist(content)
|
||||||
|
if len(readings) != 3 {
|
||||||
|
t.Fatalf("expected 3 readings, got %d", len(readings))
|
||||||
|
}
|
||||||
|
if readings[0].Name != "Inlet_Temp" || readings[0].Value != 21 || readings[0].Unit != "degrees C" {
|
||||||
|
t.Fatalf("unexpected Inlet_Temp reading: %+v", readings[0])
|
||||||
|
}
|
||||||
|
if readings[2].RawValue != "" {
|
||||||
|
t.Fatalf("expected no reading to leave RawValue empty, got %q", readings[2].RawValue)
|
||||||
|
}
|
||||||
|
}
|
||||||
+121
@@ -0,0 +1,121 @@
|
|||||||
|
// Parses log/bmc/diagnose/OtrdDiagnoseComponent.json, an HGX BMC structural
|
||||||
|
// snapshot of PCIe topology (present GPUs/NICs/NVMe, negotiated link state).
|
||||||
|
// This is a direct point-in-time source for GPU presence and PCIe link
|
||||||
|
// degradation, independent of SEL/IDL alarm history.
|
||||||
|
package inspur
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"git.mchus.pro/mchus/logpile/internal/models"
|
||||||
|
"git.mchus.pro/mchus/logpile/internal/parser/vendors/pciids"
|
||||||
|
)
|
||||||
|
|
||||||
|
// otrdDiagnosePCIeEntry mirrors the relevant fields of "Pcie Device Info"
|
||||||
|
// entries in OtrdDiagnoseComponent.json.
|
||||||
|
type otrdDiagnosePCIeEntry struct {
|
||||||
|
LocString string `json:"LocString"`
|
||||||
|
PcieSlot int `json:"PcieSlot"`
|
||||||
|
PresentStatus int `json:"PresentStatus"`
|
||||||
|
VendorId int `json:"VendorId"`
|
||||||
|
DeviceId int `json:"DeviceId"`
|
||||||
|
BusNumber int `json:"BusNumber"`
|
||||||
|
DeviceNumber int `json:"DeviceNumber"`
|
||||||
|
FunctionNumber int `json:"FunctionNumber"`
|
||||||
|
CurrentLinkSpeed int `json:"CurrentLinkSpeed"`
|
||||||
|
MaxLinkSpeed int `json:"MaxLinkSpeed"`
|
||||||
|
NegotiatedLinkWidth int `json:"NegotiatedLinkWidth"`
|
||||||
|
MaxLinkWidth int `json:"MaxLinkWidth"`
|
||||||
|
SerialNumber *string `json:"SerialNumber"`
|
||||||
|
PartNumber *string `json:"PartNumber"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type otrdDiagnoseComponent struct {
|
||||||
|
PcieDeviceInfo []otrdDiagnosePCIeEntry `json:"Pcie Device Info"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseOtrdDiagnosePCIe parses the "Pcie Device Info" array from
|
||||||
|
// OtrdDiagnoseComponent.json into PCIe device inventory entries.
|
||||||
|
func ParseOtrdDiagnosePCIe(content []byte) []models.PCIeDevice {
|
||||||
|
var doc otrdDiagnoseComponent
|
||||||
|
if err := json.Unmarshal(content, &doc); err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
devices := make([]models.PCIeDevice, 0, len(doc.PcieDeviceInfo))
|
||||||
|
for _, e := range doc.PcieDeviceInfo {
|
||||||
|
present := e.PresentStatus == 1
|
||||||
|
_, deviceName := pciids.DeviceInfo(e.VendorId, e.DeviceId)
|
||||||
|
deviceClass := normalizeModelLabel(deviceName)
|
||||||
|
if strings.Contains(strings.ToUpper(deviceName), "NVIDIA") {
|
||||||
|
deviceClass = "GPU (" + deviceName + ")"
|
||||||
|
}
|
||||||
|
|
||||||
|
serial := ""
|
||||||
|
if e.SerialNumber != nil {
|
||||||
|
serial = strings.TrimSpace(*e.SerialNumber)
|
||||||
|
}
|
||||||
|
partNum := ""
|
||||||
|
if e.PartNumber != nil {
|
||||||
|
partNum = strings.TrimSpace(*e.PartNumber)
|
||||||
|
}
|
||||||
|
|
||||||
|
status := ""
|
||||||
|
if present && isLinkDegraded(e) {
|
||||||
|
status = "Link Degraded"
|
||||||
|
}
|
||||||
|
|
||||||
|
devices = append(devices, models.PCIeDevice{
|
||||||
|
Slot: e.LocString,
|
||||||
|
VendorID: e.VendorId,
|
||||||
|
DeviceID: e.DeviceId,
|
||||||
|
BDF: formatBDF(e.BusNumber, e.DeviceNumber, e.FunctionNumber),
|
||||||
|
DeviceClass: deviceClass,
|
||||||
|
Manufacturer: normalizeModelLabel(pciids.VendorName(e.VendorId)),
|
||||||
|
LinkWidth: e.NegotiatedLinkWidth,
|
||||||
|
LinkSpeed: fmt.Sprintf("GEN%d", e.CurrentLinkSpeed),
|
||||||
|
MaxLinkWidth: e.MaxLinkWidth,
|
||||||
|
MaxLinkSpeed: fmt.Sprintf("GEN%d", e.MaxLinkSpeed),
|
||||||
|
PartNumber: partNum,
|
||||||
|
SerialNumber: serial,
|
||||||
|
Present: &present,
|
||||||
|
Status: status,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
return devices
|
||||||
|
}
|
||||||
|
|
||||||
|
func isLinkDegraded(e otrdDiagnosePCIeEntry) bool {
|
||||||
|
if e.MaxLinkSpeed > 0 && e.CurrentLinkSpeed < e.MaxLinkSpeed {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if e.MaxLinkWidth > 0 && e.NegotiatedLinkWidth < e.MaxLinkWidth {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// BuildPCIeLinkDegradationEvents emits a Warning event for every PCIe device
|
||||||
|
// present in OtrdDiagnoseComponent.json but running below its negotiated max
|
||||||
|
// link speed/width (e.g. a GPU baseboard degrading from Gen5 x16 to a lower
|
||||||
|
// state), so the log viewer surfaces the same signal the IDL alarms would.
|
||||||
|
func BuildPCIeLinkDegradationEvents(devices []models.PCIeDevice) []models.Event {
|
||||||
|
var events []models.Event
|
||||||
|
for _, d := range devices {
|
||||||
|
if d.Present == nil || !*d.Present || d.Status != "Link Degraded" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
events = append(events, models.Event{
|
||||||
|
ID: fmt.Sprintf("pcie_link_degraded_%s", strings.TrimPrefix(d.Slot, "#")),
|
||||||
|
Source: "PCIe Diagnose",
|
||||||
|
SensorType: "pcie_link",
|
||||||
|
EventType: "Link Degraded",
|
||||||
|
Severity: models.SeverityWarning,
|
||||||
|
Description: fmt.Sprintf("%s: link degraded to %s x%d (max %s x%d)", d.Slot, d.LinkSpeed, d.LinkWidth, d.MaxLinkSpeed, d.MaxLinkWidth),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return events
|
||||||
|
}
|
||||||
+70
@@ -0,0 +1,70 @@
|
|||||||
|
package inspur
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
const diagnoseJSONFixture = `{
|
||||||
|
"Pcie Device Info": [
|
||||||
|
{
|
||||||
|
"LocString": "#GPU0",
|
||||||
|
"PcieSlot": 100,
|
||||||
|
"PresentStatus": 1,
|
||||||
|
"VendorId": 4318,
|
||||||
|
"DeviceId": 10497,
|
||||||
|
"BusNumber": 23,
|
||||||
|
"DeviceNumber": 0,
|
||||||
|
"FunctionNumber": 0,
|
||||||
|
"CurrentLinkSpeed": 5,
|
||||||
|
"MaxLinkSpeed": 5,
|
||||||
|
"NegotiatedLinkWidth": 16,
|
||||||
|
"MaxLinkWidth": 16
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"LocString": "#GPU6",
|
||||||
|
"PcieSlot": 106,
|
||||||
|
"PresentStatus": 1,
|
||||||
|
"VendorId": 4318,
|
||||||
|
"DeviceId": 10497,
|
||||||
|
"BusNumber": 220,
|
||||||
|
"DeviceNumber": 0,
|
||||||
|
"FunctionNumber": 0,
|
||||||
|
"CurrentLinkSpeed": 2,
|
||||||
|
"MaxLinkSpeed": 5,
|
||||||
|
"NegotiatedLinkWidth": 8,
|
||||||
|
"MaxLinkWidth": 16
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}`
|
||||||
|
|
||||||
|
func TestParseOtrdDiagnosePCIe(t *testing.T) {
|
||||||
|
devices := ParseOtrdDiagnosePCIe([]byte(diagnoseJSONFixture))
|
||||||
|
if len(devices) != 2 {
|
||||||
|
t.Fatalf("expected 2 devices, got %d", len(devices))
|
||||||
|
}
|
||||||
|
|
||||||
|
gpu0 := devices[0]
|
||||||
|
if gpu0.Slot != "#GPU0" || gpu0.Present == nil || !*gpu0.Present {
|
||||||
|
t.Fatalf("expected GPU0 present, got %+v", gpu0)
|
||||||
|
}
|
||||||
|
if gpu0.Status == "Link Degraded" {
|
||||||
|
t.Fatalf("GPU0 should not be flagged as degraded: %+v", gpu0)
|
||||||
|
}
|
||||||
|
if gpu0.LinkSpeed != "GEN5" || gpu0.MaxLinkSpeed != "GEN5" || gpu0.LinkWidth != 16 {
|
||||||
|
t.Fatalf("unexpected GPU0 link info: %+v", gpu0)
|
||||||
|
}
|
||||||
|
|
||||||
|
gpu6 := devices[1]
|
||||||
|
if gpu6.Status != "Link Degraded" {
|
||||||
|
t.Fatalf("expected GPU6 to be flagged as degraded, got %+v", gpu6)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildPCIeLinkDegradationEvents(t *testing.T) {
|
||||||
|
devices := ParseOtrdDiagnosePCIe([]byte(diagnoseJSONFixture))
|
||||||
|
events := BuildPCIeLinkDegradationEvents(devices)
|
||||||
|
if len(events) != 1 {
|
||||||
|
t.Fatalf("expected 1 degradation event, got %d", len(events))
|
||||||
|
}
|
||||||
|
if events[0].Severity != "warning" {
|
||||||
|
t.Fatalf("expected warning severity, got %q", events[0].Severity)
|
||||||
|
}
|
||||||
|
}
|
||||||
+8
-1
@@ -72,9 +72,16 @@ func ParseFRU(content []byte) []models.FRUInfo {
|
|||||||
current.ProductName = fieldValue
|
current.ProductName = fieldValue
|
||||||
}
|
}
|
||||||
case "Board Serial", "Product Serial":
|
case "Board Serial", "Product Serial":
|
||||||
|
// Modules with no product-level FRU (e.g. SCM_FRU) report
|
||||||
|
// "Product Serial : 0" after a real "Board Serial" value;
|
||||||
|
// don't let the placeholder overwrite the real serial.
|
||||||
|
if fieldValue != "0" && fieldValue != "NULL" {
|
||||||
current.SerialNumber = fieldValue
|
current.SerialNumber = fieldValue
|
||||||
|
}
|
||||||
case "Board Part Number", "Product Part Number":
|
case "Board Part Number", "Product Part Number":
|
||||||
if fieldValue != "0" {
|
// See "Board Serial" above: placeholder "NULL"/"0" product-level
|
||||||
|
// fields must not overwrite a real board-level part number.
|
||||||
|
if fieldValue != "0" && fieldValue != "NULL" {
|
||||||
current.PartNumber = fieldValue
|
current.PartNumber = fieldValue
|
||||||
}
|
}
|
||||||
case "Product Version":
|
case "Product Version":
|
||||||
|
|||||||
+48
-2
@@ -16,7 +16,7 @@ import (
|
|||||||
|
|
||||||
// parserVersion - version of this parser module
|
// parserVersion - version of this parser module
|
||||||
// IMPORTANT: Increment this version when making changes to parser logic!
|
// IMPORTANT: Increment this version when making changes to parser logic!
|
||||||
const parserVersion = "2.2"
|
const parserVersion = "2.3"
|
||||||
|
|
||||||
func init() {
|
func init() {
|
||||||
parser.Register(&Parser{})
|
parser.Register(&Parser{})
|
||||||
@@ -203,6 +203,26 @@ func (p *Parser) Parse(files []parser.ExtractedFile) (*models.AnalysisResult, er
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// New dump_<serial>_<timestamp>/ layout has no devicefrusdr.log: FRU and
|
||||||
|
// sensors live in separate ipmitool-output text files under component/.
|
||||||
|
if parser.FindFileByName(files, "devicefrusdr.log") == nil {
|
||||||
|
if f := parser.FindFileByName(files, "fru.txt"); f != nil && len(result.FRU) == 0 {
|
||||||
|
result.FRU = ParseFRU(f.Content)
|
||||||
|
extractBoardInfo(result.FRU, result.Hardware)
|
||||||
|
}
|
||||||
|
if f := parser.FindFileByName(files, "sensor.txt"); f != nil && len(result.Sensors) == 0 {
|
||||||
|
result.Sensors = mergeSensorReadings(result.Sensors, ParseSensorList(f.Content))
|
||||||
|
} else if f := parser.FindFileByName(files, "sdr.txt"); f != nil && len(result.Sensors) == 0 {
|
||||||
|
result.Sensors = mergeSensorReadings(result.Sensors, ParseSDRElist(f.Content))
|
||||||
|
}
|
||||||
|
// Structural PCIe snapshot (GPU presence/link state) from the HGX diagnose dump.
|
||||||
|
if f := parser.FindFileByName(files, "OtrdDiagnoseComponent.json"); f != nil && result.Hardware != nil {
|
||||||
|
diagPCIe := ParseOtrdDiagnosePCIe(f.Content)
|
||||||
|
result.Hardware.PCIeDevices = MergePCIeDevices(result.Hardware.PCIeDevices, diagPCIe)
|
||||||
|
result.Events = append(result.Events, BuildPCIeLinkDegradationEvents(diagPCIe)...)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Enrich runtime component data from Redis snapshot (serials, FW, telemetry),
|
// Enrich runtime component data from Redis snapshot (serials, FW, telemetry),
|
||||||
// when text logs miss these fields.
|
// when text logs miss these fields.
|
||||||
if f := parser.FindFileByName(files, "redis-dump.rdb"); f != nil && result.Hardware != nil {
|
if f := parser.FindFileByName(files, "redis-dump.rdb"); f != nil && result.Hardware != nil {
|
||||||
@@ -216,10 +236,14 @@ func (p *Parser) Parse(files []parser.ExtractedFile) (*models.AnalysisResult, er
|
|||||||
result.Events = append(result.Events, idlEvents...)
|
result.Events = append(result.Events, idlEvents...)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Parse SEL list (selelist.csv)
|
// Parse SEL list (selelist.csv), falling back to log/sel.csv on the
|
||||||
|
// dump_<serial>_<timestamp>/ layout that has no selelist.csv.
|
||||||
if f := parser.FindFileByName(files, "selelist.csv"); f != nil {
|
if f := parser.FindFileByName(files, "selelist.csv"); f != nil {
|
||||||
selEvents := ParseSELListWithLocation(f.Content, selLocation)
|
selEvents := ParseSELListWithLocation(f.Content, selLocation)
|
||||||
result.Events = append(result.Events, selEvents...)
|
result.Events = append(result.Events, selEvents...)
|
||||||
|
} else if f := parser.FindFileByName(files, "sel.csv"); f != nil {
|
||||||
|
selEvents := ParseSELListWithLocation(f.Content, selLocation)
|
||||||
|
result.Events = append(result.Events, selEvents...)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Parse syslog files
|
// Parse syslog files
|
||||||
@@ -229,6 +253,14 @@ func (p *Parser) Parse(files []parser.ExtractedFile) (*models.AnalysisResult, er
|
|||||||
result.Events = append(result.Events, events...)
|
result.Events = append(result.Events, events...)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Parse BMC component failure logs (log/bmc/commer-comp/*): HGX management
|
||||||
|
// controller, switch VR/power, switch CPLD and slot presence failures.
|
||||||
|
result.Events = append(result.Events, ParseCommerCompEvents(files, selLocation)...)
|
||||||
|
|
||||||
|
// Same SEL event can be reported twice by the BMC (e.g. once via sel.csv,
|
||||||
|
// once via idl.log); collapse exact duplicates so they don't double-count.
|
||||||
|
result.Events = dedupSELEvents(result.Events)
|
||||||
|
|
||||||
// Fallback for archives where board serial is missing in parsed FRU/asset data:
|
// Fallback for archives where board serial is missing in parsed FRU/asset data:
|
||||||
// recover it from log content, never from archive filename.
|
// recover it from log content, never from archive filename.
|
||||||
if strings.TrimSpace(result.Hardware.BoardInfo.SerialNumber) == "" {
|
if strings.TrimSpace(result.Hardware.BoardInfo.SerialNumber) == "" {
|
||||||
@@ -261,6 +293,20 @@ func (p *Parser) Parse(files []parser.ExtractedFile) (*models.AnalysisResult, er
|
|||||||
parser.ApplyManufacturedYearWeekFromFRU(result.FRU, result.Hardware)
|
parser.ApplyManufacturedYearWeekFromFRU(result.FRU, result.Hardware)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if len(result.FRU) == 0 || len(result.Sensors) == 0 {
|
||||||
|
var missing []string
|
||||||
|
if len(result.FRU) == 0 {
|
||||||
|
missing = append(missing, "FRU")
|
||||||
|
}
|
||||||
|
if len(result.Sensors) == 0 {
|
||||||
|
missing = append(missing, "sensors")
|
||||||
|
}
|
||||||
|
result.CollectionErrors = append(result.CollectionErrors, models.CollectionError{
|
||||||
|
Section: "inventory",
|
||||||
|
Message: fmt.Sprintf("inventory sources not found: devicefrusdr.log absent, fell back to component/fru.txt+sensor.txt/sdr.txt, still missing: %s", strings.Join(missing, ", ")),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+6
@@ -314,6 +314,12 @@ func enrichPCIeDevice(dst *models.PCIeDevice, src models.PCIeDevice) {
|
|||||||
if isGenericPCIeClass(dst.DeviceClass) && !isGenericPCIeClass(src.DeviceClass) {
|
if isGenericPCIeClass(dst.DeviceClass) && !isGenericPCIeClass(src.DeviceClass) {
|
||||||
dst.DeviceClass = src.DeviceClass
|
dst.DeviceClass = src.DeviceClass
|
||||||
}
|
}
|
||||||
|
if dst.Present == nil {
|
||||||
|
dst.Present = src.Present
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(dst.Status) == "" {
|
||||||
|
dst.Status = src.Status
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizePCIeBDF(bdf string) string {
|
func normalizePCIeBDF(bdf string) string {
|
||||||
|
|||||||
+26
@@ -176,6 +176,32 @@ func determineSELSeverity(sensorStr, eventDesc, status string) models.Severity {
|
|||||||
return models.SeverityInfo
|
return models.SeverityInfo
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// dedupSELEvents collapses events reported more than once with the same
|
||||||
|
// (timestamp, event_type, description) triple. Unlike ParseIDLLog's
|
||||||
|
// dedup (which must keep recurring alarms with distinct timestamps), this
|
||||||
|
// only removes true duplicates: the same SEL entry surfacing through more
|
||||||
|
// than one source file for the exact same moment.
|
||||||
|
func dedupSELEvents(events []models.Event) []models.Event {
|
||||||
|
if len(events) == 0 {
|
||||||
|
return events
|
||||||
|
}
|
||||||
|
seen := make(map[string]struct{}, len(events))
|
||||||
|
out := make([]models.Event, 0, len(events))
|
||||||
|
for _, e := range events {
|
||||||
|
if e.Source != "SEL" {
|
||||||
|
out = append(out, e)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
key := e.Timestamp.String() + "|" + e.EventType + "|" + e.Description
|
||||||
|
if _, ok := seen[key]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[key] = struct{}{}
|
||||||
|
out = append(out, e)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
// buildSELDescription builds human-readable description
|
// buildSELDescription builds human-readable description
|
||||||
func buildSELDescription(eventDesc, status string) string {
|
func buildSELDescription(eventDesc, status string) string {
|
||||||
if status == "Asserted" || status == "Deasserted" {
|
if status == "Asserted" || status == "Deasserted" {
|
||||||
|
|||||||
+17
@@ -3,8 +3,25 @@ package inspur
|
|||||||
import (
|
import (
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"git.mchus.pro/mchus/logpile/internal/models"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func TestDedupSELEvents(t *testing.T) {
|
||||||
|
ts := time.Date(2026, 7, 27, 4, 52, 55, 0, time.UTC)
|
||||||
|
events := []models.Event{
|
||||||
|
{Source: "SEL", Timestamp: ts, EventType: "Asserted", Description: "PSU0 fail"},
|
||||||
|
{Source: "SEL", Timestamp: ts, EventType: "Asserted", Description: "PSU0 fail"},
|
||||||
|
{Source: "BMC", Timestamp: ts, EventType: "Assert", Description: "recurring alarm"},
|
||||||
|
{Source: "BMC", Timestamp: ts, EventType: "Assert", Description: "recurring alarm"},
|
||||||
|
}
|
||||||
|
|
||||||
|
out := dedupSELEvents(events)
|
||||||
|
if len(out) != 3 {
|
||||||
|
t.Fatalf("expected 3 events (1 SEL dup removed, BMC untouched), got %d: %+v", len(out), out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestParseSELListWithLocation_UsesProvidedTimezone(t *testing.T) {
|
func TestParseSELListWithLocation_UsesProvidedTimezone(t *testing.T) {
|
||||||
content := []byte("sel elist:\n1,02/28/2026,04:18:18,Sensor X,Event,Asserted\n")
|
content := []byte("sel elist:\n1,02/28/2026,04:18:18,Sensor X,Event,Asserted\n")
|
||||||
shanghai, err := time.LoadLocation("Asia/Shanghai")
|
shanghai, err := time.LoadLocation("Asia/Shanghai")
|
||||||
|
|||||||
Reference in New Issue
Block a user