diff --git a/bible-local/06-parsers.md b/bible-local/06-parsers.md index 69fa946..fd896a2 100644 --- a/bible-local/06-parsers.md +++ b/bible-local/06-parsers.md @@ -45,6 +45,12 @@ If a source embeds ` - XX:XX:XX:XX:XX:XX` in a model/name field, remove that suf When `vendor_id` and `device_id` are known but the model name is missing or generic, resolve the name via `internal/parser/vendors/pciids`. +### Preserve source-backed CPU identity + +Keep a source PPIN in `CPU.PPIN`. If the source has no separate processor serial number, use +`models.ResolveCPUSerialNumber` to expose the valid PPIN as `CPU.SerialNumber`. Never derive a CPU +serial from its socket, model, board serial, or another component; reject source placeholders. + ## Active vendor coverage | Vendor ID | Input family | Notes | @@ -54,10 +60,10 @@ When `vendor_id` and `device_id` are known but the model name is missing or gene | `h3c_g5` | H3C SDS G5 bundles | INI/XML/CSV-driven hardware and event parsing | | `h3c_g6` | H3C SDS G6 bundles | Similar flow with G6-specific files | | `hpe_ilo_ahs` | HPE iLO Active Health System (`.ahs`) | Proprietary `ABJR` container with gzip-compressed `zbb` members; parser combines SMBIOS-style inventory strings and embedded Redfish storage JSON | -| `inspur` | onekeylog archives (classic `component.log` and per-file `component/*.txt` D-Bus layouts) | FRU/SDR plus optional Redis enrichment | +| `inspur` | onekeylog archives (classic `component.log` and per-file `component/*.txt` D-Bus layouts) | FRU/SDR, optional Redis enrichment, normalized IDL/SEL/syslog events (see ADL-056) | | `lenovo_xcc` | Lenovo XCC mini-log ZIP archives | JSON inventory + platform event logs | | `nvidia` | HGX Field Diagnostics | GPU- and fabric-heavy diagnostic input | -| `nvidia_bug_report` | `nvidia-bug-report-*.log.gz` | dmidecode, lspci, NVIDIA driver sections | +| `nvidia_bug_report` | `nvidia-bug-report-*.log.gz` | dmidecode, lspci, NVIDIA driver sections; Xid/SXid GPU error events (see ADL-055, `bible-local/docs/nvidia-bug-report-analysis.md`) | | `redfish_walk` | Any archive carrying a captured Redfish directory-tree walk (tar.gz/zip of `/index.json` docs) | Low-confidence (35) vendor-agnostic fallback; see `dell` for the first known source and `redfishtree` helper package | | `unraid` | Unraid diagnostics/log bundles | Server and storage-focused parsing | | `xfusion` | xFusion iBMC `tar.gz` dump / file export | AppDump + RTOSDump + LogDump merge for hardware and firmware | @@ -87,7 +93,7 @@ with content markers (e.g. `Unraid kernel build`, parity data markers). ### H3C SDS G5 (`h3c_g5`) -**Status:** Ready (v1.0.0). Tested on H3C UniServer R4900 G5 SDS archives. +**Status:** Ready (v2.2). Tested on H3C UniServer R4900 G5 SDS archives. **Archive format:** `.sds` (tar archive) @@ -107,7 +113,7 @@ with content markers (e.g. `Unraid kernel build`, parity data markers). ### H3C SDS G6 (`h3c_g6`) -**Status:** Ready (v1.0.0). Tested on H3C UniServer R4700 G6 SDS archives. +**Status:** Ready (v2.2). Tested on H3C UniServer R4700 G6 SDS archives. **Archive format:** `.sds` (tar archive) @@ -219,7 +225,7 @@ and `LogDump/` trees. ### Dell TSR (`dell`) -**Status:** Ready (v3.0). Two inventory source generations, both handled by the same parser. +**Status:** Ready (v3.1). Two inventory source generations, both handled by the same parser. **Archive format:** TSR nested zip (`signature` + `TSR*.pl.zip`). diff --git a/bible-local/07-exporters.md b/bible-local/07-exporters.md index 00c2520..4e73346 100644 --- a/bible-local/07-exporters.md +++ b/bible-local/07-exporters.md @@ -92,3 +92,21 @@ Behavior: `GET /api/export/csv` uses the same merged canonical inventory as Reanimator export, with legacy network-card fallback kept only for records that still have no canonical device match. + +## CPU identity audit CLI + +`go run ./cmd/logpile-cpu-audit -input -output ` recursively parses +supported dump files and writes one `*.cpu-audit.json` report per input plus `_summary.json`. +Reports contain only parser provenance and CPU identity fields needed to audit the PPIN-to-serial +fallback. Parse failures are isolated per input and recorded in both the file report and summary. + +Directory scans exclude standalone `.txt` and `.log` files by default so unpacked dump contents are +not treated as separate dumps. Use `-include-plain` when the input tree intentionally contains whole +single-file diagnostic logs. `-workers` controls bounded concurrent parsing and defaults to at most +four workers. + +For direct Reanimator ingest, add `-reanimator -prefix `. This mode writes a flat set of +`*.reanimator.json` files and skips inputs without CPUs or without the required board identity. Each +payload keeps the required top-level `collected_at`, `filename`, and `hardware.board`, while +`hardware` contains no component section except `cpus`. No audit/summary JSON is written into that +directory, so every JSON file there is an ingest payload. diff --git a/bible-local/10-decisions.md b/bible-local/10-decisions.md index 6d519ef..702a82b 100644 --- a/bible-local/10-decisions.md +++ b/bible-local/10-decisions.md @@ -1487,3 +1487,183 @@ strings for the same physical device into one registry entry, same as it already `TestParseAssetJSON_HddEnrichedWithPcieVendorDeviceID` (inspur), and `TestConvertToReanimator_StorageVendorDeviceIDSurvivesRoundTrip` (exporter, full export→marshal→ reimport→re-export round trip). + +--- + +## ADL-054 — Single-file gzip decompression is no longer capped at a fixed byte count + +**Date:** 2026-08-26 +**Context:** A real `nvidia-bug-report-*.log.gz` (HGX B200, 53.25MB decompressed) was silently +truncated by `extractTarGzFromReader`'s non-tar branch, which hard-capped decompressed content at +`maxGzipDecompressedSize` (50MB) via `io.LimitReader`. The tail lost in this specific case was +low-value `sar`-style telemetry, but the truncation was silent-by-default (only surfaced as a +generic warning event) and the fixed cap has no relationship to what a legitimate bug-report bundle +can contain — larger dumps would lose structured sections (nvidia-smi -q, dmesg tail) outright. +**Decision:** Replaced the fixed cap with streaming decompression plus a decompression-*ratio* +guard (`internal/parser/archive.go`): a `countingReader` tracks bytes actually consumed from the +compressed source, and `readGzipWithBombGuard` reads the decompressed stream in 256KB chunks, +aborting only if the ratio exceeds `gzipBombRatio` (300:1, evaluated only after +`gzipBombMinCompressed` = 4KB of compressed input to avoid false positives from gzip header +overhead) or an absolute `gzipAbsoluteCeiling` (1GB, matching the existing `maxSingleFileSizeLarge` +used for large tar-in-gz) is hit. This only touches the plain-single-gzipped-file branch — tar.gz +archives already use the generous 1GB `maxSingleFileSizeLarge` limit and were unaffected. +**Consequences:** +- Legitimate large single-file logs (any size, low compression ratio) now come through whole instead + of being silently cut off; the extraction is still fully buffered into memory afterward (`[]byte`), + matching every other extractor in this file — no `VendorParser` interface change. +- Zip/gzip-bomb protection is now ratio-based instead of absolute-size-based, catching bombs of any + target size while still allowing large legitimate files through. +- Added `TestExtractArchiveFromReaderGZ_NoLongerCapsAt50MB` and + `TestExtractArchiveFromReaderGZ_AbortsOnDecompressionBomb` (`archive_test.go`). + +--- + +## ADL-055 — `nvidia_bug_report` parser now extracts Xid/SXid GPU error events + +**Date:** 2026-08-26 +**Context:** The parser only ever extracted hardware inventory (board/CPU/memory/GPU/NIC/PSU) plus +three fixed informational events (memory config, GPU detection, driver version). All GPU/NVLink +fault triage (Xid/SXid errors, "GPU recovery action changed", "fell off the bus") had to be done by +hand with `grep` against the raw decompressed log — see `bible-local/docs/nvidia-bug-report-analysis.md` +for the manual checklist this was based on, itself informed by NVIDIA's GPU Debug Guidelines and +Lambda Labs' `check-nvidia-bug-report.sh`. +**Decision:** Added `internal/parser/vendors/nvidia_bug_report/errors.go` (`parseXidAndFaultEvents`), +wired into `Parser.Parse`. It scans syslog-style kernel lines for `NVRM: Xid (PCI:...)` / +`NVRM: SXid (PCI:...)` and emits one `models.Event` per occurrence, classified `critical` vs +`warning` via a hardcoded set of hardware-leaning Xid codes (`hardwareLeaningXidCodes`: 48, 63, 64, +74, 79, 92, 94, 95, 119, 120, 122, 123, 134, 150, 154, 155 — ECC/NVLink/bus-fault-associated per +NVIDIA's Xid catalog) or a "GPU Reset Required" recovery-action transition, everything else +`warning`. A separate `fellOffBusRegex` catches the same fault when logged outside an Xid line. +Timestamps are syslog-style with no year; `parseSyslogTimestamp` assumes `CollectedAt`'s year and +rolls back one year if that guess would land implausibly after the report's collection time (>48h), +mirroring the existing year-inference pattern in `unraid`/`xigmanas`. +One specific NVLink topology-discovery retry message +(`knvlink*RxDetect*: Failed to update Rx Detect Link mask`) was observed flooding a real dump +114,440 times in a single burst — logging one event per line would blow up the event list, so it's +aggregated into a single summary event (count + first/last timestamp), severity `critical` above +100 occurrences. +**Consequences:** +- Real HGX B200 dump used for validation now surfaces 8 Xid-31 warnings + 6 Xid-150/154 criticals + + 1 aggregated NVLink-flood critical, matching the manual grep-based analysis from this session + exactly, with no event-count explosion. +- Severity classification is a heuristic triage hint, not a verdict — same caveat as documented in + `bible-local/docs/nvidia-bug-report-analysis.md`; Xid meaning is context-dependent (app bug vs + hardware) even for codes not in the hardware-leaning set. +- Bumped `parserVersion` to `1.3`. Added `errors_test.go`: code-based severity classification, + SXid event-type distinction, flood aggregation (500 synthetic lines → 1 event), fell-off-bus + detection, and year-rollover timestamp handling. + +--- + +## ADL-056 — Inspur event ingestion uses source PRI and an explicit-offset timezone timeline + +**Date:** 2026-08-27 +**Context:** Manual validation of an NF5688M7 one-key log (S/N 23E102624) found that inventory and +the primary NVMe fault were correct, but `crit.log*` was not ingested (dropping 346 MCTP timeouts and +one ME self-test failure), severity was inferred only from rotated filename, and naive SEL timestamps +were parsed with stale `timezone.conf=Asia/Shanghai`. The retained history actually changes from +`+08:00` to `+03:00`; the July SEL copy of an IDL event was consequently five hours wrong. +**Decision:** Inspur parser v2.4 now: +- ingests emerg/alert/crit/error/warning/notice/info syslog families and maps severity from RFC 5424 + PRI, with narrow content exceptions for four known-benign AMI/KVM status strings; +- omits fabricated 1970 syslog records whose only usable timestamp is seconds-since-boot; +- resolves offset-less SEL rows against a bounded timeline sampled from explicit-offset IDL records, + using `timezone.conf` only as fallback and preferring the embedded IDL timestamp over its outer + syslog envelope; +- treats deassert records as informational, deduplicates equivalent IDL/SEL copies at the same + normalized instant (preferring IDL), and stable-sorts the combined event stream by timestamp. +**Consequences:** The real NF5688M7 dump now exports all 346 MCTP timeouts and the ME self-test code +195, the July `Sys_Health` event appears once at `2026-07-23T23:20:38Z`, known benign driver/KVM +strings no longer inflate Critical/Warning, no 1970 event remains, and `event_logs` is chronological. +Synthetic regression coverage was added to `event_logs_test.go` and `sel_test.go`. + +--- + +## ADL-057 — Inspur CPU PPIN is exported as source-backed serial; active NVMe faults update storage status + +**Date:** 2026-08-27 +**Context:** NF5688M7 S/N 23E102624 exposes stable per-socket CPU identifiers only as `PPIN` in +both `asset.json` and RESTful CPU info. The same values and socket mapping were confirmed in the +older 2026-03-26 dump. Lenovo paper LP1890 describes PPIN as the processor-assigned serial/physical +CPU identifier; Intel documents it as a 64-bit physical-processor inventory identifier. LOGPile +parsed PPIN but left `CPU.SerialNumber` empty, so Reanimator dropped it. The same case also had an +active `NvmeIndex:6 drive fault` while every exported storage record remained `Unknown`. +**Decision:** Inspur parser v2.5: +- copies non-empty BMC PPIN into both `CPU.PPIN` and `CPU.SerialNumber`; this is source-backed identity, + not a generated fallback, and preserves CPU0/CPU1 socket mapping; +- projects IDL NVMe drive-fault Assert/Deassert transitions onto `hardware.storage` by the observed + zero-based `NvmeIndex:N` → one-based `OB(N+1)` mapping; +- uses `dev_status.log` as the authoritative active-fault snapshot when present, records transition + time/history/details, and exports an active fault as `Critical`; +- leaves unaffected drives `Unknown` unless the source provides an affirmative health result; mere + presence is not treated as proof of `OK`. +**Consequences:** On the real dump CPU0 exports serial `D46E5D6B1D3E40E1`, CPU1 exports +`D44F8D6B9155EE0E`, and only `OB07` exports `Critical` with the `Status_Flags error` description; +OB01–OB06 remain `Unknown`. Regression tests cover asset/component PPIN mapping, targeted NVMe fault, +deassert clearing, and rejection of unrelated NVMe inventory-change events. + +--- + +## ADL-058 — PPIN is the common source-backed CPU serial fallback + +**Date:** 2026-08-27 +**Context:** The NF5688M7 fix initially copied PPIN into `CPU.SerialNumber` only inside the Inspur +parser. Dell TSR, H3C INI/XML, and Redfish also expose PPIN (including OEM public processor serial), +but their parsed CPU serial could remain empty. Canonical devices and Reanimator could additionally +drop a PPIN already present in the model or device details. +**Decision:** CPU identity resolution is centralized in `models.ResolveCPUSerialNumber`. A valid, +explicit source serial has priority; otherwise a valid source PPIN is used. Empty values and known +placeholders (`N/A`, `Unknown`, and equivalents) are rejected. Dell, H3C, Inspur, and Redfish apply +the resolver while parsing, and API/canonical-device and Reanimator conversion apply it again at the +output boundary. No identifier is generated from socket, model, board serial, or another component. +**Consequences:** All current parsers that expose PPIN now return it consistently as CPU +`serial_number`; CSV, API, and Reanimator retain the same identity. Parsers with a genuine separate +CPU serial preserve that value. Regression tests cover vendor inputs, common output paths, priority, +fallback, and placeholder rejection. + +--- + +## ADL-059 — CPU identity regression audits use an offline batch CLI + +**Date:** 2026-08-27 +**Context:** The PPIN-to-CPU-serial fix must be checked once against a large directory of historical +dumps. The existing HTTP batch converter emits complete Reanimator payloads and requires multipart +uploads, while this audit needs small per-dump CPU-only JSON records and must continue past malformed +or unsupported archives. +**Decision:** Add `cmd/logpile-cpu-audit` as a non-server entry point over the existing parser +registry. It recursively discovers supported dump files, parses them with bounded concurrency, and +writes one path-mirrored CPU identity report per input plus a deterministic summary. Each CPU report +contains the parsed PPIN and serial, resolver expectation, promotion flag, and regression finding. +Failures are isolated and represented as JSON. Plain `.txt`/`.log` inputs are opt-in for directory +scans to avoid treating files inside unpacked dumps as independent dumps. +**Consequences:** Historical audit runs need no HTTP server or full hardware export, retain source +path and parser/version provenance, and can be reviewed or diffed as ordinary JSON. The tool shares +the production parser and CPU resolver, so its result reflects the same behavior as API and export +paths without duplicating vendor parsing logic. + +**Amendment (2026-08-27):** The helper also supports `-reanimator` mode for a flat, directly +ingestible CPU-only dataset. Audit reports are not Reanimator payloads and must not be sent to +`POST /ingest/hardware`. Reanimator mode converts through the production exporter, retains mandatory +`collected_at` and `hardware.board`, removes non-CPU hardware sections, skips empty inputs, and writes +no non-ingest JSON summaries alongside payload files. + +--- + +## ADL-060 — HPE AHS DIMM inventory dedupes per slot, newest snapshot wins + +**Date:** 2026-08-27 +**Context:** The AHS blackbox stores a full DIMM inventory snapshot for every POST +cycle, and `parseDIMMs` flattens every snapshot from every blackbox record into one +token stream. `dedupeMemory` keyed on `SerialNumber` (falling back to `slot|part`), +so when a module was physically swapped between captures the same physical slot +survived once per historical occupant. On a real bench dump where `PROC 1/2 DIMM 7` +was swapped mid-session this reported 26 modules for a 24-slot board and made the +"same P/N" and "mixed P/N" configurations look identical. +**Decision:** `dedupeMemory` collapses to one entry per non-empty `Slot`, and the +last occurrence for a slot wins. Token order follows blackbox-record order, which is +chronological, so the last occurrence is the most recent POST snapshot. Entries with +no slot label keep the previous serial/part fallback dedupe. +**Consequences:** AHS memory inventory always matches the physical slot count and +reflects the latest observed population. Historical swaps are no longer surfaced as +extra modules (if a per-slot history is wanted later it must be built explicitly, +not inferred from dedupe residue). Regression test `TestDedupeMemorySlotSwap`.