docs(bible-local): record ADL-054..060 and parser/exporter notes
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
fc68134ed9
commit
70c939ea4b
@@ -1487,3 +1487,183 @@ strings for the same physical device into one registry entry, same as it already
|
||||
`TestParseAssetJSON_HddEnrichedWithPcieVendorDeviceID` (inspur), and
|
||||
`TestConvertToReanimator_StorageVendorDeviceIDSurvivesRoundTrip` (exporter, full export→marshal→
|
||||
reimport→re-export round trip).
|
||||
|
||||
---
|
||||
|
||||
## ADL-054 — Single-file gzip decompression is no longer capped at a fixed byte count
|
||||
|
||||
**Date:** 2026-08-26
|
||||
**Context:** A real `nvidia-bug-report-*.log.gz` (HGX B200, 53.25MB decompressed) was silently
|
||||
truncated by `extractTarGzFromReader`'s non-tar branch, which hard-capped decompressed content at
|
||||
`maxGzipDecompressedSize` (50MB) via `io.LimitReader`. The tail lost in this specific case was
|
||||
low-value `sar`-style telemetry, but the truncation was silent-by-default (only surfaced as a
|
||||
generic warning event) and the fixed cap has no relationship to what a legitimate bug-report bundle
|
||||
can contain — larger dumps would lose structured sections (nvidia-smi -q, dmesg tail) outright.
|
||||
**Decision:** Replaced the fixed cap with streaming decompression plus a decompression-*ratio*
|
||||
guard (`internal/parser/archive.go`): a `countingReader` tracks bytes actually consumed from the
|
||||
compressed source, and `readGzipWithBombGuard` reads the decompressed stream in 256KB chunks,
|
||||
aborting only if the ratio exceeds `gzipBombRatio` (300:1, evaluated only after
|
||||
`gzipBombMinCompressed` = 4KB of compressed input to avoid false positives from gzip header
|
||||
overhead) or an absolute `gzipAbsoluteCeiling` (1GB, matching the existing `maxSingleFileSizeLarge`
|
||||
used for large tar-in-gz) is hit. This only touches the plain-single-gzipped-file branch — tar.gz
|
||||
archives already use the generous 1GB `maxSingleFileSizeLarge` limit and were unaffected.
|
||||
**Consequences:**
|
||||
- Legitimate large single-file logs (any size, low compression ratio) now come through whole instead
|
||||
of being silently cut off; the extraction is still fully buffered into memory afterward (`[]byte`),
|
||||
matching every other extractor in this file — no `VendorParser` interface change.
|
||||
- Zip/gzip-bomb protection is now ratio-based instead of absolute-size-based, catching bombs of any
|
||||
target size while still allowing large legitimate files through.
|
||||
- Added `TestExtractArchiveFromReaderGZ_NoLongerCapsAt50MB` and
|
||||
`TestExtractArchiveFromReaderGZ_AbortsOnDecompressionBomb` (`archive_test.go`).
|
||||
|
||||
---
|
||||
|
||||
## ADL-055 — `nvidia_bug_report` parser now extracts Xid/SXid GPU error events
|
||||
|
||||
**Date:** 2026-08-26
|
||||
**Context:** The parser only ever extracted hardware inventory (board/CPU/memory/GPU/NIC/PSU) plus
|
||||
three fixed informational events (memory config, GPU detection, driver version). All GPU/NVLink
|
||||
fault triage (Xid/SXid errors, "GPU recovery action changed", "fell off the bus") had to be done by
|
||||
hand with `grep` against the raw decompressed log — see `bible-local/docs/nvidia-bug-report-analysis.md`
|
||||
for the manual checklist this was based on, itself informed by NVIDIA's GPU Debug Guidelines and
|
||||
Lambda Labs' `check-nvidia-bug-report.sh`.
|
||||
**Decision:** Added `internal/parser/vendors/nvidia_bug_report/errors.go` (`parseXidAndFaultEvents`),
|
||||
wired into `Parser.Parse`. It scans syslog-style kernel lines for `NVRM: Xid (PCI:...)` /
|
||||
`NVRM: SXid (PCI:...)` and emits one `models.Event` per occurrence, classified `critical` vs
|
||||
`warning` via a hardcoded set of hardware-leaning Xid codes (`hardwareLeaningXidCodes`: 48, 63, 64,
|
||||
74, 79, 92, 94, 95, 119, 120, 122, 123, 134, 150, 154, 155 — ECC/NVLink/bus-fault-associated per
|
||||
NVIDIA's Xid catalog) or a "GPU Reset Required" recovery-action transition, everything else
|
||||
`warning`. A separate `fellOffBusRegex` catches the same fault when logged outside an Xid line.
|
||||
Timestamps are syslog-style with no year; `parseSyslogTimestamp` assumes `CollectedAt`'s year and
|
||||
rolls back one year if that guess would land implausibly after the report's collection time (>48h),
|
||||
mirroring the existing year-inference pattern in `unraid`/`xigmanas`.
|
||||
One specific NVLink topology-discovery retry message
|
||||
(`knvlink*RxDetect*: Failed to update Rx Detect Link mask`) was observed flooding a real dump
|
||||
114,440 times in a single burst — logging one event per line would blow up the event list, so it's
|
||||
aggregated into a single summary event (count + first/last timestamp), severity `critical` above
|
||||
100 occurrences.
|
||||
**Consequences:**
|
||||
- Real HGX B200 dump used for validation now surfaces 8 Xid-31 warnings + 6 Xid-150/154 criticals +
|
||||
1 aggregated NVLink-flood critical, matching the manual grep-based analysis from this session
|
||||
exactly, with no event-count explosion.
|
||||
- Severity classification is a heuristic triage hint, not a verdict — same caveat as documented in
|
||||
`bible-local/docs/nvidia-bug-report-analysis.md`; Xid meaning is context-dependent (app bug vs
|
||||
hardware) even for codes not in the hardware-leaning set.
|
||||
- Bumped `parserVersion` to `1.3`. Added `errors_test.go`: code-based severity classification,
|
||||
SXid event-type distinction, flood aggregation (500 synthetic lines → 1 event), fell-off-bus
|
||||
detection, and year-rollover timestamp handling.
|
||||
|
||||
---
|
||||
|
||||
## ADL-056 — Inspur event ingestion uses source PRI and an explicit-offset timezone timeline
|
||||
|
||||
**Date:** 2026-08-27
|
||||
**Context:** Manual validation of an NF5688M7 one-key log (S/N 23E102624) found that inventory and
|
||||
the primary NVMe fault were correct, but `crit.log*` was not ingested (dropping 346 MCTP timeouts and
|
||||
one ME self-test failure), severity was inferred only from rotated filename, and naive SEL timestamps
|
||||
were parsed with stale `timezone.conf=Asia/Shanghai`. The retained history actually changes from
|
||||
`+08:00` to `+03:00`; the July SEL copy of an IDL event was consequently five hours wrong.
|
||||
**Decision:** Inspur parser v2.4 now:
|
||||
- ingests emerg/alert/crit/error/warning/notice/info syslog families and maps severity from RFC 5424
|
||||
PRI, with narrow content exceptions for four known-benign AMI/KVM status strings;
|
||||
- omits fabricated 1970 syslog records whose only usable timestamp is seconds-since-boot;
|
||||
- resolves offset-less SEL rows against a bounded timeline sampled from explicit-offset IDL records,
|
||||
using `timezone.conf` only as fallback and preferring the embedded IDL timestamp over its outer
|
||||
syslog envelope;
|
||||
- treats deassert records as informational, deduplicates equivalent IDL/SEL copies at the same
|
||||
normalized instant (preferring IDL), and stable-sorts the combined event stream by timestamp.
|
||||
**Consequences:** The real NF5688M7 dump now exports all 346 MCTP timeouts and the ME self-test code
|
||||
195, the July `Sys_Health` event appears once at `2026-07-23T23:20:38Z`, known benign driver/KVM
|
||||
strings no longer inflate Critical/Warning, no 1970 event remains, and `event_logs` is chronological.
|
||||
Synthetic regression coverage was added to `event_logs_test.go` and `sel_test.go`.
|
||||
|
||||
---
|
||||
|
||||
## ADL-057 — Inspur CPU PPIN is exported as source-backed serial; active NVMe faults update storage status
|
||||
|
||||
**Date:** 2026-08-27
|
||||
**Context:** NF5688M7 S/N 23E102624 exposes stable per-socket CPU identifiers only as `PPIN` in
|
||||
both `asset.json` and RESTful CPU info. The same values and socket mapping were confirmed in the
|
||||
older 2026-03-26 dump. Lenovo paper LP1890 describes PPIN as the processor-assigned serial/physical
|
||||
CPU identifier; Intel documents it as a 64-bit physical-processor inventory identifier. LOGPile
|
||||
parsed PPIN but left `CPU.SerialNumber` empty, so Reanimator dropped it. The same case also had an
|
||||
active `NvmeIndex:6 drive fault` while every exported storage record remained `Unknown`.
|
||||
**Decision:** Inspur parser v2.5:
|
||||
- copies non-empty BMC PPIN into both `CPU.PPIN` and `CPU.SerialNumber`; this is source-backed identity,
|
||||
not a generated fallback, and preserves CPU0/CPU1 socket mapping;
|
||||
- projects IDL NVMe drive-fault Assert/Deassert transitions onto `hardware.storage` by the observed
|
||||
zero-based `NvmeIndex:N` → one-based `OB(N+1)` mapping;
|
||||
- uses `dev_status.log` as the authoritative active-fault snapshot when present, records transition
|
||||
time/history/details, and exports an active fault as `Critical`;
|
||||
- leaves unaffected drives `Unknown` unless the source provides an affirmative health result; mere
|
||||
presence is not treated as proof of `OK`.
|
||||
**Consequences:** On the real dump CPU0 exports serial `D46E5D6B1D3E40E1`, CPU1 exports
|
||||
`D44F8D6B9155EE0E`, and only `OB07` exports `Critical` with the `Status_Flags error` description;
|
||||
OB01–OB06 remain `Unknown`. Regression tests cover asset/component PPIN mapping, targeted NVMe fault,
|
||||
deassert clearing, and rejection of unrelated NVMe inventory-change events.
|
||||
|
||||
---
|
||||
|
||||
## ADL-058 — PPIN is the common source-backed CPU serial fallback
|
||||
|
||||
**Date:** 2026-08-27
|
||||
**Context:** The NF5688M7 fix initially copied PPIN into `CPU.SerialNumber` only inside the Inspur
|
||||
parser. Dell TSR, H3C INI/XML, and Redfish also expose PPIN (including OEM public processor serial),
|
||||
but their parsed CPU serial could remain empty. Canonical devices and Reanimator could additionally
|
||||
drop a PPIN already present in the model or device details.
|
||||
**Decision:** CPU identity resolution is centralized in `models.ResolveCPUSerialNumber`. A valid,
|
||||
explicit source serial has priority; otherwise a valid source PPIN is used. Empty values and known
|
||||
placeholders (`N/A`, `Unknown`, and equivalents) are rejected. Dell, H3C, Inspur, and Redfish apply
|
||||
the resolver while parsing, and API/canonical-device and Reanimator conversion apply it again at the
|
||||
output boundary. No identifier is generated from socket, model, board serial, or another component.
|
||||
**Consequences:** All current parsers that expose PPIN now return it consistently as CPU
|
||||
`serial_number`; CSV, API, and Reanimator retain the same identity. Parsers with a genuine separate
|
||||
CPU serial preserve that value. Regression tests cover vendor inputs, common output paths, priority,
|
||||
fallback, and placeholder rejection.
|
||||
|
||||
---
|
||||
|
||||
## ADL-059 — CPU identity regression audits use an offline batch CLI
|
||||
|
||||
**Date:** 2026-08-27
|
||||
**Context:** The PPIN-to-CPU-serial fix must be checked once against a large directory of historical
|
||||
dumps. The existing HTTP batch converter emits complete Reanimator payloads and requires multipart
|
||||
uploads, while this audit needs small per-dump CPU-only JSON records and must continue past malformed
|
||||
or unsupported archives.
|
||||
**Decision:** Add `cmd/logpile-cpu-audit` as a non-server entry point over the existing parser
|
||||
registry. It recursively discovers supported dump files, parses them with bounded concurrency, and
|
||||
writes one path-mirrored CPU identity report per input plus a deterministic summary. Each CPU report
|
||||
contains the parsed PPIN and serial, resolver expectation, promotion flag, and regression finding.
|
||||
Failures are isolated and represented as JSON. Plain `.txt`/`.log` inputs are opt-in for directory
|
||||
scans to avoid treating files inside unpacked dumps as independent dumps.
|
||||
**Consequences:** Historical audit runs need no HTTP server or full hardware export, retain source
|
||||
path and parser/version provenance, and can be reviewed or diffed as ordinary JSON. The tool shares
|
||||
the production parser and CPU resolver, so its result reflects the same behavior as API and export
|
||||
paths without duplicating vendor parsing logic.
|
||||
|
||||
**Amendment (2026-08-27):** The helper also supports `-reanimator` mode for a flat, directly
|
||||
ingestible CPU-only dataset. Audit reports are not Reanimator payloads and must not be sent to
|
||||
`POST /ingest/hardware`. Reanimator mode converts through the production exporter, retains mandatory
|
||||
`collected_at` and `hardware.board`, removes non-CPU hardware sections, skips empty inputs, and writes
|
||||
no non-ingest JSON summaries alongside payload files.
|
||||
|
||||
---
|
||||
|
||||
## ADL-060 — HPE AHS DIMM inventory dedupes per slot, newest snapshot wins
|
||||
|
||||
**Date:** 2026-08-27
|
||||
**Context:** The AHS blackbox stores a full DIMM inventory snapshot for every POST
|
||||
cycle, and `parseDIMMs` flattens every snapshot from every blackbox record into one
|
||||
token stream. `dedupeMemory` keyed on `SerialNumber` (falling back to `slot|part`),
|
||||
so when a module was physically swapped between captures the same physical slot
|
||||
survived once per historical occupant. On a real bench dump where `PROC 1/2 DIMM 7`
|
||||
was swapped mid-session this reported 26 modules for a 24-slot board and made the
|
||||
"same P/N" and "mixed P/N" configurations look identical.
|
||||
**Decision:** `dedupeMemory` collapses to one entry per non-empty `Slot`, and the
|
||||
last occurrence for a slot wins. Token order follows blackbox-record order, which is
|
||||
chronological, so the last occurrence is the most recent POST snapshot. Entries with
|
||||
no slot label keep the previous serial/part fallback dedupe.
|
||||
**Consequences:** AHS memory inventory always matches the physical slot count and
|
||||
reflects the latest observed population. Historical swaps are no longer surfaced as
|
||||
extra modules (if a per-slot history is wanted later it must be built explicitly,
|
||||
not inferred from dedupe residue). Regression test `TestDedupeMemorySlotSwap`.
|
||||
|
||||
Reference in New Issue
Block a user