feat(parser): linux-generic content-driven parser and RAR archive support
Field bundles from ad-hoc service scripts (first sample: Giga G894-SD3 .rar with dmidecode/lscpu/lspci/nvme/lsblk output, no fixed layout). ADL-078: - linux-generic: recognizes utility output by content, merges records by stable key, low detect confidence (20-30) below all vendor parsers - pciids.LookupByDescription: resolve a textual lspci description back to vendor/device IDs and manufacturer/model - archive: accept .rar via the in-repo internal/rar5 reader (RAR 5 / RAR 7, single volume, solid; encrypted, multi-volume, SFX and RAR 4.x rejected), same per-file and total limits as tar/zip - UI: .rar in the upload accept list Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ff5b99fa5c
commit
b4b15e58a2
25 files changed
+4901
-5
No files matched your search
@@ -13,6 +13,13 @@ Core behavior:
|
||||
`inspur` parser on shared `onekeylog/` archives (see ADL-065)
|
||||
- generic fallback stays last and low-confidence
|
||||
|
||||
Archive extraction (`internal/parser/archive.go`) is shared by all parsers and selected by file
|
||||
extension: `.tar`/`.sds`, `.tar.gz`/`.tgz`/`.gz`, `.zip`, `.rar`, `.ahs`, and plain `.txt`/`.log`.
|
||||
|
||||
`.rar` is read by the in-repo `internal/rar5` package (ADL-078): RAR 5 / RAR 7 format only, single
|
||||
volume, including solid archives. Password-protected archives, multi-volume sets, self-extracting
|
||||
archives and the legacy RAR 4.x format are rejected with an explicit error.
|
||||
|
||||
`VendorParser` contract:
|
||||
|
||||
```go
|
||||
@@ -66,6 +73,7 @@ serial from its socket, model, board serial, or another component; reject source
|
||||
| `inspur` | onekeylog archives (classic `component.log` and per-file `component/*.txt` D-Bus layouts) | FRU/SDR, optional Redis enrichment, normalized IDL/SEL/syslog events (see ADL-056) |
|
||||
| `inspur_legacy` | Legacy AMI-BMC onekeylog (NF5466M5/NF5280M5 gen: `Inspur_AssetInfoInventory.log`, binary `FRU.bin`, `Inspur_*_IDL`, `sel.log`, flat `<sev>.log`) | Binary IPMI FRU decode + text inventory + IDL/SEL/blackbox/MegaRAID/syslog events; no live sensors. Wins the `onekeylog/` detect tie via `DetectPriority`. See ADL-065 |
|
||||
| `lenovo_xcc` | Lenovo XCC mini-log ZIP archives | JSON inventory + platform event logs |
|
||||
| `linux-generic` | Any archive of standard Linux utility output (`dmidecode`, `lscpu`, `lspci`, `nvme list`, `nvme smart-log`, `smartctl`, `lsblk`) with no fixed layout | Low-confidence (20-30) content-driven fallback: files are recognized by what they contain, not by name. See ADL-078 |
|
||||
| `nvidia` | HGX Field Diagnostics | GPU- and fabric-heavy diagnostic input |
|
||||
| `nvidia_bug_report` | `nvidia-bug-report-*.log.gz` | dmidecode, lspci, NVIDIA driver sections; Xid/SXid GPU error events (see ADL-055, `bible-local/docs/nvidia-bug-report-analysis.md`) |
|
||||
| `redfish_walk` | Any archive carrying a captured Redfish directory-tree walk (tar.gz/zip of `<path>/index.json` docs) | Low-confidence (35) vendor-agnostic fallback; see `dell` for the first known source and `redfishtree` helper package |
|
||||
@@ -279,6 +287,60 @@ tree — board identity, firmware, CPUs, memory, storage, PCIe, GPUs, NICs, PSUs
|
||||
|
||||
---
|
||||
|
||||
### Linux utility output (`linux-generic`)
|
||||
|
||||
**Status:** Ready (v1.0). Tested on a service-script `.rar` bundle from a Giga Computing
|
||||
G894-SD3 (`System-info.txt`, `component-info.txt`, `service-log/*.txt`).
|
||||
|
||||
**Confidence:** 20 / 25 / 30 for one / two / three or more recognized utilities — above `generic`
|
||||
(15), below `redfish_walk` (35) and every dedicated vendor parser, because BMC dumps and support
|
||||
bundles embed the same utilities' output. `lsblk` or an NVMe SMART log alone scores 0: at least
|
||||
one identity-bearing utility (`dmidecode`, `lscpu`, `lspci`, `nvme list`, `smartctl`) is required.
|
||||
|
||||
**Archive format:** none. Bundles are produced by ad-hoc scripts, so file names, directory layout
|
||||
and the set of captured utilities differ between sources. Package `linux_generic` therefore never
|
||||
looks at file names for recognition: every text file is matched against a table of per-utility
|
||||
content signatures (`utilities` in `parser.go`), and one file may hold the output of several
|
||||
utilities. Adding a utility = one table entry plus its parse function.
|
||||
|
||||
**Extracted data (current):**
|
||||
|
||||
| Utility | Recognized by | Produces |
|
||||
|---------|---------------|----------|
|
||||
| `dmidecode` (full, `-t`, or grep-filtered) | a known section title followed by indented `Key: Value` lines; `Handle` lines optional | board identity (System Information), BIOS firmware, CPUs (type 4), DIMMs (type 17), PSUs (type 39), Base Board / Chassis as FRU entries |
|
||||
| `lscpu` | `Socket(s)` / `Core(s) per socket` / `Thread(s) per core` | one CPU per socket when dmidecode has no processor records; otherwise fills their gaps |
|
||||
| `lspci` (plain, `-nn`, `-v`/`-vv`, grep-filtered excerpts) | `BB:DD.F <class>: <description>` lines | NICs per PCI function, GPUs, storage/FC/accelerator controller functions; link speed/width, VPD part/serial number, physical slot, NUMA node |
|
||||
| `nvme list` | `Node … SN … Model` header | NVMe drives: model, serial, firmware, size, LBA format |
|
||||
| `nvme smart-log` (+ appended `id-ctrl` fields) | `Smart Log for NVME device:` | wear, health status, power-on hours and related counters, exact capacity (`tnvmcap`) |
|
||||
| `smartctl -i/-a/-x` | `=== START OF INFORMATION SECTION ===` | ATA / SAS / NVMe drives: model, serial, firmware, capacity, media type, interface, sector sizes, health |
|
||||
| `lsblk` | `NAME … SIZE … TYPE` header | disks no other utility described (name + size only); removable and virtual devices skipped |
|
||||
|
||||
**Merge rules:**
|
||||
- The same component is usually described by several files. Records are merged by a stable key —
|
||||
PCI functions by BDF, drives by serial then kernel device / NVMe controller name, DIMMs by
|
||||
serial or locator — and the first non-empty value of each field wins. Files are processed in
|
||||
path order so the result does not depend on archive member order.
|
||||
- Drive capacity takes the most precise source: exact bytes (`tnvmcap`, smartctl), then the
|
||||
`nvme list` usage column, then the `lsblk` SIZE column.
|
||||
- Plain `lspci` prints no numeric IDs. `pciids.LookupByDescription` splits the
|
||||
`<vendor> <device>` string back into manufacturer and model and recovers vendor/device IDs; an
|
||||
`lspci` `Device 1a2b` placeholder (device unknown to the host's `pci.ids`) is replaced with the
|
||||
bundled `pci.ids` name. Functions that also appear in an `-nn` excerpt keep the printed IDs.
|
||||
- VPD part/serial number captured for one function of a card is shared with its sibling functions
|
||||
(same bus:device, same class): scripts typically dump `lspci -vv` for function 0 only.
|
||||
- A display-class function with a BMC VGA vendor ID (ASPEED, Matrox) is not a GPU. Bridges, system
|
||||
peripherals and other chipset functions are not inventory.
|
||||
- SMART health is snapshot state and goes on the drive (`status`, `error_description`), not into
|
||||
the event log. The parser emits no events and never sets `collected_at` (ADL-070); the framework
|
||||
infers it from archive member mtimes.
|
||||
|
||||
**Known gaps:** no MAC addresses (`ip`/`ethtool` output is not parsed), no GPU serials
|
||||
(`nvidia-smi` is not parsed), no BMC/FRU data (`ipmitool` is not parsed), no sensors. Script-specific
|
||||
summary files (e.g. a localized `component-info.txt`) are read only where they embed a recognized
|
||||
utility's output.
|
||||
|
||||
---
|
||||
|
||||
### Generic text fallback (`generic`)
|
||||
|
||||
**Status:** Ready (v1.0.0).
|
||||
@@ -310,4 +372,5 @@ tree — board identity, firmware, CPUs, memory, storage, PCIe, GPUs, NICs, PSUs
|
||||
| XigmaNAS | `xigmanas` | Ready | FreeBSD NAS logs |
|
||||
| H3C SDS G5 | `h3c_g5` | Ready | H3C UniServer R4900 G5 SDS archives |
|
||||
| H3C SDS G6 | `h3c_g6` | Ready | H3C UniServer R4700 G6 SDS archives |
|
||||
| Linux utility output | `linux-generic` | Ready (fallback) | Service-script `.rar` bundle (G894-SD3): dmidecode, lscpu, lspci, nvme, lsblk |
|
||||
| Generic fallback | `generic` | Ready | Any text file |
|
||||
@@ -2272,3 +2272,59 @@ maps OS block devices by HCTL. Separately, all 402 snapshot `event_logs` were lo
|
||||
- The MegaRAID is still absent from the exported `pcie_devices` (ADL-062 stands); a controller
|
||||
with no VD and no HCTL-mapped disk stays invisible in the topology too.
|
||||
- Bee runtime/service/interface state events remain UI-only (no contract source).
|
||||
|
||||
## ADL-078 — `linux-generic` parser recognizes Linux utility output by content; RAR archives are supported
|
||||
|
||||
**Date:** 2026-10-02
|
||||
**Context:** Field engineers send bundles produced by ad-hoc service scripts: a `.rar` holding
|
||||
`dmidecode`, `lscpu`, `lspci`, `nvme` and `lsblk` output split over files such as
|
||||
`System-info.txt` and `service-log/ethernet_controller_2.txt` (first sample: Giga Computing
|
||||
G894-SD3, ER9132823B0003). There is no vendor, no manifest and no stable layout — file names,
|
||||
language, directory structure and the set of utilities change from script to script, and outputs are
|
||||
often grep-filtered excerpts. Such uploads failed twice: `.rar` was not an accepted archive format,
|
||||
and no parser matched the content.
|
||||
|
||||
**Decision:**
|
||||
- Add parser `linux-generic` (package `internal/parser/vendors/linux_generic`). It recognizes
|
||||
files by content only: a table of utilities, each with a content signature and a parse function,
|
||||
is applied to every text file, and a file may match several utilities. Supported: `dmidecode`,
|
||||
`lscpu`, `lspci`, `nvme list`, `nvme smart-log`, `smartctl`, `lsblk`.
|
||||
- Detect confidence is 20-30 by the number of recognized utilities and requires at least one
|
||||
identity-bearing utility. It must stay below `redfish_walk` (35) and all vendor parsers, since
|
||||
vendor dumps embed the same output.
|
||||
- Records from different files are merged by stable key (BDF; drive serial, then device name; DIMM
|
||||
serial or locator), first non-empty value wins, files processed in path order.
|
||||
- `pciids.LookupByDescription` resolves a textual lspci description back to vendor/device IDs and
|
||||
to a manufacturer/model split, instead of a hardcoded vendor-name list (plain `lspci` prints no
|
||||
numeric IDs).
|
||||
- The parser emits no events; SMART health is component state (ADL-070, ADL-071).
|
||||
- Add `.rar` to the shared archive extractor, read by a new in-repo package `internal/rar5`
|
||||
instead of a third-party module, so the main module keeps zero external Go dependencies and
|
||||
the single static binary (`CGO_ENABLED=0`) is unchanged. Shelling out to `unrar` was rejected
|
||||
for the same reason.
|
||||
- `internal/rar5` is deliberately narrow: RAR 5 container (and the RAR 7 dictionary extension),
|
||||
sequential read from a stream, stored and compressed members, solid archives, CRC32
|
||||
verification. Not supported, each rejected with its own error: encryption (file or header),
|
||||
multi-volume sets, self-extracting archives, RAR 1.5-4.x. The container reader is written for
|
||||
this project; the decompression core (`bitreader.go`, `huffman.go`, `decode.go`) is derived from
|
||||
`github.com/nwaples/rardecode` (BSD 2-Clause), whose notice is kept in `internal/rar5/LICENSE`
|
||||
and must stay there.
|
||||
- Extraction limits match tar/zip: members over 10 MB are skipped, extraction stops at 1 GB
|
||||
total. A member's sliding window is capped at 1 GiB and, for a non-solid file of known size, is
|
||||
no larger than the file, so a small archive cannot request a large allocation.
|
||||
- There is no freely available RAR compressor, so `internal/rar5` tests generate archives with a
|
||||
minimal test-only RAR 5 encoder (`encode_test.go`). Its output was checked against the
|
||||
reference `unrar` (`RAR5_TEST_DUMP_DIR=<dir> go test ./internal/rar5`, then `unrar t -mt1`),
|
||||
and the first field sample extracts byte-identical to `unrar`.
|
||||
|
||||
**Consequences:**
|
||||
- `.rar` is accepted by upload and batch convert for every parser, not only `linux-generic`.
|
||||
`internal/sanitize` cannot rewrite RAR (no writer), so `.rar` stays outside `CanSanitize`.
|
||||
- A RAR 4.x archive fails with "RAR 4.x and older archives are not supported"; it has to be
|
||||
repacked. Adding it would mean porting the RAR 3 decoder (LZ + PPMd + filter VM), roughly three
|
||||
times the current package.
|
||||
- `internal/rar5` is now ours to maintain; upstream fixes are not picked up automatically.
|
||||
- A previously rejected text-only archive that happens to contain one of the utilities' output
|
||||
now parses as `linux-generic` with partial inventory instead of failing with "no parser found".
|
||||
- Not parsed yet: `ip`/`ethtool` (MAC addresses), `nvidia-smi` (GPU serials), `ipmitool`
|
||||
(FRU/sensors). Each is one more entry in the utility table.
|
||||
Reference in new issue
Block a user