feat(parser): linux-generic content-driven parser and RAR archive support

Field bundles from ad-hoc service scripts (first sample: Giga G894-SD3 .rar
with dmidecode/lscpu/lspci/nvme/lsblk output, no fixed layout). ADL-078:
- linux-generic: recognizes utility output by content, merges records by
  stable key, low detect confidence (20-30) below all vendor parsers
- pciids.LookupByDescription: resolve a textual lspci description back to
  vendor/device IDs and manufacturer/model
- archive: accept .rar via the in-repo internal/rar5 reader (RAR 5 / RAR 7,
  single volume, solid; encrypted, multi-volume, SFX and RAR 4.x rejected),
  same per-file and total limits as tar/zip
- UI: .rar in the upload accept list

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Mikhail ChusavitinandClaude Opus 5.5 committed 2026-10-02 15:00:38 +03:00
1 parent ff5b99fa5c
commit b4b15e58a2
25 files changed
+4901 -5

No files matched your search

+63
View File
@@ -13,6 +13,13 @@ Core behavior:
`inspur` parser on shared `onekeylog/` archives (see ADL-065)
- generic fallback stays last and low-confidence
Archive extraction (`internal/parser/archive.go`) is shared by all parsers and selected by file
extension: `.tar`/`.sds`, `.tar.gz`/`.tgz`/`.gz`, `.zip`, `.rar`, `.ahs`, and plain `.txt`/`.log`.
`.rar` is read by the in-repo `internal/rar5` package (ADL-078): RAR 5 / RAR 7 format only, single
volume, including solid archives. Password-protected archives, multi-volume sets, self-extracting
archives and the legacy RAR 4.x format are rejected with an explicit error.
`VendorParser` contract:
```go
@@ -66,6 +73,7 @@ serial from its socket, model, board serial, or another component; reject source
| `inspur` | onekeylog archives (classic `component.log` and per-file `component/*.txt` D-Bus layouts) | FRU/SDR, optional Redis enrichment, normalized IDL/SEL/syslog events (see ADL-056) |
| `inspur_legacy` | Legacy AMI-BMC onekeylog (NF5466M5/NF5280M5 gen: `Inspur_AssetInfoInventory.log`, binary `FRU.bin`, `Inspur_*_IDL`, `sel.log`, flat `<sev>.log`) | Binary IPMI FRU decode + text inventory + IDL/SEL/blackbox/MegaRAID/syslog events; no live sensors. Wins the `onekeylog/` detect tie via `DetectPriority`. See ADL-065 |
| `lenovo_xcc` | Lenovo XCC mini-log ZIP archives | JSON inventory + platform event logs |
| `linux-generic` | Any archive of standard Linux utility output (`dmidecode`, `lscpu`, `lspci`, `nvme list`, `nvme smart-log`, `smartctl`, `lsblk`) with no fixed layout | Low-confidence (20-30) content-driven fallback: files are recognized by what they contain, not by name. See ADL-078 |
| `nvidia` | HGX Field Diagnostics | GPU- and fabric-heavy diagnostic input |
| `nvidia_bug_report` | `nvidia-bug-report-*.log.gz` | dmidecode, lspci, NVIDIA driver sections; Xid/SXid GPU error events (see ADL-055, `bible-local/docs/nvidia-bug-report-analysis.md`) |
| `redfish_walk` | Any archive carrying a captured Redfish directory-tree walk (tar.gz/zip of `<path>/index.json` docs) | Low-confidence (35) vendor-agnostic fallback; see `dell` for the first known source and `redfishtree` helper package |
@@ -279,6 +287,60 @@ tree — board identity, firmware, CPUs, memory, storage, PCIe, GPUs, NICs, PSUs
---
### Linux utility output (`linux-generic`)
**Status:** Ready (v1.0). Tested on a service-script `.rar` bundle from a Giga Computing
G894-SD3 (`System-info.txt`, `component-info.txt`, `service-log/*.txt`).
**Confidence:** 20 / 25 / 30 for one / two / three or more recognized utilities — above `generic`
(15), below `redfish_walk` (35) and every dedicated vendor parser, because BMC dumps and support
bundles embed the same utilities' output. `lsblk` or an NVMe SMART log alone scores 0: at least
one identity-bearing utility (`dmidecode`, `lscpu`, `lspci`, `nvme list`, `smartctl`) is required.
**Archive format:** none. Bundles are produced by ad-hoc scripts, so file names, directory layout
and the set of captured utilities differ between sources. Package `linux_generic` therefore never
looks at file names for recognition: every text file is matched against a table of per-utility
content signatures (`utilities` in `parser.go`), and one file may hold the output of several
utilities. Adding a utility = one table entry plus its parse function.
**Extracted data (current):**
| Utility | Recognized by | Produces |
|---------|---------------|----------|
| `dmidecode` (full, `-t`, or grep-filtered) | a known section title followed by indented `Key: Value` lines; `Handle` lines optional | board identity (System Information), BIOS firmware, CPUs (type 4), DIMMs (type 17), PSUs (type 39), Base Board / Chassis as FRU entries |
| `lscpu` | `Socket(s)` / `Core(s) per socket` / `Thread(s) per core` | one CPU per socket when dmidecode has no processor records; otherwise fills their gaps |
| `lspci` (plain, `-nn`, `-v`/`-vv`, grep-filtered excerpts) | `BB:DD.F <class>: <description>` lines | NICs per PCI function, GPUs, storage/FC/accelerator controller functions; link speed/width, VPD part/serial number, physical slot, NUMA node |
| `nvme list` | `Node … SN … Model` header | NVMe drives: model, serial, firmware, size, LBA format |
| `nvme smart-log` (+ appended `id-ctrl` fields) | `Smart Log for NVME device:` | wear, health status, power-on hours and related counters, exact capacity (`tnvmcap`) |
| `smartctl -i/-a/-x` | `=== START OF INFORMATION SECTION ===` | ATA / SAS / NVMe drives: model, serial, firmware, capacity, media type, interface, sector sizes, health |
| `lsblk` | `NAME … SIZE … TYPE` header | disks no other utility described (name + size only); removable and virtual devices skipped |
**Merge rules:**
- The same component is usually described by several files. Records are merged by a stable key —
PCI functions by BDF, drives by serial then kernel device / NVMe controller name, DIMMs by
serial or locator — and the first non-empty value of each field wins. Files are processed in
path order so the result does not depend on archive member order.
- Drive capacity takes the most precise source: exact bytes (`tnvmcap`, smartctl), then the
`nvme list` usage column, then the `lsblk` SIZE column.
- Plain `lspci` prints no numeric IDs. `pciids.LookupByDescription` splits the
`<vendor> <device>` string back into manufacturer and model and recovers vendor/device IDs; an
`lspci` `Device 1a2b` placeholder (device unknown to the host's `pci.ids`) is replaced with the
bundled `pci.ids` name. Functions that also appear in an `-nn` excerpt keep the printed IDs.
- VPD part/serial number captured for one function of a card is shared with its sibling functions
(same bus:device, same class): scripts typically dump `lspci -vv` for function 0 only.
- A display-class function with a BMC VGA vendor ID (ASPEED, Matrox) is not a GPU. Bridges, system
peripherals and other chipset functions are not inventory.
- SMART health is snapshot state and goes on the drive (`status`, `error_description`), not into
the event log. The parser emits no events and never sets `collected_at` (ADL-070); the framework
infers it from archive member mtimes.
**Known gaps:** no MAC addresses (`ip`/`ethtool` output is not parsed), no GPU serials
(`nvidia-smi` is not parsed), no BMC/FRU data (`ipmitool` is not parsed), no sensors. Script-specific
summary files (e.g. a localized `component-info.txt`) are read only where they embed a recognized
utility's output.
---
### Generic text fallback (`generic`)
**Status:** Ready (v1.0.0).
@@ -310,4 +372,5 @@ tree — board identity, firmware, CPUs, memory, storage, PCIe, GPUs, NICs, PSUs
| XigmaNAS | `xigmanas` | Ready | FreeBSD NAS logs |
| H3C SDS G5 | `h3c_g5` | Ready | H3C UniServer R4900 G5 SDS archives |
| H3C SDS G6 | `h3c_g6` | Ready | H3C UniServer R4700 G6 SDS archives |
| Linux utility output | `linux-generic` | Ready (fallback) | Service-script `.rar` bundle (G894-SD3): dmidecode, lscpu, lspci, nvme, lsblk |
| Generic fallback | `generic` | Ready | Any text file |
+56
View File
@@ -2272,3 +2272,59 @@ maps OS block devices by HCTL. Separately, all 402 snapshot `event_logs` were lo
- The MegaRAID is still absent from the exported `pcie_devices` (ADL-062 stands); a controller
with no VD and no HCTL-mapped disk stays invisible in the topology too.
- Bee runtime/service/interface state events remain UI-only (no contract source).
## ADL-078 — `linux-generic` parser recognizes Linux utility output by content; RAR archives are supported
**Date:** 2026-10-02
**Context:** Field engineers send bundles produced by ad-hoc service scripts: a `.rar` holding
`dmidecode`, `lscpu`, `lspci`, `nvme` and `lsblk` output split over files such as
`System-info.txt` and `service-log/ethernet_controller_2.txt` (first sample: Giga Computing
G894-SD3, ER9132823B0003). There is no vendor, no manifest and no stable layout — file names,
language, directory structure and the set of utilities change from script to script, and outputs are
often grep-filtered excerpts. Such uploads failed twice: `.rar` was not an accepted archive format,
and no parser matched the content.
**Decision:**
- Add parser `linux-generic` (package `internal/parser/vendors/linux_generic`). It recognizes
files by content only: a table of utilities, each with a content signature and a parse function,
is applied to every text file, and a file may match several utilities. Supported: `dmidecode`,
`lscpu`, `lspci`, `nvme list`, `nvme smart-log`, `smartctl`, `lsblk`.
- Detect confidence is 20-30 by the number of recognized utilities and requires at least one
identity-bearing utility. It must stay below `redfish_walk` (35) and all vendor parsers, since
vendor dumps embed the same output.
- Records from different files are merged by stable key (BDF; drive serial, then device name; DIMM
serial or locator), first non-empty value wins, files processed in path order.
- `pciids.LookupByDescription` resolves a textual lspci description back to vendor/device IDs and
to a manufacturer/model split, instead of a hardcoded vendor-name list (plain `lspci` prints no
numeric IDs).
- The parser emits no events; SMART health is component state (ADL-070, ADL-071).
- Add `.rar` to the shared archive extractor, read by a new in-repo package `internal/rar5`
instead of a third-party module, so the main module keeps zero external Go dependencies and
the single static binary (`CGO_ENABLED=0`) is unchanged. Shelling out to `unrar` was rejected
for the same reason.
- `internal/rar5` is deliberately narrow: RAR 5 container (and the RAR 7 dictionary extension),
sequential read from a stream, stored and compressed members, solid archives, CRC32
verification. Not supported, each rejected with its own error: encryption (file or header),
multi-volume sets, self-extracting archives, RAR 1.5-4.x. The container reader is written for
this project; the decompression core (`bitreader.go`, `huffman.go`, `decode.go`) is derived from
`github.com/nwaples/rardecode` (BSD 2-Clause), whose notice is kept in `internal/rar5/LICENSE`
and must stay there.
- Extraction limits match tar/zip: members over 10 MB are skipped, extraction stops at 1 GB
total. A member's sliding window is capped at 1 GiB and, for a non-solid file of known size, is
no larger than the file, so a small archive cannot request a large allocation.
- There is no freely available RAR compressor, so `internal/rar5` tests generate archives with a
minimal test-only RAR 5 encoder (`encode_test.go`). Its output was checked against the
reference `unrar` (`RAR5_TEST_DUMP_DIR=<dir> go test ./internal/rar5`, then `unrar t -mt1`),
and the first field sample extracts byte-identical to `unrar`.
**Consequences:**
- `.rar` is accepted by upload and batch convert for every parser, not only `linux-generic`.
`internal/sanitize` cannot rewrite RAR (no writer), so `.rar` stays outside `CanSanitize`.
- A RAR 4.x archive fails with "RAR 4.x and older archives are not supported"; it has to be
repacked. Adding it would mean porting the RAR 3 decoder (LZ + PPMd + filter VM), roughly three
times the current package.
- `internal/rar5` is now ours to maintain; upstream fixes are not picked up automatically.
- A previously rejected text-only archive that happens to contain one of the utilities' output
now parses as `linux-generic` with partial inventory instead of failing with "no parser found".
- Not parsed yet: `ip`/`ethtool` (MAC addresses), `nvidia-smi` (GPU serials), `ipmitool`
(FRU/sensors). Each is one more entry in the utility table.