Files
logpile/bible-local/06-parsers.md
T
Mikhail ChusavitinandClaude Opus 5.5 1cc2faa567 fix(archive): open zips with a missing or truncated central directory
A BEE-SP v14.23 blackbox (Dell PowerEdge R760) streamed from the live CD was
cut off inside the central directory: all 1920 members intact, no
end-of-central-directory record, so archive/zip rejected the whole file
("zip: not a valid zip file"). ADL-079:
- when archive/zip cannot open a .zip, read it front to back by local file
  headers; intact archives are unaffected
- streamed members (data descriptors) are delimited from the data itself,
  deflate and stored; every member is CRC32/size checked
- recovery stops at the first damaged, truncated or encrypted member and
  keeps the entries before it; the original error is returned when nothing
  could be read
- same 10 MB per-file / 1 GB total limits as the regular zip path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 15:00:39 +03:00

21 KiB

06 — Parsers

Framework

Parsers live in internal/parser/ and vendor implementations live in internal/parser/vendors/.

Core behavior:

  • registration uses init() side effects
  • all registered parsers run Detect()
  • the highest-confidence parser wins
  • a confidence tie is broken by the optional PrioritizedParser interface (DetectPriority() int, default 0); used so inspur_legacy beats the broader inspur parser on shared onekeylog/ archives (see ADL-065)
  • generic fallback stays last and low-confidence

Archive extraction (internal/parser/archive.go) is shared by all parsers and selected by file extension: .tar/.sds, .tar.gz/.tgz/.gz, .zip, .rar, .ahs, and plain .txt/.log.

.rar is read by the in-repo internal/rar5 package (ADL-078): RAR 5 / RAR 7 format only, single volume, including solid archives. Password-protected archives, multi-volume sets, self-extracting archives and the legacy RAR 4.x format are rejected with an explicit error.

A .zip that archive/zip rejects (central directory missing or cut off, e.g. a streamed blackbox whose download was interrupted in the trailer) is read front to back by its local file headers (ADL-079); members are CRC-checked and recovery stops at the first damaged one.

VendorParser contract:

type VendorParser interface {
    Name() string
    Vendor() string
    Version() string
    Detect(files []ExtractedFile) int
    Parse(files []ExtractedFile) (*models.AnalysisResult, error)
}

Adding a parser

  1. Create internal/parser/vendors/<vendor>/
  2. Start from internal/parser/vendors/template/parser.go.template
  3. Implement Detect() and Parse()
  4. Add a blank import in internal/parser/vendors/vendors.go
  5. Add at least one positive and one negative detection test

Data quality rules

System firmware only in hardware.firmware

hardware.firmware must contain system-level firmware only. Device-bound firmware belongs on the device record and must not be duplicated at the top level.

Strip embedded MAC addresses from model names

If a source embeds - XX:XX:XX:XX:XX:XX in a model/name field, remove that suffix before storing it.

Use pci.ids for empty or generic PCI model names

When vendor_id and device_id are known but the model name is missing or generic, resolve the name via internal/parser/vendors/pciids.

Preserve source-backed CPU identity

Keep a source PPIN in CPU.PPIN. If the source has no separate processor serial number, use models.ResolveCPUSerialNumber to expose the valid PPIN as CPU.SerialNumber. Never derive a CPU serial from its socket, model, board serial, or another component; reject source placeholders.

Active vendor coverage

Vendor ID Input family Notes
dell TSR ZIP archives Broad hardware, firmware, sensors, lifecycle events
easy_bee bee-support-*.tar.gz, blackbox .zip Imports the embedded hardware snapshot (inventory, event_logs with source-native SEL/dmesg timestamps, platform_config) and static DIMM/storage/storcli/NVIDIA evidence used to build topology.json (ADL-077); without bee-audit.json (audit unfinished at capture) falls back to the bundled nvidia-bug-report.txt (ADL-076)
h3c_g5 H3C SDS G5 bundles INI/XML/CSV-driven hardware and event parsing
h3c_g6 H3C SDS G6 bundles Similar flow with G6-specific files
hpe_ilo_ahs HPE iLO Active Health System (.ahs) Proprietary ABJR container with gzip-compressed zbb members; parser combines SMBIOS-style inventory strings and embedded Redfish storage JSON
inspur onekeylog archives (classic component.log and per-file component/*.txt D-Bus layouts) FRU/SDR, optional Redis enrichment, normalized IDL/SEL/syslog events (see ADL-056)
inspur_legacy Legacy AMI-BMC onekeylog (NF5466M5/NF5280M5 gen: Inspur_AssetInfoInventory.log, binary FRU.bin, Inspur_*_IDL, sel.log, flat <sev>.log) Binary IPMI FRU decode + text inventory + IDL/SEL/blackbox/MegaRAID/syslog events; no live sensors. Wins the onekeylog/ detect tie via DetectPriority. See ADL-065
lenovo_xcc Lenovo XCC mini-log ZIP archives JSON inventory + platform event logs
linux-generic Any archive of standard Linux utility output (dmidecode, lscpu, lspci, nvme list, nvme smart-log, smartctl, lsblk) with no fixed layout Low-confidence (20-30) content-driven fallback: files are recognized by what they contain, not by name. See ADL-078
nvidia HGX Field Diagnostics GPU- and fabric-heavy diagnostic input
nvidia_bug_report nvidia-bug-report-*.log.gz dmidecode, lspci, NVIDIA driver sections; Xid/SXid GPU error events (see ADL-055, bible-local/docs/nvidia-bug-report-analysis.md)
redfish_walk Any archive carrying a captured Redfish directory-tree walk (tar.gz/zip of <path>/index.json docs) Low-confidence (35) vendor-agnostic fallback; see dell for the first known source and redfishtree helper package
unraid Unraid diagnostics/log bundles Server and storage-focused parsing
xfusion xFusion iBMC tar.gz dump / file export AppDump + RTOSDump + LogDump merge for hardware and firmware
xigmanas XigmaNAS plain logs FreeBSD/NAS-oriented inventory
generic fallback Low-confidence text fallback when nothing else matches

Practical guidance

  • Be conservative with high detect scores
  • Prefer filling missing fields over overwriting stronger source data
  • Keep parser version constants current when behavior changes
  • Any new vendor-specific filtering or dedup logic must ship with tests for that vendor format

Archive format: Unraid diagnostics archive contents (text-heavy diagnostics directories).

Detection: Combines filename/path markers (diagnostics-*, unraid-*.txt, vars.txt) with content markers (e.g. Unraid kernel build, parity data markers).

Extracted data (current):

  • Board / BIOS metadata (from motherboard/system files)
  • CPU summary (from lscpu.txt)
  • Memory modules (from diagnostics memory file)
  • Storage devices (from vars.txt + SMART files)
  • Syslog events

H3C SDS G5 (h3c_g5)

Status: Ready (v2.2). Tested on H3C UniServer R4900 G5 SDS archives.

Archive format: .sds (tar archive)

Detection: hardware_info.ini, hardware.info, firmware_version.ini, user/test*.csv, plus H3C markers.

Extracted data (current):

  • Board/FRU inventory (FRUInfo.ini, board_info.ini)
  • Firmware list (firmware_version.ini)
  • CPU inventory (hardware_info.ini)
  • Memory DIMM inventory (hardware_info.ini)
  • Storage inventory (hardware.info, storage_disk.ini, NVMe_info.txt, RAID text enrichments)
  • Logical RAID volumes (raid.json, Storage_RAID-*.txt)
  • Sensor snapshot (sensor_info.ini)
  • SEL events (user/test.csv, user/test1.csv, fallback Sel.json / sel_list.txt)

H3C SDS G6 (h3c_g6)

Status: Ready (v2.2). Tested on H3C UniServer R4700 G6 SDS archives.

Archive format: .sds (tar archive)

Detection: CPUDetailInfo.xml, MemoryDetailInfo.xml, firmware_version.json, Sel.json, plus H3C markers.

Extracted data (current):

  • Board/FRU inventory (FRUInfo.ini, board_info.ini)
  • Firmware list (firmware_version.json)
  • CPU inventory (CPUDetailInfo.xml)
  • Memory DIMM inventory (MemoryDetailInfo.xml)
  • Storage inventory + capacity/model/interface (storage_disk.ini, Storage_RAID-*.txt, NVMe_info.txt)
  • Logical RAID volumes (raid.json, fallback from Storage_RAID-*.txt when available)
  • Sensor snapshot (sensor_info.ini)
  • SEL events (user/Sel.json, fallback user/sel_list.txt)

HPE iLO AHS (hpe_ilo_ahs)

Status: Ready (v1.0.0). Tested on HPE ProLiant Gen11 .ahs export from iLO 6.

Archive format: .ahs single-file Active Health System export.

Detection: Single-file input with ABJR container header and HPE AHS member names such as CUST_INFO.DAT, *.zbb, ilo_boot_support.zbb.

Extracted data (current):

  • System board identity (manufacturer, model, serial, part number)
  • iLO / System ROM / SPS top-level firmware
  • CPU inventory (model-level)
  • Memory DIMM inventory for populated slots
  • PSU inventory
  • PCIe / OCP NIC inventory from SMBIOS-style slot records
  • Storage controller and physical drives from embedded Redfish JSON inside zbb members
  • Basic iLO event log entries with timestamps when present

Implementation note: The format is proprietary. Parser support is intentionally hybrid: container parsing (ABJR + gzip) plus structured extraction from embedded Redfish objects and printable SMBIOS/FRU payloads. This is sufficient for inventory-grade parsing without decoding the entire internal zbb schema.


Inspur / Kaytus (inspur)

Status: Ready (v2.2). Tested on Inspur NF5468M7 / Kaytus KR4268X2 onekeylog archives.

Archive format: onekeylog BMC diagnostic dump. Two known on-disk layouts:

  1. Classic layout: top-level onekeylog/ directory, single combined component/component.log with RESTful <Section> info: JSON blocks (PSU, fan, HDD, PCIe, network, ...).
  2. Per-file layout (seen on newer/OEM firmware): archive root is dump_<serial>_<timestamp>/ (not onekeylog/), and component.log is absent. Each component is instead dumped separately under component/*.txt as a raw D-Bus GetAll transcript: GETALL <object> OBJect blocks with tab-separated "field"\t"type":"x"\t"data":value triples. This is not valid JSON and needs its own line-oriented extraction (internal/parser/vendors/inspur/component_dbus.go).

Detection: onekeylog/ path segment, devicefrusdr.log, component/component.log, onekeylog_dreport.log filename (per-file layout marker), component/PowerSupplyInfo.txt (per-file layout marker), plus asset.json content markers (VersionInfo + CpuInfo + MemInfo).

Extracted data (current):

  • CPU / Memory / PCIe / Storage inventory from asset.json (both layouts)
  • FRU + SDR from devicefrusdr.log (classic layout)
  • PSU inventory: from component.log's RESTful PSU info: block (classic layout) or component/PowerSupplyInfo.txt GETALL transcript (per-file layout)
  • Fan RPM/PWM sensors: from component.log's RESTful fan info: block (classic layout) or component/FanInfo.txt GETALL transcript (per-file layout)
  • Redis snapshot enrichment for serials/firmware/telemetry when redis-dump.rdb is present

Known gaps (per-file layout only):

  • component/NetworkAdapter.txt and component/HDDBpListInfo.txt use a different busctl --verbose object-tree dump (not the GETALL format) and are not currently parsed. NIC identity/MAC data is still available via PCIe device inventory from asset.json, so this is not a data-loss gap for NICs.
  • component/RAID.txt mixes formats per RAID controller family (getLsiStorageCollection, getPMCStorageCollection); not currently parsed.

Implementation note: the same GETALL object name can recur across multiple command sections in one file with different field subsets (e.g. Pwm_N appears once under the FanPWM sensor query with a real Value reading, and again later under FanControl with only a Target setpoint). parseDBusGetAllObjects unions fields across recurring object names, first-seen wins per field, so a later content-free duplicate cannot silently blank out an earlier real reading.


xFusion iBMC Dump / File Export (xfusion)

Status: Ready (v1.1.0). Tested on xFusion G5500 V7 tar.gz exports.

Archive format: tar.gz dump exported from the iBMC UI, including AppDump/, RTOSDump/, and LogDump/ trees.

Detection: AppDump/FruData/fruinfo.txt, AppDump/card_manage/card_info, RTOSDump/versioninfo/app_revision.txt, and LogDump/netcard/netcard_info.txt.

Extracted data (current):

  • Board / FRU inventory from fruinfo.txt
  • CPU inventory from CpuMem/cpu_info
  • Memory DIMM inventory from CpuMem/mem_info
  • GPU inventory from card_info
  • OCP NIC inventory by merging card_info with LogDump/netcard/netcard_info.txt
  • PSU inventory from BMC/psu_info.txt
  • Physical storage from StorageMgnt/PhysicalDrivesInfo/*/disk_info
  • System firmware entries from RTOSDump/versioninfo/app_revision.txt
  • Maintenance events from LogDump/maintenance_log

Dell TSR (dell)

Status: Ready (v3.1). Two inventory source generations, both handled by the same parser.

Archive format: TSR nested zip (signature + TSR*.pl.zip).

Extracted data:

  • Older/DCIM-XML generation: sysinfo_dcim_view.xml (board/CPU/NIC/etc.), sysinfo_dcim_softwareidentity.xml (firmware), sysinfo_cim_sensor.xml (sensors), curr_lclog.xml (Lifecycle Controller events), tsr/metadata.json (board identity fallback).
  • iDRAC10-generation: same metadata.json/curr_lclog.xml, but no DCIM-XML files — instead tsr/hardware/sysinfo/inventory/redfishidracwalk.tar.gz holds a captured Redfish directory-tree walk. Parsed via the shared redfishtree helper package and replayed through collector.ReplayRedfishFromRawPayloads (internal/parser/vendors/dell/redfish_walk.go), then append-merged into the same Hardware/Sensors/FRU/Events slices the DCIM-XML path fills, so the existing dedupe passes resolve any overlap in favor of DCIM-derived data. See ADL-048.
  • Licenses (hardware.licenses[], contract v2.12): the Redfish-walk replay reads the standard DMTF /redfish/v1/LicenseService/Licenses collection (internal/collector/redfish_replay_licenses.go), present on iDRAC10-generation firmware — feature-on-demand/advanced licenses such as "iDRAC10 17G Enterprise License", "Secure Enterprise Key Manager". System-level licenses have no component_ref; a license with AuthorizationScope: "Device" gets component_ref from Links.AuthorizedDevices. Not sourced from DCIM-XML — only available on the Redfish-walk path. See ADL-050.

Generic Redfish walk fallback (redfish_walk)

Status: Ready (v1.0.0).

Confidence: 35 (above the generic fallback's 15, below every dedicated vendor parser — wins Detect() only when no dedicated parser also recognizes the archive).

Purpose: Vendor-agnostic catch-all for any archive that carries a captured Redfish directory-tree walk (one JSON document per resource, e.g. redfish/v1/Systems/System.Embedded.1/index.json) but isn't otherwise claimed by a dedicated vendor parser. First known source is Dell iDRAC10 TSR bundles (handled directly by dell, see above); this exists so any other vendor shipping the same kind of raw Redfish walk is picked up automatically.

Detection (internal/parser/vendors/redfishtree): two-step, vendor-independent —

  1. Path hint: an archive member path containing redfish and ending in .tar.gz/.tgz/.zip.
  2. Structural confirmation: the unpacked tree must contain a document whose own @odata.id is exactly /redfish/v1, plus a /redfish/v1/Systems or /redfish/v1/Chassis collection.

Documents are keyed by their own @odata.id, not the on-disk directory name (some resource names are URL-encoded on disk, e.g. Assembly%23, but not in the JSON payload).

Extracted data: whatever collector.ReplayRedfishFromRawPayloads produces from the replayed tree — board identity, firmware, CPUs, memory, storage, PCIe, GPUs, NICs, PSUs, sensors, events.


Linux utility output (linux-generic)

Status: Ready (v1.0). Tested on a service-script .rar bundle from a Giga Computing G894-SD3 (System-info.txt, component-info.txt, service-log/*.txt).

Confidence: 20 / 25 / 30 for one / two / three or more recognized utilities — above generic (15), below redfish_walk (35) and every dedicated vendor parser, because BMC dumps and support bundles embed the same utilities' output. lsblk or an NVMe SMART log alone scores 0: at least one identity-bearing utility (dmidecode, lscpu, lspci, nvme list, smartctl) is required.

Archive format: none. Bundles are produced by ad-hoc scripts, so file names, directory layout and the set of captured utilities differ between sources. Package linux_generic therefore never looks at file names for recognition: every text file is matched against a table of per-utility content signatures (utilities in parser.go), and one file may hold the output of several utilities. Adding a utility = one table entry plus its parse function.

Extracted data (current):

Utility Recognized by Produces
dmidecode (full, -t, or grep-filtered) a known section title followed by indented Key: Value lines; Handle lines optional board identity (System Information), BIOS firmware, CPUs (type 4), DIMMs (type 17), PSUs (type 39), Base Board / Chassis as FRU entries
lscpu Socket(s) / Core(s) per socket / Thread(s) per core one CPU per socket when dmidecode has no processor records; otherwise fills their gaps
lspci (plain, -nn, -v/-vv, grep-filtered excerpts) BB:DD.F <class>: <description> lines NICs per PCI function, GPUs, storage/FC/accelerator controller functions; link speed/width, VPD part/serial number, physical slot, NUMA node
nvme list Node … SN … Model header NVMe drives: model, serial, firmware, size, LBA format
nvme smart-log (+ appended id-ctrl fields) Smart Log for NVME device: wear, health status, power-on hours and related counters, exact capacity (tnvmcap)
smartctl -i/-a/-x === START OF INFORMATION SECTION === ATA / SAS / NVMe drives: model, serial, firmware, capacity, media type, interface, sector sizes, health
lsblk NAME … SIZE … TYPE header disks no other utility described (name + size only); removable and virtual devices skipped

Merge rules:

  • The same component is usually described by several files. Records are merged by a stable key — PCI functions by BDF, drives by serial then kernel device / NVMe controller name, DIMMs by serial or locator — and the first non-empty value of each field wins. Files are processed in path order so the result does not depend on archive member order.
  • Drive capacity takes the most precise source: exact bytes (tnvmcap, smartctl), then the nvme list usage column, then the lsblk SIZE column.
  • Plain lspci prints no numeric IDs. pciids.LookupByDescription splits the <vendor> <device> string back into manufacturer and model and recovers vendor/device IDs; an lspci Device 1a2b placeholder (device unknown to the host's pci.ids) is replaced with the bundled pci.ids name. Functions that also appear in an -nn excerpt keep the printed IDs.
  • VPD part/serial number captured for one function of a card is shared with its sibling functions (same bus:device, same class): scripts typically dump lspci -vv for function 0 only.
  • A display-class function with a BMC VGA vendor ID (ASPEED, Matrox) is not a GPU. Bridges, system peripherals and other chipset functions are not inventory.
  • SMART health is snapshot state and goes on the drive (status, error_description), not into the event log. The parser emits no events and never sets collected_at (ADL-070); the framework infers it from archive member mtimes.

Known gaps: no MAC addresses (ip/ethtool output is not parsed), no GPU serials (nvidia-smi is not parsed), no BMC/FRU data (ipmitool is not parsed), no sensors. Script-specific summary files (e.g. a localized component-info.txt) are read only where they embed a recognized utility's output.


Generic text fallback (generic)

Status: Ready (v1.0.0).

Confidence: 15 (lowest — only matches if no other parser scores higher)

Purpose: Fallback for any text file or single .gz file not matching a specific vendor.

Behavior:

  • If filename matches nvidia-bug-report-*.log.gz: extracts driver version and GPU list.
  • Otherwise: confirms file is text (not binary) and records a basic "Text File" event.

Supported vendor matrix

Vendor ID Status Tested on
Dell TSR dell Ready TSR nested zip archives (DCIM-XML and iDRAC10 Redfish-walk generations)
Generic Redfish walk redfish_walk Ready (fallback) Any archive with a captured Redfish tree dump
Reanimator Easy Bee easy_bee Ready bee-support-*.tar.gz support bundles
HPE iLO AHS hpe_ilo_ahs Ready iLO 6 .ahs exports
Inspur / Kaytus inspur Ready KR4268X2 onekeylog
Lenovo XCC mini-log lenovo_xcc Ready ThinkSystem SR650 V3 XCC mini-log ZIP
NVIDIA HGX Field Diag nvidia Ready Various HGX servers
NVIDIA Bug Report nvidia_bug_report Ready H100 systems
Unraid unraid Ready Unraid diagnostics archives
xFusion iBMC dump xfusion Ready G5500 V7 file-export tar.gz bundles
XigmaNAS xigmanas Ready FreeBSD NAS logs
H3C SDS G5 h3c_g5 Ready H3C UniServer R4900 G5 SDS archives
H3C SDS G6 h3c_g6 Ready H3C UniServer R4700 G6 SDS archives
Linux utility output linux-generic Ready (fallback) Service-script .rar bundle (G894-SD3): dmidecode, lscpu, lspci, nvme, lsblk
Generic fallback generic Ready Any text file