A BEE-SP v14.23 blackbox (Dell PowerEdge R760) streamed from the live CD was
cut off inside the central directory: all 1920 members intact, no
end-of-central-directory record, so archive/zip rejected the whole file
("zip: not a valid zip file"). ADL-079:
- when archive/zip cannot open a .zip, read it front to back by local file
headers; intact archives are unaffected
- streamed members (data descriptors) are delimited from the data itself,
deflate and stored; every member is CRC32/size checked
- recovery stops at the first damaged, truncated or encrypted member and
keeps the entries before it; the original error is returned when nothing
could be read
- same 10 MB per-file / 1 GB total limits as the regular zip path
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
21 KiB
06 — Parsers
Framework
Parsers live in internal/parser/ and vendor implementations live in internal/parser/vendors/.
Core behavior:
- registration uses
init()side effects - all registered parsers run
Detect() - the highest-confidence parser wins
- a confidence tie is broken by the optional
PrioritizedParserinterface (DetectPriority() int, default 0); used soinspur_legacybeats the broaderinspurparser on sharedonekeylog/archives (see ADL-065) - generic fallback stays last and low-confidence
Archive extraction (internal/parser/archive.go) is shared by all parsers and selected by file
extension: .tar/.sds, .tar.gz/.tgz/.gz, .zip, .rar, .ahs, and plain .txt/.log.
.rar is read by the in-repo internal/rar5 package (ADL-078): RAR 5 / RAR 7 format only, single
volume, including solid archives. Password-protected archives, multi-volume sets, self-extracting
archives and the legacy RAR 4.x format are rejected with an explicit error.
A .zip that archive/zip rejects (central directory missing or cut off, e.g. a streamed
blackbox whose download was interrupted in the trailer) is read front to back by its local file
headers (ADL-079); members are CRC-checked and recovery stops at the first damaged one.
VendorParser contract:
type VendorParser interface {
Name() string
Vendor() string
Version() string
Detect(files []ExtractedFile) int
Parse(files []ExtractedFile) (*models.AnalysisResult, error)
}
Adding a parser
- Create
internal/parser/vendors/<vendor>/ - Start from
internal/parser/vendors/template/parser.go.template - Implement
Detect()andParse() - Add a blank import in
internal/parser/vendors/vendors.go - Add at least one positive and one negative detection test
Data quality rules
System firmware only in hardware.firmware
hardware.firmware must contain system-level firmware only.
Device-bound firmware belongs on the device record and must not be duplicated at the top level.
Strip embedded MAC addresses from model names
If a source embeds - XX:XX:XX:XX:XX:XX in a model/name field, remove that suffix before storing it.
Use pci.ids for empty or generic PCI model names
When vendor_id and device_id are known but the model name is missing or generic, resolve the name via internal/parser/vendors/pciids.
Preserve source-backed CPU identity
Keep a source PPIN in CPU.PPIN. If the source has no separate processor serial number, use
models.ResolveCPUSerialNumber to expose the valid PPIN as CPU.SerialNumber. Never derive a CPU
serial from its socket, model, board serial, or another component; reject source placeholders.
Active vendor coverage
| Vendor ID | Input family | Notes |
|---|---|---|
dell |
TSR ZIP archives | Broad hardware, firmware, sensors, lifecycle events |
easy_bee |
bee-support-*.tar.gz, blackbox .zip |
Imports the embedded hardware snapshot (inventory, event_logs with source-native SEL/dmesg timestamps, platform_config) and static DIMM/storage/storcli/NVIDIA evidence used to build topology.json (ADL-077); without bee-audit.json (audit unfinished at capture) falls back to the bundled nvidia-bug-report.txt (ADL-076) |
h3c_g5 |
H3C SDS G5 bundles | INI/XML/CSV-driven hardware and event parsing |
h3c_g6 |
H3C SDS G6 bundles | Similar flow with G6-specific files |
hpe_ilo_ahs |
HPE iLO Active Health System (.ahs) |
Proprietary ABJR container with gzip-compressed zbb members; parser combines SMBIOS-style inventory strings and embedded Redfish storage JSON |
inspur |
onekeylog archives (classic component.log and per-file component/*.txt D-Bus layouts) |
FRU/SDR, optional Redis enrichment, normalized IDL/SEL/syslog events (see ADL-056) |
inspur_legacy |
Legacy AMI-BMC onekeylog (NF5466M5/NF5280M5 gen: Inspur_AssetInfoInventory.log, binary FRU.bin, Inspur_*_IDL, sel.log, flat <sev>.log) |
Binary IPMI FRU decode + text inventory + IDL/SEL/blackbox/MegaRAID/syslog events; no live sensors. Wins the onekeylog/ detect tie via DetectPriority. See ADL-065 |
lenovo_xcc |
Lenovo XCC mini-log ZIP archives | JSON inventory + platform event logs |
linux-generic |
Any archive of standard Linux utility output (dmidecode, lscpu, lspci, nvme list, nvme smart-log, smartctl, lsblk) with no fixed layout |
Low-confidence (20-30) content-driven fallback: files are recognized by what they contain, not by name. See ADL-078 |
nvidia |
HGX Field Diagnostics | GPU- and fabric-heavy diagnostic input |
nvidia_bug_report |
nvidia-bug-report-*.log.gz |
dmidecode, lspci, NVIDIA driver sections; Xid/SXid GPU error events (see ADL-055, bible-local/docs/nvidia-bug-report-analysis.md) |
redfish_walk |
Any archive carrying a captured Redfish directory-tree walk (tar.gz/zip of <path>/index.json docs) |
Low-confidence (35) vendor-agnostic fallback; see dell for the first known source and redfishtree helper package |
unraid |
Unraid diagnostics/log bundles | Server and storage-focused parsing |
xfusion |
xFusion iBMC tar.gz dump / file export |
AppDump + RTOSDump + LogDump merge for hardware and firmware |
xigmanas |
XigmaNAS plain logs | FreeBSD/NAS-oriented inventory |
generic |
fallback | Low-confidence text fallback when nothing else matches |
Practical guidance
- Be conservative with high detect scores
- Prefer filling missing fields over overwriting stronger source data
- Keep parser version constants current when behavior changes
- Any new vendor-specific filtering or dedup logic must ship with tests for that vendor format
Archive format: Unraid diagnostics archive contents (text-heavy diagnostics directories).
Detection: Combines filename/path markers (diagnostics-*, unraid-*.txt, vars.txt)
with content markers (e.g. Unraid kernel build, parity data markers).
Extracted data (current):
- Board / BIOS metadata (from motherboard/system files)
- CPU summary (from
lscpu.txt) - Memory modules (from diagnostics memory file)
- Storage devices (from
vars.txt+ SMART files) - Syslog events
H3C SDS G5 (h3c_g5)
Status: Ready (v2.2). Tested on H3C UniServer R4900 G5 SDS archives.
Archive format: .sds (tar archive)
Detection: hardware_info.ini, hardware.info, firmware_version.ini, user/test*.csv, plus H3C markers.
Extracted data (current):
- Board/FRU inventory (
FRUInfo.ini,board_info.ini) - Firmware list (
firmware_version.ini) - CPU inventory (
hardware_info.ini) - Memory DIMM inventory (
hardware_info.ini) - Storage inventory (
hardware.info,storage_disk.ini,NVMe_info.txt, RAID text enrichments) - Logical RAID volumes (
raid.json,Storage_RAID-*.txt) - Sensor snapshot (
sensor_info.ini) - SEL events (
user/test.csv,user/test1.csv, fallbackSel.json/sel_list.txt)
H3C SDS G6 (h3c_g6)
Status: Ready (v2.2). Tested on H3C UniServer R4700 G6 SDS archives.
Archive format: .sds (tar archive)
Detection: CPUDetailInfo.xml, MemoryDetailInfo.xml, firmware_version.json, Sel.json, plus H3C markers.
Extracted data (current):
- Board/FRU inventory (
FRUInfo.ini,board_info.ini) - Firmware list (
firmware_version.json) - CPU inventory (
CPUDetailInfo.xml) - Memory DIMM inventory (
MemoryDetailInfo.xml) - Storage inventory + capacity/model/interface (
storage_disk.ini,Storage_RAID-*.txt,NVMe_info.txt) - Logical RAID volumes (
raid.json, fallback fromStorage_RAID-*.txtwhen available) - Sensor snapshot (
sensor_info.ini) - SEL events (
user/Sel.json, fallbackuser/sel_list.txt)
HPE iLO AHS (hpe_ilo_ahs)
Status: Ready (v1.0.0). Tested on HPE ProLiant Gen11 .ahs export from iLO 6.
Archive format: .ahs single-file Active Health System export.
Detection: Single-file input with ABJR container header and HPE AHS member names
such as CUST_INFO.DAT, *.zbb, ilo_boot_support.zbb.
Extracted data (current):
- System board identity (manufacturer, model, serial, part number)
- iLO / System ROM / SPS top-level firmware
- CPU inventory (model-level)
- Memory DIMM inventory for populated slots
- PSU inventory
- PCIe / OCP NIC inventory from SMBIOS-style slot records
- Storage controller and physical drives from embedded Redfish JSON inside
zbbmembers - Basic iLO event log entries with timestamps when present
Implementation note: The format is proprietary. Parser support is intentionally hybrid:
container parsing (ABJR + gzip) plus structured extraction from embedded Redfish objects and
printable SMBIOS/FRU payloads. This is sufficient for inventory-grade parsing without decoding the
entire internal zbb schema.
Inspur / Kaytus (inspur)
Status: Ready (v2.2). Tested on Inspur NF5468M7 / Kaytus KR4268X2 onekeylog archives.
Archive format: onekeylog BMC diagnostic dump. Two known on-disk layouts:
- Classic layout: top-level
onekeylog/directory, single combinedcomponent/component.logwithRESTful <Section> info:JSON blocks (PSU, fan, HDD, PCIe, network, ...). - Per-file layout (seen on newer/OEM firmware): archive root is
dump_<serial>_<timestamp>/(notonekeylog/), andcomponent.logis absent. Each component is instead dumped separately undercomponent/*.txtas a raw D-BusGetAlltranscript:GETALL <object> OBJectblocks with tab-separated"field"\t"type":"x"\t"data":valuetriples. This is not valid JSON and needs its own line-oriented extraction (internal/parser/vendors/inspur/component_dbus.go).
Detection: onekeylog/ path segment, devicefrusdr.log, component/component.log,
onekeylog_dreport.log filename (per-file layout marker), component/PowerSupplyInfo.txt
(per-file layout marker), plus asset.json content markers (VersionInfo + CpuInfo +
MemInfo).
Extracted data (current):
- CPU / Memory / PCIe / Storage inventory from
asset.json(both layouts) - FRU + SDR from
devicefrusdr.log(classic layout) - PSU inventory: from
component.log'sRESTful PSU info:block (classic layout) orcomponent/PowerSupplyInfo.txtGETALL transcript (per-file layout) - Fan RPM/PWM sensors: from
component.log'sRESTful fan info:block (classic layout) orcomponent/FanInfo.txtGETALL transcript (per-file layout) - Redis snapshot enrichment for serials/firmware/telemetry when
redis-dump.rdbis present
Known gaps (per-file layout only):
component/NetworkAdapter.txtandcomponent/HDDBpListInfo.txtuse a differentbusctl --verboseobject-tree dump (not the GETALL format) and are not currently parsed. NIC identity/MAC data is still available via PCIe device inventory fromasset.json, so this is not a data-loss gap for NICs.component/RAID.txtmixes formats per RAID controller family (getLsiStorageCollection,getPMCStorageCollection); not currently parsed.
Implementation note: the same GETALL object name can recur across multiple command sections in
one file with different field subsets (e.g. Pwm_N appears once under the FanPWM sensor query
with a real Value reading, and again later under FanControl with only a Target setpoint).
parseDBusGetAllObjects unions fields across recurring object names, first-seen wins per field, so
a later content-free duplicate cannot silently blank out an earlier real reading.
xFusion iBMC Dump / File Export (xfusion)
Status: Ready (v1.1.0). Tested on xFusion G5500 V7 tar.gz exports.
Archive format: tar.gz dump exported from the iBMC UI, including AppDump/, RTOSDump/,
and LogDump/ trees.
Detection: AppDump/FruData/fruinfo.txt, AppDump/card_manage/card_info,
RTOSDump/versioninfo/app_revision.txt, and LogDump/netcard/netcard_info.txt.
Extracted data (current):
- Board / FRU inventory from
fruinfo.txt - CPU inventory from
CpuMem/cpu_info - Memory DIMM inventory from
CpuMem/mem_info - GPU inventory from
card_info - OCP NIC inventory by merging
card_infowithLogDump/netcard/netcard_info.txt - PSU inventory from
BMC/psu_info.txt - Physical storage from
StorageMgnt/PhysicalDrivesInfo/*/disk_info - System firmware entries from
RTOSDump/versioninfo/app_revision.txt - Maintenance events from
LogDump/maintenance_log
Dell TSR (dell)
Status: Ready (v3.1). Two inventory source generations, both handled by the same parser.
Archive format: TSR nested zip (signature + TSR*.pl.zip).
Extracted data:
- Older/DCIM-XML generation:
sysinfo_dcim_view.xml(board/CPU/NIC/etc.),sysinfo_dcim_softwareidentity.xml(firmware),sysinfo_cim_sensor.xml(sensors),curr_lclog.xml(Lifecycle Controller events),tsr/metadata.json(board identity fallback). - iDRAC10-generation: same
metadata.json/curr_lclog.xml, but no DCIM-XML files — insteadtsr/hardware/sysinfo/inventory/redfishidracwalk.tar.gzholds a captured Redfish directory-tree walk. Parsed via the sharedredfishtreehelper package and replayed throughcollector.ReplayRedfishFromRawPayloads(internal/parser/vendors/dell/redfish_walk.go), then append-merged into the sameHardware/Sensors/FRU/Eventsslices the DCIM-XML path fills, so the existing dedupe passes resolve any overlap in favor of DCIM-derived data. See ADL-048. - Licenses (
hardware.licenses[], contract v2.12): the Redfish-walk replay reads the standard DMTF/redfish/v1/LicenseService/Licensescollection (internal/collector/redfish_replay_licenses.go), present on iDRAC10-generation firmware — feature-on-demand/advanced licenses such as "iDRAC10 17G Enterprise License", "Secure Enterprise Key Manager". System-level licenses have nocomponent_ref; a license withAuthorizationScope: "Device"getscomponent_reffromLinks.AuthorizedDevices. Not sourced from DCIM-XML — only available on the Redfish-walk path. See ADL-050.
Generic Redfish walk fallback (redfish_walk)
Status: Ready (v1.0.0).
Confidence: 35 (above the generic fallback's 15, below every dedicated vendor parser — wins
Detect() only when no dedicated parser also recognizes the archive).
Purpose: Vendor-agnostic catch-all for any archive that carries a captured Redfish
directory-tree walk (one JSON document per resource, e.g.
redfish/v1/Systems/System.Embedded.1/index.json) but isn't otherwise claimed by a dedicated vendor
parser. First known source is Dell iDRAC10 TSR bundles (handled directly by dell, see above); this
exists so any other vendor shipping the same kind of raw Redfish walk is picked up automatically.
Detection (internal/parser/vendors/redfishtree): two-step, vendor-independent —
- Path hint: an archive member path containing
redfishand ending in.tar.gz/.tgz/.zip. - Structural confirmation: the unpacked tree must contain a document whose own
@odata.idis exactly/redfish/v1, plus a/redfish/v1/Systemsor/redfish/v1/Chassiscollection.
Documents are keyed by their own @odata.id, not the on-disk directory name (some resource names
are URL-encoded on disk, e.g. Assembly%23, but not in the JSON payload).
Extracted data: whatever collector.ReplayRedfishFromRawPayloads produces from the replayed
tree — board identity, firmware, CPUs, memory, storage, PCIe, GPUs, NICs, PSUs, sensors, events.
Linux utility output (linux-generic)
Status: Ready (v1.0). Tested on a service-script .rar bundle from a Giga Computing
G894-SD3 (System-info.txt, component-info.txt, service-log/*.txt).
Confidence: 20 / 25 / 30 for one / two / three or more recognized utilities — above generic
(15), below redfish_walk (35) and every dedicated vendor parser, because BMC dumps and support
bundles embed the same utilities' output. lsblk or an NVMe SMART log alone scores 0: at least
one identity-bearing utility (dmidecode, lscpu, lspci, nvme list, smartctl) is required.
Archive format: none. Bundles are produced by ad-hoc scripts, so file names, directory layout
and the set of captured utilities differ between sources. Package linux_generic therefore never
looks at file names for recognition: every text file is matched against a table of per-utility
content signatures (utilities in parser.go), and one file may hold the output of several
utilities. Adding a utility = one table entry plus its parse function.
Extracted data (current):
| Utility | Recognized by | Produces |
|---|---|---|
dmidecode (full, -t, or grep-filtered) |
a known section title followed by indented Key: Value lines; Handle lines optional |
board identity (System Information), BIOS firmware, CPUs (type 4), DIMMs (type 17), PSUs (type 39), Base Board / Chassis as FRU entries |
lscpu |
Socket(s) / Core(s) per socket / Thread(s) per core |
one CPU per socket when dmidecode has no processor records; otherwise fills their gaps |
lspci (plain, -nn, -v/-vv, grep-filtered excerpts) |
BB:DD.F <class>: <description> lines |
NICs per PCI function, GPUs, storage/FC/accelerator controller functions; link speed/width, VPD part/serial number, physical slot, NUMA node |
nvme list |
Node … SN … Model header |
NVMe drives: model, serial, firmware, size, LBA format |
nvme smart-log (+ appended id-ctrl fields) |
Smart Log for NVME device: |
wear, health status, power-on hours and related counters, exact capacity (tnvmcap) |
smartctl -i/-a/-x |
=== START OF INFORMATION SECTION === |
ATA / SAS / NVMe drives: model, serial, firmware, capacity, media type, interface, sector sizes, health |
lsblk |
NAME … SIZE … TYPE header |
disks no other utility described (name + size only); removable and virtual devices skipped |
Merge rules:
- The same component is usually described by several files. Records are merged by a stable key — PCI functions by BDF, drives by serial then kernel device / NVMe controller name, DIMMs by serial or locator — and the first non-empty value of each field wins. Files are processed in path order so the result does not depend on archive member order.
- Drive capacity takes the most precise source: exact bytes (
tnvmcap, smartctl), then thenvme listusage column, then thelsblkSIZE column. - Plain
lspciprints no numeric IDs.pciids.LookupByDescriptionsplits the<vendor> <device>string back into manufacturer and model and recovers vendor/device IDs; anlspciDevice 1a2bplaceholder (device unknown to the host'spci.ids) is replaced with the bundledpci.idsname. Functions that also appear in an-nnexcerpt keep the printed IDs. - VPD part/serial number captured for one function of a card is shared with its sibling functions
(same bus:device, same class): scripts typically dump
lspci -vvfor function 0 only. - A display-class function with a BMC VGA vendor ID (ASPEED, Matrox) is not a GPU. Bridges, system peripherals and other chipset functions are not inventory.
- SMART health is snapshot state and goes on the drive (
status,error_description), not into the event log. The parser emits no events and never setscollected_at(ADL-070); the framework infers it from archive member mtimes.
Known gaps: no MAC addresses (ip/ethtool output is not parsed), no GPU serials
(nvidia-smi is not parsed), no BMC/FRU data (ipmitool is not parsed), no sensors. Script-specific
summary files (e.g. a localized component-info.txt) are read only where they embed a recognized
utility's output.
Generic text fallback (generic)
Status: Ready (v1.0.0).
Confidence: 15 (lowest — only matches if no other parser scores higher)
Purpose: Fallback for any text file or single .gz file not matching a specific vendor.
Behavior:
- If filename matches
nvidia-bug-report-*.log.gz: extracts driver version and GPU list. - Otherwise: confirms file is text (not binary) and records a basic "Text File" event.
Supported vendor matrix
| Vendor | ID | Status | Tested on |
|---|---|---|---|
| Dell TSR | dell |
Ready | TSR nested zip archives (DCIM-XML and iDRAC10 Redfish-walk generations) |
| Generic Redfish walk | redfish_walk |
Ready (fallback) | Any archive with a captured Redfish tree dump |
| Reanimator Easy Bee | easy_bee |
Ready | bee-support-*.tar.gz support bundles |
| HPE iLO AHS | hpe_ilo_ahs |
Ready | iLO 6 .ahs exports |
| Inspur / Kaytus | inspur |
Ready | KR4268X2 onekeylog |
| Lenovo XCC mini-log | lenovo_xcc |
Ready | ThinkSystem SR650 V3 XCC mini-log ZIP |
| NVIDIA HGX Field Diag | nvidia |
Ready | Various HGX servers |
| NVIDIA Bug Report | nvidia_bug_report |
Ready | H100 systems |
| Unraid | unraid |
Ready | Unraid diagnostics archives |
| xFusion iBMC dump | xfusion |
Ready | G5500 V7 file-export tar.gz bundles |
| XigmaNAS | xigmanas |
Ready | FreeBSD NAS logs |
| H3C SDS G5 | h3c_g5 |
Ready | H3C UniServer R4900 G5 SDS archives |
| H3C SDS G6 | h3c_g6 |
Ready | H3C UniServer R4700 G6 SDS archives |
| Linux utility output | linux-generic |
Ready (fallback) | Service-script .rar bundle (G894-SD3): dmidecode, lscpu, lspci, nvme, lsblk |
| Generic fallback | generic |
Ready | Any text file |