Michael Chus
78806f9fa0
- Add bible.git as submodule at bible/ - Move docs/bible/ → bible-local/ (project-specific architecture) - Update CLAUDE.md to reference both bible/ and bible-local/ - Add AGENTS.md for Codex with same structure Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
367 lines
11 KiB
Markdown
367 lines
11 KiB
Markdown
# 07 — Exporters & Reanimator Integration
|
|
|
|
## Export endpoints summary
|
|
|
|
| Endpoint | Format | Filename pattern |
|
|
|----------|--------|-----------------|
|
|
| `GET /api/export/csv` | CSV — serial numbers | `YYYY-MM-DD (MODEL) - SN.csv` |
|
|
| `GET /api/export/json` | **Raw export package** (JSON or ZIP bundle) for reopen/re-analysis | `YYYY-MM-DD (MODEL) - SN.(json|zip)` |
|
|
| `GET /api/export/reanimator` | Reanimator hardware JSON | `YYYY-MM-DD (MODEL) - SN.json` |
|
|
|
|
---
|
|
|
|
## Raw Export (`Export Raw Data`)
|
|
|
|
### Purpose
|
|
|
|
Preserve enough source data to reproduce parsing later after parser fixes, without requiring
|
|
another live collection from the target system.
|
|
|
|
### Format
|
|
|
|
`/api/export/json` returns a **raw export package**:
|
|
- JSON package (machine-readable), or
|
|
- ZIP bundle containing:
|
|
- `raw_export.json` — machine-readable package
|
|
- `collect.log` — human-readable collection + parsing summary
|
|
- `parser_fields.json` — structured parsed field snapshot for diffs between parser versions
|
|
|
|
### Import / reopen behavior
|
|
|
|
When a raw export package is uploaded back into LOGPile:
|
|
- the app **re-analyzes from raw source**
|
|
- it does **not** trust embedded parsed output as source of truth
|
|
|
|
For Redfish, this means replay from `raw_payloads.redfish_tree`.
|
|
|
|
### Design rule
|
|
|
|
Raw export is a **re-analysis artifact**, not a final report dump. Keep it self-contained and
|
|
forward-compatible where possible (versioned package format, additive fields only).
|
|
|
|
---
|
|
|
|
## Reanimator Export
|
|
|
|
### Purpose
|
|
|
|
Exports hardware inventory data in the format expected by the Reanimator asset tracking
|
|
system. Enables one-click push from LOGPile to an external asset management platform.
|
|
|
|
### Implementation files
|
|
|
|
| File | Role |
|
|
|------|------|
|
|
| `internal/exporter/reanimator_models.go` | Go structs for Reanimator JSON |
|
|
| `internal/exporter/reanimator_converter.go` | `ConvertToReanimator()` and helpers |
|
|
| `internal/server/handlers.go` | `handleExportReanimator()` HTTP handler |
|
|
|
|
### Conversion rules
|
|
|
|
- Source: canonical `hardware.devices` repository (see [`04-data-models.md`](04-data-models.md))
|
|
- CPU manufacturer inferred from model string (Intel / AMD / ARM / Ampere)
|
|
- PCIe serial number generated when absent: `{board_serial}-PCIE-{slot}`
|
|
- Status values normalized to: `OK`, `Warning`, `Critical`, `Unknown` (`Empty` only for memory slots)
|
|
- Timestamps in RFC3339 format
|
|
- `target_host` derived from `filename` field (`redfish://…`, `ipmi://…`) if not in source; omitted if undeterminable
|
|
- `board.manufacturer` and `board.product_name` values of `"NULL"` treated as absent
|
|
|
|
### LOGPile → Reanimator field mapping
|
|
|
|
| LOGPile type | Reanimator section | Notes |
|
|
|---|---|---|
|
|
| `BoardInfo` | `board` | Direct mapping |
|
|
| `CPU` | `cpus` | + manufacturer (inferred) |
|
|
| `MemoryDIMM` | `memory` | Direct; empty slots included (`present=false`) |
|
|
| `Storage` | `storage` | Excluded if no `serial_number` |
|
|
| `PCIeDevice` | `pcie_devices` | Serial generated if missing |
|
|
| `GPU` | `pcie_devices` | `device_class=DisplayController` |
|
|
| `NetworkAdapter` | `pcie_devices` | `device_class=NetworkController` |
|
|
| `PSU` | `power_supplies` | Excluded if no serial or `present=false` |
|
|
| `FirmwareInfo` | `firmware` | Direct mapping |
|
|
|
|
### Inclusion / exclusion rules
|
|
|
|
**Included:**
|
|
- Memory slots with `present=false` (as Empty slots)
|
|
- PCIe devices without serial number (serial is generated)
|
|
|
|
**Excluded:**
|
|
- Storage without `serial_number`
|
|
- PSU without `serial_number` or with `present=false`
|
|
- NetworkAdapters with `present=false`
|
|
|
|
---
|
|
|
|
## Reanimator Integration Guide
|
|
|
|
This section documents the Reanimator receiver-side JSON format (what the Reanimator
|
|
system expects when it ingests a LOGPile export).
|
|
|
|
> **Important:** The Reanimator endpoint uses a strict JSON decoder (`DisallowUnknownFields`).
|
|
> Any unknown field — including nested ones — causes `400 Bad Request`.
|
|
> Use only `snake_case` keys listed here.
|
|
|
|
### Top-level structure
|
|
|
|
```json
|
|
{
|
|
"filename": "redfish://10.10.10.103",
|
|
"source_type": "api",
|
|
"protocol": "redfish",
|
|
"target_host": "10.10.10.103",
|
|
"collected_at": "2026-02-10T15:30:00Z",
|
|
"hardware": {
|
|
"board": {...},
|
|
"firmware": [...],
|
|
"cpus": [...],
|
|
"memory": [...],
|
|
"storage": [...],
|
|
"pcie_devices": [...],
|
|
"power_supplies": [...]
|
|
}
|
|
}
|
|
```
|
|
|
|
**Required:** `collected_at`, `hardware.board.serial_number`
|
|
**Optional:** `target_host`, `source_type`, `protocol`, `filename`
|
|
|
|
`source_type` values: `api`, `logfile`, `manual`
|
|
`protocol` values: `redfish`, `ipmi`, `snmp`, `ssh`
|
|
|
|
### Component status fields (all component sections)
|
|
|
|
Each component may carry:
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `status` | string | `OK`, `Warning`, `Critical`, `Unknown`, `Empty` |
|
|
| `status_checked_at` | RFC3339 | When status was last verified |
|
|
| `status_changed_at` | RFC3339 | When status last changed |
|
|
| `status_at_collection` | object | `{ "status": "...", "at": "..." }` — snapshot-time status |
|
|
| `status_history` | array | `[{ "status": "...", "changed_at": "...", "details": "..." }]` |
|
|
| `error_description` | string | Human-readable error for Warning/Critical |
|
|
|
|
### Board
|
|
|
|
```json
|
|
{
|
|
"board": {
|
|
"manufacturer": "Supermicro",
|
|
"product_name": "X12DPG-QT6",
|
|
"serial_number": "21D634101",
|
|
"part_number": "X12DPG-QT6-REV1.01",
|
|
"uuid": "d7ef2fe5-2fd0-11f0-910a-346f11040868"
|
|
}
|
|
}
|
|
```
|
|
|
|
`serial_number` required. `manufacturer` / `product_name` of `"NULL"` treated as absent.
|
|
|
|
### CPUs
|
|
|
|
```json
|
|
{
|
|
"socket": 0,
|
|
"model": "INTEL(R) XEON(R) GOLD 6530",
|
|
"cores": 32,
|
|
"threads": 64,
|
|
"frequency_mhz": 2100,
|
|
"max_frequency_mhz": 4000,
|
|
"manufacturer": "Intel",
|
|
"status": "OK"
|
|
}
|
|
```
|
|
|
|
`socket` (int) and `model` required. Serial generated: `{board_serial}-CPU-{socket}`.
|
|
|
|
LOT format: `CPU_{VENDOR}_{MODEL_NORMALIZED}` → e.g. `CPU_INTEL_XEON_GOLD_6530`
|
|
|
|
### Memory
|
|
|
|
```json
|
|
{
|
|
"slot": "CPU0_C0D0",
|
|
"location": "CPU0_C0D0",
|
|
"present": true,
|
|
"size_mb": 32768,
|
|
"type": "DDR5",
|
|
"max_speed_mhz": 4800,
|
|
"current_speed_mhz": 4800,
|
|
"manufacturer": "Hynix",
|
|
"serial_number": "80AD032419E17CEEC1",
|
|
"part_number": "HMCG88AGBRA191N",
|
|
"status": "OK"
|
|
}
|
|
```
|
|
|
|
`slot` and `present` required. `serial_number` required when `present=true`.
|
|
Empty slots (`present=false`, `status="Empty"`) are included but no component created.
|
|
|
|
LOT format: `DIMM_{TYPE}_{SIZE_GB}GB` → e.g. `DIMM_DDR5_32GB`
|
|
|
|
### Storage
|
|
|
|
```json
|
|
{
|
|
"slot": "OB01",
|
|
"type": "NVMe",
|
|
"model": "INTEL SSDPF2KX076T1",
|
|
"size_gb": 7680,
|
|
"serial_number": "BTAX41900GF87P6DGN",
|
|
"manufacturer": "Intel",
|
|
"firmware": "9CV10510",
|
|
"interface": "NVMe",
|
|
"present": true,
|
|
"status": "OK"
|
|
}
|
|
```
|
|
|
|
`slot`, `model`, `serial_number`, `present` required.
|
|
|
|
LOT format: `{TYPE}_{INTERFACE}_{SIZE_TB}TB` → e.g. `SSD_NVME_07.68TB`
|
|
|
|
### Power Supplies
|
|
|
|
```json
|
|
{
|
|
"slot": "0",
|
|
"present": true,
|
|
"model": "GW-CRPS3000LW",
|
|
"vendor": "Great Wall",
|
|
"wattage_w": 3000,
|
|
"serial_number": "2P06C102610",
|
|
"part_number": "V0310C9000000000",
|
|
"firmware": "00.03.05",
|
|
"status": "OK",
|
|
"input_power_w": 137,
|
|
"output_power_w": 104,
|
|
"input_voltage": 215.25
|
|
}
|
|
```
|
|
|
|
`slot`, `present` required. `serial_number` required when `present=true`.
|
|
Telemetry fields (`input_power_w`, `output_power_w`, `input_voltage`) stored in observation only.
|
|
|
|
LOT format: `PSU_{WATTAGE}W_{VENDOR_NORMALIZED}` → e.g. `PSU_3000W_GREAT_WALL`
|
|
|
|
### PCIe Devices
|
|
|
|
```json
|
|
{
|
|
"slot": "PCIeCard1",
|
|
"vendor_id": 32902,
|
|
"device_id": 2912,
|
|
"bdf": "0000:18:00.0",
|
|
"device_class": "MassStorageController",
|
|
"manufacturer": "Intel",
|
|
"model": "RAID Controller RSP3DD080F",
|
|
"link_width": 8,
|
|
"link_speed": "Gen3",
|
|
"max_link_width": 8,
|
|
"max_link_speed": "Gen3",
|
|
"serial_number": "RAID-001-12345",
|
|
"firmware": "50.9.1-4296",
|
|
"status": "OK"
|
|
}
|
|
```
|
|
|
|
`slot` required. Serial generated if absent: `{board_serial}-PCIE-{slot}`.
|
|
|
|
`device_class` values: `NetworkController`, `MassStorageController`, `DisplayController`, etc.
|
|
|
|
LOT format: `PCIE_{DEVICE_CLASS}_{MODEL_NORMALIZED}` → e.g. `PCIE_NETWORK_CONNECTX5`
|
|
|
|
### Firmware
|
|
|
|
```json
|
|
[
|
|
{ "device_name": "BIOS", "version": "06.08.05" },
|
|
{ "device_name": "BMC", "version": "5.17.00" }
|
|
]
|
|
```
|
|
|
|
Both fields required. Changes trigger `FIRMWARE_CHANGED` timeline events.
|
|
|
|
---
|
|
|
|
### Import process (Reanimator side)
|
|
|
|
1. Validate `collected_at` (RFC3339) and `hardware.board.serial_number`.
|
|
2. Find or create Asset by `board.serial_number` → `vendor_serial`.
|
|
3. For each component: filter `present=false`, auto-determine LOT, find or create Component,
|
|
create Observation, update Installations.
|
|
4. Detect removed components (present in previous snapshot, absent in current) → close Installation.
|
|
5. Generate timeline events: `LOG_COLLECTED`, `INSTALLED`, `REMOVED`, `FIRMWARE_CHANGED`.
|
|
|
|
**Idempotency:** Repeated import of the same snapshot (same content hash) returns `200 OK`
|
|
with `"duplicate": true` and does not create duplicate records.
|
|
|
|
### Reanimator API endpoint
|
|
|
|
```http
|
|
POST /ingest/hardware
|
|
Content-Type: application/json
|
|
```
|
|
|
|
**Success (201):**
|
|
```json
|
|
{
|
|
"status": "success",
|
|
"bundle_id": "lb_01J...",
|
|
"asset_id": "mach_01J...",
|
|
"collected_at": "2026-02-10T15:30:00Z",
|
|
"duplicate": false,
|
|
"summary": {
|
|
"parts_observed": 15,
|
|
"parts_created": 2,
|
|
"installations_created": 2,
|
|
"timeline_events_created": 9
|
|
}
|
|
}
|
|
```
|
|
|
|
**Duplicate (200):**
|
|
```json
|
|
{ "status": "success", "duplicate": true, "message": "LogBundle with this content hash already exists" }
|
|
```
|
|
|
|
**Error (400):**
|
|
```json
|
|
{ "status": "error", "error": "validation_failed", "details": { "field": "...", "message": "..." } }
|
|
```
|
|
|
|
Common `400` causes:
|
|
- Unknown JSON field (strict decoder)
|
|
- Wrong key name (e.g. `targetHost` instead of `target_host`)
|
|
- Invalid `collected_at` format (must be RFC3339)
|
|
- Empty `hardware.board.serial_number`
|
|
|
|
### LOT normalization rules
|
|
|
|
1. Remove special chars `( ) - ® ™`; replace spaces with `_`
|
|
2. Uppercase all
|
|
3. Collapse multiple underscores to one
|
|
4. Strip common prefixes like `MODEL:`, `PN:`
|
|
|
|
### Status values
|
|
|
|
| Value | Meaning | Action |
|
|
|-------|---------|--------|
|
|
| `OK` | Normal | — |
|
|
| `Warning` | Degraded | Create `COMPONENT_WARNING` event (optional) |
|
|
| `Critical` | Failed | Auto-create `failure_event`, create `COMPONENT_FAILED` event |
|
|
| `Unknown` | Not determinable | Treat as working |
|
|
| `Empty` | Slot unpopulated | No component created (memory/PCIe only) |
|
|
|
|
### Missing field handling
|
|
|
|
| Field | Fallback |
|
|
|-------|---------|
|
|
| CPU serial | Generated: `{board_serial}-CPU-{socket}` |
|
|
| PCIe serial | Generated: `{board_serial}-PCIE-{slot}` |
|
|
| Other serial | Component skipped if absent |
|
|
| manufacturer (PCIe) | Looked up from `vendor_id` (8086→Intel, 10de→NVIDIA, 15b3→Mellanox…) |
|
|
| status | Treated as `Unknown` |
|
|
| firmware | No `FIRMWARE_CHANGED` event |
|