refactor: unified ingest pipeline + modular Redfish profile framework

Implement the full architectural plan: unified ingest.Service entry point
for archive and Redfish payloads, modular redfishprofile package with
composable profiles (generic, ami-family, msi, supermicro, dell,
hgx-topology), score-based profile matching with fallback expansion mode,
and profile-driven acquisition/analysis plans.

Vendor-specific logic moved out of common executors and into profile hooks.
GPU chassis lookup strategies and known storage recovery collections
(IntelVROC/HA-RAID/MRVL) now live in ResolvedAnalysisPlan, populated by
profiles at analysis time. Replay helpers read from the plan; no hardcoded
path lists remain in generic code.

Also splits redfish_replay.go into domain modules (gpu, storage, inventory,
fru, profiles) and adds full fixture/matcher/directive test coverage
including Dell, AMI, unknown-vendor fallback, and deterministic ordering.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-03-18 08:48:58 +03:00
parent d8d3d8c524
commit d650a6ba1c
45 changed files with 5231 additions and 1011 deletions
+218
View File
@@ -274,6 +274,188 @@ for `Enclosure`, `RackMount`, and any unrecognised type (fail-safe).
both the excluded types and the storage-capable types (see `TestChassisTypeCanHaveNVMe`
and `TestNVMePostProbeSkipsNonStorageChassis`).
## ADL-019 — Redfish post-probe recovery is profile-owned acquisition policy
**Date:** 2026-03-18
**Context:** Numeric collection post-probe and direct NVMe `Disk.Bay` recovery were still
controlled by collector-core heuristics, which kept platform-specific acquisition behavior in
`redfish.go` and made vendor/topology refactoring incomplete.
**Decision:** Move expensive Redfish post-probe enablement into profile-owned acquisition policy.
The collector core may execute bounded post-probe loops, but profiles must explicitly enable:
- numeric collection post-probe
- direct NVMe `Disk.Bay` recovery
- sensor collection post-probe
**Consequences:**
- Generic collector flow no longer implicitly turns on storage/NVMe recovery for every platform.
- Supermicro-specific direct NVMe recovery and generic numeric collection recovery are now
regression-tested through profile fixtures.
- Future platform storage/post-probe behavior must be added through profile tuning, not new
vendor-shaped `if` branches in collector core.
## ADL-020 — Redfish critical plan-B activation is profile-owned recovery policy
**Date:** 2026-03-18
**Context:** `critical plan-B` and `profile plan-B` were still effectively always-on collector
behavior once paths were present, including critical collection member retry and slow numeric
child probing. That kept acquisition recovery semantics in `redfish.go` instead of the profile
layer.
**Decision:** Move plan-B activation into profile-owned recovery policy. Profiles must explicitly
enable:
- critical collection member retry
- slow numeric probing during critical plan-B
- profile-specific plan-B pass
**Consequences:**
- Recovery behavior is now observable in raw Redfish diagnostics alongside other tuning.
- Generic/fallback recovery remains available through profile policy instead of implicit collector
defaults.
- Future platform-specific plan-B behavior must be introduced through profile tuning and tests,
not through new unconditional collector branches.
## ADL-021 — Extra discovered-path storage seeds must be profile-scoped, not core-baseline
**Date:** 2026-03-18
**Context:** The collector core baseline seed list still contained storage-specific discovered-path
suffixes such as `SimpleStorage` and `Storage/IntelVROC/*`. These are useful on some platforms,
but they are acquisition extensions layered on top of discovered `Systems/*` resources, not part
of the minimal vendor-neutral Redfish baseline.
**Decision:** Move such discovered-path expansions into profile-owned scoped path policy. The
collector core keeps the vendor-neutral baseline; profiles may add extra system/chassis/manager
suffixes that are expanded over discovered members during acquisition planning.
**Consequences:**
- Platform-shaped storage discovery no longer lives in `redfish.go` baseline seed construction.
- Extra discovered-path branches are visible in plan diagnostics and fixture regression tests.
- Future model/vendor storage path expansions must be added through scoped profile policy instead
of editing the shared baseline seed list.
## ADL-022 — Adaptive prefetch eligibility is profile-owned policy
**Date:** 2026-03-18
**Context:** The adaptive prefetch executor was still driven by hardcoded include/exclude path
rules in `redfish.go`. That made GPU/storage/network prefetch shaping part of collector-core
knowledge rather than profile-owned acquisition policy.
**Decision:** Move prefetch eligibility rules into profile tuning. The collector core still runs
adaptive prefetch, but profiles provide:
- `IncludeSuffixes` for critical paths eligible for prefetch
- `ExcludeContains` for path shapes that must never be prefetched
**Consequences:**
- Prefetch behavior is now visible in raw Redfish diagnostics and test fixtures.
- Platform- or topology-specific prefetch shaping no longer requires editing collector-core
string lists.
- Future prefetch tuning must be introduced through profiles and regression tests.
## ADL-023 — Core critical baseline is roots-only; critical shaping is profile-owned
**Date:** 2026-03-18
**Context:** `redfishCriticalEndpoints(...)` still encoded a broad set of system/chassis/manager
critical branches directly in collector core. This mixed minimal crawl invariants with profile-
specific acquisition shaping.
**Decision:** Reduce collector-core critical baseline to vendor-neutral roots only:
- `/redfish/v1`
- discovered `Systems/*`
- discovered `Chassis/*`
- discovered `Managers/*`
Profiles now own additional critical shaping through:
- scoped critical suffix policy for discovered resources
- explicit top-level `CriticalPaths`
**Consequences:**
- Critical inventory breadth is now explained by the acquisition plan, not hidden in collector
helper defaults.
- Generic profile still provides the previous broad critical coverage, so behavior stays stable.
- Future critical-path tuning must be implemented in profiles and regression-tested there.
## ADL-024 — Live Redfish execution plans are resolved inside redfishprofile
**Date:** 2026-03-18
**Context:** Even after moving seeds, scoped paths, critical shaping, recovery, and prefetch
policy into profiles, `redfish.go` still manually merged discovered resources with those policy
fragments. That left acquisition-plan resolution logic in collector core.
**Decision:** Introduce `redfishprofile.ResolveAcquisitionPlan(...)` as the boundary between
profile planning and collector execution. `redfishprofile` now resolves:
- baseline seeds
- baseline critical roots
- scoped path expansions
- explicit profile seed/critical/plan-B paths
The collector core consumes the resolved plan and executes it.
**Consequences:**
- Acquisition planning logic is now testable in `redfishprofile` without going through the live
collector.
- `redfish.go` no longer owns path-resolution helpers for seeds/critical planning.
- This creates a clean next step toward true per-profile acquisition hooks beyond static policy
fragments.
## ADL-025 — Post-discovery acquisition refinement belongs to profile hooks
**Date:** 2026-03-18
**Context:** Some acquisition behavior depends not only on vendor/model hints, but on what the
lightweight Redfish discovery actually returned. Static absolute path lists in profile plans are
too rigid for such cases and reintroduce guessed platform knowledge.
**Decision:** Add a post-discovery acquisition refinement hook to Redfish profiles. Profiles may
mutate the resolved execution plan after discovered `Systems/*`, `Chassis/*`, and `Managers/*`
are known.
First concrete use:
- MSI now derives GPU chassis seeds and `.../Sensors` critical/plan-B paths from discovered
`Chassis/GPU*` resources instead of hardcoded `GPU1..GPU4` absolute paths in the static plan.
Additional use:
- Supermicro now derives `UpdateService/Oem/Supermicro/FirmwareInventory` critical/plan-B paths
from resource hints instead of carrying that absolute path in the static plan.
Additional use:
- Dell now derives `Managers/iDRAC.Embedded.*` acquisition paths from discovered manager
resources instead of carrying `Managers/iDRAC.Embedded.1` as a static absolute path.
**Consequences:**
- Profile modules can react to actual discovery results without pushing conditional logic back
into `redfish.go`.
- Diagnostics still show the final refined plan because the collector stores the refined plan,
not only the pre-refinement template.
- Future vendor-specific discovery-dependent acquisition behavior should be implemented through
this hook rather than new collector-core branches.
## ADL-026 — Replay analysis uses a resolved profile plan, not ad-hoc directives only
**Date:** 2026-03-18
**Context:** Replay still relied on a flat `AnalysisDirectives` struct assembled centrally,
while vendor-specific conditions often depended on the actual snapshot shape. That made analysis
behavior harder to explain and kept too much vendor logic in generic replay collectors.
**Decision:** Introduce `redfishprofile.ResolveAnalysisPlan(...)` for replay. The resolved
analysis plan contains:
- active match result
- resolved analysis directives
- analysis notes explaining snapshot-aware hook activation
Profiles may refine this plan using the snapshot and discovered resources before replay collectors
run.
First concrete uses:
- MSI enables processor-GPU fallback and MSI chassis lookup only when the snapshot actually
contains GPU processors and `Chassis/GPU*`
- HGX enables processor-GPU alias fallback from actual HGX/GPU_SXM topology signals in the snapshot
- Supermicro enables NVMe backplane and known-controller recovery from actual snapshot paths
**Consequences:**
- Replay behavior is now closer to the acquisition architecture: a resolved profile plan feeds the
executor.
- `redfish_analysis_plan` is stored in raw payload metadata for offline debugging.
- Future analysis-side vendor logic should move into profile refinement hooks instead of growing the
central directive builder.
## ADL-027 — Replay GPU/storage executors consume resolved analysis plans
**Date:** 2026-03-18
**Context:** Even after introducing `ResolveAnalysisPlan(...)`, replay GPU/storage collectors still
accepted a raw `AnalysisDirectives` struct. That preserved an implicit shortcut from the old design
and weakened the plan/executor boundary.
**Decision:** Replay GPU/storage executors now accept `redfishprofile.ResolvedAnalysisPlan`
directly. The executor reads resolved directives from the plan instead of being passed a standalone
directive bundle.
**Consequences:**
- GPU and storage replay execution now follows the same architectural pattern as acquisition:
resolve plan first, execute second.
- Future profile-owned execution helpers can use plan notes or additional resolved fields without
changing the executor API again.
- Remaining replay areas should migrate the same way instead of continuing to accept raw directive
structs.
## ADL-019 — isDeviceBoundFirmwareName must cover vendor-specific naming patterns per vendor
**Date:** 2026-03-12
@@ -604,3 +786,39 @@ presentation drift and duplicated UI logic.
- The host UI becomes a service shell around the viewer instead of maintaining its own
field-by-field tabs.
- `internal/chart` must be updated explicitly as a git submodule when the viewer changes.
---
## ADL-031 — Redfish uses profile-driven acquisition and unified ingest entrypoints
**Date:** 2026-03-17
**Context:**
Redfish collection had accumulated platform-specific probing in the shared collector path, while
upload and raw-export replay still entered analysis through direct handler branches. This made
vendor/model tuning harder to contain and increased regression risk when one topology needed a
special acquisition strategy.
**Decision:**
- Introduce `internal/ingest.Service` as the internal source-family entrypoint for archive parsing
and Redfish raw replay.
- Introduce `internal/collector/redfishprofile/` for Redfish profile matching and modular hooks.
- Split Redfish behavior into coordinated phases:
- acquisition planning during live collection
- analysis hooks during snapshot replay
- Use score-based profile matching. If confidence is low, enter fallback acquisition mode and
aggregate only safe additive profile probes.
- Allow profile modules to provide bounded acquisition tuning hints such as crawl cap, prefetch
behavior, and expensive post-probe toggles.
- Allow profile modules to own model-specific `CriticalPaths` and bounded `PlanBPaths` so vendor
recovery targets stop leaking into the collector core.
- Expose Redfish profile matching as structured diagnostics during live collection: logs must
contain all module scores, and collect job status must expose active modules for the UI.
**Consequences:**
- Server handlers stop owning parser-vs-replay branching details directly.
- Vendor/model-specific Redfish logic gets an explicit module boundary.
- Unknown-vendor Redfish collection becomes slower but more complete by design.
- Tactical Redfish fixes should move into profile modules instead of widening generic replay logic.
- Repo-owned compact fixtures under `internal/collector/redfishprofile/testdata/`, derived from
representative raw-export snapshots, are used to lock profile matching and acquisition tuning
for known MSI and Supermicro-family shapes.