feat(privacy): scan ingested sources for customer-identifying data

Detection-only scan (internal/privacy) attached to every AnalysisResult:
a customer-domain guess plus a findings list (category, file, line, match,
hint), ported from the KB grep playbook. Runs on archive uploads and the
serialized Redfish tree; gated by LOGPILE_PRIVACY_SCAN (default on).

Surfaced at GET /api/privacy-scan, in the "Customer data" UI panel, and as
privacy_report.json in the raw-export bundle. IP policy keeps RFC1918 and
example ranges out of findings; allowlist covers standards-body and vendor
infrastructure domains. No customer tokens in the repo. See ADL-066 and
bible-local/docs/privacy-scan.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-09-02 15:24:32 +03:00
co-authored by Claude Sonnet 5
parent 3311bafd8e
commit 4a4910f207
23 changed files with 1466 additions and 33 deletions
+44
View File
@@ -1893,3 +1893,47 @@ The `inspur` parser still detected it at confidence 100 (every path contains
`TestParseMegaRAIDLog`, `TestParseAMISyslog`, `TestDetect_*`,
`TestParse_EndToEnd`, and `TestDetectFormat_PriorityBreaksConfidenceTie` in
the `parser` package.
---
## ADL-066 — Customer-data (privacy) scan of ingested sources
**Date:** 2026-09-02
**Context:** BMC dumps routinely arrive not anonymized. DNS suffix, AD domain,
NTP/DNS/syslog host names, timezone, admin e-mails, public IPs, TLS cert CN/SAN
and FRU location fields identify the customer and the install site. Before a
dump is forwarded to a vendor, a public repo, or the LOGPile test corpus an
operator had to run a grep playbook by hand.
**Decision:** Every ingested dataset is scanned by `internal/privacy`. The scan
is **detection only** - it never rewrites the source. It produces a
`models.PrivacyScan` on `AnalysisResult`: a customer guess (registrable domain
with confidence + evidence) and a findings list (category, file, line, matched
token, remediation hint).
- Rule catalogue (`rules.go`) is a port of the KB grep playbook. Categories:
domain, resolv, ad_ldap, timezone, email, public_ip, collector, cert,
fru_location, hostname, dhcp, nsupdate, mgmt_subdomain.
- IP policy (`ip.go`): RFC1918/ULA, loopback, link-local, multicast and the
RFC5737/RFC2544/RFC6598 example ranges plus well-known public resolvers are
**not** findings. Only routable public addresses are.
- Allowlist (`allowlist.go`) is reference data - RFC 2606/5737 names, NTP pools,
standards-body and BMC-vendor infrastructure domains, and vendor factory
template file-name markers. It is not vendor-detection logic.
- No real customer domain or token is committed. Tests use `acme.ru` as the
customer stand-in; `example.*` is reserved for the allowlist (it is the
sanitization target).
- Hook points: `parser.BMCParser.parseFiles` for archives,
`ingest.Service.AnalyzeRedfishRawPayloads` for the serialized Redfish tree.
Gated by `LOGPILE_PRIVACY_SCAN` (default on).
- Surfaced at `GET /api/privacy-scan`, in the "Customer data" UI panel, and as
`privacy_report.json` in the raw-export bundle.
**Consequences:**
- Sanitization stays a manual follow-up guided by the report (no re-archiving).
- The customer guess uses a small built-in public-suffix list, not the full PSL.
- Binary and over-size files are skipped, so the scan is a floor, not a proof of
cleanliness.
- Details and the full rule table live in
[`docs/privacy-scan.md`](docs/privacy-scan.md).
- Tests: `internal/privacy` (`TestScan_*`, `TestIsSensitiveIP`,
`TestIsAllowlistedDomain`, `TestRegistrableDomain`, `TestGuessCustomers_*`),
`internal/server` (`TestHandleGetPrivacyScan_*`,
`TestBuildRawExportBundle_*PrivacyReport*`).