feat(privacy): scan ingested sources for customer-identifying data
Detection-only scan (internal/privacy) attached to every AnalysisResult: a customer-domain guess plus a findings list (category, file, line, match, hint), ported from the KB grep playbook. Runs on archive uploads and the serialized Redfish tree; gated by LOGPILE_PRIVACY_SCAN (default on). Surfaced at GET /api/privacy-scan, in the "Customer data" UI panel, and as privacy_report.json in the raw-export bundle. IP policy keeps RFC1918 and example ranges out of findings; allowlist covers standards-body and vendor infrastructure domains. No customer tokens in the repo. See ADL-066 and bible-local/docs/privacy-scan.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
3311bafd8e
commit
4a4910f207
@@ -1893,3 +1893,47 @@ The `inspur` parser still detected it at confidence 100 (every path contains
|
||||
`TestParseMegaRAIDLog`, `TestParseAMISyslog`, `TestDetect_*`,
|
||||
`TestParse_EndToEnd`, and `TestDetectFormat_PriorityBreaksConfidenceTie` in
|
||||
the `parser` package.
|
||||
|
||||
---
|
||||
|
||||
## ADL-066 — Customer-data (privacy) scan of ingested sources
|
||||
|
||||
**Date:** 2026-09-02
|
||||
**Context:** BMC dumps routinely arrive not anonymized. DNS suffix, AD domain,
|
||||
NTP/DNS/syslog host names, timezone, admin e-mails, public IPs, TLS cert CN/SAN
|
||||
and FRU location fields identify the customer and the install site. Before a
|
||||
dump is forwarded to a vendor, a public repo, or the LOGPile test corpus an
|
||||
operator had to run a grep playbook by hand.
|
||||
**Decision:** Every ingested dataset is scanned by `internal/privacy`. The scan
|
||||
is **detection only** - it never rewrites the source. It produces a
|
||||
`models.PrivacyScan` on `AnalysisResult`: a customer guess (registrable domain
|
||||
with confidence + evidence) and a findings list (category, file, line, matched
|
||||
token, remediation hint).
|
||||
- Rule catalogue (`rules.go`) is a port of the KB grep playbook. Categories:
|
||||
domain, resolv, ad_ldap, timezone, email, public_ip, collector, cert,
|
||||
fru_location, hostname, dhcp, nsupdate, mgmt_subdomain.
|
||||
- IP policy (`ip.go`): RFC1918/ULA, loopback, link-local, multicast and the
|
||||
RFC5737/RFC2544/RFC6598 example ranges plus well-known public resolvers are
|
||||
**not** findings. Only routable public addresses are.
|
||||
- Allowlist (`allowlist.go`) is reference data - RFC 2606/5737 names, NTP pools,
|
||||
standards-body and BMC-vendor infrastructure domains, and vendor factory
|
||||
template file-name markers. It is not vendor-detection logic.
|
||||
- No real customer domain or token is committed. Tests use `acme.ru` as the
|
||||
customer stand-in; `example.*` is reserved for the allowlist (it is the
|
||||
sanitization target).
|
||||
- Hook points: `parser.BMCParser.parseFiles` for archives,
|
||||
`ingest.Service.AnalyzeRedfishRawPayloads` for the serialized Redfish tree.
|
||||
Gated by `LOGPILE_PRIVACY_SCAN` (default on).
|
||||
- Surfaced at `GET /api/privacy-scan`, in the "Customer data" UI panel, and as
|
||||
`privacy_report.json` in the raw-export bundle.
|
||||
**Consequences:**
|
||||
- Sanitization stays a manual follow-up guided by the report (no re-archiving).
|
||||
- The customer guess uses a small built-in public-suffix list, not the full PSL.
|
||||
- Binary and over-size files are skipped, so the scan is a floor, not a proof of
|
||||
cleanliness.
|
||||
- Details and the full rule table live in
|
||||
[`docs/privacy-scan.md`](docs/privacy-scan.md).
|
||||
- Tests: `internal/privacy` (`TestScan_*`, `TestIsSensitiveIP`,
|
||||
`TestIsAllowlistedDomain`, `TestRegistrableDomain`, `TestGuessCustomers_*`),
|
||||
`internal/server` (`TestHandleGetPrivacyScan_*`,
|
||||
`TestBuildRawExportBundle_*PrivacyReport*`).
|
||||
|
||||
Reference in New Issue
Block a user