feat(privacy): scan ingested sources for customer-identifying data
Detection-only scan (internal/privacy) attached to every AnalysisResult: a customer-domain guess plus a findings list (category, file, line, match, hint), ported from the KB grep playbook. Runs on archive uploads and the serialized Redfish tree; gated by LOGPILE_PRIVACY_SCAN (default on). Surfaced at GET /api/privacy-scan, in the "Customer data" UI panel, and as privacy_report.json in the raw-export bundle. IP policy keeps RFC1918 and example ranges out of findings; allowlist covers standards-body and vendor infrastructure domains. No customer tokens in the repo. See ADL-066 and bible-local/docs/privacy-scan.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
3311bafd8e
commit
4a4910f207
@@ -23,6 +23,7 @@ All modes converge on the same normalized hardware model and exporter pipeline.
|
||||
- Reopenable raw export bundles for future re-analysis
|
||||
- Reanimator export and batch conversion workflows
|
||||
- Embedded `pci.ids` lookup for vendor/device name enrichment
|
||||
- Customer-data / anonymization scan of ingested sources (detection only; see `docs/privacy-scan.md`)
|
||||
|
||||
## Current vendor coverage
|
||||
|
||||
|
||||
@@ -152,6 +152,13 @@ Returns normalized parse and collection issues combined from:
|
||||
- raw-export collect logs
|
||||
- derived partial-inventory warnings
|
||||
|
||||
### `GET /api/privacy-scan`
|
||||
|
||||
Returns the customer-data scan for the current dataset (`models.PrivacyScan`:
|
||||
`files_scanned`, `customers[]`, `findings[]`, `summary`).
|
||||
Returns `{ "loaded": false }` when nothing is loaded or the scan produced
|
||||
nothing. Detection only; see `docs/privacy-scan.md`.
|
||||
|
||||
### `GET /api/parsers`
|
||||
|
||||
Returns registered parser metadata.
|
||||
@@ -184,6 +191,7 @@ Current implementation emits a ZIP bundle containing:
|
||||
- `raw_export.json`
|
||||
- `collect.log`
|
||||
- `parser_fields.json`
|
||||
- `privacy_report.json` (only when the customer-data scan produced findings)
|
||||
|
||||
### `GET /api/export/reanimator`
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ Current bundle contents:
|
||||
- `raw_export.json`
|
||||
- `collect.log`
|
||||
- `parser_fields.json`
|
||||
- `privacy_report.json` (only when the customer-data scan produced findings; see `docs/privacy-scan.md`)
|
||||
|
||||
Design rules:
|
||||
- raw source is authoritative
|
||||
|
||||
@@ -1893,3 +1893,47 @@ The `inspur` parser still detected it at confidence 100 (every path contains
|
||||
`TestParseMegaRAIDLog`, `TestParseAMISyslog`, `TestDetect_*`,
|
||||
`TestParse_EndToEnd`, and `TestDetectFormat_PriorityBreaksConfidenceTie` in
|
||||
the `parser` package.
|
||||
|
||||
---
|
||||
|
||||
## ADL-066 — Customer-data (privacy) scan of ingested sources
|
||||
|
||||
**Date:** 2026-09-02
|
||||
**Context:** BMC dumps routinely arrive not anonymized. DNS suffix, AD domain,
|
||||
NTP/DNS/syslog host names, timezone, admin e-mails, public IPs, TLS cert CN/SAN
|
||||
and FRU location fields identify the customer and the install site. Before a
|
||||
dump is forwarded to a vendor, a public repo, or the LOGPile test corpus an
|
||||
operator had to run a grep playbook by hand.
|
||||
**Decision:** Every ingested dataset is scanned by `internal/privacy`. The scan
|
||||
is **detection only** - it never rewrites the source. It produces a
|
||||
`models.PrivacyScan` on `AnalysisResult`: a customer guess (registrable domain
|
||||
with confidence + evidence) and a findings list (category, file, line, matched
|
||||
token, remediation hint).
|
||||
- Rule catalogue (`rules.go`) is a port of the KB grep playbook. Categories:
|
||||
domain, resolv, ad_ldap, timezone, email, public_ip, collector, cert,
|
||||
fru_location, hostname, dhcp, nsupdate, mgmt_subdomain.
|
||||
- IP policy (`ip.go`): RFC1918/ULA, loopback, link-local, multicast and the
|
||||
RFC5737/RFC2544/RFC6598 example ranges plus well-known public resolvers are
|
||||
**not** findings. Only routable public addresses are.
|
||||
- Allowlist (`allowlist.go`) is reference data - RFC 2606/5737 names, NTP pools,
|
||||
standards-body and BMC-vendor infrastructure domains, and vendor factory
|
||||
template file-name markers. It is not vendor-detection logic.
|
||||
- No real customer domain or token is committed. Tests use `acme.ru` as the
|
||||
customer stand-in; `example.*` is reserved for the allowlist (it is the
|
||||
sanitization target).
|
||||
- Hook points: `parser.BMCParser.parseFiles` for archives,
|
||||
`ingest.Service.AnalyzeRedfishRawPayloads` for the serialized Redfish tree.
|
||||
Gated by `LOGPILE_PRIVACY_SCAN` (default on).
|
||||
- Surfaced at `GET /api/privacy-scan`, in the "Customer data" UI panel, and as
|
||||
`privacy_report.json` in the raw-export bundle.
|
||||
**Consequences:**
|
||||
- Sanitization stays a manual follow-up guided by the report (no re-archiving).
|
||||
- The customer guess uses a small built-in public-suffix list, not the full PSL.
|
||||
- Binary and over-size files are skipped, so the scan is a floor, not a proof of
|
||||
cleanliness.
|
||||
- Details and the full rule table live in
|
||||
[`docs/privacy-scan.md`](docs/privacy-scan.md).
|
||||
- Tests: `internal/privacy` (`TestScan_*`, `TestIsSensitiveIP`,
|
||||
`TestIsAllowlistedDomain`, `TestRegistrableDomain`, `TestGuessCustomers_*`),
|
||||
`internal/server` (`TestHandleGetPrivacyScan_*`,
|
||||
`TestBuildRawExportBundle_*PrivacyReport*`).
|
||||
|
||||
@@ -22,6 +22,7 @@ Keep top-level docs minimal and put maintained architecture/API contracts here.
|
||||
| [06-parsers.md](06-parsers.md) | Archive parser framework and vendor coverage |
|
||||
| [07-exporters.md](07-exporters.md) | Raw export, Reanimator export, batch convert |
|
||||
| [docs/hardware-ingest-contract.md](docs/hardware-ingest-contract.md) | Reanimator ingest schema mirrored locally |
|
||||
| [docs/privacy-scan.md](docs/privacy-scan.md) | Customer-data / anonymization scan of ingested sources |
|
||||
| [08-build-release.md](08-build-release.md) | Build and release workflow |
|
||||
| [09-testing.md](09-testing.md) | Test expectations and regression rules |
|
||||
| [10-decisions.md](10-decisions.md) | Architectural Decision Log |
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# Privacy / customer-data scan
|
||||
|
||||
`internal/privacy` scans ingested source files for customer-identifying and
|
||||
site-identifying data and attaches a `models.PrivacyScan` to `AnalysisResult`.
|
||||
Detection only - it never modifies the source. See ADL-066.
|
||||
|
||||
## Entry point
|
||||
|
||||
```go
|
||||
privacy.Scan([]privacy.File{{Path, Content}}) *models.PrivacyScan
|
||||
```
|
||||
|
||||
Returns `nil` when there is nothing to scan (no text files). Callers:
|
||||
|
||||
| Source | Hook |
|
||||
|--------|------|
|
||||
| Archive upload | `parser.BMCParser.parseFiles` (`internal/parser/parser.go`) |
|
||||
| Live Redfish / replay | `ingest.Service.AnalyzeRedfishRawPayloads` scans the JSON-serialized `raw_payloads.redfish_tree` |
|
||||
|
||||
Gated by `LOGPILE_PRIVACY_SCAN` (`0`/`false`/`off`/`no` disable; default on),
|
||||
via `parser.PrivacyScanEnabled()`.
|
||||
|
||||
## Output
|
||||
|
||||
- `GET /api/privacy-scan` - the `PrivacyScan` object, or `{"loaded": false}`.
|
||||
- UI: the "Customer data" panel (`#privacy-section`), collapsible, hidden when
|
||||
there are no findings and no customer guess.
|
||||
- `privacy_report.json` in the raw-export ZIP (`GET /api/export/json`), omitted
|
||||
when the scan produced nothing.
|
||||
|
||||
## Scan mechanics
|
||||
|
||||
- Files whose name contains a vendor factory-template marker are skipped whole
|
||||
(`_tencent`, `_jingdong`, `_pdd`, `_baidu`, `_kuaishou`, `_tianyiyun`, `_jd.`,
|
||||
`syslog_jd`, `snmptrapcfg`, `.json_bak`, `ntp_auto`).
|
||||
- Binary content (NUL byte / invalid UTF-8 in the first 8 KiB) is skipped.
|
||||
- Line-oriented; lines longer than 8 KiB are truncated for matching.
|
||||
- Findings are deduped by `category|match|path`; global cap 1000.
|
||||
- A bare-FQDN or e-mail match is dropped when a more precise rule already
|
||||
matched the same token on the same line (e.g. `resolv.conf` `domain x` yields
|
||||
a `resolv` finding, not also a `domain` one).
|
||||
- A match that parses as an IP but is not sensitive (see IP policy) is dropped
|
||||
regardless of the rule that produced it.
|
||||
|
||||
## Rule catalogue (`rules.go`)
|
||||
|
||||
| Category | Severity | Matches | Remediation target |
|
||||
|----------|----------|---------|--------------------|
|
||||
| `domain` | high | bare FQDN with a known TLD | replace with `example.local` |
|
||||
| `resolv` | high | `domain` / `search` / `nameserver` lines | `resolv.conf` |
|
||||
| `ad_ldap` | high | `racdomain=`, `adfilterdcN=`, `rolegroupN{name,domain}=`, `binddn`, `bindpw` | `activedir.conf`, `ldap.conf` |
|
||||
| `mgmt_subdomain` | high | `*.mgmt/oob/ipmi/drac/idrac/ilo/bmc.*` | management-network subdomain |
|
||||
| `nsupdate` | high | `update add|delete <fqdn> ... A|AAAA|PTR|CNAME` | `runningdata/var/tmp/nsupdate_temp` |
|
||||
| `cert` | high | `Subject:`/`Issuer:`/`CN=`/`DNS:` in `*.pem`/`*.csr`/`*.crt`/`*.cer` | certificates |
|
||||
| `timezone` | medium/low | `TimeZone=`, `SELTimeUTCOffset=`, `Continent/City`, localized TZ abbreviation before a year | `Etc/UTC`, offset 0 |
|
||||
| `email` | medium | `local@host.tld` (OData/Redfish annotations and non-TLD hosts filtered) | EventService / LDAP / certs |
|
||||
| `public_ip` | medium | routable IPv4/IPv6 (see IP policy) | RFC 5737 doc ranges |
|
||||
| `collector` | medium | `SyslogHostname=`, `@ip:port`, JSON `"Destination"` | syslog / SNMP trap receiver |
|
||||
| `fru_location` | medium | `Asset Tag`, `Product Location`, `Chassis Location`, `Board Extra` | FRU fields |
|
||||
| `dhcp` | medium | `option domain-name` | customer DHCP lease |
|
||||
| `hostname` | low | `sn-`/`bmc-`/`ilo-`/`idrac-`/`drac-`/`srv-` prefixed names | customer naming scheme |
|
||||
|
||||
## IP policy (`ip.go`)
|
||||
|
||||
Not sensitive: RFC1918 / RFC4193 ULA, loopback, link-local, unspecified,
|
||||
multicast, `0.0.0.0/8`, `255.0.0.0/8` and up; RFC 5737 (`192.0.2/24`,
|
||||
`198.51.100/24`, `203.0.113/24`), RFC 2544 (`198.18/15`), RFC 6598 CGNAT
|
||||
(`100.64/10`), 6to4 relay anycast; well-known public resolvers
|
||||
(`8.8.8.8`, `1.1.1.1`, `9.9.9.9`, `208.67.222.222`, `1.2.3.4`, ...).
|
||||
|
||||
Everything else that is global unicast is a finding.
|
||||
|
||||
## Allowlist (`allowlist.go`)
|
||||
|
||||
Reference data, not vendor-detection logic: RFC 2606/5737 names, `pool.ntp.org`
|
||||
/ `nist.gov`, standards bodies (`dmtf.org`, `oasis-open.org`, `w3.org`,
|
||||
`iana.org`, `ietf.org`), BMC-stack and vendor infrastructure
|
||||
(`megarac.com`, `ami.com`, `openssh.com`, `libssh.org`, `rsyslog.com`,
|
||||
`inspur.com`, `inservice-iq.com`, `kaytus.com`, `jd.com`, `jdcloud.com`),
|
||||
`foobar.edu` / `issue.net` (stock `hosts.allow` / sshd banner), and factory
|
||||
default values (`Asia/Shanghai`, `To Be Filled By O.E.M.`, `NULL`, `0.0.0.0`).
|
||||
|
||||
## Customer guess (`customer.go`)
|
||||
|
||||
Findings in `domain`, `resolv`, `ad_ldap`, `cert`, `nsupdate`,
|
||||
`mgmt_subdomain`, `dhcp`, `email` are reduced to a registrable domain
|
||||
(`eTLD+1`, using a small built-in multi-label suffix list). Score =
|
||||
`hits + 3*distinct_files (+10 if seen in a strong category)`. Confidence:
|
||||
`high` = strong category and >=2 files; `medium` = one of those; `low`
|
||||
otherwise. Top 3 returned, weak single-hit domains dropped unless nothing else
|
||||
qualifies. Up to 3 `path:line: excerpt` evidence strings per domain - the domain
|
||||
itself is kept in the output (identifying the customer is the point); nothing is
|
||||
written to the repo.
|
||||
|
||||
## Limitations
|
||||
|
||||
- No sanitization. The report guides a manual cleanup
|
||||
(KB: "Очистка BMC-дампа от данных заказчика").
|
||||
- Public-suffix handling is a heuristic, not the full PSL.
|
||||
- Binary and over-size (>10 MB extractor cap) files are not scanned - a clean
|
||||
report is a floor, not a proof.
|
||||
- The Redfish path scans only the serialized tree, the one text corpus a
|
||||
live/replayed collection carries.
|
||||
Reference in New Issue
Block a user