feat(privacy): scan ingested sources for customer-identifying data
Detection-only scan (internal/privacy) attached to every AnalysisResult: a customer-domain guess plus a findings list (category, file, line, match, hint), ported from the KB grep playbook. Runs on archive uploads and the serialized Redfish tree; gated by LOGPILE_PRIVACY_SCAN (default on). Surfaced at GET /api/privacy-scan, in the "Customer data" UI panel, and as privacy_report.json in the raw-export bundle. IP policy keeps RFC1918 and example ranges out of findings; allowlist covers standards-body and vendor infrastructure domains. No customer tokens in the repo. See ADL-066 and bible-local/docs/privacy-scan.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
3311bafd8e
commit
4a4910f207
@@ -0,0 +1,103 @@
|
||||
# Privacy / customer-data scan
|
||||
|
||||
`internal/privacy` scans ingested source files for customer-identifying and
|
||||
site-identifying data and attaches a `models.PrivacyScan` to `AnalysisResult`.
|
||||
Detection only - it never modifies the source. See ADL-066.
|
||||
|
||||
## Entry point
|
||||
|
||||
```go
|
||||
privacy.Scan([]privacy.File{{Path, Content}}) *models.PrivacyScan
|
||||
```
|
||||
|
||||
Returns `nil` when there is nothing to scan (no text files). Callers:
|
||||
|
||||
| Source | Hook |
|
||||
|--------|------|
|
||||
| Archive upload | `parser.BMCParser.parseFiles` (`internal/parser/parser.go`) |
|
||||
| Live Redfish / replay | `ingest.Service.AnalyzeRedfishRawPayloads` scans the JSON-serialized `raw_payloads.redfish_tree` |
|
||||
|
||||
Gated by `LOGPILE_PRIVACY_SCAN` (`0`/`false`/`off`/`no` disable; default on),
|
||||
via `parser.PrivacyScanEnabled()`.
|
||||
|
||||
## Output
|
||||
|
||||
- `GET /api/privacy-scan` - the `PrivacyScan` object, or `{"loaded": false}`.
|
||||
- UI: the "Customer data" panel (`#privacy-section`), collapsible, hidden when
|
||||
there are no findings and no customer guess.
|
||||
- `privacy_report.json` in the raw-export ZIP (`GET /api/export/json`), omitted
|
||||
when the scan produced nothing.
|
||||
|
||||
## Scan mechanics
|
||||
|
||||
- Files whose name contains a vendor factory-template marker are skipped whole
|
||||
(`_tencent`, `_jingdong`, `_pdd`, `_baidu`, `_kuaishou`, `_tianyiyun`, `_jd.`,
|
||||
`syslog_jd`, `snmptrapcfg`, `.json_bak`, `ntp_auto`).
|
||||
- Binary content (NUL byte / invalid UTF-8 in the first 8 KiB) is skipped.
|
||||
- Line-oriented; lines longer than 8 KiB are truncated for matching.
|
||||
- Findings are deduped by `category|match|path`; global cap 1000.
|
||||
- A bare-FQDN or e-mail match is dropped when a more precise rule already
|
||||
matched the same token on the same line (e.g. `resolv.conf` `domain x` yields
|
||||
a `resolv` finding, not also a `domain` one).
|
||||
- A match that parses as an IP but is not sensitive (see IP policy) is dropped
|
||||
regardless of the rule that produced it.
|
||||
|
||||
## Rule catalogue (`rules.go`)
|
||||
|
||||
| Category | Severity | Matches | Remediation target |
|
||||
|----------|----------|---------|--------------------|
|
||||
| `domain` | high | bare FQDN with a known TLD | replace with `example.local` |
|
||||
| `resolv` | high | `domain` / `search` / `nameserver` lines | `resolv.conf` |
|
||||
| `ad_ldap` | high | `racdomain=`, `adfilterdcN=`, `rolegroupN{name,domain}=`, `binddn`, `bindpw` | `activedir.conf`, `ldap.conf` |
|
||||
| `mgmt_subdomain` | high | `*.mgmt/oob/ipmi/drac/idrac/ilo/bmc.*` | management-network subdomain |
|
||||
| `nsupdate` | high | `update add|delete <fqdn> ... A|AAAA|PTR|CNAME` | `runningdata/var/tmp/nsupdate_temp` |
|
||||
| `cert` | high | `Subject:`/`Issuer:`/`CN=`/`DNS:` in `*.pem`/`*.csr`/`*.crt`/`*.cer` | certificates |
|
||||
| `timezone` | medium/low | `TimeZone=`, `SELTimeUTCOffset=`, `Continent/City`, localized TZ abbreviation before a year | `Etc/UTC`, offset 0 |
|
||||
| `email` | medium | `local@host.tld` (OData/Redfish annotations and non-TLD hosts filtered) | EventService / LDAP / certs |
|
||||
| `public_ip` | medium | routable IPv4/IPv6 (see IP policy) | RFC 5737 doc ranges |
|
||||
| `collector` | medium | `SyslogHostname=`, `@ip:port`, JSON `"Destination"` | syslog / SNMP trap receiver |
|
||||
| `fru_location` | medium | `Asset Tag`, `Product Location`, `Chassis Location`, `Board Extra` | FRU fields |
|
||||
| `dhcp` | medium | `option domain-name` | customer DHCP lease |
|
||||
| `hostname` | low | `sn-`/`bmc-`/`ilo-`/`idrac-`/`drac-`/`srv-` prefixed names | customer naming scheme |
|
||||
|
||||
## IP policy (`ip.go`)
|
||||
|
||||
Not sensitive: RFC1918 / RFC4193 ULA, loopback, link-local, unspecified,
|
||||
multicast, `0.0.0.0/8`, `255.0.0.0/8` and up; RFC 5737 (`192.0.2/24`,
|
||||
`198.51.100/24`, `203.0.113/24`), RFC 2544 (`198.18/15`), RFC 6598 CGNAT
|
||||
(`100.64/10`), 6to4 relay anycast; well-known public resolvers
|
||||
(`8.8.8.8`, `1.1.1.1`, `9.9.9.9`, `208.67.222.222`, `1.2.3.4`, ...).
|
||||
|
||||
Everything else that is global unicast is a finding.
|
||||
|
||||
## Allowlist (`allowlist.go`)
|
||||
|
||||
Reference data, not vendor-detection logic: RFC 2606/5737 names, `pool.ntp.org`
|
||||
/ `nist.gov`, standards bodies (`dmtf.org`, `oasis-open.org`, `w3.org`,
|
||||
`iana.org`, `ietf.org`), BMC-stack and vendor infrastructure
|
||||
(`megarac.com`, `ami.com`, `openssh.com`, `libssh.org`, `rsyslog.com`,
|
||||
`inspur.com`, `inservice-iq.com`, `kaytus.com`, `jd.com`, `jdcloud.com`),
|
||||
`foobar.edu` / `issue.net` (stock `hosts.allow` / sshd banner), and factory
|
||||
default values (`Asia/Shanghai`, `To Be Filled By O.E.M.`, `NULL`, `0.0.0.0`).
|
||||
|
||||
## Customer guess (`customer.go`)
|
||||
|
||||
Findings in `domain`, `resolv`, `ad_ldap`, `cert`, `nsupdate`,
|
||||
`mgmt_subdomain`, `dhcp`, `email` are reduced to a registrable domain
|
||||
(`eTLD+1`, using a small built-in multi-label suffix list). Score =
|
||||
`hits + 3*distinct_files (+10 if seen in a strong category)`. Confidence:
|
||||
`high` = strong category and >=2 files; `medium` = one of those; `low`
|
||||
otherwise. Top 3 returned, weak single-hit domains dropped unless nothing else
|
||||
qualifies. Up to 3 `path:line: excerpt` evidence strings per domain - the domain
|
||||
itself is kept in the output (identifying the customer is the point); nothing is
|
||||
written to the repo.
|
||||
|
||||
## Limitations
|
||||
|
||||
- No sanitization. The report guides a manual cleanup
|
||||
(KB: "Очистка BMC-дампа от данных заказчика").
|
||||
- Public-suffix handling is a heuristic, not the full PSL.
|
||||
- Binary and over-size (>10 MB extractor cap) files are not scanned - a clean
|
||||
report is a floor, not a proof.
|
||||
- The Redfish path scans only the serialized tree, the one text corpus a
|
||||
live/replayed collection carries.
|
||||
Reference in New Issue
Block a user