Mikhail ChusavitinandClaude Sonnet 5
f38fb2de69
Scan now reports PrivacyScan.Sanitized {detected, markers, strong, files,
evidence}. SanitizationMarkers recognises a value slot filled with one
repeated placeholder + separators (xxxxx.xxxx.xx, x@xxxx.xxxx.xx,
000.00.00.0, a decoy timezone) - it matches the shape, not the literal "x",
so evolving the redaction mechanism still trips it.
detected requires corroboration: strong>=2, or strong>=1 && markers>=3, or
markers>=4. A single filler-looking token is reported (markers:1) but never
asserted as sanitized, so a partial future pass or a coincidence does not
read as "done". 0.0.0.0 / 000 / UTC / Etc/UTC are too plausibly intentional
and do not count.
UI: the Customer-data panel shows "файл уже обезличен" and hides the
sanitize button when detected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jDYM1nnoZZ3vFz23DDaV1
156 lines
8.3 KiB
Markdown
156 lines
8.3 KiB
Markdown
# Privacy / customer-data scan
|
|
|
|
`internal/privacy` scans ingested source files for customer-identifying and
|
|
site-identifying data and attaches a `models.PrivacyScan` to `AnalysisResult`.
|
|
Detection only - it never modifies the source. See ADL-066.
|
|
|
|
## Entry point
|
|
|
|
```go
|
|
privacy.Scan([]privacy.File{{Path, Content}}) *models.PrivacyScan
|
|
```
|
|
|
|
Returns `nil` when there is nothing to scan (no text files). Callers:
|
|
|
|
| Source | Hook |
|
|
|--------|------|
|
|
| Archive upload | `parser.BMCParser.parseFiles` (`internal/parser/parser.go`) |
|
|
| Live Redfish / replay | `ingest.Service.AnalyzeRedfishRawPayloads` scans the JSON-serialized `raw_payloads.redfish_tree` |
|
|
|
|
Gated by `LOGPILE_PRIVACY_SCAN` (`0`/`false`/`off`/`no` disable; default on),
|
|
via `parser.PrivacyScanEnabled()`.
|
|
|
|
## Output
|
|
|
|
- `GET /api/privacy-scan` - the `PrivacyScan` object, or `{"loaded": false}`.
|
|
- UI: the "Customer data" panel (`#privacy-section`) at the top of the data
|
|
section, above the chart iframe. Header + customer guess are always shown; the
|
|
findings table is collapsed by default and expands on click. Hidden entirely
|
|
when there are no findings and no customer guess.
|
|
- The chart viewer also gets a top notice (above Board/CPUs) summarizing the
|
|
scan, via the standard `viewer.RenderOptions.NoticeTitle/NoticeBody`
|
|
(`privacyNotice` in `internal/server/handlers.go`) - interim until the chart
|
|
"custom panels" mechanism ships (see `chart-custom-panels-spec.md`).
|
|
- `privacy_report.json` in the raw-export ZIP (`GET /api/export/json`), omitted
|
|
when the scan produced nothing.
|
|
|
|
## Scan mechanics
|
|
|
|
- Files whose name contains a vendor factory-template marker are skipped whole
|
|
(`_tencent`, `_jingdong`, `_pdd`, `_baidu`, `_kuaishou`, `_tianyiyun`, `_jd.`,
|
|
`syslog_jd`, `snmptrapcfg`, `.json_bak`, `ntp_auto`).
|
|
- Binary content (NUL byte / invalid UTF-8 in the first 8 KiB) is skipped.
|
|
- Line-oriented; lines longer than 8 KiB are truncated for matching.
|
|
- Findings are deduped by `category|match|path`; global cap 1000.
|
|
- A bare-FQDN or e-mail match is dropped when a more precise rule already
|
|
matched the same token on the same line (e.g. `resolv.conf` `domain x` yields
|
|
a `resolv` finding, not also a `domain` one).
|
|
- A match that parses as an IP but is not sensitive (see IP policy) is dropped
|
|
regardless of the rule that produced it.
|
|
- Noise guards (calibrated against a 130k-file RMA corpus, see ADL-066):
|
|
- **Real-TLD gate** (`tld.go`): a bare FQDN / e-mail host must end in a
|
|
curated TLD (common gTLDs + regional + major-western ccTLDs) or a
|
|
pseudo-TLD (`.local`/`.lan`/... with a >=3-char label). Two-letter
|
|
file-extension / code-identifier suffixes (`.sh`, `.so`, `.md`, `.id`,
|
|
`.service`, ...) are a denylist that always loses. Kills `0.linux`,
|
|
`mountall.sh`, `libc.so`, `serial-getty@ttyAMA0.service`, `@odata.id`.
|
|
- **Clean-token boundary**: an FQDN glued to surrounding identifier chars
|
|
(`auth.backend.gssapi.store-creds`) or Title-cased (`OS.It`) is a code
|
|
fragment, not a host.
|
|
- Kernel ring-buffer lines (`[ 8.07]...`, `[ 8.07][ T1]...`) and Go
|
|
stack-trace lines (`/go-mod/`, `file.go:123`) are skipped for FQDN/email.
|
|
- `resolv` `domain`/`search` values must contain a dot.
|
|
- IPv4: skipped on `#`/`;` comment lines and version/spec-clause lines
|
|
(`X.Org 1.21.1.7`, `IEEE Std ... 8.4.2.97`, `l0fw_ver:[13.16.30.16]`);
|
|
inside a longer run (`18:6.1.4.5`, `v1.2.3.4`); `0.` / `1.` / `.0` quads;
|
|
Yandex/Google/... public resolvers and RFC 3849 `2001:db8::/32`.
|
|
- `fru_location` values that echo the field name, are all-digit, or look
|
|
like the chassis serial are dropped; the `hostname` rule was removed
|
|
(zero real hits, only `bmc-state-manager`-style noise).
|
|
- Customer guess: a single-hit low-confidence domain is not reported at all
|
|
(better "unidentified" than guessing `nvidia.com` from a driver comment or
|
|
`smartmontools.org` from a SMART copyright line).
|
|
|
|
## Rule catalogue (`rules.go`)
|
|
|
|
| Category | Severity | Matches | Remediation target |
|
|
|----------|----------|---------|--------------------|
|
|
| `domain` | low / medium | bare FQDN with a real TLD (medium at 3+ labels) | replace with `example.local` |
|
|
| `resolv` | high | `domain` / `search` / `nameserver` lines | `resolv.conf` |
|
|
| `ad_ldap` | high | `racdomain=`, `adfilterdcN=`, `rolegroupN{name,domain}=`, `binddn`, `bindpw` | `activedir.conf`, `ldap.conf` |
|
|
| `mgmt_subdomain` | high | `*.mgmt/oob/ipmi/drac/idrac/ilo/bmc.*` | management-network subdomain |
|
|
| `nsupdate` | high | `update add|delete <fqdn> ... A|AAAA|PTR|CNAME` | `runningdata/var/tmp/nsupdate_temp` |
|
|
| `cert` | high | `Subject:`/`Issuer:`/`CN=`/`DNS:` in `*.pem`/`*.csr`/`*.crt`/`*.cer` | certificates |
|
|
| `timezone` | medium/low | `TimeZone=`, `SELTimeUTCOffset=`, `Continent/City`, localized TZ abbreviation before a year | `Etc/UTC`, offset 0 |
|
|
| `email` | medium | `local@host.tld` (OData/Redfish annotations and non-TLD hosts filtered) | EventService / LDAP / certs |
|
|
| `public_ip` | medium | routable IPv4/IPv6 (see IP policy) | RFC 5737 doc ranges |
|
|
| `collector` | medium | `SyslogHostname=`, `@ip:port`, JSON `"Destination"` | syslog / SNMP trap receiver |
|
|
| `fru_location` | medium | `Asset Tag`, `Product Location`, `Chassis Location`, `Board Extra` | FRU fields |
|
|
| `dhcp` | medium | `option domain-name` | customer DHCP lease |
|
|
|
|
## IP policy (`ip.go`)
|
|
|
|
Not sensitive: RFC1918 / RFC4193 ULA, loopback, link-local, unspecified,
|
|
multicast, `0.0.0.0/8`, `255.0.0.0/8` and up; RFC 5737 (`192.0.2/24`,
|
|
`198.51.100/24`, `203.0.113/24`), RFC 2544 (`198.18/15`), RFC 6598 CGNAT
|
|
(`100.64/10`), 6to4 relay anycast; well-known public resolvers
|
|
(`8.8.8.8`, `1.1.1.1`, `9.9.9.9`, `208.67.222.222`, `1.2.3.4`, ...).
|
|
|
|
Everything else that is global unicast is a finding.
|
|
|
|
## Allowlist (`allowlist.go`)
|
|
|
|
Reference data, not vendor-detection logic: RFC 2606/5737 names, `pool.ntp.org`
|
|
/ `nist.gov`, standards bodies (`dmtf.org`, `oasis-open.org`, `w3.org`,
|
|
`iana.org`, `ietf.org`), BMC-stack and vendor infrastructure
|
|
(`megarac.com`, `ami.com`, `openssh.com`, `libssh.org`, `rsyslog.com`,
|
|
`inspur.com`, `inservice-iq.com`, `kaytus.com`, `jd.com`, `jdcloud.com`),
|
|
`foobar.edu` / `issue.net` (stock `hosts.allow` / sshd banner), and factory
|
|
default values (`Asia/Shanghai`, `To Be Filled By O.E.M.`, `NULL`, `0.0.0.0`).
|
|
|
|
## Redaction
|
|
|
|
`privacy.FindSpans(line, certFile)` is the matcher shared with
|
|
`internal/sanitize`, which rewrites the same spans in place with same-length
|
|
neutral fillers (see `log-sanitization.md`). The fillers are recognised by
|
|
`isRedactionFiller` so a scan of a sanitized file is clean.
|
|
|
|
## Already-sanitized detection (`sanitized.go`)
|
|
|
|
`Scan` also reports whether the source already looks de-identified, in
|
|
`PrivacyScan.Sanitized` (`{detected, markers, strong, files, evidence}`).
|
|
|
|
`SanitizationMarkers(line)` recognises a **value slot filled with one repeated
|
|
placeholder + separators** - `xxxxx.xxxx.xx`, `x@xxxx.xxxx.xx`, `000.00.00.0`,
|
|
a decoy timezone (`Antarctica/McMurdo`, `Etc/Universal`, ...). It matches the
|
|
*shape*, not the literal `x`, so changing the filler character later still
|
|
trips it. `0.0.0.0`, `000`, `UTC`, `Etc/UTC` are too plausibly intentional and
|
|
do not count.
|
|
|
|
`detected` requires corroboration - `strong >= 2`, or `strong >= 1 && markers
|
|
>= 3`, or `markers >= 4`. A single filler-looking token is reported
|
|
(`markers: 1`) but never asserted as sanitized, so a partial future redaction
|
|
pass or a coincidence does not read as "done".
|
|
|
|
## Customer guess (`customer.go`)
|
|
|
|
Findings in `domain`, `resolv`, `ad_ldap`, `cert`, `nsupdate`,
|
|
`mgmt_subdomain`, `dhcp`, `email` are reduced to a registrable domain
|
|
(`eTLD+1`, using a small built-in multi-label suffix list). Score =
|
|
`hits + 3*distinct_files (+10 if seen in a strong category)`. Confidence:
|
|
`high` = strong category and >=2 files; `medium` = one of those; `low`
|
|
otherwise. Top 3 returned, weak single-hit domains dropped unless nothing else
|
|
qualifies. Up to 3 `path:line: excerpt` evidence strings per domain - the domain
|
|
itself is kept in the output (identifying the customer is the point); nothing is
|
|
written to the repo.
|
|
|
|
## Limitations
|
|
|
|
- No sanitization. The report guides a manual cleanup
|
|
(KB: "Очистка BMC-дампа от данных заказчика").
|
|
- Public-suffix handling is a heuristic, not the full PSL.
|
|
- Binary and over-size (>10 MB extractor cap) files are not scanned - a clean
|
|
report is a floor, not a proof.
|
|
- The Redfish path scans only the serialized tree, the one text corpus a
|
|
live/replayed collection carries.
|