# Privacy / customer-data scan

`internal/privacy` scans ingested source files for customer-identifying and
site-identifying data and attaches a `models.PrivacyScan` to `AnalysisResult`.
Detection only - it never modifies the source. See ADL-066.

## Entry point

```go
privacy.Scan([]privacy.File{{Path, Content}}) *models.PrivacyScan
```

Returns `nil` when there is nothing to scan (no text files). Callers:

| Source | Hook |
|--------|------|
| Archive upload | `parser.BMCParser.parseFiles` (`internal/parser/parser.go`) |
| Live Redfish / replay | `ingest.Service.AnalyzeRedfishRawPayloads` scans the JSON-serialized `raw_payloads.redfish_tree` |

Gated by `LOGPILE_PRIVACY_SCAN` (`0`/`false`/`off`/`no` disable; default on),
via `parser.PrivacyScanEnabled()`.

## Output

- `GET /api/privacy-scan` - the `PrivacyScan` object, or `{"loaded": false}`.
- UI: the "Customer data" panel (`#privacy-section`) at the top of the data
  section, above the chart iframe. Header + customer guess are always shown; the
  findings table is collapsed by default and expands on click. Hidden entirely
  when there are no findings and no customer guess.
- The chart viewer also gets a top notice (above Board/CPUs) summarizing the
  scan, via the standard `viewer.RenderOptions.NoticeTitle/NoticeBody`
  (`privacyNotice` in `internal/server/handlers.go`) - interim until the chart
  "custom panels" mechanism ships (see `chart-custom-panels-spec.md`).
- `privacy_report.json` in the raw-export ZIP (`GET /api/export/json`), omitted
  when the scan produced nothing.

## Scan mechanics

- Files whose name contains a vendor factory-template marker are skipped whole
  (`_tencent`, `_jingdong`, `_pdd`, `_baidu`, `_kuaishou`, `_tianyiyun`, `_jd.`,
  `syslog_jd`, `snmptrapcfg`, `.json_bak`, `ntp_auto`).
- Binary content (NUL byte / invalid UTF-8 in the first 8 KiB) is skipped.
- Line-oriented; lines longer than 8 KiB are truncated for matching.
- Findings are deduped by `category|match|path`; global cap 1000.
- A bare-FQDN or e-mail match is dropped when a more precise rule already
  matched the same token on the same line (e.g. `resolv.conf` `domain x` yields
  a `resolv` finding, not also a `domain` one).
- A match that parses as an IP but is not sensitive (see IP policy) is dropped
  regardless of the rule that produced it.
- Noise guards (calibrated against a 130k-file RMA corpus, see ADL-066):
  - **Real-TLD gate** (`tld.go`): a bare FQDN / e-mail host must end in a
    curated TLD (common gTLDs + regional + major-western ccTLDs) or a
    pseudo-TLD (`.local`/`.lan`/... with a >=3-char label). Two-letter
    file-extension / code-identifier suffixes (`.sh`, `.so`, `.md`, `.id`,
    `.service`, ...) are a denylist that always loses. Kills `0.linux`,
    `mountall.sh`, `libc.so`, `serial-getty@ttyAMA0.service`, `@odata.id`.
  - **Clean-token boundary**: an FQDN glued to surrounding identifier chars
    (`auth.backend.gssapi.store-creds`) or Title-cased (`OS.It`) is a code
    fragment, not a host.
  - Kernel ring-buffer lines (`[ 8.07]...`, `[ 8.07][ T1]...`) and Go
    stack-trace lines (`/go-mod/`, `file.go:123`) are skipped for FQDN/email.
  - `resolv` `domain`/`search` values must contain a dot.
  - IPv4: skipped on `#`/`;` comment lines and version/spec-clause lines
    (`X.Org 1.21.1.7`, `IEEE Std ... 8.4.2.97`, `l0fw_ver:[13.16.30.16]`);
    inside a longer run (`18:6.1.4.5`, `v1.2.3.4`); `0.` / `1.` / `.0` quads;
    Yandex/Google/... public resolvers and RFC 3849 `2001:db8::/32`.
  - `fru_location` values that echo the field name, are all-digit, or look
    like the chassis serial are dropped; the `hostname` rule was removed
    (zero real hits, only `bmc-state-manager`-style noise).
- Customer guess: a single-hit low-confidence domain is not reported at all
  (better "unidentified" than guessing `nvidia.com` from a driver comment or
  `smartmontools.org` from a SMART copyright line).

## Rule catalogue (`rules.go`)

| Category | Severity | Matches | Remediation target |
|----------|----------|---------|--------------------|
| `domain` | low / medium | bare FQDN with a real TLD (medium at 3+ labels) | replace with `example.local` |
| `resolv` | high | `domain` / `search` / `nameserver` lines | `resolv.conf` |
| `ad_ldap` | high | `racdomain=`, `adfilterdcN=`, `rolegroupN{name,domain}=`, `binddn`, `bindpw` | `activedir.conf`, `ldap.conf` |
| `mgmt_subdomain` | high | `*.mgmt/oob/ipmi/drac/idrac/ilo/bmc.*` | management-network subdomain |
| `nsupdate` | high | `update add|delete <fqdn> ... A|AAAA|PTR|CNAME` | `runningdata/var/tmp/nsupdate_temp` |
| `cert` | high | `Subject:`/`Issuer:`/`CN=`/`DNS:` in `*.pem`/`*.csr`/`*.crt`/`*.cer` | certificates |
| `timezone` | medium/low | `TimeZone=`, `SELTimeUTCOffset=`, `Continent/City`, localized TZ abbreviation before a year | `Etc/UTC`, offset 0 |
| `email` | medium | `local@host.tld` (OData/Redfish annotations and non-TLD hosts filtered) | EventService / LDAP / certs |
| `public_ip` | medium | routable IPv4/IPv6 (see IP policy) | RFC 5737 doc ranges |
| `collector` | medium | `SyslogHostname=`, `@ip:port`, JSON `"Destination"` | syslog / SNMP trap receiver |
| `fru_location` | medium | `Asset Tag`, `Product Location`, `Chassis Location`, `Board Extra` | FRU fields |
| `dhcp` | medium | `option domain-name` | customer DHCP lease |

## IP policy (`ip.go`)

Not sensitive: RFC1918 / RFC4193 ULA, loopback, link-local, unspecified,
multicast, `0.0.0.0/8`, `255.0.0.0/8` and up; RFC 5737 (`192.0.2/24`,
`198.51.100/24`, `203.0.113/24`), RFC 2544 (`198.18/15`), RFC 6598 CGNAT
(`100.64/10`), 6to4 relay anycast; well-known public resolvers
(`8.8.8.8`, `1.1.1.1`, `9.9.9.9`, `208.67.222.222`, `1.2.3.4`, ...).

Everything else that is global unicast is a finding.

## Allowlist (`allowlist.go`)

Reference data, not vendor-detection logic: RFC 2606/5737 names, `pool.ntp.org`
/ `nist.gov`, standards bodies (`dmtf.org`, `oasis-open.org`, `w3.org`,
`iana.org`, `ietf.org`), BMC-stack and vendor infrastructure
(`megarac.com`, `ami.com`, `openssh.com`, `libssh.org`, `rsyslog.com`,
`inspur.com`, `inservice-iq.com`, `kaytus.com`, `jd.com`, `jdcloud.com`),
`foobar.edu` / `issue.net` (stock `hosts.allow` / sshd banner), and factory
default values (`Asia/Shanghai`, `To Be Filled By O.E.M.`, `NULL`, `0.0.0.0`).

## Redaction

`privacy.FindSpans(line, certFile)` is the matcher shared with
`internal/sanitize`, which rewrites the same spans in place with same-length
neutral fillers (see `log-sanitization.md`). The fillers are recognised by
`isRedactionFiller` so a scan of a sanitized file is clean.

## Already-sanitized detection (`sanitized.go`)

`Scan` also reports whether the source already looks de-identified, in
`PrivacyScan.Sanitized` (`{detected, markers, strong, files, evidence}`).

`SanitizationMarkers(line)` recognises a **value slot filled with one repeated
placeholder + separators** - `xxxxx.xxxx.xx`, `x@xxxx.xxxx.xx`, `000.00.00.0`,
a decoy timezone (`Antarctica/McMurdo`, `Etc/Universal`, ...). It matches the
*shape*, not the literal `x`, so changing the filler character later still
trips it. `0.0.0.0`, `000`, `UTC`, `Etc/UTC` are too plausibly intentional and
do not count.

`detected` requires corroboration - `strong >= 2`, or `strong >= 1 && markers
>= 3`, or `markers >= 4`. A single filler-looking token is reported
(`markers: 1`) but never asserted as sanitized, so a partial future redaction
pass or a coincidence does not read as "done".

## Customer guess (`customer.go`)

Findings in `domain`, `resolv`, `ad_ldap`, `cert`, `nsupdate`,
`mgmt_subdomain`, `dhcp`, `email` are reduced to a registrable domain
(`eTLD+1`, using a small built-in multi-label suffix list). Score =
`hits + 3*distinct_files (+10 if seen in a strong category)`. Confidence:
`high` = strong category and >=2 files; `medium` = one of those; `low`
otherwise. Top 3 returned, weak single-hit domains dropped unless nothing else
qualifies. Up to 3 `path:line: excerpt` evidence strings per domain - the domain
itself is kept in the output (identifying the customer is the point); nothing is
written to the repo.

## Limitations

- No sanitization. The report guides a manual cleanup
  (KB: "Очистка BMC-дампа от данных заказчика").
- Public-suffix handling is a heuristic, not the full PSL.
- Binary and over-size (>10 MB extractor cap) files are not scanned - a clean
  report is a floor, not a proof.
- The Redfish path scans only the serialized tree, the one text corpus a
  live/replayed collection carries.