fix(privacy): calibrate against the RMA log corpus (130k files)

Cross-checked internal/privacy against the batch companion-app report over
project/rma. Fixes for the dominant false-positive classes:

- Real-TLD gate (tld.go): FQDN/e-mail must end in a curated TLD or a
  pseudo-TLD with a >=3-char label; two-letter file/code suffixes
  (.sh .so .md .id .service ...) are a hard denylist. Kills "0.linux"
  (45996 hits), "mountall.sh", "libc.so", "@odata.id",
  "serial-getty@ttyAMA0.service".
- Clean-token boundary + Title-case reject: "auth.backend.gssapi.store-creds",
  "OS.It" are code, not hosts.
- Kernel ring-buffer ("[ 8.07][ T1] ...") and Go stack-trace lines skipped.
- resolv domain/search values must contain a dot ("domain 53" -> out).
- IPv4: skip comment lines, version/spec lines (X.Org, IEEE Std, l0fw_ver),
  "0."/"1."/".0" quads; allowlist Yandex resolvers + RFC3849 2001:db8::/32.
- fru_location: drop all-digit / serial-like / field-name-echo values.
- Drop the hostname rule (zero real hits, only "bmc-state-manager" noise).
- domain category: high -> low, medium at 3+ labels. Real customer signal
  now comes from resolv/nsupdate/ad_ldap/mgmt, which the corpus confirms
  catches every actual customer (netwell.local, tcsbank.ru).
- Allowlist smartmontools.org, openib.org, apache.org, freebsd.org,
  golang.org, ipxe.org, nvidia.com and other FOSS/vendor infra; skip
  LOGPile's own raw_export.json / parser_fields.json / collect.log members.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-09-02 16:45:35 +03:00
co-authored by Claude Sonnet 5
parent 551d8e450d
commit 2be215fca1
7 changed files with 296 additions and 75 deletions
+23 -8
View File
@@ -47,19 +47,35 @@ via `parser.PrivacyScanEnabled()`.
a `resolv` finding, not also a `domain` one).
- A match that parses as an IP but is not sensitive (see IP policy) is dropped
regardless of the rule that produced it.
- Noise guards: matches shorter than 2 chars or with no alphanumeric are
dropped; `fru_location` values that echo the field name (`Base Board Asset
Tag`, `P1-DIMMA1_AssetTag`) are dropped; IPv4 inside a longer dotted-number
run (`18:6.1.4.5`) is not an address; e-mail on a kernel ring-buffer line
(`[ 8.07] ...`) is driver/copyright text, not customer contact.
- Noise guards (calibrated against a 130k-file RMA corpus, see ADL-066):
- **Real-TLD gate** (`tld.go`): a bare FQDN / e-mail host must end in a
curated TLD (common gTLDs + regional + major-western ccTLDs) or a
pseudo-TLD (`.local`/`.lan`/... with a >=3-char label). Two-letter
file-extension / code-identifier suffixes (`.sh`, `.so`, `.md`, `.id`,
`.service`, ...) are a denylist that always loses. Kills `0.linux`,
`mountall.sh`, `libc.so`, `serial-getty@ttyAMA0.service`, `@odata.id`.
- **Clean-token boundary**: an FQDN glued to surrounding identifier chars
(`auth.backend.gssapi.store-creds`) or Title-cased (`OS.It`) is a code
fragment, not a host.
- Kernel ring-buffer lines (`[ 8.07]...`, `[ 8.07][ T1]...`) and Go
stack-trace lines (`/go-mod/`, `file.go:123`) are skipped for FQDN/email.
- `resolv` `domain`/`search` values must contain a dot.
- IPv4: skipped on `#`/`;` comment lines and version/spec-clause lines
(`X.Org 1.21.1.7`, `IEEE Std ... 8.4.2.97`, `l0fw_ver:[13.16.30.16]`);
inside a longer run (`18:6.1.4.5`, `v1.2.3.4`); `0.` / `1.` / `.0` quads;
Yandex/Google/... public resolvers and RFC 3849 `2001:db8::/32`.
- `fru_location` values that echo the field name, are all-digit, or look
like the chassis serial are dropped; the `hostname` rule was removed
(zero real hits, only `bmc-state-manager`-style noise).
- Customer guess: a single-hit low-confidence domain is not reported at all
(better "unidentified" than guessing `nvidia.com` from a driver comment).
(better "unidentified" than guessing `nvidia.com` from a driver comment or
`smartmontools.org` from a SMART copyright line).
## Rule catalogue (`rules.go`)
| Category | Severity | Matches | Remediation target |
|----------|----------|---------|--------------------|
| `domain` | high | bare FQDN with a known TLD | replace with `example.local` |
| `domain` | low / medium | bare FQDN with a real TLD (medium at 3+ labels) | replace with `example.local` |
| `resolv` | high | `domain` / `search` / `nameserver` lines | `resolv.conf` |
| `ad_ldap` | high | `racdomain=`, `adfilterdcN=`, `rolegroupN{name,domain}=`, `binddn`, `bindpw` | `activedir.conf`, `ldap.conf` |
| `mgmt_subdomain` | high | `*.mgmt/oob/ipmi/drac/idrac/ilo/bmc.*` | management-network subdomain |
@@ -71,7 +87,6 @@ via `parser.PrivacyScanEnabled()`.
| `collector` | medium | `SyslogHostname=`, `@ip:port`, JSON `"Destination"` | syslog / SNMP trap receiver |
| `fru_location` | medium | `Asset Tag`, `Product Location`, `Chassis Location`, `Board Extra` | FRU fields |
| `dhcp` | medium | `option domain-name` | customer DHCP lease |
| `hostname` | low | `sn-`/`bmc-`/`ilo-`/`idrac-`/`drac-`/`srv-` prefixed names | customer naming scheme |
## IP policy (`ip.go`)