fix(privacy): calibrate against the RMA log corpus (130k files)

Cross-checked internal/privacy against the batch companion-app report over
project/rma. Fixes for the dominant false-positive classes:

- Real-TLD gate (tld.go): FQDN/e-mail must end in a curated TLD or a
  pseudo-TLD with a >=3-char label; two-letter file/code suffixes
  (.sh .so .md .id .service ...) are a hard denylist. Kills "0.linux"
  (45996 hits), "mountall.sh", "libc.so", "@odata.id",
  "serial-getty@ttyAMA0.service".
- Clean-token boundary + Title-case reject: "auth.backend.gssapi.store-creds",
  "OS.It" are code, not hosts.
- Kernel ring-buffer ("[ 8.07][ T1] ...") and Go stack-trace lines skipped.
- resolv domain/search values must contain a dot ("domain 53" -> out).
- IPv4: skip comment lines, version/spec lines (X.Org, IEEE Std, l0fw_ver),
  "0."/"1."/".0" quads; allowlist Yandex resolvers + RFC3849 2001:db8::/32.
- fru_location: drop all-digit / serial-like / field-name-echo values.
- Drop the hostname rule (zero real hits, only "bmc-state-manager" noise).
- domain category: high -> low, medium at 3+ labels. Real customer signal
  now comes from resolv/nsupdate/ad_ldap/mgmt, which the corpus confirms
  catches every actual customer (netwell.local, tcsbank.ru).
- Allowlist smartmontools.org, openib.org, apache.org, freebsd.org,
  golang.org, ipxe.org, nvidia.com and other FOSS/vendor infra; skip
  LOGPile's own raw_export.json / parser_fields.json / collect.log members.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-09-02 16:45:35 +03:00
co-authored by Claude Sonnet 5
parent 551d8e450d
commit 2be215fca1
7 changed files with 296 additions and 75 deletions
+3 -1
View File
@@ -12,6 +12,7 @@ var nonSensitiveNets = func() []*net.IPNet {
"198.18.0.0/15", // RFC 2544 benchmarking
"100.64.0.0/10", // RFC 6598 CGNAT
"192.88.99.0/24", // RFC 7526 6to4 relay anycast
"2001:db8::/32", // RFC 3849 documentation
}
out := make([]*net.IPNet, 0, len(cidrs))
for _, c := range cidrs {
@@ -24,8 +25,9 @@ var nonSensitiveNets = func() []*net.IPNet {
var nonSensitiveExact = map[string]struct{}{
"8.8.8.8": {}, "8.8.4.4": {}, "1.1.1.1": {}, "1.0.0.1": {},
"4.2.2.2": {}, "4.2.2.1": {}, "9.9.9.9": {}, "1.2.3.4": {},
"4.2.2.2": {}, "4.2.2.1": {}, "9.9.9.9": {}, "1.2.3.4": {}, "2.4.6.8": {},
"208.67.222.222": {}, "208.67.220.220": {},
"77.88.8.8": {}, "77.88.8.1": {}, "77.88.8.88": {}, // Yandex public DNS
}
// isSensitiveIP reports whether s is a routable address that could identify the