fix(privacy): calibrate against the RMA log corpus (130k files)
Cross-checked internal/privacy against the batch companion-app report over project/rma. Fixes for the dominant false-positive classes: - Real-TLD gate (tld.go): FQDN/e-mail must end in a curated TLD or a pseudo-TLD with a >=3-char label; two-letter file/code suffixes (.sh .so .md .id .service ...) are a hard denylist. Kills "0.linux" (45996 hits), "mountall.sh", "libc.so", "@odata.id", "serial-getty@ttyAMA0.service". - Clean-token boundary + Title-case reject: "auth.backend.gssapi.store-creds", "OS.It" are code, not hosts. - Kernel ring-buffer ("[ 8.07][ T1] ...") and Go stack-trace lines skipped. - resolv domain/search values must contain a dot ("domain 53" -> out). - IPv4: skip comment lines, version/spec lines (X.Org, IEEE Std, l0fw_ver), "0."/"1."/".0" quads; allowlist Yandex resolvers + RFC3849 2001:db8::/32. - fru_location: drop all-digit / serial-like / field-name-echo values. - Drop the hostname rule (zero real hits, only "bmc-state-manager" noise). - domain category: high -> low, medium at 3+ labels. Real customer signal now comes from resolv/nsupdate/ad_ldap/mgmt, which the corpus confirms catches every actual customer (netwell.local, tcsbank.ru). - Allowlist smartmontools.org, openib.org, apache.org, freebsd.org, golang.org, ipxe.org, nvidia.com and other FOSS/vendor infra; skip LOGPile's own raw_export.json / parser_fields.json / collect.log members. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
551d8e450d
commit
2be215fca1
@@ -47,19 +47,35 @@ via `parser.PrivacyScanEnabled()`.
|
||||
a `resolv` finding, not also a `domain` one).
|
||||
- A match that parses as an IP but is not sensitive (see IP policy) is dropped
|
||||
regardless of the rule that produced it.
|
||||
- Noise guards: matches shorter than 2 chars or with no alphanumeric are
|
||||
dropped; `fru_location` values that echo the field name (`Base Board Asset
|
||||
Tag`, `P1-DIMMA1_AssetTag`) are dropped; IPv4 inside a longer dotted-number
|
||||
run (`18:6.1.4.5`) is not an address; e-mail on a kernel ring-buffer line
|
||||
(`[ 8.07] ...`) is driver/copyright text, not customer contact.
|
||||
- Noise guards (calibrated against a 130k-file RMA corpus, see ADL-066):
|
||||
- **Real-TLD gate** (`tld.go`): a bare FQDN / e-mail host must end in a
|
||||
curated TLD (common gTLDs + regional + major-western ccTLDs) or a
|
||||
pseudo-TLD (`.local`/`.lan`/... with a >=3-char label). Two-letter
|
||||
file-extension / code-identifier suffixes (`.sh`, `.so`, `.md`, `.id`,
|
||||
`.service`, ...) are a denylist that always loses. Kills `0.linux`,
|
||||
`mountall.sh`, `libc.so`, `serial-getty@ttyAMA0.service`, `@odata.id`.
|
||||
- **Clean-token boundary**: an FQDN glued to surrounding identifier chars
|
||||
(`auth.backend.gssapi.store-creds`) or Title-cased (`OS.It`) is a code
|
||||
fragment, not a host.
|
||||
- Kernel ring-buffer lines (`[ 8.07]...`, `[ 8.07][ T1]...`) and Go
|
||||
stack-trace lines (`/go-mod/`, `file.go:123`) are skipped for FQDN/email.
|
||||
- `resolv` `domain`/`search` values must contain a dot.
|
||||
- IPv4: skipped on `#`/`;` comment lines and version/spec-clause lines
|
||||
(`X.Org 1.21.1.7`, `IEEE Std ... 8.4.2.97`, `l0fw_ver:[13.16.30.16]`);
|
||||
inside a longer run (`18:6.1.4.5`, `v1.2.3.4`); `0.` / `1.` / `.0` quads;
|
||||
Yandex/Google/... public resolvers and RFC 3849 `2001:db8::/32`.
|
||||
- `fru_location` values that echo the field name, are all-digit, or look
|
||||
like the chassis serial are dropped; the `hostname` rule was removed
|
||||
(zero real hits, only `bmc-state-manager`-style noise).
|
||||
- Customer guess: a single-hit low-confidence domain is not reported at all
|
||||
(better "unidentified" than guessing `nvidia.com` from a driver comment).
|
||||
(better "unidentified" than guessing `nvidia.com` from a driver comment or
|
||||
`smartmontools.org` from a SMART copyright line).
|
||||
|
||||
## Rule catalogue (`rules.go`)
|
||||
|
||||
| Category | Severity | Matches | Remediation target |
|
||||
|----------|----------|---------|--------------------|
|
||||
| `domain` | high | bare FQDN with a known TLD | replace with `example.local` |
|
||||
| `domain` | low / medium | bare FQDN with a real TLD (medium at 3+ labels) | replace with `example.local` |
|
||||
| `resolv` | high | `domain` / `search` / `nameserver` lines | `resolv.conf` |
|
||||
| `ad_ldap` | high | `racdomain=`, `adfilterdcN=`, `rolegroupN{name,domain}=`, `binddn`, `bindpw` | `activedir.conf`, `ldap.conf` |
|
||||
| `mgmt_subdomain` | high | `*.mgmt/oob/ipmi/drac/idrac/ilo/bmc.*` | management-network subdomain |
|
||||
@@ -71,7 +87,6 @@ via `parser.PrivacyScanEnabled()`.
|
||||
| `collector` | medium | `SyslogHostname=`, `@ip:port`, JSON `"Destination"` | syslog / SNMP trap receiver |
|
||||
| `fru_location` | medium | `Asset Tag`, `Product Location`, `Chassis Location`, `Board Extra` | FRU fields |
|
||||
| `dhcp` | medium | `option domain-name` | customer DHCP lease |
|
||||
| `hostname` | low | `sn-`/`bmc-`/`ilo-`/`idrac-`/`drac-`/`srv-` prefixed names | customer naming scheme |
|
||||
|
||||
## IP policy (`ip.go`)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user