logpile/bible-local/docs/privacy-scan.md

Privacy / customer-data scan

internal/privacy scans ingested source files for customer-identifying and site-identifying data and attaches a models.PrivacyScan to AnalysisResult. Detection only - it never modifies the source. See ADL-066.

Entry point

privacy.Scan([]privacy.File{{Path, Content}}) *models.PrivacyScan

Returns nil when there is nothing to scan (no text files). Callers:

Source	Hook
Archive upload	parser.BMCParser.parseFiles (internal/parser/parser.go)
Live Redfish / replay	ingest.Service.AnalyzeRedfishRawPayloads scans the JSON-serialized raw_payloads.redfish_tree

Gated by LOGPILE_PRIVACY_SCAN (0/false/off/no disable; default on), via parser.PrivacyScanEnabled().

Output

• GET /api/privacy-scan - the PrivacyScan object, or {"loaded": false}.
• UI: the "Customer data" panel (#privacy-section) at the top of the data section, above the chart iframe. Header + customer guess are always shown; the findings table is collapsed by default and expands on click. Hidden entirely when there are no findings and no customer guess.
• The chart viewer also gets a top notice (above Board/CPUs) summarizing the scan, via the standard viewer.RenderOptions.NoticeTitle/NoticeBody (privacyNotice in internal/server/handlers.go) - interim until the chart "custom panels" mechanism ships (see chart-custom-panels-spec.md).
• privacy_report.json in the raw-export ZIP (GET /api/export/json), omitted when the scan produced nothing.

Scan mechanics

• Files whose name contains a vendor factory-template marker are skipped whole (_tencent, _jingdong, _pdd, _baidu, _kuaishou, _tianyiyun, _jd., syslog_jd, snmptrapcfg, .json_bak, ntp_auto).
• Binary content (NUL byte / invalid UTF-8 in the first 8 KiB) is skipped.
• Line-oriented; lines longer than 8 KiB are truncated for matching.
• Findings are deduped by category|match|path; global cap 1000.
• A bare-FQDN or e-mail match is dropped when a more precise rule already matched the same token on the same line (e.g. resolv.conf domain x yields a resolv finding, not also a domain one).
• A match that parses as an IP but is not sensitive (see IP policy) is dropped regardless of the rule that produced it.
• Noise guards (calibrated against a 130k-file RMA corpus, see ADL-066):
  • Real-TLD gate (tld.go): a bare FQDN / e-mail host must end in a curated TLD (common gTLDs + regional + major-western ccTLDs) or a pseudo-TLD (.local/.lan/... with a >=3-char label). Two-letter file-extension / code-identifier suffixes (.sh, .so, .md, .id, .service, ...) are a denylist that always loses. Kills 0.linux, mountall.sh, libc.so, serial-getty@ttyAMA0.service, @odata.id.
  • Clean-token boundary: an FQDN glued to surrounding identifier chars (auth.backend.gssapi.store-creds) or Title-cased (OS.It) is a code fragment, not a host.
  • Kernel ring-buffer lines ([ 8.07]..., [ 8.07][ T1]...) and Go stack-trace lines (/go-mod/, file.go:123) are skipped for FQDN/email.
  • resolv domain/search values must contain a dot.
  • IPv4: skipped on #/; comment lines and version/spec-clause lines (X.Org 1.21.1.7, IEEE Std ... 8.4.2.97, l0fw_ver:[13.16.30.16]); inside a longer run (18:6.1.4.5, v1.2.3.4); 0. / 1. / .0 quads; Yandex/Google/... public resolvers and RFC 3849 2001:db8::/32.
  • fru_location values that echo the field name, are all-digit, or look like the chassis serial are dropped; the hostname rule was removed (zero real hits, only bmc-state-manager-style noise).
• Customer guess: a single-hit low-confidence domain is not reported at all (better "unidentified" than guessing nvidia.com from a driver comment or smartmontools.org from a SMART copyright line).

Rule catalogue (rules.go)

Category	Severity	Matches	Remediation target
domain	low / medium	bare FQDN with a real TLD (medium at 3+ labels)	replace with example.local
resolv	high	domain / search / nameserver lines	resolv.conf
ad_ldap	high	racdomain=, adfilterdcN=, rolegroupN{name,domain}=, binddn, bindpw	activedir.conf, ldap.conf
mgmt_subdomain	high	*.mgmt/oob/ipmi/drac/idrac/ilo/bmc.*	management-network subdomain
nsupdate	high	`update add	delete ... A
cert	high	Subject:/Issuer:/CN=/DNS: in *.pem/*.csr/*.crt/*.cer	certificates
timezone	medium/low	TimeZone=, SELTimeUTCOffset=, Continent/City, localized TZ abbreviation before a year	Etc/UTC, offset 0
email	medium	local@host.tld (OData/Redfish annotations and non-TLD hosts filtered)	EventService / LDAP / certs
public_ip	medium	routable IPv4/IPv6 (see IP policy)	RFC 5737 doc ranges
collector	medium	SyslogHostname=, @ip:port, JSON "Destination"	syslog / SNMP trap receiver
fru_location	medium	Asset Tag, Product Location, Chassis Location, Board Extra	FRU fields
dhcp	medium	option domain-name	customer DHCP lease

IP policy (ip.go)

Not sensitive: RFC1918 / RFC4193 ULA, loopback, link-local, unspecified, multicast, 0.0.0.0/8, 255.0.0.0/8 and up; RFC 5737 (192.0.2/24, 198.51.100/24, 203.0.113/24), RFC 2544 (198.18/15), RFC 6598 CGNAT (100.64/10), 6to4 relay anycast; well-known public resolvers (8.8.8.8, 1.1.1.1, 9.9.9.9, 208.67.222.222, 1.2.3.4, ...).

Everything else that is global unicast is a finding.

Allowlist (allowlist.go)

Reference data, not vendor-detection logic: RFC 2606/5737 names, pool.ntp.org / nist.gov, standards bodies (dmtf.org, oasis-open.org, w3.org, iana.org, ietf.org), BMC-stack and vendor infrastructure (megarac.com, ami.com, openssh.com, libssh.org, rsyslog.com, inspur.com, inservice-iq.com, kaytus.com, jd.com, jdcloud.com), foobar.edu / issue.net (stock hosts.allow / sshd banner), and factory default values (Asia/Shanghai, To Be Filled By O.E.M., NULL, 0.0.0.0).

Redaction

privacy.FindSpans(line, certFile) is the matcher shared with internal/sanitize, which rewrites the same spans in place with same-length neutral fillers (see log-sanitization.md). The fillers are recognised by isRedactionFiller so a scan of a sanitized file is clean.

Already-sanitized detection (sanitized.go)

Scan also reports whether the source already looks de-identified, in PrivacyScan.Sanitized ({detected, markers, strong, files, evidence}).

SanitizationMarkers(line) recognises a value slot filled with one repeated placeholder + separators - xxxxx.xxxx.xx, x@xxxx.xxxx.xx, 000.00.00.0, a decoy timezone (Antarctica/McMurdo, Etc/Universal, ...). It matches the shape, not the literal x, so changing the filler character later still trips it. 0.0.0.0, 000, UTC, Etc/UTC are too plausibly intentional and do not count.

detected requires corroboration - strong >= 2, or `strong >= 1 && markers

= 3, or markers >= 4. A single filler-looking token is reported (markers: 1`) but never asserted as sanitized, so a partial future redaction pass or a coincidence does not read as "done".

Customer guess (customer.go)

Findings in domain, resolv, ad_ldap, cert, nsupdate, mgmt_subdomain, dhcp, email are reduced to a registrable domain (eTLD+1, using a small built-in multi-label suffix list). Score = hits + 3*distinct_files (+10 if seen in a strong category). Confidence: high = strong category and >=2 files; medium = one of those; low otherwise. Top 3 returned, weak single-hit domains dropped unless nothing else qualifies. Up to 3 path:line: excerpt evidence strings per domain - the domain itself is kept in the output (identifying the customer is the point); nothing is written to the repo.

Limitations

• No sanitization. The report guides a manual cleanup (KB: "Очистка BMC-дампа от данных заказчика").
• Public-suffix handling is a heuristic, not the full PSL.
• Binary and over-size (>10 MB extractor cap) files are not scanned - a clean report is a floor, not a proof.
• The Redfish path scans only the serialized tree, the one text corpus a live/replayed collection carries.