logpile/bible-local/docs/privacy-scan.md

Privacy / customer-data scan

internal/privacy scans ingested source files for customer-identifying and site-identifying data and attaches a models.PrivacyScan to AnalysisResult. Detection only - it never modifies the source. See ADL-066.

Entry point

privacy.Scan([]privacy.File{{Path, Content}}) *models.PrivacyScan

Returns nil when there is nothing to scan (no text files). Callers:

Source	Hook
Archive upload	parser.BMCParser.parseFiles (internal/parser/parser.go)
Live Redfish / replay	ingest.Service.AnalyzeRedfishRawPayloads scans the JSON-serialized raw_payloads.redfish_tree

Gated by LOGPILE_PRIVACY_SCAN (0/false/off/no disable; default on), via parser.PrivacyScanEnabled().

Output

• GET /api/privacy-scan - the PrivacyScan object, or {"loaded": false}.
• UI: the "Customer data" panel (#privacy-section), collapsible, hidden when there are no findings and no customer guess.
• privacy_report.json in the raw-export ZIP (GET /api/export/json), omitted when the scan produced nothing.

Scan mechanics

• Files whose name contains a vendor factory-template marker are skipped whole (_tencent, _jingdong, _pdd, _baidu, _kuaishou, _tianyiyun, _jd., syslog_jd, snmptrapcfg, .json_bak, ntp_auto).
• Binary content (NUL byte / invalid UTF-8 in the first 8 KiB) is skipped.
• Line-oriented; lines longer than 8 KiB are truncated for matching.
• Findings are deduped by category|match|path; global cap 1000.
• A bare-FQDN or e-mail match is dropped when a more precise rule already matched the same token on the same line (e.g. resolv.conf domain x yields a resolv finding, not also a domain one).
• A match that parses as an IP but is not sensitive (see IP policy) is dropped regardless of the rule that produced it.

Rule catalogue (rules.go)

Category	Severity	Matches	Remediation target
domain	high	bare FQDN with a known TLD	replace with example.local
resolv	high	domain / search / nameserver lines	resolv.conf
ad_ldap	high	racdomain=, adfilterdcN=, rolegroupN{name,domain}=, binddn, bindpw	activedir.conf, ldap.conf
mgmt_subdomain	high	*.mgmt/oob/ipmi/drac/idrac/ilo/bmc.*	management-network subdomain
nsupdate	high	`update add	delete ... A
cert	high	Subject:/Issuer:/CN=/DNS: in *.pem/*.csr/*.crt/*.cer	certificates
timezone	medium/low	TimeZone=, SELTimeUTCOffset=, Continent/City, localized TZ abbreviation before a year	Etc/UTC, offset 0
email	medium	local@host.tld (OData/Redfish annotations and non-TLD hosts filtered)	EventService / LDAP / certs
public_ip	medium	routable IPv4/IPv6 (see IP policy)	RFC 5737 doc ranges
collector	medium	SyslogHostname=, @ip:port, JSON "Destination"	syslog / SNMP trap receiver
fru_location	medium	Asset Tag, Product Location, Chassis Location, Board Extra	FRU fields
dhcp	medium	option domain-name	customer DHCP lease
hostname	low	sn-/bmc-/ilo-/idrac-/drac-/srv- prefixed names	customer naming scheme

IP policy (ip.go)

Not sensitive: RFC1918 / RFC4193 ULA, loopback, link-local, unspecified, multicast, 0.0.0.0/8, 255.0.0.0/8 and up; RFC 5737 (192.0.2/24, 198.51.100/24, 203.0.113/24), RFC 2544 (198.18/15), RFC 6598 CGNAT (100.64/10), 6to4 relay anycast; well-known public resolvers (8.8.8.8, 1.1.1.1, 9.9.9.9, 208.67.222.222, 1.2.3.4, ...).

Everything else that is global unicast is a finding.

Allowlist (allowlist.go)

Reference data, not vendor-detection logic: RFC 2606/5737 names, pool.ntp.org / nist.gov, standards bodies (dmtf.org, oasis-open.org, w3.org, iana.org, ietf.org), BMC-stack and vendor infrastructure (megarac.com, ami.com, openssh.com, libssh.org, rsyslog.com, inspur.com, inservice-iq.com, kaytus.com, jd.com, jdcloud.com), foobar.edu / issue.net (stock hosts.allow / sshd banner), and factory default values (Asia/Shanghai, To Be Filled By O.E.M., NULL, 0.0.0.0).

Customer guess (customer.go)

Findings in domain, resolv, ad_ldap, cert, nsupdate, mgmt_subdomain, dhcp, email are reduced to a registrable domain (eTLD+1, using a small built-in multi-label suffix list). Score = hits + 3*distinct_files (+10 if seen in a strong category). Confidence: high = strong category and >=2 files; medium = one of those; low otherwise. Top 3 returned, weak single-hit domains dropped unless nothing else qualifies. Up to 3 path:line: excerpt evidence strings per domain - the domain itself is kept in the output (identifying the customer is the point); nothing is written to the repo.

Limitations

• No sanitization. The report guides a manual cleanup (KB: "Очистка BMC-дампа от данных заказчика").
• Public-suffix handling is a heuristic, not the full PSL.
• Binary and over-size (>10 MB extractor cap) files are not scanned - a clean report is a floor, not a proof.
• The Redfish path scans only the serialized tree, the one text corpus a live/replayed collection carries.