diff --git a/bible-local/01-overview.md b/bible-local/01-overview.md index 469af0c..2bebcfe 100644 --- a/bible-local/01-overview.md +++ b/bible-local/01-overview.md @@ -23,6 +23,7 @@ All modes converge on the same normalized hardware model and exporter pipeline. - Reopenable raw export bundles for future re-analysis - Reanimator export and batch conversion workflows - Embedded `pci.ids` lookup for vendor/device name enrichment +- Customer-data / anonymization scan of ingested sources (detection only; see `docs/privacy-scan.md`) ## Current vendor coverage diff --git a/bible-local/03-api.md b/bible-local/03-api.md index eaaaac2..de53852 100644 --- a/bible-local/03-api.md +++ b/bible-local/03-api.md @@ -152,6 +152,13 @@ Returns normalized parse and collection issues combined from: - raw-export collect logs - derived partial-inventory warnings +### `GET /api/privacy-scan` + +Returns the customer-data scan for the current dataset (`models.PrivacyScan`: +`files_scanned`, `customers[]`, `findings[]`, `summary`). +Returns `{ "loaded": false }` when nothing is loaded or the scan produced +nothing. Detection only; see `docs/privacy-scan.md`. + ### `GET /api/parsers` Returns registered parser metadata. @@ -184,6 +191,7 @@ Current implementation emits a ZIP bundle containing: - `raw_export.json` - `collect.log` - `parser_fields.json` +- `privacy_report.json` (only when the customer-data scan produced findings) ### `GET /api/export/reanimator` diff --git a/bible-local/07-exporters.md b/bible-local/07-exporters.md index 4e73346..036d9d3 100644 --- a/bible-local/07-exporters.md +++ b/bible-local/07-exporters.md @@ -19,6 +19,7 @@ Current bundle contents: - `raw_export.json` - `collect.log` - `parser_fields.json` +- `privacy_report.json` (only when the customer-data scan produced findings; see `docs/privacy-scan.md`) Design rules: - raw source is authoritative diff --git a/bible-local/10-decisions.md b/bible-local/10-decisions.md index 67fa846..c98cf2b 100644 --- a/bible-local/10-decisions.md +++ b/bible-local/10-decisions.md @@ -1893,3 +1893,47 @@ The `inspur` parser still detected it at confidence 100 (every path contains `TestParseMegaRAIDLog`, `TestParseAMISyslog`, `TestDetect_*`, `TestParse_EndToEnd`, and `TestDetectFormat_PriorityBreaksConfidenceTie` in the `parser` package. + +--- + +## ADL-066 — Customer-data (privacy) scan of ingested sources + +**Date:** 2026-09-02 +**Context:** BMC dumps routinely arrive not anonymized. DNS suffix, AD domain, +NTP/DNS/syslog host names, timezone, admin e-mails, public IPs, TLS cert CN/SAN +and FRU location fields identify the customer and the install site. Before a +dump is forwarded to a vendor, a public repo, or the LOGPile test corpus an +operator had to run a grep playbook by hand. +**Decision:** Every ingested dataset is scanned by `internal/privacy`. The scan +is **detection only** - it never rewrites the source. It produces a +`models.PrivacyScan` on `AnalysisResult`: a customer guess (registrable domain +with confidence + evidence) and a findings list (category, file, line, matched +token, remediation hint). +- Rule catalogue (`rules.go`) is a port of the KB grep playbook. Categories: + domain, resolv, ad_ldap, timezone, email, public_ip, collector, cert, + fru_location, hostname, dhcp, nsupdate, mgmt_subdomain. +- IP policy (`ip.go`): RFC1918/ULA, loopback, link-local, multicast and the + RFC5737/RFC2544/RFC6598 example ranges plus well-known public resolvers are + **not** findings. Only routable public addresses are. +- Allowlist (`allowlist.go`) is reference data - RFC 2606/5737 names, NTP pools, + standards-body and BMC-vendor infrastructure domains, and vendor factory + template file-name markers. It is not vendor-detection logic. +- No real customer domain or token is committed. Tests use `acme.ru` as the + customer stand-in; `example.*` is reserved for the allowlist (it is the + sanitization target). +- Hook points: `parser.BMCParser.parseFiles` for archives, + `ingest.Service.AnalyzeRedfishRawPayloads` for the serialized Redfish tree. + Gated by `LOGPILE_PRIVACY_SCAN` (default on). +- Surfaced at `GET /api/privacy-scan`, in the "Customer data" UI panel, and as + `privacy_report.json` in the raw-export bundle. +**Consequences:** +- Sanitization stays a manual follow-up guided by the report (no re-archiving). +- The customer guess uses a small built-in public-suffix list, not the full PSL. +- Binary and over-size files are skipped, so the scan is a floor, not a proof of + cleanliness. +- Details and the full rule table live in + [`docs/privacy-scan.md`](docs/privacy-scan.md). +- Tests: `internal/privacy` (`TestScan_*`, `TestIsSensitiveIP`, + `TestIsAllowlistedDomain`, `TestRegistrableDomain`, `TestGuessCustomers_*`), + `internal/server` (`TestHandleGetPrivacyScan_*`, + `TestBuildRawExportBundle_*PrivacyReport*`). diff --git a/bible-local/README.md b/bible-local/README.md index f66186d..07acd7a 100644 --- a/bible-local/README.md +++ b/bible-local/README.md @@ -22,6 +22,7 @@ Keep top-level docs minimal and put maintained architecture/API contracts here. | [06-parsers.md](06-parsers.md) | Archive parser framework and vendor coverage | | [07-exporters.md](07-exporters.md) | Raw export, Reanimator export, batch convert | | [docs/hardware-ingest-contract.md](docs/hardware-ingest-contract.md) | Reanimator ingest schema mirrored locally | +| [docs/privacy-scan.md](docs/privacy-scan.md) | Customer-data / anonymization scan of ingested sources | | [08-build-release.md](08-build-release.md) | Build and release workflow | | [09-testing.md](09-testing.md) | Test expectations and regression rules | | [10-decisions.md](10-decisions.md) | Architectural Decision Log | diff --git a/bible-local/docs/privacy-scan.md b/bible-local/docs/privacy-scan.md new file mode 100644 index 0000000..76e0d19 --- /dev/null +++ b/bible-local/docs/privacy-scan.md @@ -0,0 +1,103 @@ +# Privacy / customer-data scan + +`internal/privacy` scans ingested source files for customer-identifying and +site-identifying data and attaches a `models.PrivacyScan` to `AnalysisResult`. +Detection only - it never modifies the source. See ADL-066. + +## Entry point + +```go +privacy.Scan([]privacy.File{{Path, Content}}) *models.PrivacyScan +``` + +Returns `nil` when there is nothing to scan (no text files). Callers: + +| Source | Hook | +|--------|------| +| Archive upload | `parser.BMCParser.parseFiles` (`internal/parser/parser.go`) | +| Live Redfish / replay | `ingest.Service.AnalyzeRedfishRawPayloads` scans the JSON-serialized `raw_payloads.redfish_tree` | + +Gated by `LOGPILE_PRIVACY_SCAN` (`0`/`false`/`off`/`no` disable; default on), +via `parser.PrivacyScanEnabled()`. + +## Output + +- `GET /api/privacy-scan` - the `PrivacyScan` object, or `{"loaded": false}`. +- UI: the "Customer data" panel (`#privacy-section`), collapsible, hidden when + there are no findings and no customer guess. +- `privacy_report.json` in the raw-export ZIP (`GET /api/export/json`), omitted + when the scan produced nothing. + +## Scan mechanics + +- Files whose name contains a vendor factory-template marker are skipped whole + (`_tencent`, `_jingdong`, `_pdd`, `_baidu`, `_kuaishou`, `_tianyiyun`, `_jd.`, + `syslog_jd`, `snmptrapcfg`, `.json_bak`, `ntp_auto`). +- Binary content (NUL byte / invalid UTF-8 in the first 8 KiB) is skipped. +- Line-oriented; lines longer than 8 KiB are truncated for matching. +- Findings are deduped by `category|match|path`; global cap 1000. +- A bare-FQDN or e-mail match is dropped when a more precise rule already + matched the same token on the same line (e.g. `resolv.conf` `domain x` yields + a `resolv` finding, not also a `domain` one). +- A match that parses as an IP but is not sensitive (see IP policy) is dropped + regardless of the rule that produced it. + +## Rule catalogue (`rules.go`) + +| Category | Severity | Matches | Remediation target | +|----------|----------|---------|--------------------| +| `domain` | high | bare FQDN with a known TLD | replace with `example.local` | +| `resolv` | high | `domain` / `search` / `nameserver` lines | `resolv.conf` | +| `ad_ldap` | high | `racdomain=`, `adfilterdcN=`, `rolegroupN{name,domain}=`, `binddn`, `bindpw` | `activedir.conf`, `ldap.conf` | +| `mgmt_subdomain` | high | `*.mgmt/oob/ipmi/drac/idrac/ilo/bmc.*` | management-network subdomain | +| `nsupdate` | high | `update add|delete ... A|AAAA|PTR|CNAME` | `runningdata/var/tmp/nsupdate_temp` | +| `cert` | high | `Subject:`/`Issuer:`/`CN=`/`DNS:` in `*.pem`/`*.csr`/`*.crt`/`*.cer` | certificates | +| `timezone` | medium/low | `TimeZone=`, `SELTimeUTCOffset=`, `Continent/City`, localized TZ abbreviation before a year | `Etc/UTC`, offset 0 | +| `email` | medium | `local@host.tld` (OData/Redfish annotations and non-TLD hosts filtered) | EventService / LDAP / certs | +| `public_ip` | medium | routable IPv4/IPv6 (see IP policy) | RFC 5737 doc ranges | +| `collector` | medium | `SyslogHostname=`, `@ip:port`, JSON `"Destination"` | syslog / SNMP trap receiver | +| `fru_location` | medium | `Asset Tag`, `Product Location`, `Chassis Location`, `Board Extra` | FRU fields | +| `dhcp` | medium | `option domain-name` | customer DHCP lease | +| `hostname` | low | `sn-`/`bmc-`/`ilo-`/`idrac-`/`drac-`/`srv-` prefixed names | customer naming scheme | + +## IP policy (`ip.go`) + +Not sensitive: RFC1918 / RFC4193 ULA, loopback, link-local, unspecified, +multicast, `0.0.0.0/8`, `255.0.0.0/8` and up; RFC 5737 (`192.0.2/24`, +`198.51.100/24`, `203.0.113/24`), RFC 2544 (`198.18/15`), RFC 6598 CGNAT +(`100.64/10`), 6to4 relay anycast; well-known public resolvers +(`8.8.8.8`, `1.1.1.1`, `9.9.9.9`, `208.67.222.222`, `1.2.3.4`, ...). + +Everything else that is global unicast is a finding. + +## Allowlist (`allowlist.go`) + +Reference data, not vendor-detection logic: RFC 2606/5737 names, `pool.ntp.org` +/ `nist.gov`, standards bodies (`dmtf.org`, `oasis-open.org`, `w3.org`, +`iana.org`, `ietf.org`), BMC-stack and vendor infrastructure +(`megarac.com`, `ami.com`, `openssh.com`, `libssh.org`, `rsyslog.com`, +`inspur.com`, `inservice-iq.com`, `kaytus.com`, `jd.com`, `jdcloud.com`), +`foobar.edu` / `issue.net` (stock `hosts.allow` / sshd banner), and factory +default values (`Asia/Shanghai`, `To Be Filled By O.E.M.`, `NULL`, `0.0.0.0`). + +## Customer guess (`customer.go`) + +Findings in `domain`, `resolv`, `ad_ldap`, `cert`, `nsupdate`, +`mgmt_subdomain`, `dhcp`, `email` are reduced to a registrable domain +(`eTLD+1`, using a small built-in multi-label suffix list). Score = +`hits + 3*distinct_files (+10 if seen in a strong category)`. Confidence: +`high` = strong category and >=2 files; `medium` = one of those; `low` +otherwise. Top 3 returned, weak single-hit domains dropped unless nothing else +qualifies. Up to 3 `path:line: excerpt` evidence strings per domain - the domain +itself is kept in the output (identifying the customer is the point); nothing is +written to the repo. + +## Limitations + +- No sanitization. The report guides a manual cleanup + (KB: "Очистка BMC-дампа от данных заказчика"). +- Public-suffix handling is a heuristic, not the full PSL. +- Binary and over-size (>10 MB extractor cap) files are not scanned - a clean + report is a floor, not a proof. +- The Redfish path scans only the serialized tree, the one text corpus a + live/replayed collection carries. diff --git a/internal/ingest/service.go b/internal/ingest/service.go index 035937f..acefff9 100644 --- a/internal/ingest/service.go +++ b/internal/ingest/service.go @@ -2,12 +2,14 @@ package ingest import ( "bytes" + "encoding/json" "fmt" "strings" "git.mchus.pro/mchus/logpile/internal/collector" "git.mchus.pro/mchus/logpile/internal/models" "git.mchus.pro/mchus/logpile/internal/parser" + "git.mchus.pro/mchus/logpile/internal/privacy" ) type Service struct{} @@ -59,5 +61,25 @@ func (s *Service) AnalyzeRedfishRawPayloads(rawPayloads map[string]any, meta Red result.Filename = "redfish://snapshot" } } + if scan := scanRedfishTreePrivacy(rawPayloads); scan != nil { + result.PrivacyScan = scan + } return result, "redfish", nil } + +// scanRedfishTreePrivacy runs the customer-data scan over the serialized Redfish +// tree (the only text corpus a live/replayed collection carries). +func scanRedfishTreePrivacy(rawPayloads map[string]any) *models.PrivacyScan { + if !parser.PrivacyScanEnabled() || rawPayloads == nil { + return nil + } + tree, ok := rawPayloads["redfish_tree"] + if !ok { + return nil + } + body, err := json.Marshal(tree) + if err != nil || len(body) == 0 { + return nil + } + return privacy.Scan([]privacy.File{{Path: "redfish_tree.json", Content: body}}) +} diff --git a/internal/models/models.go b/internal/models/models.go index 66efe3d..b847326 100644 --- a/internal/models/models.go +++ b/internal/models/models.go @@ -9,19 +9,60 @@ const ( // AnalysisResult contains all parsed data from an archive type AnalysisResult struct { - Filename string `json:"filename"` - SourceType string `json:"source_type,omitempty"` // archive | api - Protocol string `json:"protocol,omitempty"` // redfish | ipmi - TargetHost string `json:"target_host,omitempty"` // BMC host for live collect - SourceTimezone string `json:"source_timezone,omitempty"` // Source timezone/offset used during collection (e.g. +08:00) - CollectedAt time.Time `json:"collected_at,omitempty"` // Collection/upload timestamp - InventoryLastModifiedAt time.Time `json:"inventory_last_modified_at,omitempty"` // Redfish inventory last modified (InventoryData/Status) - RawPayloads map[string]any `json:"raw_payloads,omitempty"` // Additional source payloads (e.g. Redfish tree) - CollectionErrors []CollectionError `json:"collection_errors,omitempty"` // BMC-reported failures to collect specific sections - Events []Event `json:"events"` - FRU []FRUInfo `json:"fru"` - Sensors []SensorReading `json:"sensors"` - Hardware *HardwareConfig `json:"hardware"` + Filename string `json:"filename"` + SourceType string `json:"source_type,omitempty"` // archive | api + Protocol string `json:"protocol,omitempty"` // redfish | ipmi + TargetHost string `json:"target_host,omitempty"` // BMC host for live collect + SourceTimezone string `json:"source_timezone,omitempty"` // Source timezone/offset used during collection (e.g. +08:00) + CollectedAt time.Time `json:"collected_at,omitempty"` // Collection/upload timestamp + InventoryLastModifiedAt time.Time `json:"inventory_last_modified_at,omitempty"` // Redfish inventory last modified (InventoryData/Status) + RawPayloads map[string]any `json:"raw_payloads,omitempty"` // Additional source payloads (e.g. Redfish tree) + CollectionErrors []CollectionError `json:"collection_errors,omitempty"` // BMC-reported failures to collect specific sections + Events []Event `json:"events"` + FRU []FRUInfo `json:"fru"` + Sensors []SensorReading `json:"sensors"` + Hardware *HardwareConfig `json:"hardware"` + PrivacyScan *PrivacyScan `json:"privacy_scan,omitempty"` // customer-data / anonymization scan of the source files +} + +// PrivacyScan is the result of scanning the source files for customer-identifying +// and site-identifying data (DNS suffix, AD domain, NTP/DNS/syslog host names, +// timezone, e-mail, public IPs, cert CN/SAN, FRU location fields). Detection only: +// it never modifies the source. See bible-local/docs/privacy-scan.md. +type PrivacyScan struct { + FilesScanned int `json:"files_scanned"` + Customers []CustomerGuess `json:"customers,omitempty"` + Findings []PrivacyFinding `json:"findings,omitempty"` + Summary PrivacySummary `json:"summary"` +} + +// CustomerGuess is a registrable domain that most likely identifies the customer, +// derived from the scan findings. +type CustomerGuess struct { + Domain string `json:"domain"` + Confidence string `json:"confidence"` // high | medium | low + Hits int `json:"hits"` + Evidence []string `json:"evidence,omitempty"` // up to 3 "path:line: excerpt" strings +} + +// PrivacyFinding is one matched sensitive token. +type PrivacyFinding struct { + Category string `json:"category"` + Severity string `json:"severity"` // high | medium | low + Path string `json:"path"` + Line int `json:"line,omitempty"` + Match string `json:"match"` + Excerpt string `json:"excerpt,omitempty"` + Hint string `json:"hint,omitempty"` +} + +// PrivacySummary is the finding count breakdown. +type PrivacySummary struct { + Total int `json:"total"` + High int `json:"high"` + Medium int `json:"medium"` + Low int `json:"low"` + ByCategory map[string]int `json:"by_category,omitempty"` } // CollectionError represents a BMC-reported failure to collect a specific data section. @@ -263,26 +304,26 @@ type MemoryDIMM struct { // Storage represents a storage device type Storage struct { - Slot string `json:"slot"` - Type string `json:"type"` - Model string `json:"model"` - Description string `json:"description,omitempty"` - SizeGB int `json:"size_gb"` - SerialNumber string `json:"serial_number,omitempty"` - Manufacturer string `json:"manufacturer,omitempty"` - Firmware string `json:"firmware,omitempty"` - Interface string `json:"interface,omitempty"` - Present bool `json:"present"` - Location string `json:"location,omitempty"` // Front/Rear - BackplaneID int `json:"backplane_id,omitempty"` - VendorID int `json:"vendor_id,omitempty"` // PCI Vendor ID (decimal), NVMe drives only - DeviceID int `json:"device_id,omitempty"` // PCI Device ID (decimal), NVMe drives only - RemainingEndurancePct *int `json:"remaining_endurance_pct,omitempty"` // 0-100 %; nil = not reported - LogicalBlockSizeBytes int64 `json:"logical_block_size_bytes,omitempty"` - PhysicalBlockSizeBytes int64 `json:"physical_block_size_bytes,omitempty"` - MetadataBytesPerBlock int64 `json:"metadata_bytes_per_block,omitempty"` - Status string `json:"status,omitempty"` - Details map[string]any `json:"details,omitempty"` + Slot string `json:"slot"` + Type string `json:"type"` + Model string `json:"model"` + Description string `json:"description,omitempty"` + SizeGB int `json:"size_gb"` + SerialNumber string `json:"serial_number,omitempty"` + Manufacturer string `json:"manufacturer,omitempty"` + Firmware string `json:"firmware,omitempty"` + Interface string `json:"interface,omitempty"` + Present bool `json:"present"` + Location string `json:"location,omitempty"` // Front/Rear + BackplaneID int `json:"backplane_id,omitempty"` + VendorID int `json:"vendor_id,omitempty"` // PCI Vendor ID (decimal), NVMe drives only + DeviceID int `json:"device_id,omitempty"` // PCI Device ID (decimal), NVMe drives only + RemainingEndurancePct *int `json:"remaining_endurance_pct,omitempty"` // 0-100 %; nil = not reported + LogicalBlockSizeBytes int64 `json:"logical_block_size_bytes,omitempty"` + PhysicalBlockSizeBytes int64 `json:"physical_block_size_bytes,omitempty"` + MetadataBytesPerBlock int64 `json:"metadata_bytes_per_block,omitempty"` + Status string `json:"status,omitempty"` + Details map[string]any `json:"details,omitempty"` StatusCheckedAt *time.Time `json:"status_checked_at,omitempty"` StatusChangedAt *time.Time `json:"status_changed_at,omitempty"` diff --git a/internal/parser/parser.go b/internal/parser/parser.go index ce8ab92..57c6245 100644 --- a/internal/parser/parser.go +++ b/internal/parser/parser.go @@ -3,10 +3,12 @@ package parser import ( "fmt" "io" + "os" "strings" "time" "git.mchus.pro/mchus/logpile/internal/models" + "git.mchus.pro/mchus/logpile/internal/privacy" ) // BMCParser parses BMC diagnostic archives using vendor-specific parsers @@ -66,6 +68,9 @@ func (p *BMCParser) parseFiles() error { result.Filename = p.result.Filename appendExtractionWarnings(result, p.files) + if scan := runPrivacyScan(p.files); scan != nil { + result.PrivacyScan = scan + } if result.CollectedAt.IsZero() { if ts := inferCollectedAtFromExtractedFiles(p.files); !ts.IsZero() { result.CollectedAt = ts.UTC() @@ -76,6 +81,28 @@ func (p *BMCParser) parseFiles() error { return nil } +// PrivacyScanEnabled reports whether the customer-data scan runs. It is on by +// default and disabled by LOGPILE_PRIVACY_SCAN=0 / false. +func PrivacyScanEnabled() bool { + switch strings.ToLower(strings.TrimSpace(os.Getenv("LOGPILE_PRIVACY_SCAN"))) { + case "0", "false", "off", "no": + return false + default: + return true + } +} + +func runPrivacyScan(files []ExtractedFile) *models.PrivacyScan { + if !PrivacyScanEnabled() { + return nil + } + in := make([]privacy.File, 0, len(files)) + for _, f := range files { + in = append(in, privacy.File{Path: f.Path, Content: f.Content}) + } + return privacy.Scan(in) +} + func inferCollectedAtFromExtractedFiles(files []ExtractedFile) time.Time { var latestReliable time.Time var latestAny time.Time diff --git a/internal/privacy/allowlist.go b/internal/privacy/allowlist.go new file mode 100644 index 0000000..dc219fc --- /dev/null +++ b/internal/privacy/allowlist.go @@ -0,0 +1,81 @@ +package privacy + +import ( + "path" + "strings" +) + +// Public infrastructure, documentation, and vendor-default values that are not +// customer leaks. These are reference data (RFC 2606 / 5737 names, well-known +// NTP pools, standards-body domains, factory defaults), not vendor-detection +// logic. +var ( + allowlistedZones = []string{ + "example.com", "example.net", "example.org", "example.local", "example.edu", + "foobar.edu", "issue.net", + "localhost", "localdomain", "local.lan", + "pool.ntp.org", "ntp.org", "nist.gov", "windows.com", "microsoft.com", + "dmtf.org", "iana.org", "openssl.org", "openssh.com", "openssh.org", + "libssh.org", "rsyslog.com", "adiscon.com", "redhat.com", "kernel.org", + "megarac.com", "ami.com", "commond.com", + "oasis-open.org", "w3.org", "xmlsoap.org", "purl.org", "ietf.org", + "inspur.com", "inspurcloud.com", "inservice-iq.com", "kaytus.com", + "jd.com", "jd.local", "jdcloud.com", "in-addr.arpa", "ip6.arpa", "arpa", + } + + allowlistedValues = map[string]struct{}{ + "asia/shanghai": {}, + "etc/utc": {}, + "utc": {}, + "to be filled by o.e.m.": {}, + "default string": {}, + "unknown": {}, + "n/a": {}, + "none": {}, + "null": {}, + "0": {}, + "0.0.0.0": {}, + } + + // Archive members that are vendor factory templates, not the active config. + allowlistedFilenameParts = []string{ + "_tencent", "_jingdong", "_pdd", "_baidu", "_kuaishou", "_tianyiyun", + "_jd.", "syslog_jd", "snmptrapcfg", ".json_bak", "ntp_auto", + } +) + +func isAllowlistedDomain(domain string) bool { + d := strings.ToLower(strings.TrimSuffix(strings.TrimSpace(domain), ".")) + for _, z := range allowlistedZones { + if d == z || strings.HasSuffix(d, "."+z) { + return true + } + } + return false +} + +func isAllowlistedValue(v string) bool { + lv := strings.ToLower(strings.TrimSpace(v)) + if _, ok := allowlistedValues[lv]; ok { + return true + } + return isAllowlistedDomain(v) +} + +func isAllowlistedFilename(p string) bool { + lp := strings.ToLower(p) + for _, part := range allowlistedFilenameParts { + if strings.Contains(lp, part) { + return true + } + } + return false +} + +func isCertFilename(p string) bool { + switch strings.ToLower(path.Ext(p)) { + case ".pem", ".csr", ".crt", ".cer": + return true + } + return false +} diff --git a/internal/privacy/customer.go b/internal/privacy/customer.go new file mode 100644 index 0000000..e37f733 --- /dev/null +++ b/internal/privacy/customer.go @@ -0,0 +1,174 @@ +package privacy + +import ( + "sort" + "strconv" + "strings" + + "git.mchus.pro/mchus/logpile/internal/models" +) + +// Multi-label public suffixes handled explicitly; every other final label is +// treated as the suffix. This is a heuristic, not a full PSL - good enough to +// collapse "sn-x.mgmt.corp.example.co.uk" to "example.co.uk". +var multiLabelSuffixes = []string{ + "co.uk", "org.uk", "gov.uk", "ac.uk", "me.uk", + "com.cn", "net.cn", "org.cn", "gov.cn", "edu.cn", + "com.au", "net.au", "org.au", "co.jp", "co.kr", "com.br", "com.tr", "com.ua", +} + +// Categories whose findings feed the customer guess, and whether a hit in that +// category is strong evidence on its own. +var customerCategories = map[string]bool{ + catDomain: false, + catResolv: true, + catADLDAP: true, + catCert: true, + catNSUpdate: true, + catMgmtSubdomain: true, + catDHCP: false, + catEmail: false, +} + +type customerTally struct { + domain string + hits int + paths map[string]struct{} + strong bool + evidence []string + evidenceSeen map[string]struct{} +} + +func guessCustomers(findings []models.PrivacyFinding) []models.CustomerGuess { + tallies := map[string]*customerTally{} + + for _, f := range findings { + strongCat, ok := customerCategories[f.Category] + if !ok { + continue + } + host := f.Match + if f.Category == catEmail { + if at := strings.IndexByte(host, '@'); at >= 0 { + host = host[at+1:] + } + } + reg := registrableDomain(host) + if reg == "" || isAllowlistedDomain(reg) { + continue + } + t := tallies[reg] + if t == nil { + t = &customerTally{domain: reg, paths: map[string]struct{}{}, evidenceSeen: map[string]struct{}{}} + tallies[reg] = t + } + t.hits++ + if f.Path != "" { + t.paths[f.Path] = struct{}{} + } + if strongCat { + t.strong = true + } + if len(t.evidence) < 3 && f.Excerpt != "" { + line := evidenceLine(f) + if _, dup := t.evidenceSeen[line]; !dup { + t.evidenceSeen[line] = struct{}{} + t.evidence = append(t.evidence, line) + } + } + } + + guesses := make([]models.CustomerGuess, 0, len(tallies)) + for _, t := range tallies { + guesses = append(guesses, models.CustomerGuess{ + Domain: t.domain, + Confidence: confidence(t), + Hits: t.hits, + Evidence: t.evidence, + }) + } + + sort.Slice(guesses, func(i, j int) bool { + si, sj := score(tallies[guesses[i].Domain]), score(tallies[guesses[j].Domain]) + if si != sj { + return si > sj + } + return guesses[i].Domain < guesses[j].Domain + }) + + // Keep the strongest few; drop weak single-hit noise unless it is all we have. + out := make([]models.CustomerGuess, 0, 3) + for _, g := range guesses { + if len(out) >= 3 { + break + } + if g.Hits < 2 && g.Confidence == "low" && len(out) > 0 { + continue + } + out = append(out, g) + } + return out +} + +func score(t *customerTally) int { + s := t.hits + 3*len(t.paths) + if t.strong { + s += 10 + } + return s +} + +func confidence(t *customerTally) string { + switch { + case t.strong && len(t.paths) >= 2: + return "high" + case t.strong || len(t.paths) >= 2: + return "medium" + default: + return "low" + } +} + +func evidenceLine(f models.PrivacyFinding) string { + loc := f.Path + if f.Line > 0 { + loc += ":" + strconv.Itoa(f.Line) + } + return loc + ": " + f.Excerpt +} + +// registrableDomain returns the registrable ("eTLD+1") portion of a host name, +// or "" if host is an IP, has no dot, or is a bare TLD. +func registrableDomain(host string) string { + host = strings.ToLower(strings.Trim(strings.TrimSpace(host), ".*")) + host = strings.TrimPrefix(host, "*.") + if host == "" || strings.ContainsAny(host, ":/ ") { + return "" + } + if strings.Count(host, ".") == 3 { + allDigits := true + for _, r := range host { + if r != '.' && (r < '0' || r > '9') { + allDigits = false + break + } + } + if allDigits { + return "" + } + } + labels := strings.Split(host, ".") + if len(labels) < 2 { + return "" + } + for _, suf := range multiLabelSuffixes { + if strings.HasSuffix(host, "."+suf) { + sufLabels := strings.Count(suf, ".") + 1 + if len(labels) > sufLabels { + return strings.Join(labels[len(labels)-sufLabels-1:], ".") + } + return "" + } + } + return strings.Join(labels[len(labels)-2:], ".") +} diff --git a/internal/privacy/ip.go b/internal/privacy/ip.go new file mode 100644 index 0000000..438b424 --- /dev/null +++ b/internal/privacy/ip.go @@ -0,0 +1,59 @@ +package privacy + +import "net" + +// Documentation, benchmarking, and well-known example addresses that carry no +// site information even though they are globally routable. +var nonSensitiveNets = func() []*net.IPNet { + cidrs := []string{ + "192.0.2.0/24", // RFC 5737 TEST-NET-1 + "198.51.100.0/24", // RFC 5737 TEST-NET-2 + "203.0.113.0/24", // RFC 5737 TEST-NET-3 + "198.18.0.0/15", // RFC 2544 benchmarking + "100.64.0.0/10", // RFC 6598 CGNAT + "192.88.99.0/24", // RFC 7526 6to4 relay anycast + } + out := make([]*net.IPNet, 0, len(cidrs)) + for _, c := range cidrs { + if _, n, err := net.ParseCIDR(c); err == nil { + out = append(out, n) + } + } + return out +}() + +var nonSensitiveExact = map[string]struct{}{ + "8.8.8.8": {}, "8.8.4.4": {}, "1.1.1.1": {}, "1.0.0.1": {}, + "4.2.2.2": {}, "4.2.2.1": {}, "9.9.9.9": {}, "1.2.3.4": {}, + "208.67.222.222": {}, "208.67.220.220": {}, +} + +// isSensitiveIP reports whether s is a routable address that could identify the +// customer's provider or site. Private (RFC1918/ULA), loopback, link-local, +// multicast, and the example/benchmark ranges above are not sensitive. +func isSensitiveIP(s string) bool { + ip := net.ParseIP(s) + if ip == nil { + return false + } + if _, ok := nonSensitiveExact[s]; ok { + return false + } + if ip.IsLoopback() || ip.IsPrivate() || ip.IsUnspecified() || + ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() || + ip.IsMulticast() || ip.IsInterfaceLocalMulticast() { + return false + } + if !ip.IsGlobalUnicast() { + return false + } + if v4 := ip.To4(); v4 != nil && (v4[0] == 0 || v4[0] == 255 || v4[0] >= 240) { + return false + } + for _, n := range nonSensitiveNets { + if n.Contains(ip) { + return false + } + } + return true +} diff --git a/internal/privacy/privacy.go b/internal/privacy/privacy.go new file mode 100644 index 0000000..cb5aeb3 --- /dev/null +++ b/internal/privacy/privacy.go @@ -0,0 +1,175 @@ +// Package privacy scans ingested source files for customer-identifying and +// site-identifying data: DNS suffix, AD domain, NTP/DNS/syslog host names, +// timezone, e-mail, public IPs, TLS cert CN/SAN, FRU location fields. +// +// Detection only. Nothing here modifies the source. The rule catalogue is a +// port of the KB grep playbook; see bible-local/docs/privacy-scan.md. +package privacy + +import ( + "bufio" + "bytes" + "sort" + "strings" + "unicode/utf8" + + "git.mchus.pro/mchus/logpile/internal/models" +) + +// File is one unit of scanned content (an extracted archive member, or a +// serialized Redfish tree). +type File struct { + Path string + Content []byte +} + +const ( + maxFindings = 1000 + maxExcerptRunes = 200 + maxScanLineBytes = 8192 +) + +// Scan runs every rule over every text file and returns the aggregated report. +// Returns nil when there is nothing to scan (no text files). +func Scan(files []File) *models.PrivacyScan { + scanned := 0 + seen := make(map[string]struct{}) + var findings []models.PrivacyFinding + + emit := func(f models.PrivacyFinding) { + if len(findings) >= maxFindings { + return + } + key := f.Category + "|" + f.Match + "|" + f.Path + if _, ok := seen[key]; ok { + return + } + seen[key] = struct{}{} + findings = append(findings, f) + } + + for _, file := range files { + if isAllowlistedFilename(file.Path) { + continue + } + if !looksLikeText(file.Content) { + continue + } + scanned++ + scanFileLines(file, emit) + } + + if scanned == 0 { + return nil + } + + sortFindings(findings) + + report := &models.PrivacyScan{ + FilesScanned: scanned, + Findings: findings, + Customers: guessCustomers(findings), + Summary: summarize(findings), + } + return report +} + +func scanFileLines(file File, emit func(models.PrivacyFinding)) { + certFile := isCertFilename(file.Path) + sc := bufio.NewScanner(bytes.NewReader(file.Content)) + sc.Buffer(make([]byte, 0, 64*1024), 1024*1024) + ln := 0 + for sc.Scan() { + ln++ + line := sc.Text() + if len(line) > maxScanLineBytes { + line = line[:maxScanLineBytes] + } + scanLine(file.Path, line, ln, certFile, emit) + } +} + +func looksLikeText(b []byte) bool { + if len(b) == 0 { + return false + } + head := b + if len(head) > 8192 { + head = head[:8192] + } + if bytes.IndexByte(head, 0) >= 0 { + return false + } + // Tolerate a truncated multi-byte rune at the sampled boundary. + if !utf8.Valid(head) && !utf8.Valid(trimIncompleteRune(head)) { + return false + } + return true +} + +func trimIncompleteRune(b []byte) []byte { + for i := 0; i < 3 && i < len(b); i++ { + if utf8.RuneStart(b[len(b)-1-i]) { + return b[:len(b)-1-i] + } + } + return b +} + +func excerpt(line string) string { + line = strings.Map(func(r rune) rune { + if r == 0 || (r < 0x20 && r != '\t') { + return ' ' + } + return r + }, line) + line = strings.Join(strings.Fields(line), " ") + line = strings.TrimSpace(line) + if utf8.RuneCountInString(line) <= maxExcerptRunes { + return line + } + r := []rune(line) + return string(r[:maxExcerptRunes]) + "..." +} + +func severityRank(s string) int { + switch s { + case severityHigh: + return 0 + case severityMedium: + return 1 + default: + return 2 + } +} + +func sortFindings(f []models.PrivacyFinding) { + sort.SliceStable(f, func(i, j int) bool { + if f[i].Severity != f[j].Severity { + return severityRank(f[i].Severity) < severityRank(f[j].Severity) + } + if f[i].Category != f[j].Category { + return f[i].Category < f[j].Category + } + if f[i].Path != f[j].Path { + return f[i].Path < f[j].Path + } + return f[i].Match < f[j].Match + }) +} + +func summarize(f []models.PrivacyFinding) models.PrivacySummary { + s := models.PrivacySummary{Total: len(f), ByCategory: map[string]int{}} + for _, x := range f { + switch x.Severity { + case severityHigh: + s.High++ + case severityMedium: + s.Medium++ + default: + s.Low++ + } + s.ByCategory[x.Category]++ + } + return s +} diff --git a/internal/privacy/privacy_test.go b/internal/privacy/privacy_test.go new file mode 100644 index 0000000..bda91d9 --- /dev/null +++ b/internal/privacy/privacy_test.go @@ -0,0 +1,121 @@ +package privacy + +import "testing" + +// Fixtures use "acme.ru" as a stand-in customer domain. The RFC 2606 +// "example.*" names are on the allowlist (they are the sanitization target), +// so they cannot double as the customer placeholder in detection tests. +func TestScan_DetectsAndClassifies(t *testing.T) { + files := []File{ + {Path: "onekeylog/configuration/conf/resolv.conf", Content: []byte( + "domain corp.acme.ru\nsearch corp.acme.ru\nnameserver 10.10.0.1\n")}, + {Path: "onekeylog/configuration/conf/activedir.conf", Content: []byte( + "racdomain=corp.acme.ru\nadfilterdc1=10.10.0.5\nrolegroup1name=cab-gr-CI00363277-x86bmc\n")}, + {Path: "onekeylog/configuration/conf/BMC1/misccfg.ini", Content: []byte( + "TimeZone=Europe/Moscow\nSELTimeUTCOffset=180\n")}, + {Path: "onekeylog/log/audit.log", Content: []byte( + "2026-08-24T11:47:34+03:00 login admin from ntp01.acme.ru ok\n" + + "contact ops@acme.ru for access\n" + + "outbound 45.32.10.7 established\n" + + "internal 192.168.31.4 and 10.1.2.3 and doc 203.0.113.9\n")}, + {Path: "onekeylog/runningdata/rundatainfo.log", Content: []byte( + "Mon Aug 24 11:58:48 MSK 2026\n")}, + // Vendor factory template - must be skipped entirely. + {Path: "onekeylog/configuration/conf/ntp_auto.conf", Content: []byte( + "server ntp.should-not-match.ru\n")}, + {Path: "onekeylog/bin/blob", Content: []byte{0x00, 0x01, 0x02, 0xff}}, + } + + rep := Scan(files) + if rep == nil { + t.Fatal("nil report") + } + if rep.FilesScanned != 5 { + t.Fatalf("FilesScanned = %d, want 5 (binary + template skipped)", rep.FilesScanned) + } + + has := func(cat, match string) bool { + for _, f := range rep.Findings { + if f.Category == cat && f.Match == match { + return true + } + } + return false + } + mustHave := []struct{ cat, match string }{ + {catResolv, "corp.acme.ru"}, + {catADLDAP, "corp.acme.ru"}, + {catADLDAP, "cab-gr-CI00363277-x86bmc"}, + {catTimezone, "Europe/Moscow"}, + {catTimezone, "180"}, + {catDomain, "ntp01.acme.ru"}, + {catEmail, "ops@acme.ru"}, + {catPublicIP, "45.32.10.7"}, + {catTimezone, "MSK"}, + } + for _, w := range mustHave { + if !has(w.cat, w.match) { + t.Errorf("missing finding %s / %q", w.cat, w.match) + } + } + + mustNotHave := []string{"192.168.31.4", "10.1.2.3", "203.0.113.9", "ntp.should-not-match.ru"} + for _, f := range rep.Findings { + for _, bad := range mustNotHave { + if f.Match == bad { + t.Errorf("unexpected finding for %q (%s)", bad, f.Category) + } + } + } + + if len(rep.Customers) == 0 || rep.Customers[0].Domain != "acme.ru" { + t.Fatalf("customer guess = %+v, want acme.ru first", rep.Customers) + } + if rep.Customers[0].Confidence != "high" { + t.Errorf("confidence = %s, want high", rep.Customers[0].Confidence) + } +} + +func TestScan_NothingToScan(t *testing.T) { + if Scan(nil) != nil { + t.Fatal("want nil for no files") + } + if Scan([]File{{Path: "x", Content: []byte{0}}}) != nil { + t.Fatal("want nil when only binary files") + } +} + +func TestScan_Dedupe(t *testing.T) { + rep := Scan([]File{{Path: "a.log", Content: []byte( + "host is srv-prod.acme.net\nhost is srv-prod.acme.net again\n")}}) + if rep == nil { + t.Fatal("nil report") + } + n := 0 + for _, f := range rep.Findings { + if f.Category == catDomain && f.Match == "srv-prod.acme.net" { + n++ + } + } + if n != 1 { + t.Fatalf("domain finding counted %d times, want 1", n) + } +} + +func TestScan_AllowlistedDomainNotFlagged(t *testing.T) { + rep := Scan([]File{{Path: "ntp.conf", Content: []byte( + "server 0.pool.ntp.org\nserver time.nist.gov\ncontact admin@example.com\n")}}) + if rep != nil && len(rep.Findings) > 0 { + t.Fatalf("allowlisted infra flagged: %+v", rep.Findings) + } +} + +func TestSummary(t *testing.T) { + rep := Scan([]File{{Path: "resolv.conf", Content: []byte("domain acme.ru\n")}}) + if rep.Summary.Total != len(rep.Findings) || rep.Summary.Total == 0 { + t.Fatalf("summary total mismatch: %+v", rep.Summary) + } + if rep.Summary.High == 0 { + t.Fatalf("expected a high finding, got %+v", rep.Summary) + } +} diff --git a/internal/privacy/rules.go b/internal/privacy/rules.go new file mode 100644 index 0000000..359d3da --- /dev/null +++ b/internal/privacy/rules.go @@ -0,0 +1,207 @@ +package privacy + +import ( + "net" + "regexp" + "strings" + + "git.mchus.pro/mchus/logpile/internal/models" +) + +const ( + severityHigh = "high" + severityMedium = "medium" + severityLow = "low" + + catDomain = "domain" + catResolv = "resolv" + catADLDAP = "ad_ldap" + catTimezone = "timezone" + catEmail = "email" + catPublicIP = "public_ip" + catCollector = "collector" + catCert = "cert" + catFRULocation = "fru_location" + catHostname = "hostname" + catDHCP = "dhcp" + catNSUpdate = "nsupdate" + catMgmtSubdomain = "mgmt_subdomain" +) + +// tableRule is a simple line-regexp rule. group is the submatch index used as +// the reported token (0 = whole match). +type tableRule struct { + category string + severity string + re *regexp.Regexp + group int + hint string +} + +var ( + // A conservative TLD set keeps the bare-FQDN rule from matching things like + // "foo.bar" in prose or "index.json" in paths. + fqdnTLD = `(?:ru|su|by|kz|ua|com|net|org|local|io|dev|cloud|info|biz|eu|de|uk|fr|nl|cn|jp|kr|us|gov|edu|mil|co|tech|online)` + + reFQDN = regexp.MustCompile(`(?i)\b((?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+` + fqdnTLD + `)\b`) + reMgmtSubdomain = regexp.MustCompile(`(?i)\b([a-z0-9-]+(?:\.[a-z0-9-]+)*\.(?:mgmt|oob|ipmi|drac|idrac|ilo|bmc)\.[a-z0-9.-]+)\b`) + reIPv4 = regexp.MustCompile(`\b((?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])(?:\.(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])){3})\b`) + reEmail = regexp.MustCompile(`(?i)\b([a-z0-9._%+\-]+@[a-z0-9.\-]+\.[a-z]{2,})\b`) + reSyslogTarget = regexp.MustCompile(`@((?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])(?:\.(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])){3}:[0-9]{1,5})`) + reTZName = regexp.MustCompile(`\b((?:Africa|America|Antarctica|Asia|Atlantic|Australia|Europe|Indian|Pacific)/[A-Za-z_]+(?:/[A-Za-z_]+)?)\b`) + reTZAbbr = regexp.MustCompile(`\b(MSK|MSD|EEST|EET|MDST|CEST|CET|WEST|WET)\b\s+20\d\d`) + + tableRules = []tableRule{ + {catResolv, severityHigh, regexp.MustCompile(`(?i)^\s*(?:domain|search)\s+(\S+)`), 1, + "resolv.conf DNS suffix - replace with example.local"}, + {catADLDAP, severityHigh, regexp.MustCompile(`(?i)\b(racdomain|adfilterdc[0-9]|rolegroup[0-9](?:name|domain))\s*=\s*(\S+)`), 2, + "activedir.conf - AD domain / DC address / role-group name"}, + {catADLDAP, severityHigh, regexp.MustCompile(`(?i)\b(binddn|bindpw)\s*=\s*(\S+)`), 2, + "ldap.conf - directory service-account bind"}, + {catTimezone, severityMedium, regexp.MustCompile(`(?i)\b(TimeZone|SELTimeUTCOffset)\s*=\s*(\S+)`), 2, + "timezone reveals region - set Etc/UTC / offset 0"}, + {catCollector, severityMedium, regexp.MustCompile(`(?i)\b(SyslogHostname)\s*=\s*(\S+)`), 2, + "customer syslog collector"}, + {catCollector, severityMedium, regexp.MustCompile(`(?i)"Destination"\s*:\s*"([^"]+)"`), 1, + "SNMP trap / event destination"}, + {catFRULocation, severityMedium, regexp.MustCompile(`(?i)\b(Asset Tag|Product Location|Chassis Location|Board Extra)\b\s*[:=]\s*(.+)`), 2, + "FRU site / inventory field"}, + {catNSUpdate, severityHigh, regexp.MustCompile(`(?i)\bupdate\s+(?:add|delete)\s+(\S+\.\S+)\s+.*\b(?:A|AAAA|PTR|CNAME)\b`), 1, + "DDNS update - host FQDN + BMC record"}, + {catDHCP, severityMedium, regexp.MustCompile(`(?i)option\s+domain-name\s+"?([^";]+)`), 1, + "domain-name handed out by customer DHCP"}, + {catHostname, severityLow, regexp.MustCompile(`(?i)\b((?:sn|bmc|ilo|idrac|drac|srv)-[a-z0-9][a-z0-9-]{2,})\b`), 1, + "hostname follows a customer naming scheme"}, + } + + certLineRules = []tableRule{ + {catCert, severityHigh, regexp.MustCompile(`(?i)^\s*(?:Subject|Issuer):\s*(.+)`), 1, "TLS cert subject/issuer"}, + {catCert, severityHigh, regexp.MustCompile(`(?i)\bCN\s*=\s*([^,/]+)`), 1, "TLS cert common name"}, + {catCert, severityHigh, regexp.MustCompile(`(?i)\bDNS:\s*([a-z0-9.\-*]+)`), 1, "TLS cert SAN"}, + } +) + +func scanLine(path, line string, ln int, certFile bool, emit func(models.PrivacyFinding)) { + var local []models.PrivacyFinding + specific := map[string]struct{}{} // matches from categories more precise than a bare FQDN + + add := func(cat, sev, match, hint string) { + match = strings.Trim(strings.TrimSpace(match), `"',;`) + if match == "" || isAllowlistedValue(match) { + return + } + if ip := net.ParseIP(match); ip != nil && !isSensitiveIP(match) { + return + } + if cat != catDomain && cat != catEmail && cat != catPublicIP { + specific[strings.ToLower(match)] = struct{}{} + } + local = append(local, models.PrivacyFinding{ + Category: cat, Severity: sev, Path: path, Line: ln, + Match: match, Excerpt: excerpt(line), Hint: hint, + }) + } + + for _, r := range tableRules { + for _, m := range r.re.FindAllStringSubmatch(line, -1) { + if r.group < len(m) { + add(r.category, r.severity, m[r.group], r.hint) + } + } + } + + if certFile { + for _, r := range certLineRules { + for _, m := range r.re.FindAllStringSubmatch(line, -1) { + if r.group < len(m) { + add(r.category, r.severity, m[r.group], r.hint) + } + } + } + } + + for _, m := range reMgmtSubdomain.FindAllStringSubmatch(line, -1) { + add(catMgmtSubdomain, severityHigh, m[1], "management-network subdomain") + } + for _, m := range reFQDN.FindAllStringSubmatch(line, -1) { + if isAllowlistedDomain(m[1]) { + continue + } + add(catDomain, severityHigh, m[1], "domain / FQDN reveals the customer") + } + for _, m := range reEmail.FindAllStringSubmatch(line, -1) { + host := m[1][strings.IndexByte(m[1], '@')+1:] + if isAllowlistedDomain(host) || !looksLikeMailHost(host) { + continue + } + add(catEmail, severityMedium, m[1], "e-mail address") + } + for _, m := range reSyslogTarget.FindAllStringSubmatch(line, -1) { + add(catCollector, severityMedium, m[1], "remote syslog target") + } + for _, m := range reIPv4.FindAllStringSubmatch(line, -1) { + if !isSensitiveIP(m[1]) { + continue + } + add(catPublicIP, severityMedium, m[1], "public IP reveals provider / site") + } + if strings.Contains(strings.ToLower(line), "timezone") || strings.Contains(line, "/") { + for _, m := range reTZName.FindAllStringSubmatch(line, -1) { + if isAllowlistedValue(m[1]) { + continue + } + add(catTimezone, severityMedium, m[1], "timezone reveals region - set Etc/UTC") + } + } + for _, m := range reTZAbbr.FindAllStringSubmatch(line, -1) { + add(catTimezone, severityLow, m[1], "localized timestamp reveals region") + } + + for _, f := range local { + if (f.Category == catDomain || f.Category == catEmail) && matchCoveredBySpecific(f.Match, specific) { + continue + } + emit(f) + } +} + +// looksLikeMailHost rejects the many "local@identifier.token" strings that are +// not e-mail: OData/Redfish JSON annotations (Members@odata.count), SSH +// cipher/kex names (aes256-gcm@openssh.com is handled by the domain allowlist, +// but the shape is the same). +func looksLikeMailHost(host string) bool { + h := strings.ToLower(host) + if strings.Contains(h, "odata") || strings.Contains(h, "redfish") || strings.Contains(h, "message.") { + return false + } + dot := strings.LastIndexByte(h, '.') + if dot < 0 { + return false + } + tld := h[dot+1:] + if len(tld) < 2 || len(tld) > 24 { + return false + } + for _, r := range tld { + if r < 'a' || r > 'z' { + return false + } + } + return true +} + +// matchCoveredBySpecific reports whether a bare FQDN/e-mail finding is already +// represented by a more precise finding on the same line (e.g. the resolv.conf +// "domain corp.acme.ru" line yields both a resolv and a domain hit). +func matchCoveredBySpecific(match string, specific map[string]struct{}) bool { + m := strings.ToLower(match) + if _, ok := specific[m]; ok { + return true + } + if at := strings.IndexByte(m, '@'); at >= 0 { + if _, ok := specific[m[at+1:]]; ok { + return true + } + } + return false +} diff --git a/internal/privacy/unit_test.go b/internal/privacy/unit_test.go new file mode 100644 index 0000000..e015f7c --- /dev/null +++ b/internal/privacy/unit_test.go @@ -0,0 +1,85 @@ +package privacy + +import ( + "testing" + + "git.mchus.pro/mchus/logpile/internal/models" +) + +func TestIsSensitiveIP(t *testing.T) { + cases := []struct { + ip string + want bool + }{ + {"45.32.10.7", true}, + {"93.184.216.34", true}, + {"2606:2800:220:1:248:1893:25c8:1946", true}, + {"10.1.2.3", false}, + {"172.16.5.5", false}, + {"192.168.31.4", false}, + {"127.0.0.1", false}, + {"169.254.1.1", false}, + {"0.0.0.0", false}, + {"255.255.255.255", false}, + {"224.0.0.1", false}, + {"203.0.113.9", false}, + {"198.51.100.1", false}, + {"192.0.2.7", false}, + {"8.8.8.8", false}, + {"1.1.1.1", false}, + {"100.64.0.1", false}, + {"not-an-ip", false}, + } + for _, c := range cases { + if got := isSensitiveIP(c.ip); got != c.want { + t.Errorf("isSensitiveIP(%q) = %v, want %v", c.ip, got, c.want) + } + } +} + +func TestIsAllowlistedDomain(t *testing.T) { + yes := []string{"example.com", "host.example.local", "pool.ntp.org", "0.pool.ntp.org", "redhat.com", "a.b.jd.com"} + no := []string{"corp.acme.ru", "tcs.example-bank.com", "sigma.internal.io"} + for _, d := range yes { + if !isAllowlistedDomain(d) { + t.Errorf("%q should be allowlisted", d) + } + } + for _, d := range no { + if isAllowlistedDomain(d) { + t.Errorf("%q should not be allowlisted", d) + } + } +} + +func TestRegistrableDomain(t *testing.T) { + cases := map[string]string{ + "sn-x.mgmt.corp.example.local": "example.local", + "ntp01.acme.ru": "acme.ru", + "a.b.c.example.co.uk": "example.co.uk", + "10.20.30.40": "", + "localhost": "", + "com": "", + "*.wildcard.acme.ru": "acme.ru", + } + for in, want := range cases { + if got := registrableDomain(in); got != want { + t.Errorf("registrableDomain(%q) = %q, want %q", in, got, want) + } + } +} + +func TestGuessCustomers_RanksStrongEvidence(t *testing.T) { + findings := []models.PrivacyFinding{ + {Category: catResolv, Path: "resolv.conf", Line: 1, Match: "corp.acme.ru", Excerpt: "domain corp.acme.ru"}, + {Category: catADLDAP, Path: "activedir.conf", Line: 3, Match: "corp.acme.ru", Excerpt: "racdomain=corp.acme.ru"}, + {Category: catDomain, Path: "audit.log", Line: 9, Match: "noise.other.com", Excerpt: "x noise.other.com"}, + } + got := guessCustomers(findings) + if len(got) == 0 || got[0].Domain != "acme.ru" { + t.Fatalf("got %+v, want acme.ru first", got) + } + if got[0].Confidence != "high" { + t.Errorf("confidence = %s, want high", got[0].Confidence) + } +} diff --git a/internal/server/handlers.go b/internal/server/handlers.go index d045b7e..c285c2e 100644 --- a/internal/server/handlers.go +++ b/internal/server/handlers.go @@ -1199,6 +1199,15 @@ func extractFirmwareComponentAndModel(deviceName string) (component, model strin return deviceName, "-" } +func (s *Server) handleGetPrivacyScan(w http.ResponseWriter, r *http.Request) { + result := s.GetResult() + if result == nil || result.PrivacyScan == nil { + jsonResponse(w, map[string]any{"loaded": false}) + return + } + jsonResponse(w, result.PrivacyScan) +} + func (s *Server) handleGetStatus(w http.ResponseWriter, r *http.Request) { result := s.GetResult() if result == nil { diff --git a/internal/server/privacy_scan_test.go b/internal/server/privacy_scan_test.go new file mode 100644 index 0000000..0d88b79 --- /dev/null +++ b/internal/server/privacy_scan_test.go @@ -0,0 +1,101 @@ +package server + +import ( + "archive/zip" + "bytes" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "git.mchus.pro/mchus/logpile/internal/models" +) + +func samplePrivacyScan() *models.PrivacyScan { + return &models.PrivacyScan{ + FilesScanned: 3, + Customers: []models.CustomerGuess{{Domain: "acme.ru", Confidence: "high", Hits: 4}}, + Findings: []models.PrivacyFinding{ + {Category: "resolv", Severity: "high", Path: "resolv.conf", Line: 1, Match: "corp.acme.ru"}, + }, + Summary: models.PrivacySummary{Total: 1, High: 1, ByCategory: map[string]int{"resolv": 1}}, + } +} + +func TestHandleGetPrivacyScan_NotLoaded(t *testing.T) { + srv := &Server{} + req := httptest.NewRequest("GET", "/api/privacy-scan", nil) + w := httptest.NewRecorder() + srv.handleGetPrivacyScan(w, req) + + if w.Code != http.StatusOK { + t.Fatalf("status = %d", w.Code) + } + var body map[string]any + if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil { + t.Fatal(err) + } + if body["loaded"] != false { + t.Fatalf("want loaded:false, got %v", body) + } +} + +func TestHandleGetPrivacyScan_Loaded(t *testing.T) { + srv := &Server{} + srv.SetResult(&models.AnalysisResult{PrivacyScan: samplePrivacyScan()}) + + req := httptest.NewRequest("GET", "/api/privacy-scan", nil) + w := httptest.NewRecorder() + srv.handleGetPrivacyScan(w, req) + + if w.Code != http.StatusOK { + t.Fatalf("status = %d", w.Code) + } + var got models.PrivacyScan + if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil { + t.Fatal(err) + } + if len(got.Customers) != 1 || got.Customers[0].Domain != "acme.ru" { + t.Fatalf("unexpected payload: %+v", got) + } +} + +func TestBuildRawExportBundle_IncludesPrivacyReport(t *testing.T) { + pkg := newRawExportFromUploadedFile("dump.tar.gz", "application/gzip", []byte("x"), &models.AnalysisResult{}) + result := &models.AnalysisResult{PrivacyScan: samplePrivacyScan()} + + raw, err := buildRawExportBundle(pkg, result, "test") + if err != nil { + t.Fatal(err) + } + zr, err := zip.NewReader(bytes.NewReader(raw), int64(len(raw))) + if err != nil { + t.Fatal(err) + } + found := false + for _, f := range zr.File { + if f.Name == rawExportBundlePrivacyFile { + found = true + } + } + if !found { + t.Fatalf("%s missing from bundle", rawExportBundlePrivacyFile) + } +} + +func TestBuildRawExportBundle_NoPrivacyReportWhenNil(t *testing.T) { + pkg := newRawExportFromUploadedFile("dump.tar.gz", "application/gzip", []byte("x"), &models.AnalysisResult{}) + raw, err := buildRawExportBundle(pkg, &models.AnalysisResult{}, "test") + if err != nil { + t.Fatal(err) + } + zr, err := zip.NewReader(bytes.NewReader(raw), int64(len(raw))) + if err != nil { + t.Fatal(err) + } + for _, f := range zr.File { + if f.Name == rawExportBundlePrivacyFile { + t.Fatalf("%s should be absent when PrivacyScan is nil", rawExportBundlePrivacyFile) + } + } +} diff --git a/internal/server/raw_export.go b/internal/server/raw_export.go index 83cf6be..66ab47e 100644 --- a/internal/server/raw_export.go +++ b/internal/server/raw_export.go @@ -19,6 +19,7 @@ const ( rawExportBundlePackageFile = "raw_export.json" rawExportBundleLogFile = "collect.log" rawExportBundleFieldsFile = "parser_fields.json" + rawExportBundlePrivacyFile = "privacy_report.json" ) type RawExportPackage struct { @@ -150,6 +151,20 @@ func buildRawExportBundle(pkg *RawExportPackage, result *models.AnalysisResult, return nil, err } + if result != nil && result.PrivacyScan != nil { + pf, err := zw.Create(rawExportBundlePrivacyFile) + if err != nil { + return nil, err + } + privacyJSON, err := json.MarshalIndent(result.PrivacyScan, "", " ") + if err != nil { + return nil, err + } + if _, err := pf.Write(privacyJSON); err != nil { + return nil, err + } + } + if err := zw.Close(); err != nil { return nil, err } diff --git a/internal/server/server.go b/internal/server/server.go index 4ed5ff4..fa95f9b 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -88,6 +88,7 @@ func (s *Server) setupRoutes() { s.mux.HandleFunc("GET /api/serials", s.handleGetSerials) s.mux.HandleFunc("GET /api/firmware", s.handleGetFirmware) s.mux.HandleFunc("GET /api/parse-errors", s.handleGetParseErrors) + s.mux.HandleFunc("GET /api/privacy-scan", s.handleGetPrivacyScan) s.mux.HandleFunc("GET /api/export/csv", s.handleExportCSV) s.mux.HandleFunc("GET /api/export/json", s.handleExportJSON) s.mux.HandleFunc("GET /api/export/reanimator", s.handleExportReanimator) diff --git a/web/static/css/style.css b/web/static/css/style.css index 6767027..7e07187 100644 --- a/web/static/css/style.css +++ b/web/static/css/style.css @@ -1001,3 +1001,48 @@ code { color: #7a5200; font-weight: 600; } + +.privacy-customer { + padding: 10px 12px; + border-bottom: 1px solid var(--border); + background: #fff8f0; +} + +.privacy-customer.hidden { + display: none; +} + +.privacy-customer-row { + margin-bottom: 6px; +} + +.privacy-conf { + font-size: 0.8em; + text-transform: uppercase; + padding: 1px 6px; + border-radius: 3px; + background: var(--border); +} + +.privacy-conf-high { + color: var(--crit-fg); + font-weight: 600; +} + +.privacy-conf-medium { + color: #7a5200; + font-weight: 600; +} + +.privacy-hits { + color: var(--muted); + font-size: 0.85em; +} + +.privacy-evidence { + margin: 4px 0 0; + padding-left: 18px; + font-family: monospace; + font-size: 0.82em; + color: var(--muted); +} diff --git a/web/static/js/app.js b/web/static/js/app.js index faf0348..2240966 100644 --- a/web/static/js/app.js +++ b/web/static/js/app.js @@ -1415,6 +1415,86 @@ async function loadData(vendor, filename) { loadAuditViewer(); loadParseErrors(); + loadPrivacyScan(); +} + +async function loadPrivacyScan() { + const section = document.getElementById('privacy-section'); + const rows = document.getElementById('privacy-rows'); + const title = document.getElementById('privacy-title'); + const customer = document.getElementById('privacy-customer'); + if (!section || !rows) return; + + let data; + try { + const resp = await fetch('/api/privacy-scan'); + if (!resp.ok) return; + data = await resp.json(); + } catch (e) { + return; + } + + if (!data || data.loaded === false) { + section.classList.add('hidden'); + return; + } + + const findings = Array.isArray(data.findings) ? data.findings : []; + const customers = Array.isArray(data.customers) ? data.customers : []; + if (findings.length === 0 && customers.length === 0) { + section.classList.add('hidden'); + return; + } + + const s = data.summary || {}; + const parts = []; + if (s.high) parts.push(`${s.high} high`); + if (s.medium) parts.push(`${s.medium} medium`); + if (s.low) parts.push(`${s.low} low`); + const lead = customers.length > 0 ? `likely ${customers[0].domain}` : `${findings.length} finding${findings.length > 1 ? 's' : ''}`; + title.textContent = `Customer data — ${lead}${parts.length ? ' · ' + parts.join(', ') : ''}`; + + if (customers.length > 0) { + customer.innerHTML = customers.map(c => + `
${escapeHtml(c.domain)} ` + + `${escapeHtml(c.confidence || '')} ` + + `${c.hits || 0} hit${(c.hits || 0) === 1 ? '' : 's'}` + + ((c.evidence && c.evidence.length) + ? `` + : '') + + `
` + ).join(''); + customer.classList.remove('hidden'); + } else { + customer.innerHTML = ''; + customer.classList.add('hidden'); + } + + rows.innerHTML = ''; + for (const f of findings) { + const loc = f.line ? `${f.path}:${f.line}` : (f.path || ''); + const tr = document.createElement('tr'); + tr.className = `parse-error-row parse-error-${f.severity === 'high' ? 'error' : (f.severity === 'medium' ? 'warning' : 'info')}`; + tr.innerHTML = + `${escapeHtml(f.severity || '')}` + + `${escapeHtml(f.category || '')}` + + `${escapeHtml(loc)}` + + `${escapeHtml(f.match || '')}` + + `${escapeHtml(f.hint || '')}`; + rows.appendChild(tr); + } + + section.classList.remove('hidden'); +} + +let privacyCollapsed = false; +function togglePrivacy() { + const body = document.getElementById('privacy-body'); + const toggle = document.getElementById('privacy-toggle'); + if (!body) return; + privacyCollapsed = !privacyCollapsed; + body.style.display = privacyCollapsed ? 'none' : ''; + toggle.textContent = privacyCollapsed ? '▼' : '▲'; } async function loadParseErrors() { @@ -1535,6 +1615,17 @@ async function clearData() { if (parseErrBody) parseErrBody.style.display = ''; const parseErrToggle = document.getElementById('parse-errors-toggle'); if (parseErrToggle) parseErrToggle.textContent = '▲'; + const privacySection = document.getElementById('privacy-section'); + if (privacySection) privacySection.classList.add('hidden'); + const privacyRows = document.getElementById('privacy-rows'); + if (privacyRows) privacyRows.innerHTML = ''; + const privacyCustomer = document.getElementById('privacy-customer'); + if (privacyCustomer) privacyCustomer.innerHTML = ''; + privacyCollapsed = false; + const privacyBody = document.getElementById('privacy-body'); + if (privacyBody) privacyBody.style.display = ''; + const privacyToggle = document.getElementById('privacy-toggle'); + if (privacyToggle) privacyToggle.textContent = '▲'; } catch (err) { console.error('Failed to clear data:', err); } diff --git a/web/templates/index.html b/web/templates/index.html index 7913d43..d8dea8b 100644 --- a/web/templates/index.html +++ b/web/templates/index.html @@ -190,6 +190,27 @@ +