From f38fb2de696904f32238617e893b196bedbf9ae4 Mon Sep 17 00:00:00 2001 From: Mikhail Chusavitin Date: Thu, 3 Sep 2026 10:02:02 +0300 Subject: [PATCH] feat(privacy): detect an already-sanitized source MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scan now reports PrivacyScan.Sanitized {detected, markers, strong, files, evidence}. SanitizationMarkers recognises a value slot filled with one repeated placeholder + separators (xxxxx.xxxx.xx, x@xxxx.xxxx.xx, 000.00.00.0, a decoy timezone) - it matches the shape, not the literal "x", so evolving the redaction mechanism still trips it. detected requires corroboration: strong>=2, or strong>=1 && markers>=3, or markers>=4. A single filler-looking token is reported (markers:1) but never asserted as sanitized, so a partial future pass or a coincidence does not read as "done". 0.0.0.0 / 000 / UTC / Etc/UTC are too plausibly intentional and do not count. UI: the Customer-data panel shows "файл уже обезличен" and hides the sanitize button when detected. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_014jDYM1nnoZZ3vFz23DDaV1 --- bible-local/03-api.md | 8 +- bible-local/10-decisions.md | 4 + bible-local/docs/log-sanitization.md | 5 + bible-local/docs/privacy-scan.md | 17 +++ internal/models/models.go | 12 ++ internal/privacy/privacy.go | 9 +- internal/privacy/sanitized.go | 187 +++++++++++++++++++++++++++ internal/privacy/sanitized_test.go | 69 ++++++++++ web/static/css/style.css | 8 ++ web/static/js/app.js | 25 +++- 10 files changed, 334 insertions(+), 10 deletions(-) create mode 100644 internal/privacy/sanitized.go create mode 100644 internal/privacy/sanitized_test.go diff --git a/bible-local/03-api.md b/bible-local/03-api.md index 8f01315..c0697b7 100644 --- a/bible-local/03-api.md +++ b/bible-local/03-api.md @@ -155,9 +155,11 @@ Returns normalized parse and collection issues combined from: ### `GET /api/privacy-scan` Returns the customer-data scan for the current dataset (`models.PrivacyScan`: -`files_scanned`, `customers[]`, `findings[]`, `summary`). -Returns `{ "loaded": false }` when nothing is loaded or the scan produced -nothing. Detection only; see `docs/privacy-scan.md`. +`files_scanned`, `customers[]`, `findings[]`, `summary`), plus `sanitizable` +(bool) and, when the source already looks de-identified, `sanitized` +(`{detected, markers, strong, files, evidence}`). +Returns `{ "loaded": false }` when nothing is loaded. Detection only; see +`docs/privacy-scan.md`. ### `POST /api/sanitize` diff --git a/bible-local/10-decisions.md b/bible-local/10-decisions.md index 33e41e3..c74cf8a 100644 --- a/bible-local/10-decisions.md +++ b/bible-local/10-decisions.md @@ -1974,6 +1974,10 @@ with **same-length neutral fillers**, in place. reported, not edited. - Re-parsing a sanitized dump yields the identical hardware inventory (verified on Dell TSR, xFusion, Inspur onekeylog, H3C `.sds`). +- The scan detects an already-sanitized source + (`PrivacyScan.Sanitized`, `privacy/sanitized.go`): it matches the *shape* + of a filled value slot, not the literal `x`, and needs corroborating + markers so a partial future redaction pass does not read as "done". - Full contract and rule list: `bible-local/docs/log-sanitization.md`. - Tests: `internal/sanitize` (`TestRedactText_*`, `TestSanitize_Tar*`, `TestSanitize_Zip_*`, `TestSanitize_BinaryMemberFlagged`, diff --git a/bible-local/docs/log-sanitization.md b/bible-local/docs/log-sanitization.md index bb72c88..522008b 100644 --- a/bible-local/docs/log-sanitization.md +++ b/bible-local/docs/log-sanitization.md @@ -21,6 +21,11 @@ scan (see `privacy-scan.md`). ADL-067. - **No marker.** Nothing is stamped into the file. - **Idempotent.** The fillers are on the privacy allowlist (`isRedactionFiller`), so a re-scan finds nothing and a second `Sanitize` is a no-op. +- **Detectable.** The privacy scan reports an already-sanitized file in + `PrivacyScan.Sanitized` - see `privacy-scan.md`. It needs corroborating + markers, so a single filler-looking token (or a partial future redaction + pass) does not read as "done", and the UI hides the sanitize button when a + file already looks de-identified. ## Entry point diff --git a/bible-local/docs/privacy-scan.md b/bible-local/docs/privacy-scan.md index 54054a8..c02dc1e 100644 --- a/bible-local/docs/privacy-scan.md +++ b/bible-local/docs/privacy-scan.md @@ -115,6 +115,23 @@ default values (`Asia/Shanghai`, `To Be Filled By O.E.M.`, `NULL`, `0.0.0.0`). neutral fillers (see `log-sanitization.md`). The fillers are recognised by `isRedactionFiller` so a scan of a sanitized file is clean. +## Already-sanitized detection (`sanitized.go`) + +`Scan` also reports whether the source already looks de-identified, in +`PrivacyScan.Sanitized` (`{detected, markers, strong, files, evidence}`). + +`SanitizationMarkers(line)` recognises a **value slot filled with one repeated +placeholder + separators** - `xxxxx.xxxx.xx`, `x@xxxx.xxxx.xx`, `000.00.00.0`, +a decoy timezone (`Antarctica/McMurdo`, `Etc/Universal`, ...). It matches the +*shape*, not the literal `x`, so changing the filler character later still +trips it. `0.0.0.0`, `000`, `UTC`, `Etc/UTC` are too plausibly intentional and +do not count. + +`detected` requires corroboration - `strong >= 2`, or `strong >= 1 && markers +>= 3`, or `markers >= 4`. A single filler-looking token is reported +(`markers: 1`) but never asserted as sanitized, so a partial future redaction +pass or a coincidence does not read as "done". + ## Customer guess (`customer.go`) Findings in `domain`, `resolv`, `ad_ldap`, `cert`, `nsupdate`, diff --git a/internal/models/models.go b/internal/models/models.go index b847326..bbdf360 100644 --- a/internal/models/models.go +++ b/internal/models/models.go @@ -34,6 +34,18 @@ type PrivacyScan struct { Customers []CustomerGuess `json:"customers,omitempty"` Findings []PrivacyFinding `json:"findings,omitempty"` Summary PrivacySummary `json:"summary"` + Sanitized *SanitizedReport `json:"sanitized,omitempty"` // filler markers left by internal/sanitize +} + +// SanitizedReport says whether the source already looks de-identified. Detected +// requires corroboration - a single filler-looking token is not enough, so +// evolving the redaction mechanism does not trip false positives. +type SanitizedReport struct { + Detected bool `json:"detected"` + Markers int `json:"markers"` + Strong int `json:"strong"` + Files int `json:"files"` + Evidence []string `json:"evidence,omitempty"` } // CustomerGuess is a registrable domain that most likely identifies the customer, diff --git a/internal/privacy/privacy.go b/internal/privacy/privacy.go index cb5aeb3..d90e661 100644 --- a/internal/privacy/privacy.go +++ b/internal/privacy/privacy.go @@ -48,6 +48,7 @@ func Scan(files []File) *models.PrivacyScan { findings = append(findings, f) } + san := newSanTally() for _, file := range files { if isAllowlistedFilename(file.Path) { continue @@ -56,7 +57,7 @@ func Scan(files []File) *models.PrivacyScan { continue } scanned++ - scanFileLines(file, emit) + scanFileLines(file, emit, san) } if scanned == 0 { @@ -70,11 +71,12 @@ func Scan(files []File) *models.PrivacyScan { Findings: findings, Customers: guessCustomers(findings), Summary: summarize(findings), + Sanitized: san.report(), } return report } -func scanFileLines(file File, emit func(models.PrivacyFinding)) { +func scanFileLines(file File, emit func(models.PrivacyFinding), san *sanTally) { certFile := isCertFilename(file.Path) sc := bufio.NewScanner(bytes.NewReader(file.Content)) sc.Buffer(make([]byte, 0, 64*1024), 1024*1024) @@ -86,6 +88,9 @@ func scanFileLines(file File, emit func(models.PrivacyFinding)) { line = line[:maxScanLineBytes] } scanLine(file.Path, line, ln, certFile, emit) + for _, m := range SanitizationMarkers(line) { + san.add(file.Path, ln, line, m) + } } } diff --git a/internal/privacy/sanitized.go b/internal/privacy/sanitized.go new file mode 100644 index 0000000..f0f8f83 --- /dev/null +++ b/internal/privacy/sanitized.go @@ -0,0 +1,187 @@ +package privacy + +import ( + "regexp" + "strconv" + "strings" + + "git.mchus.pro/mchus/logpile/internal/models" +) + +// A dotted quad, permissive about leading zeros so it also matches the +// zero-padded fillers ("000.00.00.0") that reIPv4 deliberately rejects. +var reLooseQuad = regexp.MustCompile(`\b\d{1,3}(?:\.\d{1,3}){3}\b`) + +// SanitizationMarker is one redaction-filler token found sitting where a real +// value (hostname, IP, e-mail, timezone) would be. It is the fingerprint left +// by internal/sanitize. +type SanitizationMarker struct { + Value string + Strong bool +} + +// Timezone names internal/sanitize substitutes for a real zone. Only the +// implausible ones count - a BMC genuinely in Antarctica, or an operator who +// deliberately wrote "UTC", must not read as "sanitized". +var decoyZones = map[string]struct{}{ + "antarctica/troll": {}, "antarctica/vostok": {}, "antarctica/mcmurdo": {}, + "pacific/tarawa": {}, "pacific/guadalcanal": {}, "pacific/bougainville": {}, + "atlantic/azores": {}, "etc/universal": {}, "us/pacific": {}, + "brazil/east": {}, "canada/yukon": {}, +} + +// SanitizationMarkers returns the redaction-filler tokens on one line. Robust to +// future filler characters: it recognises "a hostname / IP / e-mail / timezone +// slot filled entirely with a single repeated placeholder plus separators", +// not the literal string "x". +func SanitizationMarkers(line string) []SanitizationMarker { + var out []SanitizationMarker + seen := map[string]struct{}{} + push := func(v string, strong bool) { + v = strings.Trim(strings.TrimSpace(v), `"',;:`) + if v == "" { + return + } + if _, ok := seen[v]; ok { + return + } + seen[v] = struct{}{} + out = append(out, SanitizationMarker{Value: v, Strong: strong}) + } + + // A filler in a known config-key value position. + for _, r := range tableRules { + switch r.category { + case catResolv, catADLDAP, catNSUpdate, catTimezone, catCollector, catDHCP: + default: + continue + } + for _, m := range r.re.FindAllStringSubmatch(line, -1) { + if r.group >= len(m) { + continue + } + val := m[r.group] + if isGenericFiller(val) || isDecoyZone(val) { + strong := r.category == catResolv || r.category == catADLDAP || r.category == catNSUpdate + push(val, strong && isGenericFiller(val)) + } + } + } + // A bare FQDN / e-mail made entirely of filler. + for _, v := range reFQDN.FindAllString(line, -1) { + if isGenericFiller(v) && strings.Contains(v, ".") { + push(v, strings.Count(v, ".") >= 2 && len(v) >= 8) + } + } + for _, v := range reEmail.FindAllString(line, -1) { + if isGenericFiller(v) { + push(v, true) + } + } + // A zero-filled IP ("000.00.00.0"); "0.0.0.0" is a real value, not a marker. + for _, v := range reLooseQuad.FindAllString(line, -1) { + if isGenericFiller(v) { + push(v, true) + } + } + // A decoy timezone name anywhere on the line. + for _, v := range reTZName.FindAllString(line, -1) { + if isDecoyZone(v) { + push(v, false) + } + } + return out +} + +func isDecoyZone(s string) bool { + _, ok := decoyZones[strings.ToLower(strings.TrimSpace(s))] + return ok +} + +// isGenericFiller reports whether s is a value slot filled with one repeated +// placeholder character. Letter fills ("xxxxx.xxxx.xx") always qualify; digit +// fills qualify only in the IP shape with a wide group ("00.000.000.00"), so +// "0.0.0.0" and "000" do not. +func isGenericFiller(s string) bool { + parts := strings.FieldsFunc(s, func(r rune) bool { + return r == '.' || r == '-' || r == '_' || r == '@' || r == ':' || r == '/' || r == '+' + }) + if len(parts) < 2 { + return false + } + var fill rune + total, wide := 0, false + for _, p := range parts { + for _, r := range p { + if !isAlnum(r) { + return false + } + if fill == 0 { + fill = r + } else if r != fill { + return false + } + } + total += len(p) + if len(p) >= 2 { + wide = true + } + } + if total < 4 { + return false + } + if fill >= '0' && fill <= '9' { + return len(parts) == 4 && wide // IP shape with a redacted (0-padded) octet + } + return true +} + +// sanTally accumulates markers while Scan walks the files. +type sanTally struct { + strong, total int + files map[string]struct{} + seen map[string]struct{} + evidence []string +} + +func newSanTally() *sanTally { + return &sanTally{files: map[string]struct{}{}, seen: map[string]struct{}{}} +} + +func (t *sanTally) add(path string, ln int, line string, m SanitizationMarker) { + key := m.Value + "|" + path + if _, ok := t.seen[key]; ok { + return + } + t.seen[key] = struct{}{} + t.total++ + if m.Strong { + t.strong++ + } + if path != "" { + t.files[path] = struct{}{} + } + if len(t.evidence) < 3 { + loc := path + if ln > 0 { + loc += ":" + strconv.Itoa(ln) + } + t.evidence = append(t.evidence, loc+": "+excerpt(line)) + } +} + +func (t *sanTally) report() *models.SanitizedReport { + if t.total == 0 { + return nil + } + // One stray filler-looking token proves nothing (coincidence, or a partial + // future redaction pass). Require corroboration. + detected := t.strong >= 2 || (t.strong >= 1 && t.total >= 3) || t.total >= 4 + return &models.SanitizedReport{ + Detected: detected, + Markers: t.total, + Strong: t.strong, + Files: len(t.files), + Evidence: t.evidence, + } +} diff --git a/internal/privacy/sanitized_test.go b/internal/privacy/sanitized_test.go new file mode 100644 index 0000000..e889abc --- /dev/null +++ b/internal/privacy/sanitized_test.go @@ -0,0 +1,69 @@ +package privacy + +import "testing" + +func TestIsGenericFiller(t *testing.T) { + yes := []string{"xxxxx.xxxx.xx", "x@xxxx.xxxx.xx", "xxxxxx/xxxxxx", "000.00.00.0", "00.00.00.00"} + no := []string{"0.0.0.0", "000", "corp.acme.ru", "192.0.2.0", "10.20.30.40", "example.local", "x", "xx"} + for _, s := range yes { + if !isGenericFiller(s) { + t.Errorf("isGenericFiller(%q) = false, want true", s) + } + } + for _, s := range no { + if isGenericFiller(s) { + t.Errorf("isGenericFiller(%q) = true, want false", s) + } + } +} + +func TestSanitizationMarkers_Line(t *testing.T) { + strong := 0 + for _, m := range SanitizationMarkers("domain xxxxxxx.xxxxx") { + if m.Strong { + strong++ + } + } + if strong == 0 { + t.Fatal("redacted resolv domain not detected as a strong marker") + } + if len(SanitizationMarkers("domain corp.acme.ru")) != 0 { + t.Fatal("real domain flagged as a sanitization marker") + } + if len(SanitizationMarkers("gateway 0.0.0.0 unset")) != 0 { + t.Fatal("0.0.0.0 flagged as a marker") + } +} + +func TestScan_SanitizedDetectionNeedsCorroboration(t *testing.T) { + // One lone filler-looking token: reported, but not "detected". + rep := Scan([]File{{Path: "a.log", Content: []byte("host is xxxxx.xxxx.xx today\nnothing else\n")}}) + if rep == nil || rep.Sanitized == nil { + t.Fatal("expected a sanitized report") + } + if rep.Sanitized.Detected { + t.Fatalf("a single marker must not read as sanitized: %+v", rep.Sanitized) + } + if rep.Sanitized.Markers != 1 { + t.Fatalf("markers = %d, want 1", rep.Sanitized.Markers) + } + + // Two strong markers in two files -> detected. + rep2 := Scan([]File{ + {Path: "resolv.conf", Content: []byte("domain xxxxxxx.xxxxx\n")}, + {Path: "nsupdate_temp", Content: []byte("update add xxxxx.xxxxxxx.xx 0 A 10.0.0.1\n")}, + }) + if rep2 == nil || rep2.Sanitized == nil || !rep2.Sanitized.Detected { + t.Fatalf("two-file sanitized dump not detected: %+v", rep2.Sanitized) + } +} + +func TestScan_UnsanitizedHasNoSanitizedReport(t *testing.T) { + rep := Scan([]File{{Path: "resolv.conf", Content: []byte("domain corp.acme.ru\nnameserver 10.0.0.1\n")}}) + if rep == nil { + t.Fatal("nil report") + } + if rep.Sanitized != nil { + t.Fatalf("clean dump got a sanitized report: %+v", rep.Sanitized) + } +} diff --git a/web/static/css/style.css b/web/static/css/style.css index ce08ffe..b87d5e1 100644 --- a/web/static/css/style.css +++ b/web/static/css/style.css @@ -1096,3 +1096,11 @@ code { .privacy-sanitize-err { color: var(--crit-fg); } + +.privacy-sanitized-note { + padding: 6px 8px; + background: #eef7ee; + border: 1px solid #cfe6cf; + border-radius: 4px; + font-size: 0.88em; +} diff --git a/web/static/js/app.js b/web/static/js/app.js index ee9545f..854928e 100644 --- a/web/static/js/app.js +++ b/web/static/js/app.js @@ -1441,7 +1441,9 @@ async function loadPrivacyScan() { const findings = Array.isArray(data.findings) ? data.findings : []; const customers = Array.isArray(data.customers) ? data.customers : []; - if (findings.length === 0 && customers.length === 0) { + const san = data.sanitized || null; + const alreadySanitized = !!(san && san.detected); + if (findings.length === 0 && customers.length === 0 && !alreadySanitized) { section.classList.add('hidden'); return; } @@ -1451,7 +1453,8 @@ async function loadPrivacyScan() { if (s.high) parts.push(`${s.high} high`); if (s.medium) parts.push(`${s.medium} medium`); if (s.low) parts.push(`${s.low} low`); - const lead = customers.length > 0 ? `likely ${customers[0].domain}` : `${findings.length} finding${findings.length > 1 ? 's' : ''}`; + const lead = alreadySanitized ? 'файл уже обезличен' + : (customers.length > 0 ? `likely ${customers[0].domain}` : `${findings.length} finding${findings.length > 1 ? 's' : ''}`); title.textContent = `Customer data — ${lead}${parts.length ? ' · ' + parts.join(', ') : ''}`; if (customers.length > 0) { @@ -1499,11 +1502,23 @@ async function loadPrivacyScan() { const sanBox = document.getElementById('privacy-sanitize'); if (sanBox) { - sanBox.classList.toggle('hidden', !data.sanitizable); + sanBox.classList.toggle('hidden', !data.sanitizable && !alreadySanitized); const prev = document.getElementById('privacy-sanitize-preview'); - if (prev) { prev.classList.add('hidden'); prev.innerHTML = ''; } const btn = document.getElementById('privacy-sanitize-btn'); - if (btn) { btn.disabled = false; btn.textContent = 'Обезличить и скачать копию'; } + if (alreadySanitized) { + if (btn) btn.classList.add('hidden'); + if (prev) { + prev.innerHTML = `
Файл выглядит уже обезличенным: ` + + `${san.markers} маркер(ов) в ${san.files} файл(ах).` + + ((san.evidence && san.evidence.length) + ? `
    ${san.evidence.map(e => `
  • ${escapeHtml(e)}
    • `).join('')}
` + : '') + `
`; + prev.classList.remove('hidden'); + } + } else { + if (btn) { btn.classList.remove('hidden'); btn.disabled = false; btn.textContent = 'Обезличить и скачать копию'; } + if (prev) { prev.classList.add('hidden'); prev.innerHTML = ''; } + } } section.classList.remove('hidden');