feat(privacy): detect an already-sanitized source

Scan now reports PrivacyScan.Sanitized {detected, markers, strong, files,
evidence}. SanitizationMarkers recognises a value slot filled with one
repeated placeholder + separators (xxxxx.xxxx.xx, x@xxxx.xxxx.xx,
000.00.00.0, a decoy timezone) - it matches the shape, not the literal "x",
so evolving the redaction mechanism still trips it.

detected requires corroboration: strong>=2, or strong>=1 && markers>=3, or
markers>=4. A single filler-looking token is reported (markers:1) but never
asserted as sanitized, so a partial future pass or a coincidence does not
read as "done". 0.0.0.0 / 000 / UTC / Etc/UTC are too plausibly intentional
and do not count.

UI: the Customer-data panel shows "файл уже обезличен" and hides the
sanitize button when detected.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jDYM1nnoZZ3vFz23DDaV1
This commit is contained in:
Mikhail Chusavitin
2026-09-03 10:02:02 +03:00
co-authored by Claude Sonnet 5
parent a63bb17438
commit f38fb2de69
10 changed files with 334 additions and 10 deletions
+5 -3
View File
@@ -155,9 +155,11 @@ Returns normalized parse and collection issues combined from:
### `GET /api/privacy-scan`
Returns the customer-data scan for the current dataset (`models.PrivacyScan`:
`files_scanned`, `customers[]`, `findings[]`, `summary`).
Returns `{ "loaded": false }` when nothing is loaded or the scan produced
nothing. Detection only; see `docs/privacy-scan.md`.
`files_scanned`, `customers[]`, `findings[]`, `summary`), plus `sanitizable`
(bool) and, when the source already looks de-identified, `sanitized`
(`{detected, markers, strong, files, evidence}`).
Returns `{ "loaded": false }` when nothing is loaded. Detection only; see
`docs/privacy-scan.md`.
### `POST /api/sanitize`
+4
View File
@@ -1974,6 +1974,10 @@ with **same-length neutral fillers**, in place.
reported, not edited.
- Re-parsing a sanitized dump yields the identical hardware inventory
(verified on Dell TSR, xFusion, Inspur onekeylog, H3C `.sds`).
- The scan detects an already-sanitized source
(`PrivacyScan.Sanitized`, `privacy/sanitized.go`): it matches the *shape*
of a filled value slot, not the literal `x`, and needs corroborating
markers so a partial future redaction pass does not read as "done".
- Full contract and rule list: `bible-local/docs/log-sanitization.md`.
- Tests: `internal/sanitize` (`TestRedactText_*`, `TestSanitize_Tar*`,
`TestSanitize_Zip_*`, `TestSanitize_BinaryMemberFlagged`,
+5
View File
@@ -21,6 +21,11 @@ scan (see `privacy-scan.md`). ADL-067.
- **No marker.** Nothing is stamped into the file.
- **Idempotent.** The fillers are on the privacy allowlist (`isRedactionFiller`),
so a re-scan finds nothing and a second `Sanitize` is a no-op.
- **Detectable.** The privacy scan reports an already-sanitized file in
`PrivacyScan.Sanitized` - see `privacy-scan.md`. It needs corroborating
markers, so a single filler-looking token (or a partial future redaction
pass) does not read as "done", and the UI hides the sanitize button when a
file already looks de-identified.
## Entry point
+17
View File
@@ -115,6 +115,23 @@ default values (`Asia/Shanghai`, `To Be Filled By O.E.M.`, `NULL`, `0.0.0.0`).
neutral fillers (see `log-sanitization.md`). The fillers are recognised by
`isRedactionFiller` so a scan of a sanitized file is clean.
## Already-sanitized detection (`sanitized.go`)
`Scan` also reports whether the source already looks de-identified, in
`PrivacyScan.Sanitized` (`{detected, markers, strong, files, evidence}`).
`SanitizationMarkers(line)` recognises a **value slot filled with one repeated
placeholder + separators** - `xxxxx.xxxx.xx`, `x@xxxx.xxxx.xx`, `000.00.00.0`,
a decoy timezone (`Antarctica/McMurdo`, `Etc/Universal`, ...). It matches the
*shape*, not the literal `x`, so changing the filler character later still
trips it. `0.0.0.0`, `000`, `UTC`, `Etc/UTC` are too plausibly intentional and
do not count.
`detected` requires corroboration - `strong >= 2`, or `strong >= 1 && markers
>= 3`, or `markers >= 4`. A single filler-looking token is reported
(`markers: 1`) but never asserted as sanitized, so a partial future redaction
pass or a coincidence does not read as "done".
## Customer guess (`customer.go`)
Findings in `domain`, `resolv`, `ad_ldap`, `cert`, `nsupdate`,