feat(sanitize): in-place, length-preserving log de-identification

Adds internal/sanitize: rewrites the customer-identifying spans that
internal/privacy detects (domain/FQDN/e-mail/AD/public-IP/timezone) with
same-length neutral fillers, in place, without changing the file format.

- Fillers keep byte length: "sigma.sbrf.ru" -> "xxxxx.xxxx.xx", IP ->
  "00.000.000.00", "Europe/Moscow" -> "Etc/Universal" (same-length valid
  neutral IANA zone), offset "180" -> "000". Timestamps are not recomputed.
- Lossless recursive archive walk (tar/.sds/gz/tgz/zip): entry names, modes,
  and all embedded timestamps preserved; untouched zip entries copied raw;
  member payload length unchanged so tar headers stay byte-identical; only the
  .gz/.zip compression layer is rebuilt. 0 redactions -> byte-identical output.
- privacy.FindSpans is the one matcher shared by detection and redaction;
  fillers are recognised by isRedactionFiller so a re-scan / second pass is a
  no-op. New privacy FPs fixed along the way: syslog selectors (local7.info),
  "MEVersion" firmware quads, *.conf_bak vendor templates, bundled viewer
  domains.
- Binary members (FRU.bin, localtime, redis-dump.rdb, SOL captures) and
  unreadable nested archives are reported in Result.SkippedBinary, never edited.
- Surfaces: POST /api/sanitize (+ GET /api/sanitize/download), the "Обезличить
  и скачать копию" button in the Customer-data panel, and
  logpile -sanitize <file> (restores mtime/atime).

Verified: re-parsing a sanitized Dell TSR / xFusion / Inspur onekeylog / H3C
.sds yields the identical hardware inventory; re-scan is clean. ADL-067,
bible-local/docs/log-sanitization.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
2026-09-02 18:05:27 +03:00
co-authored by Claude Sonnet 5
parent e74e01ad05
commit a63bb17438
25 changed files with 1742 additions and 66 deletions
+42
View File
@@ -1937,3 +1937,45 @@ token, remediation hint).
`TestIsAllowlistedDomain`, `TestRegistrableDomain`, `TestGuessCustomers_*`),
`internal/server` (`TestHandleGetPrivacyScan_*`,
`TestBuildRawExportBundle_*PrivacyReport*`).
---
## ADL-067 — In-place, length-preserving log sanitization
**Date:** 2026-09-02
**Context:** ADL-066 detects customer-identifying data but the operator still
redacts by hand before forwarding a dump. The redaction has to keep the file
usable and unremarkable: same format, same archive structure, same embedded
timestamps, no "sanitized by" marker.
**Decision:** `internal/sanitize` rewrites the spans `internal/privacy` finds
with **same-length neutral fillers**, in place.
- Fillers: hostname/e-mail/AD/cert → letters and digits to `x`, punctuation
kept (`sigma.sbrf.ru` → `xxxxx.xxxx.xx`); public IP → digits to `0`
(`93.184.216.34` → `00.000.000.00`, `net.ParseIP` → nil); timezone name →
a same-length valid neutral IANA zone from a curated `len → zone` table
(`Europe/Moscow` → `Etc/Universal`); UTC offset → zeros (`180` → `000`).
Timestamps are never recomputed.
- Because member payload length never changes, tar/zip entry headers,
checksums, names, modes and mtimes are byte-identical; for `.gz`/`.zip` only
the compression layer is rebuilt. Uncompressed `.tar`/`.sds`/plain text with
nothing to redact come out byte-for-byte identical.
- Detection and redaction share one matcher: `privacy.FindSpans`. The fillers
are recognised by `isRedactionFiller` so a re-scan / second pass is a no-op.
- Not edited: binary members (`FRU.bin`, `localtime` tzdata, `redis-dump.rdb`,
DER certs), unreadable nested archives — listed in `Result.SkippedBinary`
for manual handling. `.gz`/`.zip` cannot be byte-identical (recompression).
ctime is not restorable on a CLI in-place edit.
- Surfaced by `POST /api/sanitize` (+ `GET /api/sanitize/download`), the
"Обезличить и скачать копию" button in the Customer-data panel, and
`logpile -sanitize <file>` (restores mtime/atime via `os.Chtimes`).
**Consequences:**
- Only formats the walker can rebuild losslessly are offered
(`sanitize.CanSanitize`): `.tar .sds .gz .tgz .zip .txt .log`. AHS is
reported, not edited.
- Re-parsing a sanitized dump yields the identical hardware inventory
(verified on Dell TSR, xFusion, Inspur onekeylog, H3C `.sds`).
- Full contract and rule list: `bible-local/docs/log-sanitization.md`.
- Tests: `internal/sanitize` (`TestRedactText_*`, `TestSanitize_Tar*`,
`TestSanitize_Zip_*`, `TestSanitize_BinaryMemberFlagged`,
`TestNeutralZonesAreValidAndSameLength`, `TestTZFiller`),
`internal/server` (`TestHandleSanitize_*`).