feat(sanitize): in-place, length-preserving log de-identification
Adds internal/sanitize: rewrites the customer-identifying spans that internal/privacy detects (domain/FQDN/e-mail/AD/public-IP/timezone) with same-length neutral fillers, in place, without changing the file format. - Fillers keep byte length: "sigma.sbrf.ru" -> "xxxxx.xxxx.xx", IP -> "00.000.000.00", "Europe/Moscow" -> "Etc/Universal" (same-length valid neutral IANA zone), offset "180" -> "000". Timestamps are not recomputed. - Lossless recursive archive walk (tar/.sds/gz/tgz/zip): entry names, modes, and all embedded timestamps preserved; untouched zip entries copied raw; member payload length unchanged so tar headers stay byte-identical; only the .gz/.zip compression layer is rebuilt. 0 redactions -> byte-identical output. - privacy.FindSpans is the one matcher shared by detection and redaction; fillers are recognised by isRedactionFiller so a re-scan / second pass is a no-op. New privacy FPs fixed along the way: syslog selectors (local7.info), "MEVersion" firmware quads, *.conf_bak vendor templates, bundled viewer domains. - Binary members (FRU.bin, localtime, redis-dump.rdb, SOL captures) and unreadable nested archives are reported in Result.SkippedBinary, never edited. - Surfaces: POST /api/sanitize (+ GET /api/sanitize/download), the "Обезличить и скачать копию" button in the Customer-data panel, and logpile -sanitize <file> (restores mtime/atime). Verified: re-parsing a sanitized Dell TSR / xFusion / Inspur onekeylog / H3C .sds yields the identical hardware inventory; re-scan is clean. ADL-067, bible-local/docs/log-sanitization.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
e74e01ad05
commit
a63bb17438
@@ -1937,3 +1937,45 @@ token, remediation hint).
|
||||
`TestIsAllowlistedDomain`, `TestRegistrableDomain`, `TestGuessCustomers_*`),
|
||||
`internal/server` (`TestHandleGetPrivacyScan_*`,
|
||||
`TestBuildRawExportBundle_*PrivacyReport*`).
|
||||
|
||||
---
|
||||
|
||||
## ADL-067 — In-place, length-preserving log sanitization
|
||||
|
||||
**Date:** 2026-09-02
|
||||
**Context:** ADL-066 detects customer-identifying data but the operator still
|
||||
redacts by hand before forwarding a dump. The redaction has to keep the file
|
||||
usable and unremarkable: same format, same archive structure, same embedded
|
||||
timestamps, no "sanitized by" marker.
|
||||
**Decision:** `internal/sanitize` rewrites the spans `internal/privacy` finds
|
||||
with **same-length neutral fillers**, in place.
|
||||
- Fillers: hostname/e-mail/AD/cert → letters and digits to `x`, punctuation
|
||||
kept (`sigma.sbrf.ru` → `xxxxx.xxxx.xx`); public IP → digits to `0`
|
||||
(`93.184.216.34` → `00.000.000.00`, `net.ParseIP` → nil); timezone name →
|
||||
a same-length valid neutral IANA zone from a curated `len → zone` table
|
||||
(`Europe/Moscow` → `Etc/Universal`); UTC offset → zeros (`180` → `000`).
|
||||
Timestamps are never recomputed.
|
||||
- Because member payload length never changes, tar/zip entry headers,
|
||||
checksums, names, modes and mtimes are byte-identical; for `.gz`/`.zip` only
|
||||
the compression layer is rebuilt. Uncompressed `.tar`/`.sds`/plain text with
|
||||
nothing to redact come out byte-for-byte identical.
|
||||
- Detection and redaction share one matcher: `privacy.FindSpans`. The fillers
|
||||
are recognised by `isRedactionFiller` so a re-scan / second pass is a no-op.
|
||||
- Not edited: binary members (`FRU.bin`, `localtime` tzdata, `redis-dump.rdb`,
|
||||
DER certs), unreadable nested archives — listed in `Result.SkippedBinary`
|
||||
for manual handling. `.gz`/`.zip` cannot be byte-identical (recompression).
|
||||
ctime is not restorable on a CLI in-place edit.
|
||||
- Surfaced by `POST /api/sanitize` (+ `GET /api/sanitize/download`), the
|
||||
"Обезличить и скачать копию" button in the Customer-data panel, and
|
||||
`logpile -sanitize <file>` (restores mtime/atime via `os.Chtimes`).
|
||||
**Consequences:**
|
||||
- Only formats the walker can rebuild losslessly are offered
|
||||
(`sanitize.CanSanitize`): `.tar .sds .gz .tgz .zip .txt .log`. AHS is
|
||||
reported, not edited.
|
||||
- Re-parsing a sanitized dump yields the identical hardware inventory
|
||||
(verified on Dell TSR, xFusion, Inspur onekeylog, H3C `.sds`).
|
||||
- Full contract and rule list: `bible-local/docs/log-sanitization.md`.
|
||||
- Tests: `internal/sanitize` (`TestRedactText_*`, `TestSanitize_Tar*`,
|
||||
`TestSanitize_Zip_*`, `TestSanitize_BinaryMemberFlagged`,
|
||||
`TestNeutralZonesAreValidAndSameLength`, `TestTZFiller`),
|
||||
`internal/server` (`TestHandleSanitize_*`).
|
||||
|
||||
Reference in New Issue
Block a user