Files

T

Mikhail ChusavitinandClaude Sonnet 5 a63bb17438 feat(sanitize): in-place, length-preserving log de-identification

Adds internal/sanitize: rewrites the customer-identifying spans that
internal/privacy detects (domain/FQDN/e-mail/AD/public-IP/timezone) with
same-length neutral fillers, in place, without changing the file format.

- Fillers keep byte length: "sigma.sbrf.ru" -> "xxxxx.xxxx.xx", IP ->
  "00.000.000.00", "Europe/Moscow" -> "Etc/Universal" (same-length valid
  neutral IANA zone), offset "180" -> "000". Timestamps are not recomputed.
- Lossless recursive archive walk (tar/.sds/gz/tgz/zip): entry names, modes,
  and all embedded timestamps preserved; untouched zip entries copied raw;
  member payload length unchanged so tar headers stay byte-identical; only the
  .gz/.zip compression layer is rebuilt. 0 redactions -> byte-identical output.
- privacy.FindSpans is the one matcher shared by detection and redaction;
  fillers are recognised by isRedactionFiller so a re-scan / second pass is a
  no-op. New privacy FPs fixed along the way: syslog selectors (local7.info),
  "MEVersion" firmware quads, *.conf_bak vendor templates, bundled viewer
  domains.
- Binary members (FRU.bin, localtime, redis-dump.rdb, SOL captures) and
  unreadable nested archives are reported in Result.SkippedBinary, never edited.
- Surfaces: POST /api/sanitize (+ GET /api/sanitize/download), the "Обезличить
  и скачать копию" button in the Customer-data panel, and
  logpile -sanitize <file> (restores mtime/atime).

Verified: re-parsing a sanitized Dell TSR / xFusion / Inspur onekeylog / H3C
.sds yields the identical hardware inventory; re-scan is clean. ADL-067,
bible-local/docs/log-sanitization.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-09-02 18:05:27 +03:00

4.3 KiB

Raw Blame History

Log sanitization

internal/sanitize produces a de-identified copy of an uploaded diagnostic file. It is the redaction counterpart of the detection-only internal/privacy scan (see privacy-scan.md). ADL-067.

Contract

Same format, same structure. Archive entry names, modes, uid/gid, and all embedded timestamps (tar ModTime/AccessTime/ChangeTime, gzip Name/ ModTime/OS, zip Modified/Extra) are preserved. Entry order is kept.
Length-preserving replacement. Every filler is the exact byte length of the token it replaces, so tar/zip member sizes - and therefore the tar headers and their checksums - are byte-identical. Untouched zip entries are copied raw ((*zip.Writer).Copy).
Byte-identical when nothing changes. An uncompressed .tar/.sds/.txt/ .log with no redactable data comes out equal to the input. .gz/.zip cannot be byte-identical because the compression layer is always rebuilt (member payloads and metadata still match; only the compressed stream and the total size differ).
No marker. Nothing is stamped into the file.
Idempotent. The fillers are on the privacy allowlist (isRedactionFiller), so a re-scan finds nothing and a second Sanitize is a no-op.

Entry point

sanitize.Sanitize(filename string, data []byte) (*sanitize.Result, error)
sanitize.CanSanitize(filename string) bool   // .tar .sds .gz .tgz .zip .txt .log

Result{ Data, Changes []Change, SkippedBinary []string, TotalReplaced }. Format is resolved by extension; nested archives (.tar.gz in a .zip, ...) are rewritten recursively.

Fillers

Category	Example	Filler
`domain`, `mgmt_subdomain`, `resolv`, `ad_ldap`, `nsupdate`, `collector`, `dhcp`, `cert`	`sigma.sbrf.ru`	letters/digits → `x`, punctuation kept → `xxxxx.xxxx.xx`
`email`	`admin@corp.acme.ru`	`xxxxx@xxxx.xxxx.xx`
`public_ip`	`93.184.216.34`	digits → `0`, dots kept → `00.000.000.00` (`net.ParseIP` → nil)
`timezone` name	`Europe/Moscow`	same-length valid neutral IANA zone from `neutralZoneByLen` (`Etc/Universal`, `Antarctica/McMurdo`, ...)
`timezone` offset	`SELTimeUTCOffset=180`	`000`
`timezone` abbr	`... MSK 2026`	`UTC` (len 3), else `x`-fill

Timezone redaction changes the config value only; event timestamps are never recomputed, so a re-analysis of the sanitized dump may read event times in the wrong zone (the same trade-off as the KB manual cleanup).

fru_location spans are not redacted (often a serial / manufacturing code, sometimes in a binary FRU area) - only reported by the scan.

Private IPs, pool.ntp.org and everything else on the privacy allowlist are never touched. Vendor factory-template members (*_tianyiyun, *.conf_bak, raw_export.json, ...) are skipped whole (privacy.IsAllowlistedFile).

Not edited (`Result.SkippedBinary`)

Reported for manual handling, never modified:

Binary members with a customer string in a printable run: FRU.bin (Asset Tag), redis-dump.rdb, SDR.dat, SOL captures with control bytes, racsessioninfo / session_token.
configuration/conf/localtime - a real UTC tzdata blob is a different length, so it cannot be swapped in place.
Truncated / mis-named nested archives (copied verbatim).
HPE .ahs (proprietary container).

Surfaces

Where	How
API	`POST /api/sanitize` runs it on the retained upload bytes and returns the preview JSON (`total_replaced`, `changes[]`, `skipped_binary[]`); `GET /api/sanitize/download` streams the file (`Content-Disposition: attachment; filename="<original>"`, `application/octet-stream`). `422` for live-Redfish / snapshot sources.
UI	"Обезличить и скачать копию" in the Customer-data panel (shown when `privacy-scan` reports `sanitizable: true`) → preview → download.
CLI	`logpile -sanitize <file> [-sanitize-out <file>]` - edits in place (or writes `-sanitize-out`), restores mtime/atime via `os.Chtimes`, prints the change summary. ctime reflects the edit.

Limits

Whole file is held in memory and rebuilt: maxInputBytes 800 MiB, maxMemberBytes 1 GiB per decompressed member.
Fixtures and tests use acme.ru / corp.acme.local - never a real customer domain, same rule as the privacy scan.

4.3 KiB Raw Blame History