Files

T

Mikhail ChusavitinandClaude Sonnet 5 f38fb2de69 feat(privacy): detect an already-sanitized source

Scan now reports PrivacyScan.Sanitized {detected, markers, strong, files,
evidence}. SanitizationMarkers recognises a value slot filled with one
repeated placeholder + separators (xxxxx.xxxx.xx, x@xxxx.xxxx.xx,
000.00.00.0, a decoy timezone) - it matches the shape, not the literal "x",
so evolving the redaction mechanism still trips it.

detected requires corroboration: strong>=2, or strong>=1 && markers>=3, or
markers>=4. A single filler-looking token is reported (markers:1) but never
asserted as sanitized, so a partial future pass or a coincidence does not
read as "done". 0.0.0.0 / 000 / UTC / Etc/UTC are too plausibly intentional
and do not count.

UI: the Customer-data panel shows "файл уже обезличен" and hides the
sanitize button when detected.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jDYM1nnoZZ3vFz23DDaV1

2026-09-03 10:02:02 +03:00

4.6 KiB

Raw Permalink Blame History

Log sanitization

internal/sanitize produces a de-identified copy of an uploaded diagnostic file. It is the redaction counterpart of the detection-only internal/privacy scan (see privacy-scan.md). ADL-067.

Contract

Same format, same structure. Archive entry names, modes, uid/gid, and all embedded timestamps (tar ModTime/AccessTime/ChangeTime, gzip Name/ ModTime/OS, zip Modified/Extra) are preserved. Entry order is kept.
Length-preserving replacement. Every filler is the exact byte length of the token it replaces, so tar/zip member sizes - and therefore the tar headers and their checksums - are byte-identical. Untouched zip entries are copied raw ((*zip.Writer).Copy).
Byte-identical when nothing changes. An uncompressed .tar/.sds/.txt/ .log with no redactable data comes out equal to the input. .gz/.zip cannot be byte-identical because the compression layer is always rebuilt (member payloads and metadata still match; only the compressed stream and the total size differ).
No marker. Nothing is stamped into the file.
Idempotent. The fillers are on the privacy allowlist (isRedactionFiller), so a re-scan finds nothing and a second Sanitize is a no-op.
Detectable. The privacy scan reports an already-sanitized file in PrivacyScan.Sanitized - see privacy-scan.md. It needs corroborating markers, so a single filler-looking token (or a partial future redaction pass) does not read as "done", and the UI hides the sanitize button when a file already looks de-identified.

Entry point

sanitize.Sanitize(filename string, data []byte) (*sanitize.Result, error)
sanitize.CanSanitize(filename string) bool   // .tar .sds .gz .tgz .zip .txt .log

Result{ Data, Changes []Change, SkippedBinary []string, TotalReplaced }. Format is resolved by extension; nested archives (.tar.gz in a .zip, ...) are rewritten recursively.

Fillers

Category	Example	Filler
`domain`, `mgmt_subdomain`, `resolv`, `ad_ldap`, `nsupdate`, `collector`, `dhcp`, `cert`	`sigma.sbrf.ru`	letters/digits → `x`, punctuation kept → `xxxxx.xxxx.xx`
`email`	`admin@corp.acme.ru`	`xxxxx@xxxx.xxxx.xx`
`public_ip`	`93.184.216.34`	digits → `0`, dots kept → `00.000.000.00` (`net.ParseIP` → nil)
`timezone` name	`Europe/Moscow`	same-length valid neutral IANA zone from `neutralZoneByLen` (`Etc/Universal`, `Antarctica/McMurdo`, ...)
`timezone` offset	`SELTimeUTCOffset=180`	`000`
`timezone` abbr	`... MSK 2026`	`UTC` (len 3), else `x`-fill

Timezone redaction changes the config value only; event timestamps are never recomputed, so a re-analysis of the sanitized dump may read event times in the wrong zone (the same trade-off as the KB manual cleanup).

fru_location spans are not redacted (often a serial / manufacturing code, sometimes in a binary FRU area) - only reported by the scan.

Private IPs, pool.ntp.org and everything else on the privacy allowlist are never touched. Vendor factory-template members (*_tianyiyun, *.conf_bak, raw_export.json, ...) are skipped whole (privacy.IsAllowlistedFile).

Not edited (`Result.SkippedBinary`)

Reported for manual handling, never modified:

Binary members with a customer string in a printable run: FRU.bin (Asset Tag), redis-dump.rdb, SDR.dat, SOL captures with control bytes, racsessioninfo / session_token.
configuration/conf/localtime - a real UTC tzdata blob is a different length, so it cannot be swapped in place.
Truncated / mis-named nested archives (copied verbatim).
HPE .ahs (proprietary container).

Surfaces

Where	How
API	`POST /api/sanitize` runs it on the retained upload bytes and returns the preview JSON (`total_replaced`, `changes[]`, `skipped_binary[]`); `GET /api/sanitize/download` streams the file (`Content-Disposition: attachment; filename="<original>"`, `application/octet-stream`). `422` for live-Redfish / snapshot sources.
UI	"Обезличить и скачать копию" in the Customer-data panel (shown when `privacy-scan` reports `sanitizable: true`) → preview → download.
CLI	`logpile -sanitize <file> [-sanitize-out <file>]` - edits in place (or writes `-sanitize-out`), restores mtime/atime via `os.Chtimes`, prints the change summary. ctime reflects the edit.

Limits

Whole file is held in memory and rebuilt: maxInputBytes 800 MiB, maxMemberBytes 1 GiB per decompressed member.
Fixtures and tests use acme.ru / corp.acme.local - never a real customer domain, same rule as the privacy scan.

4.6 KiB Raw Permalink Blame History