Mikhail ChusavitinandClaude Sonnet 5
f38fb2de69
Scan now reports PrivacyScan.Sanitized {detected, markers, strong, files,
evidence}. SanitizationMarkers recognises a value slot filled with one
repeated placeholder + separators (xxxxx.xxxx.xx, x@xxxx.xxxx.xx,
000.00.00.0, a decoy timezone) - it matches the shape, not the literal "x",
so evolving the redaction mechanism still trips it.
detected requires corroboration: strong>=2, or strong>=1 && markers>=3, or
markers>=4. A single filler-looking token is reported (markers:1) but never
asserted as sanitized, so a partial future pass or a coincidence does not
read as "done". 0.0.0.0 / 000 / UTC / Etc/UTC are too plausibly intentional
and do not count.
UI: the Customer-data panel shows "файл уже обезличен" and hides the
sanitize button when detected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jDYM1nnoZZ3vFz23DDaV1
88 lines
4.6 KiB
Markdown
88 lines
4.6 KiB
Markdown
# Log sanitization
|
|
|
|
`internal/sanitize` produces a de-identified copy of an uploaded diagnostic
|
|
file. It is the redaction counterpart of the detection-only `internal/privacy`
|
|
scan (see `privacy-scan.md`). ADL-067.
|
|
|
|
## Contract
|
|
|
|
- **Same format, same structure.** Archive entry names, modes, uid/gid, and all
|
|
embedded timestamps (tar `ModTime`/`AccessTime`/`ChangeTime`, gzip `Name`/
|
|
`ModTime`/`OS`, zip `Modified`/`Extra`) are preserved. Entry order is kept.
|
|
- **Length-preserving replacement.** Every filler is the exact byte length of
|
|
the token it replaces, so tar/zip member sizes - and therefore the tar
|
|
headers and their checksums - are byte-identical. Untouched zip entries are
|
|
copied raw (`(*zip.Writer).Copy`).
|
|
- **Byte-identical when nothing changes.** An uncompressed `.tar`/`.sds`/`.txt`/
|
|
`.log` with no redactable data comes out equal to the input. `.gz`/`.zip`
|
|
cannot be byte-identical because the compression layer is always rebuilt
|
|
(member payloads and metadata still match; only the compressed stream and the
|
|
total size differ).
|
|
- **No marker.** Nothing is stamped into the file.
|
|
- **Idempotent.** The fillers are on the privacy allowlist (`isRedactionFiller`),
|
|
so a re-scan finds nothing and a second `Sanitize` is a no-op.
|
|
- **Detectable.** The privacy scan reports an already-sanitized file in
|
|
`PrivacyScan.Sanitized` - see `privacy-scan.md`. It needs corroborating
|
|
markers, so a single filler-looking token (or a partial future redaction
|
|
pass) does not read as "done", and the UI hides the sanitize button when a
|
|
file already looks de-identified.
|
|
|
|
## Entry point
|
|
|
|
```go
|
|
sanitize.Sanitize(filename string, data []byte) (*sanitize.Result, error)
|
|
sanitize.CanSanitize(filename string) bool // .tar .sds .gz .tgz .zip .txt .log
|
|
```
|
|
|
|
`Result{ Data, Changes []Change, SkippedBinary []string, TotalReplaced }`.
|
|
Format is resolved by extension; nested archives (`.tar.gz` in a `.zip`, ...)
|
|
are rewritten recursively.
|
|
|
|
## Fillers
|
|
|
|
| Category | Example | Filler |
|
|
|----------|---------|--------|
|
|
| `domain`, `mgmt_subdomain`, `resolv`, `ad_ldap`, `nsupdate`, `collector`, `dhcp`, `cert` | `sigma.sbrf.ru` | letters/digits → `x`, punctuation kept → `xxxxx.xxxx.xx` |
|
|
| `email` | `admin@corp.acme.ru` | `xxxxx@xxxx.xxxx.xx` |
|
|
| `public_ip` | `93.184.216.34` | digits → `0`, dots kept → `00.000.000.00` (`net.ParseIP` → nil) |
|
|
| `timezone` name | `Europe/Moscow` | same-length valid neutral IANA zone from `neutralZoneByLen` (`Etc/Universal`, `Antarctica/McMurdo`, ...) |
|
|
| `timezone` offset | `SELTimeUTCOffset=180` | `000` |
|
|
| `timezone` abbr | `... MSK 2026` | `UTC` (len 3), else `x`-fill |
|
|
|
|
Timezone redaction changes the config value only; **event timestamps are never
|
|
recomputed**, so a re-analysis of the sanitized dump may read event times in the
|
|
wrong zone (the same trade-off as the KB manual cleanup).
|
|
|
|
`fru_location` spans are **not** redacted (often a serial / manufacturing code,
|
|
sometimes in a binary FRU area) - only reported by the scan.
|
|
|
|
Private IPs, `pool.ntp.org` and everything else on the privacy allowlist are
|
|
never touched. Vendor factory-template members (`*_tianyiyun`, `*.conf_bak`,
|
|
`raw_export.json`, ...) are skipped whole (`privacy.IsAllowlistedFile`).
|
|
|
|
## Not edited (`Result.SkippedBinary`)
|
|
|
|
Reported for manual handling, never modified:
|
|
- Binary members with a customer string in a printable run: `FRU.bin`
|
|
(Asset Tag), `redis-dump.rdb`, `SDR.dat`, SOL captures with control bytes,
|
|
`racsessioninfo` / `session_token`.
|
|
- `configuration/conf/localtime` - a real UTC tzdata blob is a different length,
|
|
so it cannot be swapped in place.
|
|
- Truncated / mis-named nested archives (copied verbatim).
|
|
- HPE `.ahs` (proprietary container).
|
|
|
|
## Surfaces
|
|
|
|
| Where | How |
|
|
|-------|-----|
|
|
| API | `POST /api/sanitize` runs it on the retained upload bytes and returns the preview JSON (`total_replaced`, `changes[]`, `skipped_binary[]`); `GET /api/sanitize/download` streams the file (`Content-Disposition: attachment; filename="<original>"`, `application/octet-stream`). `422` for live-Redfish / snapshot sources. |
|
|
| UI | "Обезличить и скачать копию" in the Customer-data panel (shown when `privacy-scan` reports `sanitizable: true`) → preview → download. |
|
|
| CLI | `logpile -sanitize <file> [-sanitize-out <file>]` - edits in place (or writes `-sanitize-out`), restores mtime/atime via `os.Chtimes`, prints the change summary. ctime reflects the edit. |
|
|
|
|
## Limits
|
|
|
|
- Whole file is held in memory and rebuilt: `maxInputBytes` 800 MiB,
|
|
`maxMemberBytes` 1 GiB per decompressed member.
|
|
- Fixtures and tests use `acme.ru` / `corp.acme.local` - never a real customer
|
|
domain, same rule as the privacy scan.
|