Mikhail ChusavitinandClaude Sonnet 5
a63bb17438
Adds internal/sanitize: rewrites the customer-identifying spans that internal/privacy detects (domain/FQDN/e-mail/AD/public-IP/timezone) with same-length neutral fillers, in place, without changing the file format. - Fillers keep byte length: "sigma.sbrf.ru" -> "xxxxx.xxxx.xx", IP -> "00.000.000.00", "Europe/Moscow" -> "Etc/Universal" (same-length valid neutral IANA zone), offset "180" -> "000". Timestamps are not recomputed. - Lossless recursive archive walk (tar/.sds/gz/tgz/zip): entry names, modes, and all embedded timestamps preserved; untouched zip entries copied raw; member payload length unchanged so tar headers stay byte-identical; only the .gz/.zip compression layer is rebuilt. 0 redactions -> byte-identical output. - privacy.FindSpans is the one matcher shared by detection and redaction; fillers are recognised by isRedactionFiller so a re-scan / second pass is a no-op. New privacy FPs fixed along the way: syslog selectors (local7.info), "MEVersion" firmware quads, *.conf_bak vendor templates, bundled viewer domains. - Binary members (FRU.bin, localtime, redis-dump.rdb, SOL captures) and unreadable nested archives are reported in Result.SkippedBinary, never edited. - Surfaces: POST /api/sanitize (+ GET /api/sanitize/download), the "Обезличить и скачать копию" button in the Customer-data panel, and logpile -sanitize <file> (restores mtime/atime). Verified: re-parsing a sanitized Dell TSR / xFusion / Inspur onekeylog / H3C .sds yields the identical hardware inventory; re-scan is clean. ADL-067, bible-local/docs/log-sanitization.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
83 lines
4.3 KiB
Markdown
83 lines
4.3 KiB
Markdown
# Log sanitization
|
|
|
|
`internal/sanitize` produces a de-identified copy of an uploaded diagnostic
|
|
file. It is the redaction counterpart of the detection-only `internal/privacy`
|
|
scan (see `privacy-scan.md`). ADL-067.
|
|
|
|
## Contract
|
|
|
|
- **Same format, same structure.** Archive entry names, modes, uid/gid, and all
|
|
embedded timestamps (tar `ModTime`/`AccessTime`/`ChangeTime`, gzip `Name`/
|
|
`ModTime`/`OS`, zip `Modified`/`Extra`) are preserved. Entry order is kept.
|
|
- **Length-preserving replacement.** Every filler is the exact byte length of
|
|
the token it replaces, so tar/zip member sizes - and therefore the tar
|
|
headers and their checksums - are byte-identical. Untouched zip entries are
|
|
copied raw (`(*zip.Writer).Copy`).
|
|
- **Byte-identical when nothing changes.** An uncompressed `.tar`/`.sds`/`.txt`/
|
|
`.log` with no redactable data comes out equal to the input. `.gz`/`.zip`
|
|
cannot be byte-identical because the compression layer is always rebuilt
|
|
(member payloads and metadata still match; only the compressed stream and the
|
|
total size differ).
|
|
- **No marker.** Nothing is stamped into the file.
|
|
- **Idempotent.** The fillers are on the privacy allowlist (`isRedactionFiller`),
|
|
so a re-scan finds nothing and a second `Sanitize` is a no-op.
|
|
|
|
## Entry point
|
|
|
|
```go
|
|
sanitize.Sanitize(filename string, data []byte) (*sanitize.Result, error)
|
|
sanitize.CanSanitize(filename string) bool // .tar .sds .gz .tgz .zip .txt .log
|
|
```
|
|
|
|
`Result{ Data, Changes []Change, SkippedBinary []string, TotalReplaced }`.
|
|
Format is resolved by extension; nested archives (`.tar.gz` in a `.zip`, ...)
|
|
are rewritten recursively.
|
|
|
|
## Fillers
|
|
|
|
| Category | Example | Filler |
|
|
|----------|---------|--------|
|
|
| `domain`, `mgmt_subdomain`, `resolv`, `ad_ldap`, `nsupdate`, `collector`, `dhcp`, `cert` | `sigma.sbrf.ru` | letters/digits → `x`, punctuation kept → `xxxxx.xxxx.xx` |
|
|
| `email` | `admin@corp.acme.ru` | `xxxxx@xxxx.xxxx.xx` |
|
|
| `public_ip` | `93.184.216.34` | digits → `0`, dots kept → `00.000.000.00` (`net.ParseIP` → nil) |
|
|
| `timezone` name | `Europe/Moscow` | same-length valid neutral IANA zone from `neutralZoneByLen` (`Etc/Universal`, `Antarctica/McMurdo`, ...) |
|
|
| `timezone` offset | `SELTimeUTCOffset=180` | `000` |
|
|
| `timezone` abbr | `... MSK 2026` | `UTC` (len 3), else `x`-fill |
|
|
|
|
Timezone redaction changes the config value only; **event timestamps are never
|
|
recomputed**, so a re-analysis of the sanitized dump may read event times in the
|
|
wrong zone (the same trade-off as the KB manual cleanup).
|
|
|
|
`fru_location` spans are **not** redacted (often a serial / manufacturing code,
|
|
sometimes in a binary FRU area) - only reported by the scan.
|
|
|
|
Private IPs, `pool.ntp.org` and everything else on the privacy allowlist are
|
|
never touched. Vendor factory-template members (`*_tianyiyun`, `*.conf_bak`,
|
|
`raw_export.json`, ...) are skipped whole (`privacy.IsAllowlistedFile`).
|
|
|
|
## Not edited (`Result.SkippedBinary`)
|
|
|
|
Reported for manual handling, never modified:
|
|
- Binary members with a customer string in a printable run: `FRU.bin`
|
|
(Asset Tag), `redis-dump.rdb`, `SDR.dat`, SOL captures with control bytes,
|
|
`racsessioninfo` / `session_token`.
|
|
- `configuration/conf/localtime` - a real UTC tzdata blob is a different length,
|
|
so it cannot be swapped in place.
|
|
- Truncated / mis-named nested archives (copied verbatim).
|
|
- HPE `.ahs` (proprietary container).
|
|
|
|
## Surfaces
|
|
|
|
| Where | How |
|
|
|-------|-----|
|
|
| API | `POST /api/sanitize` runs it on the retained upload bytes and returns the preview JSON (`total_replaced`, `changes[]`, `skipped_binary[]`); `GET /api/sanitize/download` streams the file (`Content-Disposition: attachment; filename="<original>"`, `application/octet-stream`). `422` for live-Redfish / snapshot sources. |
|
|
| UI | "Обезличить и скачать копию" in the Customer-data panel (shown when `privacy-scan` reports `sanitizable: true`) → preview → download. |
|
|
| CLI | `logpile -sanitize <file> [-sanitize-out <file>]` - edits in place (or writes `-sanitize-out`), restores mtime/atime via `os.Chtimes`, prints the change summary. ctime reflects the edit. |
|
|
|
|
## Limits
|
|
|
|
- Whole file is held in memory and rebuilt: `maxInputBytes` 800 MiB,
|
|
`maxMemberBytes` 1 GiB per decompressed member.
|
|
- Fixtures and tests use `acme.ru` / `corp.acme.local` - never a real customer
|
|
domain, same rule as the privacy scan.
|