feat(sanitize): in-place, length-preserving log de-identification
Adds internal/sanitize: rewrites the customer-identifying spans that internal/privacy detects (domain/FQDN/e-mail/AD/public-IP/timezone) with same-length neutral fillers, in place, without changing the file format. - Fillers keep byte length: "sigma.sbrf.ru" -> "xxxxx.xxxx.xx", IP -> "00.000.000.00", "Europe/Moscow" -> "Etc/Universal" (same-length valid neutral IANA zone), offset "180" -> "000". Timestamps are not recomputed. - Lossless recursive archive walk (tar/.sds/gz/tgz/zip): entry names, modes, and all embedded timestamps preserved; untouched zip entries copied raw; member payload length unchanged so tar headers stay byte-identical; only the .gz/.zip compression layer is rebuilt. 0 redactions -> byte-identical output. - privacy.FindSpans is the one matcher shared by detection and redaction; fillers are recognised by isRedactionFiller so a re-scan / second pass is a no-op. New privacy FPs fixed along the way: syslog selectors (local7.info), "MEVersion" firmware quads, *.conf_bak vendor templates, bundled viewer domains. - Binary members (FRU.bin, localtime, redis-dump.rdb, SOL captures) and unreadable nested archives are reported in Result.SkippedBinary, never edited. - Surfaces: POST /api/sanitize (+ GET /api/sanitize/download), the "Обезличить и скачать копию" button in the Customer-data panel, and logpile -sanitize <file> (restores mtime/atime). Verified: re-parsing a sanitized Dell TSR / xFusion / Inspur onekeylog / H3C .sds yields the identical hardware inventory; re-scan is clean. ADL-067, bible-local/docs/log-sanitization.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Mikhail Chusavitin
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
e74e01ad05
commit
a63bb17438
@@ -24,6 +24,7 @@ All modes converge on the same normalized hardware model and exporter pipeline.
|
||||
- Reanimator export and batch conversion workflows
|
||||
- Embedded `pci.ids` lookup for vendor/device name enrichment
|
||||
- Customer-data / anonymization scan of ingested sources (detection only; see `docs/privacy-scan.md`)
|
||||
- In-place, length-preserving de-identification of an uploaded dump (`docs/log-sanitization.md`)
|
||||
|
||||
## Current vendor coverage
|
||||
|
||||
|
||||
@@ -159,6 +159,27 @@ Returns the customer-data scan for the current dataset (`models.PrivacyScan`:
|
||||
Returns `{ "loaded": false }` when nothing is loaded or the scan produced
|
||||
nothing. Detection only; see `docs/privacy-scan.md`.
|
||||
|
||||
### `POST /api/sanitize`
|
||||
|
||||
Builds a de-identified copy of the current uploaded file (customer-data spans
|
||||
replaced with same-length neutral fillers, format and timestamps preserved) and
|
||||
returns the change preview:
|
||||
|
||||
```json
|
||||
{ "filename": "...", "input_size": 0, "output_size": 0,
|
||||
"total_replaced": 0, "changes": [{"path","category","count","sample_before","sample_after"}],
|
||||
"skipped_binary": ["member (binary - sanitize manually)"] }
|
||||
```
|
||||
|
||||
`422` when the current dataset is not an uploaded file (live Redfish / snapshot)
|
||||
or the format cannot be rebuilt. See `docs/log-sanitization.md`.
|
||||
|
||||
### `GET /api/sanitize/download`
|
||||
|
||||
Streams the file produced by the last `POST /api/sanitize`
|
||||
(`Content-Disposition: attachment; filename="<original>"`,
|
||||
`Content-Type: application/octet-stream`). `404` if none is ready.
|
||||
|
||||
### `GET /api/parsers`
|
||||
|
||||
Returns registered parser metadata.
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
| `GET /api/export/reanimator` | JSON | Reanimator hardware payload |
|
||||
| `GET /chart/current?print=true` | HTML (auto-print) | Print/PDF version of the report — opens in new tab, calls `window.print()` |
|
||||
| `POST /api/convert` | async ZIP artifact | Batch archive-to-Reanimator conversion |
|
||||
| `POST /api/sanitize` + `GET /api/sanitize/download` | same-format file | De-identified copy of the upload (`docs/log-sanitization.md`) |
|
||||
|
||||
## Raw export
|
||||
|
||||
|
||||
@@ -8,6 +8,8 @@ Defined in `cmd/logpile/main.go`:
|
||||
|------|---------|---------|
|
||||
| `--port` | `8082` | HTTP server port |
|
||||
| `--file` | empty | Preload archive file |
|
||||
| `--sanitize` | empty | De-identify customer data in this file and exit (`docs/log-sanitization.md`) |
|
||||
| `--sanitize-out` | empty | Where `--sanitize` writes its result (default: overwrite the input) |
|
||||
| `--version` | `false` | Print version and exit |
|
||||
| `--no-browser` | `false` | Do not auto-open browser |
|
||||
| `--hold-on-crash` | `true` on Windows | Keep console open after fatal crash |
|
||||
|
||||
@@ -1937,3 +1937,45 @@ token, remediation hint).
|
||||
`TestIsAllowlistedDomain`, `TestRegistrableDomain`, `TestGuessCustomers_*`),
|
||||
`internal/server` (`TestHandleGetPrivacyScan_*`,
|
||||
`TestBuildRawExportBundle_*PrivacyReport*`).
|
||||
|
||||
---
|
||||
|
||||
## ADL-067 — In-place, length-preserving log sanitization
|
||||
|
||||
**Date:** 2026-09-02
|
||||
**Context:** ADL-066 detects customer-identifying data but the operator still
|
||||
redacts by hand before forwarding a dump. The redaction has to keep the file
|
||||
usable and unremarkable: same format, same archive structure, same embedded
|
||||
timestamps, no "sanitized by" marker.
|
||||
**Decision:** `internal/sanitize` rewrites the spans `internal/privacy` finds
|
||||
with **same-length neutral fillers**, in place.
|
||||
- Fillers: hostname/e-mail/AD/cert → letters and digits to `x`, punctuation
|
||||
kept (`sigma.sbrf.ru` → `xxxxx.xxxx.xx`); public IP → digits to `0`
|
||||
(`93.184.216.34` → `00.000.000.00`, `net.ParseIP` → nil); timezone name →
|
||||
a same-length valid neutral IANA zone from a curated `len → zone` table
|
||||
(`Europe/Moscow` → `Etc/Universal`); UTC offset → zeros (`180` → `000`).
|
||||
Timestamps are never recomputed.
|
||||
- Because member payload length never changes, tar/zip entry headers,
|
||||
checksums, names, modes and mtimes are byte-identical; for `.gz`/`.zip` only
|
||||
the compression layer is rebuilt. Uncompressed `.tar`/`.sds`/plain text with
|
||||
nothing to redact come out byte-for-byte identical.
|
||||
- Detection and redaction share one matcher: `privacy.FindSpans`. The fillers
|
||||
are recognised by `isRedactionFiller` so a re-scan / second pass is a no-op.
|
||||
- Not edited: binary members (`FRU.bin`, `localtime` tzdata, `redis-dump.rdb`,
|
||||
DER certs), unreadable nested archives — listed in `Result.SkippedBinary`
|
||||
for manual handling. `.gz`/`.zip` cannot be byte-identical (recompression).
|
||||
ctime is not restorable on a CLI in-place edit.
|
||||
- Surfaced by `POST /api/sanitize` (+ `GET /api/sanitize/download`), the
|
||||
"Обезличить и скачать копию" button in the Customer-data panel, and
|
||||
`logpile -sanitize <file>` (restores mtime/atime via `os.Chtimes`).
|
||||
**Consequences:**
|
||||
- Only formats the walker can rebuild losslessly are offered
|
||||
(`sanitize.CanSanitize`): `.tar .sds .gz .tgz .zip .txt .log`. AHS is
|
||||
reported, not edited.
|
||||
- Re-parsing a sanitized dump yields the identical hardware inventory
|
||||
(verified on Dell TSR, xFusion, Inspur onekeylog, H3C `.sds`).
|
||||
- Full contract and rule list: `bible-local/docs/log-sanitization.md`.
|
||||
- Tests: `internal/sanitize` (`TestRedactText_*`, `TestSanitize_Tar*`,
|
||||
`TestSanitize_Zip_*`, `TestSanitize_BinaryMemberFlagged`,
|
||||
`TestNeutralZonesAreValidAndSameLength`, `TestTZFiller`),
|
||||
`internal/server` (`TestHandleSanitize_*`).
|
||||
|
||||
@@ -23,6 +23,7 @@ Keep top-level docs minimal and put maintained architecture/API contracts here.
|
||||
| [07-exporters.md](07-exporters.md) | Raw export, Reanimator export, batch convert |
|
||||
| [docs/hardware-ingest-contract.md](docs/hardware-ingest-contract.md) | Reanimator ingest schema mirrored locally |
|
||||
| [docs/privacy-scan.md](docs/privacy-scan.md) | Customer-data / anonymization scan of ingested sources |
|
||||
| [docs/log-sanitization.md](docs/log-sanitization.md) | In-place, length-preserving redaction of customer data |
|
||||
| [08-build-release.md](08-build-release.md) | Build and release workflow |
|
||||
| [09-testing.md](09-testing.md) | Test expectations and regression rules |
|
||||
| [10-decisions.md](10-decisions.md) | Architectural Decision Log |
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
# Log sanitization
|
||||
|
||||
`internal/sanitize` produces a de-identified copy of an uploaded diagnostic
|
||||
file. It is the redaction counterpart of the detection-only `internal/privacy`
|
||||
scan (see `privacy-scan.md`). ADL-067.
|
||||
|
||||
## Contract
|
||||
|
||||
- **Same format, same structure.** Archive entry names, modes, uid/gid, and all
|
||||
embedded timestamps (tar `ModTime`/`AccessTime`/`ChangeTime`, gzip `Name`/
|
||||
`ModTime`/`OS`, zip `Modified`/`Extra`) are preserved. Entry order is kept.
|
||||
- **Length-preserving replacement.** Every filler is the exact byte length of
|
||||
the token it replaces, so tar/zip member sizes - and therefore the tar
|
||||
headers and their checksums - are byte-identical. Untouched zip entries are
|
||||
copied raw (`(*zip.Writer).Copy`).
|
||||
- **Byte-identical when nothing changes.** An uncompressed `.tar`/`.sds`/`.txt`/
|
||||
`.log` with no redactable data comes out equal to the input. `.gz`/`.zip`
|
||||
cannot be byte-identical because the compression layer is always rebuilt
|
||||
(member payloads and metadata still match; only the compressed stream and the
|
||||
total size differ).
|
||||
- **No marker.** Nothing is stamped into the file.
|
||||
- **Idempotent.** The fillers are on the privacy allowlist (`isRedactionFiller`),
|
||||
so a re-scan finds nothing and a second `Sanitize` is a no-op.
|
||||
|
||||
## Entry point
|
||||
|
||||
```go
|
||||
sanitize.Sanitize(filename string, data []byte) (*sanitize.Result, error)
|
||||
sanitize.CanSanitize(filename string) bool // .tar .sds .gz .tgz .zip .txt .log
|
||||
```
|
||||
|
||||
`Result{ Data, Changes []Change, SkippedBinary []string, TotalReplaced }`.
|
||||
Format is resolved by extension; nested archives (`.tar.gz` in a `.zip`, ...)
|
||||
are rewritten recursively.
|
||||
|
||||
## Fillers
|
||||
|
||||
| Category | Example | Filler |
|
||||
|----------|---------|--------|
|
||||
| `domain`, `mgmt_subdomain`, `resolv`, `ad_ldap`, `nsupdate`, `collector`, `dhcp`, `cert` | `sigma.sbrf.ru` | letters/digits → `x`, punctuation kept → `xxxxx.xxxx.xx` |
|
||||
| `email` | `admin@corp.acme.ru` | `xxxxx@xxxx.xxxx.xx` |
|
||||
| `public_ip` | `93.184.216.34` | digits → `0`, dots kept → `00.000.000.00` (`net.ParseIP` → nil) |
|
||||
| `timezone` name | `Europe/Moscow` | same-length valid neutral IANA zone from `neutralZoneByLen` (`Etc/Universal`, `Antarctica/McMurdo`, ...) |
|
||||
| `timezone` offset | `SELTimeUTCOffset=180` | `000` |
|
||||
| `timezone` abbr | `... MSK 2026` | `UTC` (len 3), else `x`-fill |
|
||||
|
||||
Timezone redaction changes the config value only; **event timestamps are never
|
||||
recomputed**, so a re-analysis of the sanitized dump may read event times in the
|
||||
wrong zone (the same trade-off as the KB manual cleanup).
|
||||
|
||||
`fru_location` spans are **not** redacted (often a serial / manufacturing code,
|
||||
sometimes in a binary FRU area) - only reported by the scan.
|
||||
|
||||
Private IPs, `pool.ntp.org` and everything else on the privacy allowlist are
|
||||
never touched. Vendor factory-template members (`*_tianyiyun`, `*.conf_bak`,
|
||||
`raw_export.json`, ...) are skipped whole (`privacy.IsAllowlistedFile`).
|
||||
|
||||
## Not edited (`Result.SkippedBinary`)
|
||||
|
||||
Reported for manual handling, never modified:
|
||||
- Binary members with a customer string in a printable run: `FRU.bin`
|
||||
(Asset Tag), `redis-dump.rdb`, `SDR.dat`, SOL captures with control bytes,
|
||||
`racsessioninfo` / `session_token`.
|
||||
- `configuration/conf/localtime` - a real UTC tzdata blob is a different length,
|
||||
so it cannot be swapped in place.
|
||||
- Truncated / mis-named nested archives (copied verbatim).
|
||||
- HPE `.ahs` (proprietary container).
|
||||
|
||||
## Surfaces
|
||||
|
||||
| Where | How |
|
||||
|-------|-----|
|
||||
| API | `POST /api/sanitize` runs it on the retained upload bytes and returns the preview JSON (`total_replaced`, `changes[]`, `skipped_binary[]`); `GET /api/sanitize/download` streams the file (`Content-Disposition: attachment; filename="<original>"`, `application/octet-stream`). `422` for live-Redfish / snapshot sources. |
|
||||
| UI | "Обезличить и скачать копию" in the Customer-data panel (shown when `privacy-scan` reports `sanitizable: true`) → preview → download. |
|
||||
| CLI | `logpile -sanitize <file> [-sanitize-out <file>]` - edits in place (or writes `-sanitize-out`), restores mtime/atime via `os.Chtimes`, prints the change summary. ctime reflects the edit. |
|
||||
|
||||
## Limits
|
||||
|
||||
- Whole file is held in memory and rebuilt: `maxInputBytes` 800 MiB,
|
||||
`maxMemberBytes` 1 GiB per decompressed member.
|
||||
- Fixtures and tests use `acme.ru` / `corp.acme.local` - never a real customer
|
||||
domain, same rule as the privacy scan.
|
||||
@@ -108,6 +108,13 @@ Reference data, not vendor-detection logic: RFC 2606/5737 names, `pool.ntp.org`
|
||||
`foobar.edu` / `issue.net` (stock `hosts.allow` / sshd banner), and factory
|
||||
default values (`Asia/Shanghai`, `To Be Filled By O.E.M.`, `NULL`, `0.0.0.0`).
|
||||
|
||||
## Redaction
|
||||
|
||||
`privacy.FindSpans(line, certFile)` is the matcher shared with
|
||||
`internal/sanitize`, which rewrites the same spans in place with same-length
|
||||
neutral fillers (see `log-sanitization.md`). The fillers are recognised by
|
||||
`isRedactionFiller` so a scan of a sanitized file is clean.
|
||||
|
||||
## Customer guess (`customer.go`)
|
||||
|
||||
Findings in `domain`, `resolv`, `ad_ldap`, `cert`, `nsupdate`,
|
||||
|
||||
Reference in New Issue
Block a user