Skip to content

Commit

Permalink
util/log: make sensitive data redactable
Browse files Browse the repository at this point in the history
See "log redaction" RFC from #48076 for details.

General concept:

- logs stored on disk contain markers around data items potentially
  containing PII or confidential data (“unsafe data”).

- `debug zip`, `debug merge-log`, and the `Logs` and `LogFiles` RPCs
  edit the log entries upon request to replace all the unsafe data by
  `‹×›`.

See also the release notes below for a summary of UX.

Example redacted output:

```
I200426 20:06:45.558765 1 cli/start.go:671 ⋮ GEOS initialized at ‹×›
I200426 20:06:45.558772 1 cli/start.go:676 ⋮ starting cockroach node
I200426 20:06:45.562212 75 storage/rocksdb.go:606 ⋮ opening rocksdb instance at ‹×›
I200426 20:06:45.574608 75 server/server.go:750 ⋮ [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I200426 20:06:45.574782 75 storage/rocksdb.go:606 ⋮ opening rocksdb instance at ‹×›
I200426 20:06:45.581451 75 server/config.go:576 ⋮ [n?] ‹×› storage engine‹×› initialized
I200426 20:06:45.581459 75 server/config.go:579 ⋮ [n?] ‹×›
I200426 20:06:45.581466 75 server/config.go:579 ⋮ [n?] ‹×›
I200426 20:06:45.602204 75 server/server.go:1270 ⋮ [n?] **** add additional nodes by specifying --join=‹×›
I200426 20:06:45.602218 75 server/init.go:166 ⋮ [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node.
I200426 20:06:45.602234 75 server/init.go:186 ⋮ [n?] **** cluster ‹×› has been created
I200426 20:06:47.676567 117 gossip/gossip.go:1538 ⋮ [n1] node has connected to cluster via gossip
I200426 20:06:47.676620 117 kv/kvserver/stores.go:255 ⋮ [n1] wrote ‹×› node addresses to persistent storage
I200508 12:34:37.238123 1346 kv/kvserver/replica_consistency.go:255 ⋮ [n1,consistencyChecker,s2,r‹×›] triggering stats recomputation to resolve delta of ‹×›
```

This output was produced from the following raw log:

```
I200426 20:06:45.558765 1 cli/start.go:671 ⋮ GEOS initialized at ‹/data/home/kena/src/go/src/github.com/cockroachdb/cockroach/lib/libgeos_c.so›
I200426 20:06:45.558772 1 cli/start.go:676 ⋮ starting cockroach node
I200426 20:06:45.562212 75 storage/rocksdb.go:606 ⋮ opening rocksdb instance at ‹"/data/home/kena/src/go/src/github.com/cockroachdb/cockroach/cockroach-data/cockroach-temp302681397"›
I200426 20:06:45.574608 75 server/server.go:750 ⋮ [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I200426 20:06:45.574782 75 storage/rocksdb.go:606 ⋮ opening rocksdb instance at ‹"/data/home/kena/src/go/src/github.com/cockroachdb/cockroach/cockroach-data"›
I200426 20:06:45.581451 75 server/config.go:576 ⋮ [n?] ‹1› storage engine‹› initialized
I200426 20:06:45.581459 75 server/config.go:579 ⋮ [n?] ‹RocksDB cache size: 128 MiB›
I200426 20:06:45.581466 75 server/config.go:579 ⋮ [n?] ‹store 0: RocksDB, max size 0 B, max open file limit 1878853›
I200426 20:06:45.602204 75 server/server.go:1270 ⋮ [n?] **** add additional nodes by specifying --join=‹kenax:26257›
I200426 20:06:45.602218 75 server/init.go:166 ⋮ [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node.
I200426 20:06:45.602234 75 server/init.go:186 ⋮ [n?] **** cluster ‹2a7d34b5-a2aa-469b-8efd-ef7deb62ca73› has been created
I200426 20:06:47.676567 117 gossip/gossip.go:1538 ⋮ [n1] node has connected to cluster via gossip
I200426 20:06:47.676620 117 kv/kvserver/stores.go:255 ⋮ [n1] wrote ‹0› node addresses to persistent storage
I200508 12:34:37.238123 1346 kv/kvserver/replica_consistency.go:255 ⋮ [n1,consistencyChecker,s2,r‹4/1:/System{/tsd-tse}›] triggering stats recomputation to resolve delta of ‹{ContainsEstimates:1438 LastUpdateNanos:1588941276231756380 IntentAge:0 GCBytesAge:0 LiveBytes:-35159 LiveCount:-692 KeyBytes:-33660 KeyCount:-692 ValBytes:-1499 ValCount:-692 IntentBytes:0 IntentCount:0 SysBytes:0 SysCount:0}›
```

Release note (cli change): Certain kinds of advanced troubleshooting
information, for example internal details from the Go runtime, are now
output to a separate log file in the configured log directory.  This
file is named `<program>-stderr.xxx.log`.

Release note (cli change): The server CLI commands (`start`,
`start-single-node`, `demo`) now support a new flag
`--redactable-logs`, which introduces markers in generated log entries
to facilitate redaction of sensitive information by `cockroach debug
zip` and other CockroachDB APIs. This flag is currently not enabled by
default, but is recommended for all deployments and will be enabled by
default in a later version.

This feature is experimental.

Note: the flag is *advertised* (in `--help`) by `cockroach start` and
`cockroach start-single-node`, where log files are expected as a
matter of course; it is also supported, yet *hidden* (from `--help`),
by every other CLI command because every command can also produce log
entries when passed `--log-dir` or `--logtostderr`.

Release note (cli change): The commands `cockroach debug zip` and
`cockroach debug merge-logs` support a new flag `--redact-logs`. When
specified, it causes log messages that potentially contain
confidential data or PII to be redacted away.

This feature is experimental.

Note 1: At the time of this writing, this redaction is extremely
aggressive and may hinder the ability of Cockroach Labs to investigate
issues from log files, especially if `--redactable-logs` is not
enabled server-side.

Note 2: Other potentially-confidential data may still be retrieved by
`cockroach debug zip`; the new flag only applies to retrieved log
files inside the output zip file.
  • Loading branch information
knz committed Jun 6, 2020
1 parent 14a5fe7 commit f19398c
Show file tree
Hide file tree
Showing 64 changed files with 2,756 additions and 919 deletions.
131 changes: 119 additions & 12 deletions c-deps/libroach/protos/util/log/log.pb.cc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

122 changes: 116 additions & 6 deletions c-deps/libroach/protos/util/log/log.pb.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit f19398c

Please sign in to comment.