Merge #48051 #49978

48051: util/log: automatic log redaction r=tbg,bdarnell a=knz

Fixes #46031 

See "log redaction" RFC from #48076 for details.

General concept:

- logs stored on disk contain markers around data items potentially
  containing PII or confidential data (“unsafe data”).

- `debug zip`, `debug merge-log`, and the `Logs` and `LogFiles` RPCs
  edit the log entries upon request to replace all the unsafe data by
  `‹×›`.

See also the release notes below for a summary of UX.

Example redacted output:

```
I200426 20:06:45.558765 1 cli/start.go:671 ⋮ GEOS initialized at ‹×›
I200426 20:06:45.558772 1 cli/start.go:676 ⋮ starting cockroach node
I200426 20:06:45.562212 75 storage/rocksdb.go:606 ⋮ opening rocksdb instance at ‹×›
I200426 20:06:45.574608 75 server/server.go:750 ⋮ [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I200426 20:06:45.574782 75 storage/rocksdb.go:606 ⋮ opening rocksdb instance at ‹×›
I200426 20:06:45.581451 75 server/config.go:576 ⋮ [n?] ‹×› storage engine‹×› initialized
I200426 20:06:45.581459 75 server/config.go:579 ⋮ [n?] ‹×›
I200426 20:06:45.581466 75 server/config.go:579 ⋮ [n?] ‹×›
I200426 20:06:45.602204 75 server/server.go:1270 ⋮ [n?] **** add additional nodes by specifying --join=‹×›
I200426 20:06:45.602218 75 server/init.go:166 ⋮ [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node.
I200426 20:06:45.602234 75 server/init.go:186 ⋮ [n?] **** cluster ‹×› has been created
I200426 20:06:47.676567 117 gossip/gossip.go:1538 ⋮ [n1] node has connected to cluster via gossip
I200426 20:06:47.676620 117 kv/kvserver/stores.go:255 ⋮ [n1] wrote ‹×› node addresses to persistent storage
I200508 12:34:37.238123 1346 kv/kvserver/replica_consistency.go:255 ⋮ [n1,consistencyChecker,s2,r‹×›] triggering stats recomputation to resolve delta of ‹×›
```

This output was produced from the following raw log:

```
I200426 20:06:45.558765 1 cli/start.go:671 ⋮ GEOS initialized at ‹/data/home/kena/src/go/src/github.com/cockroachdb/cockroach/lib/libgeos_c.so›
I200426 20:06:45.558772 1 cli/start.go:676 ⋮ starting cockroach node
I200426 20:06:45.562212 75 storage/rocksdb.go:606 ⋮ opening rocksdb instance at ‹"/data/home/kena/src/go/src/github.com/cockroachdb/cockroach/cockroach-data/cockroach-temp302681397"›
I200426 20:06:45.574608 75 server/server.go:750 ⋮ [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I200426 20:06:45.574782 75 storage/rocksdb.go:606 ⋮ opening rocksdb instance at ‹"/data/home/kena/src/go/src/github.com/cockroachdb/cockroach/cockroach-data"›
I200426 20:06:45.581451 75 server/config.go:576 ⋮ [n?] ‹1› storage engine‹› initialized
I200426 20:06:45.581459 75 server/config.go:579 ⋮ [n?] ‹RocksDB cache size: 128 MiB›
I200426 20:06:45.581466 75 server/config.go:579 ⋮ [n?] ‹store 0: RocksDB, max size 0 B, max open file limit 1878853›
I200426 20:06:45.602204 75 server/server.go:1270 ⋮ [n?] **** add additional nodes by specifying --join=‹kenax:26257›
I200426 20:06:45.602218 75 server/init.go:166 ⋮ [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node.
I200426 20:06:45.602234 75 server/init.go:186 ⋮ [n?] **** cluster ‹2a7d34b5-a2aa-469b-8efd-ef7deb62ca73› has been created
I200426 20:06:47.676567 117 gossip/gossip.go:1538 ⋮ [n1] node has connected to cluster via gossip
I200426 20:06:47.676620 117 kv/kvserver/stores.go:255 ⋮ [n1] wrote ‹0› node addresses to persistent storage
I200508 12:34:37.238123 1346 kv/kvserver/replica_consistency.go:255 ⋮ [n1,consistencyChecker,s2,r‹4/1:/System{/tsd-tse}›] triggering stats recomputation to resolve delta of ‹{ContainsEstimates:1438 LastUpdateNanos:1588941276231756380 IntentAge:0 GCBytesAge:0 LiveBytes:-35159 LiveCount:-692 KeyBytes:-33660 KeyCount:-692 ValBytes:-1499 ValCount:-692 IntentBytes:0 IntentCount:0 SysBytes:0 SysCount:0}›
```

Release note (cli change): Certain kinds of advanced troubleshooting
information, for example internal details from the Go runtime, are now
output to a separate log file in the configured log directory.  This
file is named `<program>-stderr.xxx.log`.

Release note (cli change): The server CLI commands (`start`,
`start-single-node`, `demo`) now support a new flag
`--redactable-logs`, which introduces markers in generated log entries
to facilitate redaction of sensitive information by `cockroach debug
zip` and other CockroachDB APIs. This flag is currently not enabled by
default, but is recommended for all deployments and will be enabled by
default in a later version.

This feature is experimental.

Note: the flag is *advertised* (in `--help`) by `cockroach start` and
`cockroach start-single-node`, where log files are expected as a
matter of course; it is also supported, yet *hidden* (from `--help`),
by every other CLI command because every command can also produce log
entries when passed `--log-dir` or `--logtostderr`.

Release note (cli change): The commands `cockroach debug zip` and
`cockroach debug merge-logs` support a new flag `--redact-logs`. When
specified, it causes log messages that potentially contain
confidential data or PII to be redacted away.

This feature is experimental.

Note 1: At the time of this writing, this redaction is extremely
aggressive and may hinder the ability of Cockroach Labs to investigate
issues from log files, especially if `--redactable-logs` is not
enabled server-side.

Note 2: Other potentially-confidential data may still be retrieved by
`cockroach debug zip`; the new flag only applies to retrieved log
files inside the output zip file.

49978: importccl: add basic tests for CSV export of enum types r=miretskiy a=rohany

Work for #49971.

This PR adds some basic tests for CSV export of enum types.

Release note: None

Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net>
Co-authored-by: Rohan Yadav <rohany@alumni.cmu.edu>

Makefile

@@ -825,7 +825,7 @@ SQLPARSER_TARGETS = \
 
 PROTOBUF_TARGETS := bin/.go_protobuf_sources bin/.gw_protobuf_sources bin/.cpp_protobuf_sources bin/.cpp_ccl_protobuf_sources
 
-DOCGEN_TARGETS := bin/.docgen_bnfs bin/.docgen_functions
+DOCGEN_TARGETS := bin/.docgen_bnfs bin/.docgen_functions docs/generated/redact_safe.md
 
 EXECGEN_TARGETS = \
   pkg/col/coldata/vec.eg.go \
@@ -1528,6 +1528,19 @@ bin/.docgen_functions: bin/docgen
 	docgen functions docs/generated/sql --quiet
 	touch $@
 
+.PHONY: docs/generated/redact_safe.md
+
+docs/generated/redact_safe.md:
+	@(echo "The following types are considered always safe for reporting:"; echo; \
+	  echo "File | Type"; echo "--|--") >$@.tmp
+	@git grep '^func \(.*\) SafeValue\(\)' | \
+	  grep -v '^pkg/util/redact' | \
+	  sed -E -e 's/^([^:]*):func \(([^ ]* )?(.*)\) SafeValue.*$$/\1 | \`\3\`/g' >>$@.tmp || rm -f $@.tmp
+	@git grep 'redact\.RegisterSafeType' | \
+	  grep -v '^pkg/util/redact' | \
+	  sed -E -e 's/^([^:]*):.*redact\.RegisterSafeType\((.*)\).*/\1 | \`\2\`/g' >>$@.tmp || rm -f $@.tmp
+	@mv -f $@.tmp $@
+
 settings-doc-gen := $(if $(filter buildshort,$(MAKECMDGOALS)),$(COCKROACHSHORT),$(COCKROACH))
 
 $(SETTINGS_DOC_PAGE): $(settings-doc-gen)