tracing: NPE in SELECT from crdb_internal.cluster_inflight_traces inside tenant

[stevendanna]

demo@127.0.0.1:57912/movr> select * from crdb_internal.cluster_inflight_traces WHERE trace_id = 4;
*
* ERROR: [sql,client=127.0.0.1:57925,hostssl,user=demo] a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	/usr/local/Cellar/go/1.17.2/libexec/src/runtime/panic.go:1038
*   | runtime.panicmem
*   | 	/usr/local/Cellar/go/1.17.2/libexec/src/runtime/panic.go:221
*   | runtime.sigpanic
*   | 	/usr/local/Cellar/go/1.17.2/libexec/src/runtime/signal_unix.go:735
*   | github.com/cockroachdb/cockroach/pkg/util/tracing/collector.(*TraceCollector).StartIter
*   | 	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/collector/collector.go:96
*   | github.com/cockroachdb/cockroach/pkg/sql.glob..func78
*   | 	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/sql/crdb_internal.go:1292
*   | github.com/cockroachdb/cockroach/pkg/sql.(*virtualDefEntry).makeConstrainedRowsGenerator.func1
*   | 	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/sql/virtual_schema.go:632
*   | github.com/cockroachdb/cockroach/pkg/sql.setupGenerator.func3
*   | 	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/sql/virtual_table.go:120
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:491
*   | runtime.goexit
*   | 	/usr/local/Cellar/go/1.17.2/libexec/src/runtime/asm_amd64.s:1581
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x5e79ff6]

goroutine 10576 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc000ae7b90, {0x9b920a8, 0xc0022d94d0})
	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:254 +0xaa
panic({0x7e23fe0, 0xc6ab760})
	/usr/local/Cellar/go/1.17.2/libexec/src/runtime/panic.go:1038 +0x215
github.com/cockroachdb/cockroach/pkg/util/tracing/collector.(*TraceCollector).StartIter(0x0, {0x9b920a8, 0xc0022d9210}, 0x4)
	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/collector/collector.go:96 +0x96
github.com/cockroachdb/cockroach/pkg/sql.glob..func78({0x9b920a8, 0xc0022d9210}, {0x9d07b50, 0xc002b72f08}, 0xc003c1f8e8, {0xc002397638, 0x4682805}, 0xc004f20900)
	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/sql/crdb_internal.go:1292 +0x105
github.com/cockroachdb/cockroach/pkg/sql.(*virtualDefEntry).makeConstrainedRowsGenerator.func1({0x9af5600, 0xc002e04e00})
	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/sql/virtual_schema.go:632 +0x4d9
github.com/cockroachdb/cockroach/pkg/sql.setupGenerator.func3({0x9b920a8, 0xc0022d94d0})
	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/sql/virtual_table.go:120 +0x130
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:491 +0x12a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	/Users/ssd/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:481 +0x445

[RaduBerinde]

CC @adityamaru

[aliher1911]

We don't seem to have a trace collector on tenant nodes as it relies on liveness info for collection and it is not available. So we do explicit check for that, no liveness info - no tracing.
Do we want to make an error explicit or make traceCollector work on tenants?

[RaduBerinde]

In the short term we just need a reasonable error (or return no rows).

[aliher1911]

Something like this: #72825?