ui: loading table details page crashes server

[nvanbenschoten]

I don't know quite what's going on here, but I was recording my steps along the way so I can share exactly what I did:

roachprod create nathan-geo -n=4 --gce-machine-type=n1-standard-16 --geo --gce-zones='us-east1-b,us-west1-b,europe-west1-b'
roachprod stage nathan-geo cockroach
roachprod start nathan-geo:1-3

roachprod run nathan-geo:4 -- './cockroach workload init kv {pgurl:1}'
roachprod sql nathan-geo:1 -- -e "ALTER TABLE kv.kv CONFIGURE ZONE USING constraints = COPY FROM PARENT, lease_preferences = '[[+region=us-east1]]'"

Navigate to:
  http://nathan-geo-0001.roachprod.crdb.io:26258/#/database/kv/table/kv

This resulted in the following crash, which I hit again when trying another node:

F200527 21:01:46.243190 4485 util/log/log.go:35  unexpected panic: not implemented
goroutine 4485 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x7ced201, 0xed660cc3a, 0x0, 0xc001b46e01)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xb8
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0x7ce99c0, 0xc000000004, 0x72335d6, 0xf, 0x23, 0xc001ab2000, 0x21)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:212 +0xbcd
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x50cf220, 0xc00007a130, 0xc000000004, 0x2, 0x47ccbd5, 0x14, 0xc000cbb1b8, 0x1, 0x1)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:68 +0x2c9
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x50cf220, 0xc00007a130, 0x1, 0xc000000004, 0x47ccbd5, 0x14, 0xc000cbb1b8, 0x1, 0x1)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:44 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:148
github.com/cockroachdb/cockroach/pkg/util/log.FatalOnPanic()
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:35 +0xb8
panic(0x40fe0a0, 0x4fe4270)
        /usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/cockroachdb/cockroach/vendor/github.com/golang/protobuf/proto.(*InternalMessageInfo).Size(...)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/golang/protobuf/proto/deprecated.go:91
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-gateway/internal.(*Error).XXX_Size(0xc00075e5a0, 0x7f07e05c2400)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-gateway/internal/errors.pb.go:55 +0x39
github.com/cockroachdb/cockroach/vendor/github.com/gogo/protobuf/proto.Marshal(0x7f07e05c2400, 0xc00075e5a0, 0xc00075e5a0, 0x7f07e05c2400, 0xc00075e5a0, 0xc001d6a101, 0xc000cbb3b0)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/gogo/protobuf/proto/table_marshal.go:2937 +0x278
github.com/cockroachdb/cockroach/pkg/util/protoutil.(*ProtoPb).Marshal(0x7d3c250, 0x45f2d60, 0xc00075e5a0, 0xc0026be500, 0xc001737978, 0xc0004c4700, 0x4927598, 0x203000)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/marshaler.go:40 +0x129
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime.DefaultHTTPError(0x50cf2a0, 0xc00301a6c0, 0xc0006fdab0, 0x50f5b20, 0x7d3c250, 0x50a50e0, 0xc002a160c0, 0xc0018f7b00, 0x50435c0, 0xc0026be460)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/errors.go:139 +0x3e2
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime.MuxOrGlobalHTTPError(0x50cf2a0, 0xc00301a6c0, 0xc0006fdab0, 0x50f5b20, 0x7d3c250, 0x50a50e0, 0xc002a160c0, 0xc0018f7b00, 0x50435c0, 0xc0026be460)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/errors.go:102 +0x12a
github.com/cockroachdb/cockroach/pkg/server/serverpb.RegisterAdminHandlerClient.func4(0x50a50e0, 0xc002a160c0, 0xc0018f7b00, 0xc003138ba0)
        /go/src/github.com/cockroachdb/cockroach/pkg/server/serverpb/admin.pb.gw.go:1408 +0x32c
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime.(*ServeMux).ServeHTTP(0xc0006fdab0, 0x50a50e0, 0xc002a160c0, 0xc0018f7b00)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/mux.go:240 +0xbd9
net/http.(*ServeMux).ServeHTTP(0xc00047f678, 0x50a50e0, 0xc002a160c0, 0xc0018f7b00)
        /usr/local/go/src/net/http/server.go:2387 +0x1bd
github.com/cockroachdb/cockroach/pkg/server.(*safeServeMux).ServeHTTP(0xc00047f660, 0x50a50e0, 0xc002a160c0, 0xc0018f7b00)
        /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:127 +0x61
github.com/cockroachdb/cockroach/pkg/server.(*Server).ServeHTTP(0xc00047f200, 0x50b6e20, 0xc001062c40, 0xc0018f7b00)
        /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1954 +0x1e1
net/http.serverHandler.ServeHTTP(0xc000142540, 0x50b6e20, 0xc001062c40, 0xc0018f7b00)
        /usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc000402640, 0x50cf1e0, 0xc0016ac000)
        /usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2928 +0x384

A wild guess is that this is related to the recent upgrade to grpc-gateway.

[petermattis]

Just saw this crash myself. Exact same stack.

[pbardea]

I confirmed that this crash also happens shortly after viewing the jobs tab in the admin UI (even on an empty cluster). Same log as above.

[nvanbenschoten]

This was also detected in #49842 and has a fix incoming. Closing this issue as a duplicate.