reflect: k8s unit tests on ppc64le sees SEGV fault with tip

[Karthik-K-N]

What version of Go are you using (go version)?

$ go version
go version devel go1.18-6226020c2f Tue Sep 7 03:56:13 2021 +0000 linux/ppc64le

Does this issue reproduce with the latest release?

No

What operating system and processor architecture are you using (go env)?

Linux ppc64le

go env Output

$ go env
GO111MODULE=""
GOARCH="ppc64le"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/usr/local/go/bin/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/go/bin:/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/go/bin:/root/bin"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_ppc64le"
GOVCS=""
GOVERSION="devel go1.18-f5bdbf311c Wed Sep 8 05:32:14 2021 +0000"
GCCGO="gccgo"
GOPPC64="power8"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/kubernetes/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3272977011=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Installed go and kubernetes using below steps and ran the test for kubernetes

Install upstream dev version of golang:
    Have a stable version of go already installed.
    git clone https://github.com/golang/go.git
    cd go/src
    ./make.bash
    cd ../..
    rm -rf /usr/local/go
    cp -r ./go /usr/local
    export PATH=/usr/local/go/bin:$PATH
    export PATH=$GOPATH/bin:$PATH
Clone k8s/k8s repo and run below steps: (git clone https://github.com/kubernetes/kubernetes)
    export KUBE_TIMEOUT='--timeout=600s' export KUBE_COVER="n" export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y export LOG_LEVEL=4
    cd kubernetes
    pushd ./hack/tools
    GO111MODULE=on go install gotest.tools/gotestsum
    popd
    make test WHAT=./cmd/kubeadm/app/cmd/

What did you expect to see?

Test passing successfully

What did you see instead

Test panics with the below stack trace, Even observed similar panic on x86 as well.

[root@karthik-workspace kubernetes]# make test WHAT=./cmd/kubeadm/app/cmd/
+++ [0907 08:51:10] Running tests without code coverage and with -race
[certs] Generating "ca" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local node] and IPs [10.96.0.1 1.2.3.4]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost node] and IPs [1.2.3.4 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost node] and IPs [1.2.3.4 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[kubeconfig] Writing "admin.conf" kubeconfig file
--- FAIL: TestRunRenewCommands (0.17s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1 pc=0x10a6d9b0]

goroutine 67 [running]:
testing.tRunner.func1.2({0x12c11d20, 0x142f1640})
	/usr/local/go/src/testing/testing.go:1209 +0x36c
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1212 +0x334
panic({0x12c11d20, 0x142f1640})
	/usr/local/go/src/runtime/panic.go:820 +0x2ac
reflect.mapiterinit(0x12bf2600, 0x0, 0x1)
	/usr/local/go/src/runtime/map.go:1340 +0x3c
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/unsafe_map.go:112
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*sortKeysMapEncoder).IsEmpty(0xc00034b9b0, 0xc0001e45a8)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_map.go:333 +0xb0
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*placeholderEncoder).IsEmpty(0xc00028c8d0, 0xc0001e45a8)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect.go:336 +0x60
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structFieldEncoder).IsEmpty(0xc000363aa0, 0xc0001e4510)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:118 +0x9c
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structEncoder).Encode(0xc000363bf0, 0xc0001e4510, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:148 +0x9f4
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*placeholderEncoder).Encode(0xc00028d540, 0xc0001e4510, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect.go:332 +0x68
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000363e60, 0xc0001e4500, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:110 +0xa8
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structEncoder).Encode(0xc000363ec0, 0xc0001e4500, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:158 +0xc68
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*sliceEncoder).Encode(0xc000360720, 0xc0003501b8, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_slice.go:38 +0x7a8
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000364c30, 0xc000350160, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:110 +0xa8
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structEncoder).Encode(0xc000364d20, 0xc000350160, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:158 +0xc68
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*OptionalEncoder).Encode(0xc0003552f0, 0xc0001a6970, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_optional.go:70 +0xe0
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc000355310, 0xc000350160, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect.go:219 +0xc8
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*Stream).WriteVal(0xc0004fbe00, {0x12cd1e20, 0xc000350160})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect.go:98 +0x20c
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*frozenConfig).Marshal(0xc0002779a0, {0x12cd1e20, 0xc000350160})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/config.go:299 +0xbc
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).doEncode(0xc00018f450, {0x1327dab0, 0xc000350160}, {0x1324ff60, 0xc0002b7530})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:306 +0xb8
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Encode(0xc00018f450, {0x1327dab0, 0xc000350160}, {0x1324ff60, 0xc0002b7530})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:301 +0x19c
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).doEncode(0xc0003265a0, {0x1327da60, 0xc0004fbd40}, {0x1324ff60, 0xc0002b7530})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:244 +0xb14
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).Encode(0xc0003265a0, {0x1327da60, 0xc0004fbd40}, {0x1324ff60, 0xc0002b7530})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:184 +0x19c
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime.Encode({0x7fffc9c82118, 0xc0003265a0}, {0x1327da60, 0xc0004fbd40})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:50 +0xd8
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd.Write(...)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/loader.go:466
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd.WriteToFile({{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, 0xc0002b7470, 0xc0002b74d0, 0xc0002b74a0, {0xc000517260, ...}, ...}, ...)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/loader.go:419 +0x11c
k8s.io/kubernetes/cmd/kubeadm/app/util/kubeconfig.WriteToDisk(...)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/util/kubeconfig/kubeconfig.go:98
k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.createKubeConfigFileIfNotExists({0xc00059a571, 0xf}, {0x12e78e96, 0xa}, 0xc0004fbce0)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:270 +0x22c
k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.createKubeConfigFiles({0xc00059a571, 0xf}, 0xc00063e000, {0xc000161468, 0x1, 0x1})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:131 +0x1b0
k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.CreateKubeConfigFile({0x12e78e96, 0xa}, {0xc00059a571, 0xf}, 0xc00063e000)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:104 +0x1a8
k8s.io/kubernetes/cmd/kubeadm/app/cmd.TestRunRenewCommands(0xc000583380)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/certs_test.go:137 +0x190c
testing.tRunner(0xc000583380, 0x12f68a58)
	/usr/local/go/src/testing/testing.go:1259 +0x1b8
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1306 +0x6b8
FAIL	k8s.io/kubernetes/cmd/kubeadm/app/cmd	0.330s
FAIL
make: *** [Makefile:184: test] Error 1

When I traced the above scenario found a recent change 1b2d794 and the test runs before this commit is successfully passing.

[cuonglm]

I think modern-go/reflect2 must be changed to adopt the new change to mapiterinit: https://github.com/modern-go/reflect2/blob/333559e1834b0c1840d283d79220bf121d704022/unsafe_link.go#L32

It uses //go:linkname, thus is not Go 1 compatibility guarantee.

cc @josharian

[mkumatag]

cc @taowen

[ALTree]

Putting in the 1.18 milestone just to keep this on the radar for a decision, but if the issue is caused by go:linkname usage in a dependency -as Cuong pointed out-, it should be fixed there. As noted, that's not covered by the Go 1 backward compatibility promise.

[ianlancetaylor]

Upstream bug is modern-go/reflect2#24. I'm going to close this issue because there isn't anything to change in the Go repository. (This is an example of why I personally encourage people to avoid using reflect2. It's impossible for us to support that kind of package. Where there are deficiencies in reflect, let's fix them, rather than introducing an unsupportable third party package. Just my opinion, of course.)

[taowen]

anyone can help to maintain https://github.com/modern-go/reflect2 ? I no longer have the insight to keep up with the language evolution.

[taowen]

fixed at
https://github.com/modern-go/reflect2/releases/tag/v1.0.2
https://github.com/json-iterator/go/releases/tag/v1.1.12