Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sonobuoy panics with nil pointer dereference on Kubernetes 1.27 when prometheus adapter serves custom.metrics.k8s.io/v1beta1 resources #1910

Closed
rrpolanco opened this issue May 1, 2023 · 1 comment
Closed

Sonobuoy panics with nil pointer dereference on Kubernetes 1.27 when prometheus adapter serves custom.metrics.k8s.io/v1beta1 resources #1910

rrpolanco opened this issue May 1, 2023 · 1 comment
Labels
area/k8s-upstream

Comments

@rrpolanco
Copy link

rrpolanco commented May 1, 2023 •
edited
Loading

What steps did you take and what happened:

Installed sonobuoy and ran sonobuoy run --mode quick and received this panic:

$ sonobuoy run --mode quick
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1265ef0]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:88
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc000045c08, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:69 +0x5f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc000044f30, 0x15}, {0xc0003d0480, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:35 +0x2f8
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x23ce740?)
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:310 +0x47c
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0x0?)
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:198 +0x5c
k8s.io/client-go/discovery.ServerGroupsAndResources({0x23d4220, 0xc0002cb410})
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:392 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:356 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc00051ba10)
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:621 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:355 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x23d4220?, 0xc0002cb410?})
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/restmapper/discovery.go:148 +0x42
github.com/vmware-tanzu/sonobuoy/pkg/dynamic.NewAPIHelperFromRESTConfig(0x1608c6a?)
	/home/runner/work/sonobuoy/sonobuoy/pkg/dynamic/client.go:50 +0x5d
github.com/vmware-tanzu/sonobuoy/cmd/sonobuoy/app.getSonobuoyClient(0xc00027ab40)
	/home/runner/work/sonobuoy/sonobuoy/cmd/sonobuoy/app/common.go:15 +0x2e
github.com/vmware-tanzu/sonobuoy/cmd/sonobuoy/app.getSonobuoyClientFromKubecfg({0x0, {0x0, 0x0}})
	/home/runner/work/sonobuoy/sonobuoy/cmd/sonobuoy/app/common.go:28 +0xa6
github.com/vmware-tanzu/sonobuoy/cmd/sonobuoy/app.submitSonobuoyRun.func1(0xc0001dbb00?, {0xc0001349a0?, 0x0?, 0x2?})
	/home/runner/work/sonobuoy/sonobuoy/cmd/sonobuoy/app/run.go:74 +0x3b
rafael@rafael-k8s127-sonobuoy-ubuntu-2204-bug-repro:~$ sonobuoy run --mode quick
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1265ef0]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:88
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc000045818, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:69 +0x5f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc00012af18, 0x15}, {0xc00011eee0, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/aggregated_discovery.go:35 +0x2f8
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x23ce740?)
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:310 +0x47c
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0x0?)
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:198 +0x5c
k8s.io/client-go/discovery.ServerGroupsAndResources({0x23d4220, 0xc0002bd410})
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:392 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:356 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc00015ba10)
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:621 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/discovery/discovery_client.go:355 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x23d4220?, 0xc0002bd410?})
	/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.0/restmapper/discovery.go:148 +0x42
github.com/vmware-tanzu/sonobuoy/pkg/dynamic.NewAPIHelperFromRESTConfig(0x1608c6a?)
	/home/runner/work/sonobuoy/sonobuoy/pkg/dynamic/client.go:50 +0x5d
github.com/vmware-tanzu/sonobuoy/cmd/sonobuoy/app.getSonobuoyClient(0xc000278240)
	/home/runner/work/sonobuoy/sonobuoy/cmd/sonobuoy/app/common.go:15 +0x2e
github.com/vmware-tanzu/sonobuoy/cmd/sonobuoy/app.getSonobuoyClientFromKubecfg({0x0, {0x0, 0x0}})
	/home/runner/work/sonobuoy/sonobuoy/cmd/sonobuoy/app/common.go:28 +0xa6
github.com/vmware-tanzu/sonobuoy/cmd/sonobuoy/app.submitSonobuoyRun.func1(0xc000121b00?, {0xc0004024a0?, 0x0?, 0x2?})
	/home/runner/work/sonobuoy/sonobuoy/cmd/sonobuoy/app/run.go:74 +0x3b
github.com/spf13/cobra.(*Command).execute(0xc000121b00, {0xc000402480, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0xc00002c300)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
main.main()
	/home/runner/work/sonobuoy/sonobuoy/main.go:27 +0x1e

This only happens when using the prometheus adapter which serves the custom.metrics.k8s.io/v1beta1 API on 1.27.
It looks like the v0.26 client-go cannot handle api resources with nil group/version in convertAPIRsource() which custom.metrics.k8s.io/v1beta1 is serving. client-go was made to be more resilient when receiving these types of malformed responses in 1.27.

What did you expect to happen:

I expect for sonobuoy to run on Kubernetes 1.27 when there are custom.metrics.k8s.io/v1beta1 resources created.

Anything else you would like to add:

Updating the go-client in this project to 0.27.x as proposed in this PR should resolve this issue.

Environment:

  • Sonobuoy version: v0.56.16
  • Kubernetes version: (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:21:19Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:14:42Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes installer & version: kubeadm 1.27
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release): Ubuntu 22.04.2 LTS
  • Sonobuoy tarball (which contains * below)
@rrpolanco rrpolanco changed the title Sonobuoy panics with nil pointer dereference on Kubernetes 1.27 and prometheus adapter serving custom.metrics.k8s.io/v1beta1 Sonobuoy panics with nil pointer dereference on Kubernetes 1.27 when prometheus adapter serves custom.metrics.k8s.io/v1beta1 resources May 1, 2023
@laverya laverya mentioned this issue May 1, 2023
Merged
@dims dims mentioned this issue May 31, 2023
Closed
@franknstyle franknstyle reopened this Jun 14, 2023
@franknstyle
Copy link
Collaborator

Hi @rrpolanco,

There is a new version v0.56.17 now available if you'd like to verify as resolution to this issue.

Best Regards,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/k8s-upstream
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rrpolanco @franknstyle and others