Minikube openapi result does not contain kubernetes extensions

[pwittrock]

How to reproduce:

• Start minikube
• kubectl proxy
• curl 127.0.0.1:8001/swagger.json

Result is missing x-kubernetes tags that are present here.

@mbohlool said this is probably due to the apiserver being setup differently than the core server.

[mbohlool]

More context on this. The tags exists in v1.6.0 onward: https://github.com/kubernetes/kubernetes/blob/v1.6.0/api/openapi-spec/swagger.json

[dlorenc]

Do you have a pointer to any apiserver flags that control this behavior?

[mbohlool]

It is not flag protected. It is one of the config setting one pass ApiServer creation.

[dlorenc]

Ah thanks - do you happen to know which config setting?

[mbohlool]

k8s.io/apimachinery/pkg/openapi.Config

[r2d4]

I looked at this a bit last night. This is the relevant part of kube-apiserver. https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/server.go#L307-L310

This is where we set the server options for the apiserver in minikube
https://github.com/kubernetes/minikube/blob/master/pkg/localkube/apiserver.go

Since kube-apiserver defaults to these options, I don't think we shouldn't have to explicitly set anything else. I was pretty confused with the go2idl/openapi-gen code, it seems like you worked on all of this stuff upstream @mbohlool maybe you could explain a little?

[mbohlool]

What you referenced is on head but I assume you guys are using release 1.6 branch? The code has changed in app/server.go so I am referring to 1.6 branch:

https://github.com/kubernetes/kubernetes/blob/release-1.6/cmd/kube-apiserver/app/server.go#L368

Here is where DefaultOpenAPIConfig is called. In that function, We set definition namer:

https://github.com/kubernetes/apiserver/blob/release-1.6/pkg/server/config.go#L237

and definition namer is the one responsible for adding those x-kubernetes tag:

https://github.com/kubernetes/apiserver/blob/release-1.6/pkg/endpoints/openapi/openapi.go#L167

The only explanation I have is for some reason typeGroupVersionKinds is not populated there and that can only because of an empty list returned by AllKnownTypes here:

https://github.com/kubernetes/apiserver/blob/release-1.6/pkg/endpoints/openapi/openapi.go#L154

That means the api types are not registered or not registered yet when we create this namer.

[r2d4]

Yup sorry, we use the 1.6 branch but the code path is the same.

You're right, there seems to be a race condition where the OpenAPI is calling AllKnownTypes before they are registered. I added a deubgging line to
https://github.com/kubernetes/kubernetes/blob/v1.6.0/staging/src/k8s.io/apimachinery/pkg/runtime/scheme.go#L180
and where AllKnownTypes is called for OpenAPI
https://github.com/kubernetes/apiserver/blob/release-1.6/pkg/endpoints/openapi/openapi.go#L154
Here are the log files with debugging. Still looking into why this could be happening

minikube.txt

openapi-api-server.txt

[r2d4]

Fixed with #1693