TLS woes

[sdlarsen]

I'm unsure if this is addressed in any other issue (#132 for instance), but my setup is fairly simple:
k8s oauth to github with dex/gangway. The certificate is valid (production cert from letsencrypt).
When trying to connect with the client setup using let client = Client::try_default().await?; I get:

`Error: ReqwestError: error sending request for url (https://my-cluster/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/collections.blah.test?&force=true&fieldManager=test-operator): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913: (unable to get issuer certificate)

Caused by:
0: error sending request for url (https://my-cluster/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/collections.blah.test?&force=true&fieldManager=test-operator): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913: (unable to get issuer certificate)
1: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913: (unable to get issuer certificate)
2: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913: (unable to get issuer certificate)
3: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913:`

Any suggestions? No other tool I've tried refuses to use my current config, so I'm a bit suprised :)

[clux]

Hard to tell from just that output. Oauth providers have always been hard for us to test, but it looks like you're beyond that stage. Then again we're not really doing a lot of magic to the certs. Does the problem persist with openssl as the tls stack?

[nightkr]

FWIW, if this is for local testing then you could use kubectl proxy as a workaround, to let kubectl handle all of the OAuth/TLS stuff for you. Kube-rs' kubeconfig support is far from perfect at the moment (see also #132 for example).

For production you'd presumably use the in-cluster option instead, where all of the TLS certificates are provided statically.

[sdlarsen]

@clux I've tried with "native-tls" and "rustls-tls". It seems not to make any difference. And that output is all I get :)
@teozkr, I'll try using kubectl to proxy it for my testing, thanks!

[sdlarsen]

@teozkr Using kube proxy works fine for testing. But moving to the cluster gives me much the same situation.

Using a plain client = Client::try_default().await? gives me similar output:
`Error: ReqwestError: error sending request for url (https://10.3.0.1/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/collections.blah.test?&force=true&fieldManager=test-operator): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)

Caused by:
0: error sending request for url (https://10.3.0.1/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/collections.blah.test?&force=true&fieldManager=test-operator): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
1: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
2: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
3: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:
`

From the pod describe I see the sa is mounted:
Volumes: solr-operator-sa-token-dn6jg: Type: Secret (a volume populated by a Secret) SecretName: test-operator-sa-token-dn6jg Optional: false

And to make sure I didn't miss any permissions, the sa is bound to cluster-admin. I'm running with native-tls-vendored.

Suggestions are welcome :)

[edit] I'm on k8s 1.18, forgot to mention [/edit]

[nightkr]

@sdlarsen What does your /var/run/secrets/kubernetes.io/serviceaccount/ca.crt contain? Is curl https://10.3.0.1/ --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt able to connect?

[sdlarsen]

Yes, I can connect with curl, but only if I add --insecure as the cluster certs are self-signed, which seems to be the issue (for both environments actually).
Sorry for bothering you with this. The current solution is:

     let mut config = Config::infer().await?;
     config.accept_invalid_certs = true;
     client = Client::new(config);

Which I think is the correct way with self-signed certs?

[nightkr]

No, it should allow the connection if you specify the correct CA explicitly (via --cacert). That cURL doesn't work either makes it sound like a cluster issue, where it mounts the wrong CA into the pods for some reason, or the cert is invalid for some other reason (expired? incorrect clock?).

[sdlarsen]

True, something is off with the certs. Let me fix that and get back to you.

[sdlarsen]

True, fixing the cert setup solved that. Thank you for the assistance.