[kubernetes_logs] certificate verify failed "Hostname mismatch"

[rifler]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I'm trying to install Vector (from helm chart) as k8s daemonset in order to collect std logs from containers.
Got next error during k8s api request (checked on distroless, alpine and debian images):

2023-06-13T15:25:28.585685Z DEBUG HTTP{http.method=GET http.url=https://[fdf3:9cd8:f41c::1]/api/v1/nodes?&fieldSelector=metadata.name%3Dip-xxx.eu-west-1.compute.internal&resourceVersion=0 otel.name="list" otel.kind="client"}: kube_client::client::builder: requesting
2023-06-13T15:25:28.589988Z ERROR HTTP{http.method=GET http.url=https://[fdf3:9cd8:f41c::1]/api/v1/nodes?&fieldSelector=metadata.name%3Dip-xxx.eu-west-1.compute.internal&resourceVersion=0 otel.name="list" otel.kind="client" otel.status_code="ERROR"}: kube_client::client::builder: failed with error error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:: Hostname mismatch
2023-06-13T15:25:28.590039Z DEBUG kube_runtime::watcher: watch list error: HyperError(hyper::Error(Connect, ConnectError { error: Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 337047686, library: "SSL routines", function: "tls_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 1919 }]))) }, verify_result: X509VerifyResult { code: 62, error: "Hostname mismatch" } }))
2023-06-13T15:25:28.590064Z DEBUG kube_runtime::utils::stream_backoff: Error received, backing off deadline=Instant { tv_sec: 1734644, tv_nsec: 737317577 } duration=24.887044708s
2023-06-13T15:25:28.590076Z  WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=InitialListFailed(HyperError(hyper::Error(Connect, ConnectError { error: Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 337047686, library: "SSL routines", function: "tls_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 1919 }]))) }, verify_result: X509VerifyResult { code: 62, error: "Hostname mismatch" } })))

On the other hand, curl from pod works fine - https://gist.github.com/rifler/c13de0f4435ab480f401ab1d7c18eef9

Configuration

customConfig:
  sources:
    kubernetes_logs:
      type: kubernetes_logs
      extra_label_selector: app!=""
      data_dir: /volumes/vector
  sinks:
    aws_kinesis:
      type: aws_kinesis_streams
      inputs:
        - kubernetes_logs
      compression: none
      endpoint: https://xxx.net
      region: xxx
      stream_name: /xxx/xxx/xxx
      encoding:
        codec: json

Version

vector 0.30.0 (x86_64-unknown-linux-musl 38c3f0b 2023-05-22 17:38:48.655488673)

Debug Output

No response

Example Data

No response

Additional Context

My k8s cluster is running in AWS (eks)

References

No response

[rifler]

Fixed it by explicitly adding env variable KUBERNETES_SERVICE_HOST to values.yaml:

env:
  - name: KUBERNETES_SERVICE_HOST
    value: "kubernetes.default.svc"

For some inexplicable reason it was filled with ip address, not host:

k exec -it vector-daemonset-base-6d8bw -c vector -n vector -- sh

# before fix
env | grep KUBERNETES_SERVICE_HOST
KUBERNETES_SERVICE_HOST=fdf3:9cd8:f41c::1

# after fix
env | grep KUBERNETES_SERVICE_HOST
KUBERNETES_SERVICE_HOST=kubernetes.default.svc

Nevertheless, seems not ok that curl works with ip, but Vector doesn't

[dsmith3197]

Hi @rifler,

I'm glad you were able to find a suitable workaround.

This is an issue with the kube crate itself and is being tracked kube-rs/kube#991. We'll have to wait until an upstream fix is released.

Note that this issue has been resolved for rustls but not openssl, which is what Vector uses.

[mmerickel]

FWIW running into this on an AWS EKS 1.27 cluster with IPv6 enabled after upgrading vector from 0.28.1 to 0.32.2. Super unexpected breakage. :-( The workaround mentioned above does fix the issue.