[prometheus-kube-stack] kube-proxy metrics status with connection refused.

[shan100github]

I did set an EKS environment monitoring with prometheus-kube-stack. Notices prometheus-kube-proxy metrics are down with 10249: connect: connection refused displayed in Prometheus>Status>Targets displayed always.

Version of Helm and Kubernetes:
Helm Version:

$ helm version
version.BuildInfo{Version:"v3.3.4", GitCommit:"a61ce5633af99708171414353ed49547cf05013d", GitTreeState:"dirty", GoVersion:"go1.15.2"}

Kubernetes Version:

$ kubectl version
Client Version: v1.18.8-eks-7c9bda
Server Version: v1.18.9-eks-d1db3c

Which version of the chart:

kube-prometheus-stack-15.4.6

What happened:
Always 10249: connect: connection refused displayed in Prometheus>Status>Targets and not able to monitor metrics of kube-proxy

Changed values of values.yaml (only put values which differ from the defaults):

## Component scraping kube proxy
##
kubeProxy:
  enabled: true

  ## If your kube proxy is not deployed as a pod, specify IPs it can be found on
  ##
  endpoints: []
  # - 10.141.4.22
  # - 10.141.4.23
  # - 10.141.4.24

  service:
    enabled: true
    port: 10249
    targetPort: 10249
    selector:
      k8s-app: kube-proxy

  serviceMonitor:
    enabled: true
    ## Scrape interval. If not set, the Prometheus default scrape interval is used.
    ##
    interval: ""

    ## proxyUrl: URL of a proxy that should be used for scraping.
    ##
    proxyUrl: ""

    ## Enable scraping kube-proxy over https.
    ## Requires proper certs (not self-signed) and delegated authentication/authorization checks
    ##
    https: false

    ## 	metric relabel configs to apply to samples before ingestion.
    ##
    metricRelabelings: []
    # - action: keep
    #   regex: 'kube_(daemonset|deployment|pod|namespace|node|statefulset).+'
    #   sourceLabels: [__name__]

    # 	relabel configs to apply to samples before ingestion.
    ##
    relabelings: []
    # - action: keep
    #   regex: 'kube_(daemonset|deployment|pod|namespace|node|statefulset).+'
    #   sourceLabels: [__name__]

Helm values set after installation/upgrade related to kubeProxy setting

helm get values kube-prom

kubeProxy:
  enabled: true
  endpoints: []
  service:
    enabled: true
    port: 10249
    selector:
      k8s-app: kube-proxy
    targetPort: 10249
  serviceMonitor:
    enabled: true
    https: false
    interval: ""
    metricRelabelings: []
    proxyUrl: ""
    relabelings: []

[shan100github]

I have gone through helm/charts#16476 & it doesn't help. Let me know if anything to be checked in particular.

[katepangLiu]

I have gone through helm/charts#16476 & it doesn't help. Let me know if anything to be checked in particular.

Maybe you can check your docker http_proxy https_proxy setting [ check docker.service and .docker/config.json ]

• If you don't need proxy, Clear it
• If you need to pull image by proxy , put all your cluster node ip into proxy
  https://docs.docker.com/network/proxy/

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.

[alternaivan]

I have gone through helm/charts#16476 & it doesn't help. Let me know if anything to be checked in particular.

I checked out the proposed solution, applied the changes and it's working on EKS.
Version of the applications:
Kubernetes on EKS: Major:"1", Minor:"20+", GitVersion:"v1.20.4-eks-6b7464"
Prometheus stack version: kube-prometheus-stack-17.0.2

I've changed the following:

kubectl edit cm kube-proxy-config -n kube-system
## Change from
    metricsBindAddress: 127.0.0.1:10249 ### <--- Too secure
## Change to
    metricsBindAddress: 0.0.0.0:10249

Regards,
Marjan

[vallard]

Thanks @alternaivan ! That worked for me!

[juliohm1978]

Confirming this workaround works fine.

If anyone is concerned about the port being exposed on 0.0.0.0, kube-proxy automatically changes the protocol to HTTPS and adds in-cluster RBAC authentication to this endpoint.

prometheus-operator is configured out-of-the-box to authenticate and should pick up the metrics right away.

[fernferret]

Confirming this workaround works fine.

If anyone is concerned about the port being exposed on 0.0.0.0, kube-proxy automatically changes the protocol to HTTPS and adds in-cluster RBAC authentication to this endpoint.

prometheus-operator is configured out-of-the-box to authenticate and should pick up the metrics right away.

@juliohm1978 Where do you see kube-proxy using https for the metrics endpoint?

I read this issue and was excited to read this, but my results show that it's still serving the metrics endpoint over http after the config change of setting the metricsBindAddress: 0.0.0.0:102490. I also dove in a bit to the source. I don't think I'm wildly concerned but its curious to see that kube-proxy doesn't seem to support encryption for this endpoint.

Testing

Using curl with the http endpoint gives a valid result (and I see the metrics):

curl -sv http://192.168.1.21:10249/metrics | head
*   Trying 192.168.1.21:10249...
* Connected to 192.168.1.21 (192.168.1.21) port 10249 (#0)
> GET /metrics HTTP/1.1
> Host: 192.168.1.21:10249
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Date: Sat, 21 May 2022 12:51:28 GMT
< Transfer-Encoding: chunked
<
{ [1308 bytes data]
# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.52e-05
go_gc_duration_seconds{quantile="0.25"} 4.3184e-05

If we curl using tls:

curl -sv https://192.168.1.21:10249/metrics > /dev/null
*   Trying 192.168.1.21:10249...
* Connected to 192.168.1.21 (192.168.1.21) port 10249 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
} [296 bytes data]
* error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
* Closing connection 0
*

OpenSSL also doesn't think this is an encrypted stream:

openssl s_client -connect 192.168.1.21:10249
CONNECTED(00000003)
4555073196:error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Start Time: 1653137583
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

Investigating

I also had a look at the kube-proxy source to see if there's something I missed in the config file that others have uses but kube-proxy just calls http.ListenAndServe. Unless the authors are doing something very sneaky (which I don't think they are?) that will call directly to net/http which is the non TLS version (the tls variant is http.ListenAndServeTLS)

After reaching my own consensus that I didn't see a way this could support TLS, I decided to see if there were any issues (ie was this a bug that I couldn't make it work?), well it looks like there is a closed issue for this: kubernetes/kubernetes#106870, but it seems to be closed without resolution.

In conclusion, I don't see a way to make this work with TLS but the kubernetes community doesn't seem too concerned with this so I guess I'm not either. I've not yet investigated all of the metrics provided here but the other note is that there's no command or control from this endpoint, it's just metrics.

I guess the alternatives are to run a daemonset on each node and use the push-gateway or to use a NetworkPolicy on kube-proxy?

Edit: Also found this: kubernetes/kubernetes#74300, so I guess it was decided to keep it defaulted at 127.0.0.1.