Recommendations are too low - maybe it does not properly use Prometheus for historic data?

[spewu]

Which component are you using?: recommender - installed via https://charts.fairwinds.com/stable

What version of the component are you using?:
k8s.gcr.io/autoscaling/vpa-recommender:0.9.2
fairwinds-stable/vpa helm chart v. 0.4.4

Component version:

What k8s version are you using (kubectl version)?:

kubectl version Output

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-14T05:15:04Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"0b17c6315e806a66d507e77760a5d60ab5cccfd8", GitTreeState:"clean", BuildDate:"2021-08-30T01:42:22Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

What environment is this in?:

Azure Kubernetes Service (AKS)

What did you expect to happen?:

I expected to see recommendations inside Goldilocks that would make more sense.

What happened instead?:

The current recommendations are very low for CPU, which would break the API if I followed them. So my current theory is that the VPA recommender is not reading the historic data properly.

How to reproduce it (as minimally and precisely as possible):

1. Create new AKS cluster
2. Install Prometheus via https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack with this command:

helm upgrade --install --version 19.0.1 --namespace prometheus --create-namespace prometheus prometheus-community/kube-prometheus-stack --values values.yaml

And this values file:

# Disabling scraping of Master Nodes Components
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeEtcd:
  enabled: false
kubeProxy:
  enabled: false
kubelet:
  serviceMonitor:
    # Disbles the normal cAdvisor scraping, as we add it with the job name "kubernetes-cadvisor" under additionalScrapeConfigs
    # The reason for doing this is to enable the VPA to use the metrics for the recommender
    # https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/FAQ.md#how-can-i-use-prometheus-as-a-history-provider-for-the-vpa-recommender
    cAdvisor: false
prometheus:
  prometheusSpec:
    retention: 30d
    storageSpec:
      volumeClaimTemplate:
        spec:
          # the azurefile storage class is created automatically on AKS
          storageClassName: azurefile
          accessModes: ["ReadWriteMany"]
          resources:
            requests:
              storage: 100Gi
    additionalScrapeConfigs:
      - job_name: 'kubernetes-cadvisor'
        honor_labels: true
        honor_timestamps: true
        scrape_interval: 30s
        scrape_timeout: 10s
        metrics_path: /metrics/cadvisor
        scheme: https
        authorization:
          type: Bearer
          credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        follow_redirects: true
        relabel_configs:
        - source_labels: [job]
          separator: ;
          regex: (.*)
          target_label: __tmp_prometheus_job_name
          replacement: $1
          action: replace
        - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
          separator: ;
          regex: kubelet
          replacement: $1
          action: keep
        - source_labels: [__meta_kubernetes_service_label_k8s_app]
          separator: ;
          regex: kubelet
          replacement: $1
          action: keep
        - source_labels: [__meta_kubernetes_endpoint_port_name]
          separator: ;
          regex: https-metrics
          replacement: $1
          action: keep
        - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
          separator: ;
          regex: Node;(.*)
          target_label: node
          replacement: ${1}
          action: replace
        - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
          separator: ;
          regex: Pod;(.*)
          target_label: pod
          replacement: ${1}
          action: replace
        - source_labels: [__meta_kubernetes_namespace]
          separator: ;
          regex: (.*)
          target_label: namespace
          replacement: $1
          action: replace
        - source_labels: [__meta_kubernetes_service_name]
          separator: ;
          regex: (.*)
          target_label: service
          replacement: $1
          action: replace
        - source_labels: [__meta_kubernetes_pod_name]
          separator: ;
          regex: (.*)
          target_label: pod
          replacement: $1
          action: replace
        - source_labels: [__meta_kubernetes_pod_container_name]
          separator: ;
          regex: (.*)
          target_label: container
          replacement: $1
          action: replace
        - separator: ;
          regex: (.*)
          target_label: endpoint
          replacement: https-metrics
          action: replace
        - source_labels: [__metrics_path__]
          separator: ;
          regex: (.*)
          target_label: metrics_path
          replacement: $1
          action: replace
        - source_labels: [__address__]
          separator: ;
          regex: (.*)
          modulus: 1
          target_label: __tmp_hash
          replacement: $1
          action: hashmod
        - source_labels: [__tmp_hash]
          separator: ;
          regex: "0"
          replacement: $1
          action: keep
        kubernetes_sd_configs:
        - role: endpoints
          kubeconfig_file: ""
          follow_redirects: true
          namespaces:
            names:
            - kube-system

3. Install VPA recommender via: https://github.com/FairwindsOps/charts/tree/master/stable/vpa with this command:

helm upgrade --install --version 0.4.4 --namespace vpa --create-namespace vpa fairwinds-stable/vpa --values values.yaml

With this values file:

updater:
  enabled: false
recommender:
  extraArgs:
    storage: prometheus
    prometheus-address: http://prometheus-kube-prometheus-prometheus.prometheus.svc.cluster.local:9090

4. Install Goldilocks, to get an easy web UI for seeing the recommendations:

helm upgrade --install --version 3.2.8 --namespace goldilocks goldilocks fairwinds-stable/goldilocks --create-namespace

5. Deploy something to monitor recommendations for, and let it run for a bit (best with some varying load, so we can get a good history for the recommender to base its recommendations on)

Anything else we need to know?:

My original kube-prometheus-stack values file looked like this:

# Disabling scraping of Master Nodes Components
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeEtcd:
  enabled: false
kubeProxy:
  enabled: false
prometheus:
  prometheusSpec:
    retention: 30d
    storageSpec:
      volumeClaimTemplate:
        spec:
          # the azurefile storage class is created automatically on AKS
          storageClassName: azurefile
          accessModes: ["ReadWriteMany"]
          resources:
            requests:
              storage: 100Gi

But I followed the advice on this answer on Stack Overflow to get the job label to say kubernetes-cadvisor, and ended up with the values file listed higher up .... but this broke the Grafana dashboards, and it does not seem to have made any impact on the recommender either 🙁

After getting help on this question on Stack Overflow the Grafana dashboards now work correctly, but the recommendations inside Goldilocks still seem way too low (for CPU at least).
Example:

If I were to follow that advice, the API would break, as it needs more CPU than 15m. In general, it seems like the recommendations in Goldilocks for CPU is always 15m - for all of our services. So something looks not right.

I initially posted this issue in the VPA repo, but they advised me to post it here instead, so 🤞

[sudermanjr]

@spewu thanks for the detailed report. We do use VPA directly from the recommendations block in the VPA object. I've commented on the other issue to try to get some clarification.

[sudermanjr]

The only thing I can think at the moment is that there's a bug in our chart that is not applying the flags to the VPA recommender correctly to connect to prometheus. Can you get the VPA recommender pod specification from your cluster, as well as check the logs on the recommender to verify that prometheus is being used?

[spewu]

The only thing I can think at the moment is that there's a bug in our chart that is not applying the flags to the VPA recommender correctly to connect to prometheus. Can you get the VPA recommender pod specification from your cluster, as well as check the logs on the recommender to verify that prometheus is being used?

Thanks a lot for your quick reply, and sorry I have been so slow to get back to you!

I am not 100% sure if this is what you are asking, but this is the specification for the VPA recommender running in our production cluster right now:

apiVersion: v1
kind: Pod
metadata:
  name: vpa-recommender-7fbb79d4df-qcfhq
  generateName: vpa-recommender-7fbb79d4df-
  namespace: vpa
  uid: b3558bc0-85cc-4845-8bb9-e107f3ea921e
  resourceVersion: '149810263'
  creationTimestamp: '2021-10-05T14:58:27Z'
  labels:
    app.kubernetes.io/component: recommender
    app.kubernetes.io/instance: vpa
    app.kubernetes.io/name: vpa
    pod-template-hash: 7fbb79d4df
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: vpa-recommender-7fbb79d4df
      uid: 20d8d65d-7800-423c-a858-dd5b87421bfa
      controller: true
      blockOwnerDeletion: true
  managedFields:
    - manager: kube-controller-manager
      operation: Update
      apiVersion: v1
      time: '2021-12-04T01:28:58Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:generateName: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/component: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/name: {}
            f:pod-template-hash: {}
          f:ownerReferences:
            .: {}
            k:{"uid":"20d8d65d-7800-423c-a858-dd5b87421bfa"}:
              .: {}
              f:apiVersion: {}
              f:blockOwnerDeletion: {}
              f:controller: {}
              f:kind: {}
              f:name: {}
              f:uid: {}
        f:spec:
          f:containers:
            k:{"name":"vpa"}:
              .: {}
              f:args: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:livenessProbe:
                .: {}
                f:failureThreshold: {}
                f:httpGet:
                  .: {}
                  f:path: {}
                  f:port: {}
                  f:scheme: {}
                f:periodSeconds: {}
                f:successThreshold: {}
                f:timeoutSeconds: {}
              f:name: {}
              f:ports:
                .: {}
                k:{"containerPort":8942,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
              f:readinessProbe:
                .: {}
                f:failureThreshold: {}
                f:httpGet:
                  .: {}
                  f:path: {}
                  f:port: {}
                  f:scheme: {}
                f:periodSeconds: {}
                f:successThreshold: {}
                f:timeoutSeconds: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:securityContext: {}
              f:terminationMessagePath: {}
              f:terminationMessagePolicy: {}
          f:dnsPolicy: {}
          f:enableServiceLinks: {}
          f:restartPolicy: {}
          f:schedulerName: {}
          f:securityContext:
            .: {}
            f:runAsNonRoot: {}
            f:runAsUser: {}
          f:serviceAccount: {}
          f:serviceAccountName: {}
          f:terminationGracePeriodSeconds: {}
    - manager: kubelet
      operation: Update
      apiVersion: v1
      time: '2021-12-04T01:29:47Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:conditions:
            k:{"type":"ContainersReady"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Ready"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
          f:containerStatuses: {}
          f:hostIP: {}
          f:phase: {}
          f:podIP: {}
          f:podIPs:
            .: {}
            k:{"ip":"10.244.4.238"}:
              .: {}
              f:ip: {}
          f:startTime: {}
  selfLink: /api/v1/namespaces/vpa/pods/vpa-recommender-7fbb79d4df-qcfhq
status:
  phase: Running
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2021-10-05T14:58:27Z'
    - type: Ready
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2021-12-04T01:29:27Z'
    - type: ContainersReady
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2021-12-04T01:29:27Z'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2021-10-05T14:58:27Z'
  hostIP: 10.240.0.8
  podIP: 10.244.4.238
  podIPs:
    - ip: 10.244.4.238
  startTime: '2021-10-05T14:58:27Z'
  containerStatuses:
    - name: vpa
      state:
        running:
          startedAt: '2021-12-04T01:29:26Z'
      lastState:
        terminated:
          exitCode: 255
          reason: Unknown
          startedAt: '2021-10-05T14:58:31Z'
          finishedAt: '2021-12-04T01:29:09Z'
          containerID: >-
            containerd://a4e08d2d0804e669a2bb735de5a94be9ed889102bffc332f0f4a251de8c34146
      ready: true
      restartCount: 1
      image: k8s.gcr.io/autoscaling/vpa-recommender:0.9.2
      imageID: >-
        k8s.gcr.io/autoscaling/vpa-recommender@sha256:54d3ca854d6718e321ff99807263bb1a38ad93fd089594c605e52913dc2e445c
      containerID: >-
        containerd://15d032c6c6a788629c987b4e41e7c72351f2a193b26c299db9369909c8bb4098
      started: true
  qosClass: Burstable
spec:
  volumes:
    - name: kube-api-access-vnq9k
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: vpa
      image: k8s.gcr.io/autoscaling/vpa-recommender:0.9.2
      args:
        - '--pod-recommendation-min-cpu-millicores=15'
        - '--pod-recommendation-min-memory-mb=100'
        - >-
          --prometheus-address=http://prometheus-kube-prometheus-prometheus.prometheus.svc.cluster.local:9090
        - '--storage=prometheus'
        - '--v=4'
      ports:
        - name: metrics
          containerPort: 8942
          protocol: TCP
      resources:
        limits:
          cpu: 200m
          memory: 1000Mi
        requests:
          cpu: 50m
          memory: 500Mi
      volumeMounts:
        - name: kube-api-access-vnq9k
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      livenessProbe:
        httpGet:
          path: /health-check
          port: metrics
          scheme: HTTP
        timeoutSeconds: 3
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 6
      readinessProbe:
        httpGet:
          path: /health-check
          port: metrics
          scheme: HTTP
        timeoutSeconds: 3
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 120
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: Always
      securityContext: {}
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: vpa-recommender
  serviceAccount: vpa-recommender
  nodeName: aks-d2sv4-34779159-vmss000002
  securityContext:
    runAsUser: 65534
    runAsNonRoot: true
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/memory-pressure
      operator: Exists
      effect: NoSchedule
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

Here are the logs:
vpa-recommender-7fbb79d4df-qcfhq.log

[spewu]

I just restarted the VPA recommender to get the initialization logs output:
vpa-recommender-648c87cc86-fmwft.log

[spewu]

Ahh, I am learning some stuff on the way here. I can see in the logs that it looks like the VPA recommender will use this query to fetch CPU metrics from Prometheus:

rate(container_cpu_usage_seconds_total{job="kubernetes-cadvisor", pod_name=~".+", name!="POD", name!=""}[1h])

If I copy that query and execute it directly in Prometheus, I do not get any result back. If I modify the query to this (removing the pod_name part):

rate(container_cpu_usage_seconds_total{job="kubernetes-cadvisor", name!="POD", name!=""}[1h])

Then I do get results back, looking like this:

Example row:

{
   "container=""accounting-integration-maintainer",
   "cpu=""total",
   "endpoint=""https-metrics",
   "id=""/kubepods/burstable/podf54a63a4-a294-45bd-a3ac-e100f4aa875e/593aacf2186fc545cc5d881aaea6c2901c2f758d1c05acfa43c197e846cc6695",
   "image=""likvido.azurecr.io/accounting-integration-maintainer_production:237",
   "instance=""10.240.0.7:10250",
   "job=""kubernetes-cadvisor",
   "metrics_path=""/metrics/cadvisor",
   "name=""593aacf2186fc545cc5d881aaea6c2901c2f758d1c05acfa43c197e846cc6695",
   "namespace=""default",
   "node=""aks-d2sv4-34779159-vmss000001",
   "pod=""accounting-integration-maintainer-5d8c5f9d8f-7z9gf",
   "service=""prometheus-kube-prometheus-kubelet"
}

Could the problem be that there is no property called pod_name?

[bbetter173]

I have managed to resolve the query by applying a configuration similar to what is outlined here:

vpa:
  enabled: true
  recommender:
    extraArgs:
      prometheus-address: prom:9090
      storage: prometheus
      prometheus-cadvisor-job-name: kubelet
      pod-label-prefix: ""
      pod-namespace-label: namespace
      pod-name-label: pod
      container-pod-name-label: pod
      container-name-label: name

which generates this query for historical data:

rate(container_cpu_usage_seconds_total{job="kubelet", pod=~".+", name!="POD", name!=""}[1h])

and when I run this query manually, I'm seeing results. However, this has not resulted in any appreciable difference in the recommendations made.

[spewu]

outlined here

Thanks a lot for this input!

I updated my values.yaml to this now:

updater:
  enabled: false
recommender:
  extraArgs:
    storage: prometheus
    prometheus-address: http://prometheus-kube-prometheus-prometheus.prometheus.svc.cluster.local:9090
    pod-name-label: pod
    container-pod-name-label: pod
    container-name-label: container

And I see the VPA will now query for CPU metrics in Prometheus like this:

rate(container_cpu_usage_seconds_total{job="kubernetes-cadvisor", pod=~".+", container!="POD", container!=""}[1h])

Which does return results!

I now gets lots of errors like these in the logs though:

I1220 14:15:34.408270       1 cluster_feeder.go:235] Adding pod {default auto-start-cases-27325410-bnp65} with labels map[]
I1220 14:15:34.408305       1 cluster_feeder.go:242] Adding 3 samples for container {{default auto-start-cases-27325410-bnp65} auto-start-cases}
W1220 14:15:34.408316       1 cluster_feeder.go:249] Error adding metric sample for container {{default auto-start-cases-27325410-bnp65} auto-start-cases}: KeyError: {{default auto-start-cases-27325410-bnp65} auto-start-cases}
W1220 14:15:34.408330       1 cluster_feeder.go:249] Error adding metric sample for container {{default auto-start-cases-27325410-bnp65} auto-start-cases}: KeyError: {{default auto-start-cases-27325410-bnp65} auto-start-cases}
W1220 14:15:34.408337       1 cluster_feeder.go:249] Error adding metric sample for container {{default auto-start-cases-27325410-bnp65} auto-start-cases}: KeyError: {{default auto-start-cases-27325410-bnp65} auto-start-cases}

But it looks like these are all containers that are no longer running, so maybe that it OK.

I also now realize that all the trouble I went through to change the job name for the cAdvisor metrics on Prometheus metrics from kubelet to kubernetes-cadvisor could probably have been avoided just by specifying kubelet as the prometheus-cadvisor-job-name like you did ... doh!

The Burstable recommendations changed now ... so I guess that is a start. This is before I applied that change:

And this is after I applied that change:

[sudermanjr]

Thanks for all the details of the journey! Looks like you got it working?

[spewu]

Thanks for all the details of the journey! Looks like you got it working?

Yep, it looks like it is working, although the Guaranteed QoS still looks wrong

[patsevanton]

Are you add extraArgs with address prometheus recommender pod ?
I dont find extraArgs/prometheus/storage in https://artifacthub.io/packages/helm/fairwinds-stable/goldilocks

How are you set extraArgs/prometheus/storage for goldilocks by use helm chart?

[blackhelicopters]

@patsevanton these settings could be specified in the custom values.yaml file:

recommender:
  enabled: true
  extraArgs:
    prometheus-address: |
      http://prometheus-server.monitoring.svc.cluster.local
    storage: prometheus
    v: "4"
    pod-recommendation-min-cpu-millicores: 15
    pod-recommendation-min-memory-mb: 100

[divyanjali1]

@spewu @bbetter173 Can you share your prometheus values.yaml used to scrape caadvisor metrics ?

[patsevanton]

@blackhelicopters is you code support victoriametrics ?