Accessing default prometheus server in Openshift for Scaled Objects

[thotz]

Report

Authenticate KEDA with the Thanos Query tenancy-specific endpoint. So that it can access the metrics provided by the default Prometheus server. Currently the Thanos-query service with TLS enabled, but we cannot create proper ClusterTriggerAuthentication resource in the ScaledObject

Expected Behavior

Scaler need to access the thanos-querier endpoint for the required Prometheus metrics
Thanos

Actual Behavior

there are no defined steps to access the prometheus endpoint for Keda

Steps to Reproduce the Problem

1. Install KEDA operator in Openshift Cluster
2. Cretae the ScaledObject resource for Prometheus Scaler

Logs from KEDA operator

example

KEDA Version

2.5.0

Kubernetes Version

1.23

Platform

Red Hat OpenShift

Scaler Details

Prometheus

Anything else?

May be this is another bug, if I skip the ServerAddress from ScaledObject , resource creation succeeds but trigger for scaling is CPU than the custom metrics. I felt it as weird behaviour.

Name:         rgw-scale
Namespace:    openshift-storage
Labels:       scaledobject.keda.sh/name=rgw-scale
Annotations:  <none>
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2022-01-25T07:50:58Z
  Finalizers:
    finalizer.keda.sh
  Generation:  1
  Managed Fields:
    API Version:  keda.sh/v1alpha1
    Fields Type:  FieldsV1
    Fields V 1:
      F : Spec:
        .:
        F : Max Replica Count:
        F : Min Replica Count:
        F : Scale Target Ref:
          .:
          F : Kind:
          F : Name:
        F : Triggers:
    Manager:      oc
    Operation:    Update
    Time:         2022-01-25T07:50:58Z
    API Version:  keda.sh/v1alpha1
    Fields Type:  FieldsV1
    Fields V 1:
      F : Metadata:
        F : Finalizers:
          .:
          V :" Finalizer . Keda . Sh ":
        F : Labels:
          .:
          F : Scaledobject . Keda . Sh / Name:
    Manager:      keda
    Operation:    Update
    Time:         2022-01-25T07:50:59Z
    API Version:  keda.sh/v1alpha1
    Fields Type:  FieldsV1
    Fields V 1:
      F : Status:
        .:
        F : Conditions:
        F : Original Replica Count:
        F : Scale Target GVKR:
          .:
          F : Group:
          F : Kind:
          F : Resource:
          F : Version:
        F : Scale Target Kind:
    Manager:         keda
    Operation:       Update
    Subresource:     status
    Time:            2022-01-25T07:50:59Z
  Resource Version:  62423
  UID:               86b2f308-4664-4ef7-832d-9cc64d14a1c2
Spec:
  Max Replica Count:  5
  Min Replica Count:  1
  Scale Target Ref:
    Kind:  Deployment
    Name:  rook-ceph-rgw-my-store-a
  Triggers:
    Metadata:
      Metric Name:  ceph_rgw_put_collector
      Query:        sum(rate(ceph_rgw_put[2m]))

      Threshold:  90
    Type:         prometheus
Status:
  Conditions:
    Message:               ScaledObject is defined correctly and is ready for scaling
    Reason:                ScaledObjectReady
    Status:                True
    Type:                  Ready
    Message:               Scaling is not performed because triggers are not active
    Reason:                ScalerNotActive
    Status:                False
    Type:                  Active
    Status:                Unknown
    Type:                  Fallback
  Original Replica Count:  1
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type     Reason              Age   From           Message
  ----     ------              ----  ----           -------
  Warning  KEDAScalerFailed    18m   keda-operator  error parsing prometheus metadata: no serverAddress given
  Normal   KEDAScalersStarted  18m   keda-operator  Started scalers watch
  Normal   ScaledObjectReady   18m   keda-operator  ScaledObject is ready for scaling

Namespace:                                             openshift-storage
Labels:                                                app.kubernetes.io/managed-by=keda-operator
                                                       app.kubernetes.io/name=keda-hpa-rgw-scale
                                                       app.kubernetes.io/part-of=rgw-scale
                                                       app.kubernetes.io/version=2.5.0
                                                       scaledobject.keda.sh/name=rgw-scale
Annotations:                                           <none>
CreationTimestamp:                                     Tue, 25 Jan 2022 13:20:59 +0530
Reference:                                             Deployment/rook-ceph-rgw-my-store-a
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 80%
Min replicas:                                          1
Max replicas:                                          5
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu

[tomkerkhove]

Can you elaborate a bit more what we need to change? Is it to support the authentication with Thanos?

Can you open another bug for ServerAddress please? It should indeed be blocked.

[zroubalik]

Can you elaborate a bit more what we need to change? Is it to support the authentication with Thanos?

+1

Can you open another bug for ServerAddress please? It should indeed be blocked.

I think that this problem has been already resolved for 2.6 in: #2394

[thotz]

Can you elaborate a bit more what we need to change? Is it to support the authentication with Thanos?

+1

Sorry for the confusion.

I have read the Prometheus metrics can be read from Thanos query service, but for that, it needs to authenticate with that. So in short my request was the steps/documentation to add the default Prometheus metrics endpoint in Openshift to ScaledObject

Can you open another bug for ServerAddress please? It should indeed be blocked.

I think that this problem has been already resolved for 2.6 in: #2394

[thotz]

@zroubalik helped me in resolving the issue of accessing the thanos-query service for scaledobject in Openshift. Hence closing the issue

[ejsa13]

@zroubalik helped me in resolving the issue of accessing the thanos-query service for scaledobject in Openshift. Hence closing the issue

Hi @thotz, can you share the steps that @zroubalik provided. Thanks!

[zroubalik]

@ejsa13 please refer to this: https://github.com/zroubalik/keda-openshift-examples/tree/main/prometheus/ocp-monitoring

[sanasz91mdev]

the thanos-query service for scaledobject in Openshift. Hence closing the issue

So helpful! thanks!

@ejsa13 please refer to this: https://github.com/zroubalik/keda-openshift-examples/tree/main/prometheus/ocp-monitoring