Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Keda scale down to 0, suspected bug when metric api connection is not available. #2244

Closed
Akhisar opened this issue Nov 2, 2021 · 4 comments
Closed

Keda scale down to 0, suspected bug when metric api connection is not available. #2244

Akhisar opened this issue Nov 2, 2021 · 4 comments
Labels
bug Something isn't working stale All issues that are marked as stale due to inactivity

Comments

@Akhisar
Copy link

Akhisar commented Nov 2, 2021

Report

Hello All,

It appears that the keda is scaling down the deployment to 0 when there is an issue with metric api. Also since upgrade to 2.4 we are observing weird error messages on the keda logs.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  annotations:
    meta.helm.sh/release-name: 2021q3-aggregation-cache-base
    meta.helm.sh/release-namespace: 2021q3
  creationTimestamp: "2021-10-05T08:30:44Z"
  finalizers:
  - finalizer.keda.sh
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    scaledobject.keda.sh/name: aggregation-cache-base-scaled
  name: aggregation-cache-base-scaled
  namespace: 2021q3
  resourceVersion: "198651014"
  uid: 87d4ebea-6cd4-4f96-9d04-2748a7e2beca
spec:
  cooldownPeriod: 60
  maxReplicaCount: 8
  minReplicaCount: 0
  pollingInterval: 30
  scaleTargetRef:
    envSourceContainerName: aggregation-cache
    name: aggregation-cache-base
  triggers:
  - metadata:
      targetValue: "1"
      url: http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings
      valueLocation: BASE.replicas.current
    type: metrics-api
status:
  conditions:
  - message: ScaledObject is defined correctly and is ready for scaling
    reason: ScaledObjectReady
    status: "True"
    type: Ready
  - message: Scaling is performed because triggers are active
    reason: ScalerActive
    status: "True"
    type: Active
  - message: No fallbacks are active on this scaled object
    reason: NoFallbackFound
    status: "False"
    type: Fallback
  externalMetricNames:
  - http-http---aggregation-service-2021q3-svc-cluster-local-8080-v1-admin-cache-settings-BASE-replicas-current
  health:
    http-http---aggregation-service-2021q3-svc-cluster-local-8080-v1-admin-cache-settings-base-replicas-current:
      numberOfFailures: 0
      status: Happy
  lastActiveTime: "2021-11-02T06:18:09Z"
  originalReplicaCount: 0
  scaleTargetGVKR:
    group: apps
    kind: Deployment
    resource: deployments
    version: v1
  scaleTargetKind: apps/v1.Deployment

Expected Behavior

I assume the default behaviour when a metric api is not reachable is not to propagate metric and hence retain the current replica.

Actual Behavior

Keda looks to scale down the replica to 0 if the json endpoint is not available

Steps to Reproduce the Problem

  1. Create a metric endpoint based on json o/p
  2. Check the keda behaviour when the metric is unavailable, may be scale down the deployment providing the metric endpoint
    3.Observe the deployment getting scaled down to 0.

Logs from KEDA operator

=======================from operator logs=============

2021-10-29T21:18:15.485Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:22:44.479Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:22:48.779Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:23:23.397Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:23:56.916Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:25:14.134Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:25:26.862Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:25:33.697Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://replication-service.healthcheck1.svc.cluster.local:8080/keda/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://replication-service.healthcheck1.svc.cluster.local:8080/keda/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:26:30.491Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:26:45.676Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:29:10.499Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/keda/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/keda/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:31:22.395Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:32:09.074Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:32:24.255Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:32:25.472Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:32:42.450Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:32:59.319Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:33:28.539Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:34:19.524Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:34:34.682Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:34:46.941Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:34:47.208Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:34:52.815Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:35:07.984Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:35:36.740Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://autoheal-service.2021q3.svc.cluster.local:8080/keda/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:38.835Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:39.690Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:41.279Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:42.018Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:42.893Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-service-scaled"}
2021-10-29T21:35:43.593Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/keda/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/keda/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:45.156Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:46.635Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-cache-cvar9-scaled"}
2021-10-29T21:35:47.247Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:48.078Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:52.268Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-cache-base-scaled"}
2021-10-29T21:35:54.015Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:54.755Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:55.358Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:35:57.098Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-cache-cvar4-scaled"}
2021-10-29T21:35:58.870Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:36:00.468Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-cache-cvar5-scaled"}
2021-10-29T21:36:03.569Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-cache-cvar1-scaled"}
2021-10-29T21:36:06.650Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "2021q3", "ScaledObject.Name": "aggregation-cache-cvar7-scaled"}
2021-10-29T21:36:10.369Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:36:12.705Z	ERROR	metrics_api_scaler	Error when checking metric value: Get "http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)	{"error": "Get \"http://aggregation-service.2021q3.svc.cluster.local:8080/v1/admin/cache-settings\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
	/workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:145
2021-10-29T21:36:12.727Z	INFO	scaleexecutor	Successfully set ScaleTarget replicas count to ScaledObject minReplicaCount	{"scaledobject.Name": "aggregation-cache-base-scaled", "scaledObject.Namespace": "2021q3", "scaleTarget.Name": "aggregation-cache-base", "Original Replicas Count": 6, "New Replicas Count": 0}

========================And on metric api logs===============

E1031 20:18:04.863042       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-cache-cvar8-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 20:24:18.285001       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-service-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 20:24:21.829742       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"validation-service-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 20:24:48.403041       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-service-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 20:25:48.531131       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-service-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 20:32:13.220980       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-cache-cvar1-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 20:49:53.771169       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-cache-base-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 20:50:06.091968       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-cache-cvar11-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 21:08:17.656094       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-cache-cvar6-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 21:10:23.300021       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-service-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 21:12:37.015214       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-cache-cvar10-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 21:17:13.601482       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"validation-service-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 21:22:34.515299       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"aggregation-cache-cvar4-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null
E1031 21:30:24.459730       1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh \"validation-service-scaled\": the object has been modified; please apply your changes to the latest version and try again"  "Error"=null

KEDA Version

2.4.0

Kubernetes Version

1.21

Platform

Microsoft Azure

Scaler Details

metricapi

Anything else?

No response
@Akhisar Akhisar added the bug Something isn't working label Nov 2, 2021
@JorTurFer
Copy link
Member

JorTurFer commented Nov 2, 2021 •
edited
Loading

Could this problem be related with this fix @zroubalik ?
The error in the log sounds familiar to me 🤔

@zroubalik
Copy link
Member

Could this problem be related with this fix @zroubalik ? The error in the log sounds familiar to me 🤔

Yeah the error in the Metrics Server has been fixed.

@Akhisar you can use fallback for exactly this scenario.

@stale
Copy link

stale bot commented Jan 3, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale All issues that are marked as stale due to inactivity label Jan 3, 2022
@stale
Copy link

stale bot commented Jan 11, 2022

This issue has been automatically closed due to inactivity.

@stale stale bot closed this as completed Jan 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working stale All issues that are marked as stale due to inactivity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zroubalik @JorTurFer @Akhisar