Expose prometheus metrics for ScaledJob resources

[yoongon]

Provide a description of what has been changed

Checklist

• I have verified that my change is according to the deprecations & breaking changes policy
• Changelog has been updated and is aligned with our changelog requirements
• A PR is opened to update the documentation on (repo) (if applicable)
• Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #4798
Relates to kedacore/keda-docs#1263

[github-actions]

Thank you for your contribution! 🙏 We will review your PR as soon as possible.

🏖️ Over the summer, the response time will be longer than usual due to maintainers taking time off so please bear with us.

While you are waiting, make sure to:

• Add an entry in our changelog in alphabetical order and link related issue
• Update the documentation, if needed
• Add unit & e2e tests for your changes
• GitHub checks are passing
• Is the DCO check failing? Here is how you can fix DCO issues

Learn more about:

• Our contribution guide

[yoongon]

Hi @JorTurFer
#4817

The previous PR was broken, so I opened this new PR.
I am sorry but I couldn't roll it back.

Please take a look at this again.

Thanks.

[zroubalik]

Could you please add e2e test for this feature? You can see existing tests for Prom metrics for ScaledObjects: https://github.com/kedacore/keda/blob/main/tests/sequential/prometheus_metrics/prometheus_metrics_test.go

[yoongon]

Could you please add e2e test for this feature? You can see existing tests for Prom metrics for ScaledObjects: https://github.com/kedacore/keda/blob/main/tests/sequential/prometheus_metrics/prometheus_metrics_test.go

Hi @zroubalik I will update it together, once the above discussion is done.
Thanks for the suggestion.

[zroubalik]

Could you please add e2e test for this feature? You can see existing tests for Prom metrics for ScaledObjects: https://github.com/kedacore/keda/blob/main/tests/sequential/prometheus_metrics/prometheus_metrics_test.go

Hi @zroubalik I will update it together, once the above discussion is done. Thanks for the suggestion.

@yoongon thanks, just FYI we plan to do release tomorrow (Wednesday)

[yoongon]

Hi @zroubalik
I updated patch.
Cause I am not familiar with e2e, It takes time. However, I think we can move in parallel.
Could you please check this change first?

BTW, there are some change between this PR and master, I am rebasing it.
I will reupdate it.

Thanks

[yoongon]

@zroubalik Thanks for checking. I should have known this one. I closed that PR.
BTW, cause e2e build is not working properly on my local machine, it drags me down.
If you see any missing point on this PR, it will really be appreciated.
I am focusing on e2e now for more debugging.

[JorTurFer]

I'm deploying the code into my cluster, I'll give you some feedback about the problem soon

[JorTurFer]

You didn't register the metric in Prometheus server:
https://github.com/kedacore/keda/blob/main/pkg/metricscollector/prommetrics.go#L160-L177

That's why now it's not available

[JorTurFer]

You probably need to add also a metric record for the jobs here:

keda/pkg/scaling/scale_handler.go

Lines 821 to 822 in 6c1a67d

	metrics, isTriggerActive, _, err := cache.GetMetricsAndActivityForScaler(ctx, i, metricSpecs[0].External.Metric.Name)
	if err != nil {

You can do it adding the line metricscollector.RecordScaledJobError(scaledJob.Namespace, scaledJob.Name, err)

[zroubalik]

/run-e2e sequential
Update: You can check the progress here

[yoongon]

@JorTurFer
Thank you so much for the checking.
TBH, I had registered it but i missed after rolled it back a few times. Eventually I forgot.
Thanks again and I've just updated it

[yoongon]

e2e failed here.

I changed metric name to the original one.
We did a discussion a lot, so I might be confusing to you.

BTW, if you have any concern about the metric name, please let me know again.

[JorTurFer]

/run-e2e sequential
Update: You can check the progress here

[yoongon]

Some e2e errors appear to be related with time.Sleep between creating and deleting wrong scaled resource.
I guess it has been newly added.
I've just added this commit.
44ffcae

[JorTurFer]

/run-e2e sequential
Update: You can check the progress here

[yoongon]

Looks like final error on e2e. https://github.com//pull/4913/commits/75476df9115d6071234c970e904722577a9460c6

I think this commit will fix it. I must have missed it during the rebase.

[JorTurFer]

/run-e2e sequential
Update: You can check the progress here