feat: Add support for formula based evaluation of metrics values

[gauron99]

So I have created (this big PR) but after some comments from others decided to separate it into 2 smaller PRs. So I extracted the functionality of formula and kept it here.

WHATS NEW

This change introduces a new feature I call "formula". The definition lies in SO (see below).

// scaledObject.yaml definition
spec:
  advanced:
    scalingModifiers:
      target: "2"
      formula: "(mapi_trig + kw_trig)/2"
...
  triggers:
  - type: kubernetes-workload
    name: kw_trig
    metadata:
      podSelector: 'pod=workload-test'
      value: '1'
  - type: metrics-api
    name: mapi_trig
    metadata:
      targetValue: "2"
      url: "https://mockbin.org/bin/336a8d99-9e09-4f1f-979d-851a6d1b1423"
      valueLocation: "tasks"

• When formula is defined, it creates a new composite-scaler-metric that is passed on to HPA (hpa.go). HPA will make a request for this composite-metric ONLY instead of all external metrics. (resource triggers are untouched). This way all external metrics are collected at the same time and passed on to the formula itself (scale_handler.go). Formula is a string that specifies some mathematical/conditional statements and returns 1 number which is the final metric. Internally, it uses https://github.com/antonmedv/expr. (see language definition)
• All external triggers must be named when using the formula because the trigger name is how its referenced in the formula itself. Otherwise some parsing would be required, to check whether specific trigger is used in formula and add only those that are.
• The code above demonstrates working formula. -> takes mapi_trig and kw_trig, sums them up and divides by 2 aka creating an average of the 2.
• If fallback exists, the formula is NOT applied, its skipped.

Additional info

Formula lies within the scalingModifiers struct which is used for additional changes (the second part of the big PR mentioned at the top would go here as well). The struct could also look like this:

advanced:
 modifiers:
  target: "2"
  formula: "xxx"
  external:
  - name: grpc_one
    url: localhost
    timeout: 10

or it could be separated entirely. The problem is where to put the target because it is mandatory for formula and optional for external grpc servers. (if grpc server is given but target not, it will modify each external metric on its own by users definition of Calculate method - more in external metric calculation PR - will open this after struct is decided

Checklist

• I have verified that my change is according to the deprecations & breaking changes policy
• Tests have been added
• Changelog has been updated and is aligned with our changelog requirements
• A PR is opened to update the documentation on (repo)
• Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #2440
Fixes #4823

Relates to #4583

[github-actions]

Thank you for your contribution! 🙏 We will review your PR as soon as possible.

🏖️ Over the summer, the response time will be longer than usual due to maintainers taking time off so please bear with us.

While you are waiting, make sure to:

• Add an entry in our changelog in alphabetical order and link related issue
• Update the documentation, if needed
• Add unit & e2e tests for your changes
• GitHub checks are passing
• Is the DCO check failing? Here is how you can fix DCO issues

Learn more about:

• Our contribution guide

[gauron99]

/assign zroubalik

[matan-legit]

will this support boolean operators as well?

[gauron99]

some self-review of what i thought were important points

[gauron99]

will this support boolean operators as well?

you can see what you can with the package in here: https://expr.medv.io/docs/Language-Definition
you can use anything that returns a single value, which will be passed to hpa for scaling

[gauron99]

I rechecked everything, hopefully i didnt miss anything. I started a self-review with the structure name.

Once this PR is merged I will match the second new feature of the big parent PR of the structure name etc. and open it for a review

[JorTurFer]

First of all, I 💘 this feature! Thanks a ton for implementing it!

I know that it's not something direct, but if we'd have a way to query CPU/Memory metrics to include them in the formula would be nice. I think that using multiple scalers, including the CPU/Memory is a common pattern. Primary scaling based on events, but having CPU/Memory as fallback.

Regarding with the formula calculation, maybe we can move it to it's own file/package to group all the related stuff. Maybe its own package for all the modifiers is worth as there is too much logic only for it this formula calculation and it'll increase in the future

[tomkerkhove]

Is the doc PR up-to-date with the end-user experience @gauron99?

[gauron99]

Is the doc PR up-to-date with the end-user experience @gauron99?

its up to date with what i had in mind when i created the PR. Im expecting to change some names of structures etc based on some conversation above if thats what you mean. I have this @tomkerkhove which currently matches the code in this PR

[tomkerkhove]

Thanks, added some doc comments: kedacore/keda-docs#1198 (review)

[zroubalik]

Great stuff! I think we should mark this feature as an EXPERIMENATAL. This will allow us to change/modify the behavior after the release without the need for backward compatibility.

[gauron99]

@zroubalik @JorTurFer @tomkerkhove
Regarding the new package/file where to put the feature. I just want to start this comment on this so we can unify the decision.

Currently there are formula help functions in pkg/scaling/scale_handler.go that help to implement the feature and there are validating functions in apis/keda/v1alpha1/scaledobject_webhook.go that are used in the webhook itself and while creating hpa object in controllers/keda/hpa.go.

Im thinking of few possibilities:

1. completely separate package which can implement each feature separately (even if we want to add more in the future) called something like scaling_modifiers with files like formula.go with validation either in the file or separate like formula_validation.go. It just cannot lie within the scaling package (pkg/scaling/) because it imports kedav1alpha1 therefore the validation in webhook would cause a cyclic dep.

2. A file for the implementation called something like scaling_modifiers.go where all these new features could be but the validation would be separate in apis/keda/v1alpha1 in a file something like scaling_modifiers.go (to match the name maybe? or just add _validation.go at the end.

the question is if its ok for the implementation to be in the pkg/scaling package and validation be part of kedav1alpha1 and/or if its worth creating a whole new package for each/ both and where?

[gauron99]

@zroubalik @JorTurFer PTAL 🙏 Ive also created #4992

[zroubalik]

/run-e2e
Update: You can check the progress here

[JorTurFer]

LGTM!
Thanks for this incredible feature