Provide first-class support for HTTP-based autoscaling

[tomkerkhove]

As of today, we provide a variety of scalers which are external systems from which we get metrics. HTTP-based autoscaling, however, isn't supported while this is very often required.

We should provide support for this without having to run too much infrastructure.

Use-Case

Automatically scale HTTP-based deployments based on request rate.

Suggestion

We should introduce a new component, in our KEDA suite, that is dedicated to only scaling HTTP-based workloads and know how to get the metrics.

Options are:

• Use Prometheus (see below)
• Use Service Mesh Interface (SMI | spec)

Use Prometheus

@anirudhgarg wrote a nice post on how to do this with Prometheus for now: https://dev.to/anirudhgarg_99/scale-up-and-down-a-http-triggered-function-app-in-kubernetes-using-keda-4m42

This works fine, but not everybody is using Prometheus. We could recommend people to start using it but it's a big dependency to add to our product in my opinion.

Personally, I think Prometheus as a single option is a no-go. KEDA is super simple, I don't want to have to run Prometheus to use KEDA.

Pro(s):

• Already works today
• Prometheus is a well-known product

Con(s):

• We force customers to learn, deploy & operate Prometheus
• KEDA infrastructure becomes more complex

Use Service Mesh Interface

Service Mesh Interface (SMI | spec) is an open specification implemented by most service meshes which allows us to integrate with a lot of products out-of-the-box which allows customers to use what they already have/want to use.

The specification includes Traffic Metrics which would be the perfect metric source for KEDA.

Pro(s):

• Customers can use any service mesh that is SMI compatible
• Wide range of supported customers
• Based on open specification

Con(s):

• Customers have to use a service mesh

Why a seperate component?

For me it feels like a natural split between what we have now and what HTTP will bring. Most probably HTTP-based scaling will depend on/bring more infrastructure so customers who don't need it should not be forced to use that.

Another aspect is that we should have small blocks to run than one big monolith.

[tomkerkhove]

Created #615 which is the second part of supporting HTTP workloads.

Users should have the option to choose:

• If they are using service meshes, SMI scaler will be the one (Provide a Service Mesh Interface (SMI) scaler #615)
• If they are not using service mesh and have Prometheus around, Prometheus scaler will be their best bet.

[RichiCoder1]

This would be mostly be covering 1 to X scaling, right? As for 0 to 1, you'd need some sort of activator ala osiris?

[yaron2]

@RichiCoder1 that's correct.

[AlbertoDePena]

Maybe some guidance on auto scaling with metrics server in the interim... I have a POC working but I am not sure what is a good target CPU utilization to use. Currently it's set up as 50%.

[tomkerkhove]

Let's continue our conversation here, thanks for the cool demo @arschles!

Those who have missed it, you can check out https://github.com/osscda/kedahttp

[arschles]

You're welcome @tomkerkhove - it was my pleasure to show the demo, and I'm really pleased that folks liked the idea. As promised in the last community meeting, I've written a proposal for how to integrate it into KEDA. Please see this link: https://hackmd.io/xoJQ20vERU2sDiVhMimUQw. It should be open to comment for everybody, but you'll have to sign in with your GitHub account.

[tomkerkhove]

Thanks, I did a first run-through and maybe @zroubalik might want to have a look at it.

I'm happy to see the move to CRDs & operators, for me the biggest concern is our proxy and how it works with other ingresses; we should be open for others as well so that we don't force this upon users.

My suggestion for the HTPP repo would be keda-add-on-http, http-add-on or so.

[arschles]

@tomkerkhove agreed. I'm glad you and others brought that up in the meeting. Regarding the proxy, I think it would certainly be optional, since it alone decides whether/when it is in the critical path based on the # of replicas a target deployment has (i.e. the Activator pattern). If a user decides to use the proxy, they can do so with the ingress of their choice because the proxy will only modify the Ingress object for a given deployment

Also, I responded to your comments in the hackmd document. As you requested, I will make a diagram of the proposed architecture

[tomkerkhove]

Thanks for sharing! Can you expand a bit on what it would modify in the ingress please?

[zroubalik]

The proposal is looking good, the operator based approach is definitely the right direction!

[arschles]

@tomkerkhove as promised, here's the architecture diagram. It's a bit on the complex side because of all the communication between operators and k8s resources. I'm happy to lay it all out in the next community meeting.

@tomkerkhove the only thing we need modified in the ingress would be a way to get the length of the HTTP queue. Many of the ingresses I would assume have prometheus metrics, so KEDA could possibly just talk to that API. Otherwise, the "interceptor" in the arch diagram could be in the request path in all cases and keep track of the HTTP queue length.

[arschles]

@tomkerkhove @zroubalik I've updated the proposal to be more clear about the Operator and the CLI. It removed some of your comments though, unfortunately. I'll be at the meeting tomorrow morning, in case you have more questions or would like to discuss.

[tomkerkhove]

Thanks, certainly helps clarifying.

Will go through the updated proposal but something to keep in mind is that this should not only be compatible to be used with Ingresses but maybe HTTP more in general. So we would need to have a flexible model so that we can integrate SMI through this (instead of a scaler #615) and service-to-service comms maybe even.

Thoughts on this?

[arschles]

I originally envisioned this for scaling based on external HTTP traffic coming from outside the cluster, so I want to say that's out of scope. I do believe this is modular enough to accommodate that if it's a requirement, because the external scaler can simply read metrics from SMI. The only other pieces is having the operator indicate that the service mesh should route traffic to the app. If I remember, there is a way to do this with SMI, but I'm not very up to speed on it.

[beriberikix]

A third option might be to leverage the new Service APIs aka Ingress V2. Here's an example of a simple Gateway.

[christle]

Hi,

this sounds very interesting! Here my 5 cents:

In our current project, we are using the eventhub-scaler. But HTTP autoscaling is of course also interesting, because we have some HTTP triggered Azure Functions within the cluster (I think http scaling is one of the most interesting part for a lot of projects). We are currently integrating the azure monitor scaler to get the corresponding metrics.

I tried to sort the approaches here for myself:

For me, a scaler is a very specific connector to translate metrics of external resources for KEDA.

That's why I just find an ingress scaler just as useful as an SMI scaler. I wouldn't try to integrate this into KEDAhttp. If I had a service mesh in my project, I would only expect to add a specific smi scaler to KEDA setup. And then, not just for HTTP related metrics.

What would I expect from a KEDA first class http solution? A simple way to scale every http service (not just those with an ingress resource). Of course, a scaler is not enough here, as it is not a metric source. So for me, KEDAhttp is primarily:

• a first class sitecar only for http queue and for scale to zero similar like the implementation in osiris (as I understand the sitecar is the interceptor proxy). Maybe this could simply a new CRD?
• a KEDA scaler that gets the metrics from the http sitecar

But I would not try to equip KEDAHttp with an SMI and ingress module other other things. Like i said, that would be for me simply new KEDA scalers.

From my point of view, the interceptor site car would be sufficient to manage the http queue for every app-service and expose metrics for a new scaler. No matter it is an request from outside or not. Like a very small KEDA http mesh.

I'm not quite sure yet about the use of an additional operator, but maybe to exclude the lifecycle of the new http out of keda core.

[tomkerkhove]

You can find our experimental add-on here: https://github.com/kedacore/http-add-on

Thanks for your thorough opinion and don't be afraid, we will definitely provide an SMI scaler for sure. We are just carving things out, for now, to come up with the best solution and evaluate the next steps. It is possible that we decide to move things out and integrate with another project, or move scalers back in our core, ...

My hope actually, is to fully rely on open standards to support all traffic types, unless there are none.

This allows us to make breaking changes and ship separately, but also because it's an extension of KEDA as you were referring to as its model is different.

Osiris is, unfortunately, no longer an option since it's no longer maintained (at least last time I've checked).

[chirangaalwis]

@tomkerkhove when digging into @anirudhgarg's blog article, one concern I encountered was as to how this solution could be integrated to a scenario which involves multiple Ingresses being managed by a single NGINX Ingress Controller installation (e.g. a multi application deployment in which each application service is exposed via its own Ingress resource and each application needs to be individually auto scaled based on the request count specific to it).

In such a scenario, the request count for a unit time, scraped from the NGINX Ingress Controller obviously is for all the Ingresses and not for each Ingress (please correct me if I am wrong). Is the discussed solution capable of handling such a scenario?

I created kubernetes/kubernetes#102575 which was inspired by Skipper collector but I did not encounter a similar solution for the community NGINX Ingress Controller. Pardon me, maybe this feature request is a bit far-fetched (as I am a bit of a newbie to autoscaling aspect of deployments), but I appreciate if you could explain if this all new https://github.com/kedacore/http-add-on solution is capable of addressing the request I have made.

[tomkerkhove]

kedacore/http-add-on aims at scaling a single ingress component and not all of them combined so I'd recommend giving that one a try. @arschles can help you in case of issues, feel free to open a discussion on that repo.

The reason why we are starting kedacore/http-add-on is that we don't want end-users having to rely on external components to be able to autoscale, unless they are using open standards such as Service Mesh Interface.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[tomkerkhove]

We can close this issue when kedacore/keda-docs#548 is done

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[arschles]

@tomkerkhove the HTTP Addon has a beta release, so do you think this is ok to close?

[tomkerkhove]

I'd prefer to wait until the docs are on the website, is that OK for you?

[arschles]

yes, absolutely 😄

[jeffhollan]

Updating this issue on roadmap with latest on the discussion here: https://keda.sh/blog/2022-09-27-http-add-on-is-on-hold/