Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement connectionPool into mysql scaler based on grpc connectionPool #6314

Open
bbrala opened this issue Nov 6, 2024 · 4 comments · May be fixed by #6327
Open

Implement connectionPool into mysql scaler based on grpc connectionPool #6314

bbrala opened this issue Nov 6, 2024 · 4 comments · May be fixed by #6327
Labels
feature-request All issues for new features that have not been committed to needs-discussion

Comments

@bbrala
Copy link

bbrala commented Nov 6, 2024

Proposal

When using mysql scaler with a large amounts of namespaces, but against the same connection settings we run out of connections on the database. It seems the external scaler has a connectionPool implementation which could work for mysql also if we use the connection string as key.

Use-Case

When using a single connection string to scale multiple namespaces, for example; when you 'enable' en envoroment using a mysql database. It would be helpfull to poll using a single connection instead of ending up with 100 connections if you look at 100 namespaces/deployments.

Is this a feature you are interested in implementing yourself?

Maybe

Anything else?

Looks pretty simple to do, just not sure about the caveats of go and mysql connections. It looks like keda doesn't check parralel, but perhaps that could be an issue if multiple scalers try to use the same connection? Although this might be what the mutex is fixing in the keda grps scaler?

This also kinda assumed the queries are very very light. The ones we are using should be extremly easy (a simple where with 2 indexes, namespace and expires timestamp).

@bbrala bbrala added feature-request All issues for new features that have not been committed to needs-discussion labels Nov 6, 2024
@bbrala bbrala changed the title Implement grpc connectionPool into mysql scaler Implement connectionPool into mysql scaler based on grpc connectionPool Nov 6, 2024
@robpickerill robpickerill linked a pull request Nov 10, 2024 that will close this issue
7 tasks
@robpickerill
Copy link
Contributor

robpickerill commented Nov 10, 2024

I put together an implementation that provides a set of global connection pools for MySQL. It uses an atomic reference counted concurrent hashmap to do so. The biggest problem I found when writing this with respect to the GRPC implementation, is that the cleanup of the pools becomes problematic as the pools are shared across multiple scalers. This creates a problem whereby the close function could prematurely close a connection pool being utilized by another scaler.

This approach is still naive in its handling of the connection pool key, and its handling of the connection pool settings in that first conn pool in the cache gets settings applied, and there's no reconciliation afterwards -- solvable problems but I parked it up here for now, as the complexity was growing, and wasn't sure if the Keda project wants to move in this direction.

@bbrala
Copy link
Author

bbrala commented Nov 10, 2024

Type for trying. This could be an awesome way to make mysql more usefull.

In our setup we moved to redis since that has way less overhead, just makes the app controlling the scaling a little less confortable.

Still think this could be helpfull though

@bbrala
Copy link
Author

bbrala commented Nov 10, 2024

Didnt get to the key part (on mobile) but id assume using the connection string as key would be good and make things seperate enough. That would skip some issues with having to reauth and such.

@robpickerill
Copy link
Contributor

Yeah, I'll patch it up properly over the next few days then leave some comments on the PR for others to weigh in

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request All issues for new features that have not been committed to needs-discussion
Projects
Status: To Triage
Development

Successfully merging a pull request may close this issue.

2 participants