-
Notifications
You must be signed in to change notification settings - Fork 350
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide performance benchmarks with and without Proxy #1871
Comments
Been doing a bit more testing: Via proxy (SSL), IAM enabled:
Via proxy (SSL), username/password:
No proxy, username/password, no SSL:
This is starting to look more like a cloudsql problem rather than a proxy problem? |
Thought i'd try against pg14 local docker instances:
So now i'm questioning my test script... going to try with a different pg client and eventually a different language. Certainly one interesting observation here regardless is that IAM is 2x as slow as username/password on new connections. |
Thanks for the response @Stono. We don't provide performance benchmarks, but increasingly see a need for it. Let's repurpose this issue for our team to provide some baseline numbers for comparison. Generally, the Proxy will introduce some overhead as it's a few extra hops to your database (localhost -> proxy -> server side proxy -> localhost db). Separately, we recommend scaling the Proxy with CPU for more throughput and more memory for more connections. Finally, if you're writing an app in Node.js, we do have a connector now: https://github.com/GoogleCloudPlatform/cloud-sql-nodejs-connector. That will eliminate some of the latency and is worth trying out. |
Hey @enocom thanks for the response. Pretty sure the latency we're observing now is not the proxy anyway. Even testing against a local postgres instance with no proxy i observe similar behaviours, basically if you fire a batch of connection requests to postgres in parallel, they all take a long time to connect. Adding in TLS and Workload Identity on cloudsql exacerbate that. Saying that, providing latency numbers for the proxy is always welcome to help folks make informed decisions particularly on latency sensitive applications! You'll probably see what i'm seeing if you start doing the performance benchmarks of proxy yourself. Try 1, 10 concurrent then 50 concurrent. For context; we use the proxy for a language agnostic way to wrap up connecting to cloudsql on our internal platform (so the apps don't need to worry about it, and they're written in python, java, node, bash, whatever haha). It works really nicely, i actually much prefer this approach to getting people instrumenting their code with libraries that need to be kept up to date (we have circa 600 apps). |
This is definitely an area where we'd like to provide more guidance to help folks make their own measurements and compare against what we expect customers to see. I'll update here when we have more on the topic. |
Hi! Wanted to also mention that performance benchmarks would be appreciated.
maybe my questions can help the team think through what kinds of benchmarks are useful. thanks in advance! |
In process connectors will provide a better user experience and will likely be faster given they don't have to do the localhost hop and go straight to the database's proxy server. We haven't done any formal benchmarking of v1 vs v2, but v2 will startup faster. V2 offers a bunch of additional benefits like support for tracing (as shown above), prometheus support, and others. So we do recommend upgrading. |
@enocom thanks for the info! |
We have a migration guide, but don't explicitly compare v1 and v2 other than here. In addition to the feature list in the README, v2 will start up faster and continue to get new features. v1 meanwhile will continue to get security updates, but new features will land in v2. |
Just sharing this here as i found it interesting https://twitter.com/BdKozlovski/status/1684098236426878976?t=SWYsfn24ltvFSyEOKHjjEQ&s=19 Cloudflare use Postgres at scale and point out how expensive connections are and how they use https://www.pgbouncer.org to mitigate that. Led me down a rabbit hole wondering if cloudsql proxy could implement such a pattern. Probably scope creep but would be cool. |
For what it's worth, a person can run the Proxy behind pgbouncer. And we have an example here of how to do that in a basic way. |
Hello, sorry for bumping an old issue, but I suspect this is still relevant. Environment:
Switching to cloud-sql-proxy and IAM authentication introduced a consistent increase in PHP processing time of ~60ms. Considering, PHP processing time with direct connection via private IP was sub 30ms, that is quite concerning. Tried using UNIX socket, instead of TCP, but did not notice any difference. Is there anything to be explored here, considering there's no native connector to cloudSql for PHP? |
Have you considered connecting directly with private IP? EDIT: I see you're already using private IP. In general, if you're running a latency sensitive workload, you'll always get the best performance from a direct connection (fewer components involved). |
Yeah, I'm aware of that, the reason I opted for cloud-sql-proxy is the IAM auth and security posture. |
Bug Description
Hello!
I've been investigating some reports from some of our users around latency spikes. I've narrowed the investigation down to when an application receives a large burst of requests, its connection pool can grow quite rapidly, very quickly. What i'm observing is that parallel connections to cloudsql (via cloudsql proxy) seem to have a linear increase in latency with number of connections being made, affecting all connection attempts, not just the first.
I have a test setup that will connect, perform a query, then disconnect.
For example, here's the timings around a single connection attempt:
If i make 5 connection attempts in parallel:
And here is 10:
The pattern continues the more connections I make.
Example code (or command)
This is using the
pg
library. However all of our users are using java, so this isn't a client library issue.Steps to reproduce?
Code sample provided above
Environment
./cloud-sql-proxy --version
): 2.5.0./cloud-sql-proxy --port 5432 INSTANCE_CONNECTION_NAME
):--private-ip --prometheus --http-address 0.0.0.0 --http-port 9739 --auto-iam-authn, termination period: 30
Additional Details
The text was updated successfully, but these errors were encountered: