Db connection pooling dev

[MSDrao]

Pull Request Checklist:

• Positive Test Case Written by Dev

• Automated Testing

• Sufficient User and Developer Documentation

• Passing Jenkins Build

• Peer Code review and approval

Positive Test Case

1. [Enter positive test case here]

[github-actions]

Test Results

    2 files  ±0      2 suites  ±0   1h 34m 18s ⏱️ + 2m 26s
1 415 tests ±0  1 399 ✅ ±0  16 💤 ±0  0 ❌ ±0 
1 527 runs  ±0  1 509 ✅ ±0  18 💤 ±0  0 ❌ ±0 

Results for commit a858959. ± Comparison against base commit 69f2ddc.

♻️ This comment has been updated with latest results.

[devincowan]

@MSDrao looks like you have some broken tests. I will review once those are resolved.

[devincowan]

Also noting that this work will take some tooling to get pgbouncer running in k8s

[devincowan]

just curious, why not https://hub.docker.com/r/bitnami/pgbouncer/ ?

[devincowan]

this looks good!
next step is some load testing...

[devincowan]

@MSDrao I committed your locustfile.py here:
0469853
and bumping MAX_CLIENT_CONNECTIONS to 1000 seemed to work for me:
Out of ~1600 hits to /user I had 0% failure

[devincowan]

Posting some initial load tests here...not looking promising at this point but I'm going to keep at it.

1. Run without pgbouncer: https://github.com/devincowan/hydroshare/actions/runs/12751630824
2. With pgbouncer: https://github.com/devincowan/hydroshare/actions/runs/12751427298 (max connections 1000)
3. W/ pgbouncer: https://github.com/devincowan/hydroshare/actions/runs/12752226259 (max conn 300)

Hope to show something more promising in a bit...

UPDATE with some additional tests:

1. With pgbouncer: https://github.com/devincowan/hydroshare/actions/runs/12753100485
   w-pg-bounce-dcowan-locust-results.zip
   

2. Without pgbouncer: https://github.com/devincowan/hydroshare/actions/runs/12753260321
   no-pg-bounce-dcowan-locust-results(1).zip
   

There isn't a huge benefit here, but it does look like there is some improvement (pgbouncer test allows higher # of users before failure)

[devincowan]

ahhh here we go!
Adding more settings: https://github.com/CUAHSI/platform-recipes/pull/71/commits/b5463ea8e0a870358df935dbe198270b7b01ab83
And then re-running the tests:
https://github.com/devincowan/hydroshare/actions/runs/12753590880
with-pg-and-settings-locust-results(11).zip

☝️ we can get to much higher numbers of users before getting 502s and we can also achieve significantly higher RPS!

[devincowan]

after load testing on dev-1 I think this looks good!