DEVOPS-29207 close idle connections after 1 minute

[kd7lxl]

By default, each sql.DB will keep 2 idle connections open indefinitely. Adding SetConnMaxIdleTime(time.Minute) instructs go to close these idle connections after one minute of inactivity. The next time this sql.DB needs a connection, it will have to open a new one. This adds some latency, but for an administrative process like db-controller that's not a problem and is preferred over having many open idle connections.

[drewwells]

DEMO

 make test-e2e
kubectl config current-context | awk -F@ '{print $1}' > .env
NOCLEANUP= NODEPLOY= NAMESPACE=dwells ENV=box-3 go test ./test/e2e/ -v --ginkgo.v --ginkgo.timeout=1h -timeout=1h
=== RUN   TestE2E
  Starting E2E suite
Running Suite: e2e suite - /home/drew/src/github.com/infobloxopen/db-controller/test/e2e
[FailFast]
========================================================================================
Random Seed: 1730308070

Will run 3 of 3 specs
[BeforeSuite]
/home/drew/src/github.com/infobloxopen/db-controller/test/e2e/e2e_suite_test.go:78
  running: kubectl config current-context
  STEP: Building image @ 10/30/24 12:07:50.666
  running: make build-images
  STEP: Pushing the operator images @ 10/30/24 12:08:10.244
  running: make push-images
  STEP: Helm upgrading the manager @ 10/30/24 12:08:25.691
  running: make deploy
  STEP: validating that the controller-manager pod is running as expected @ 10/30/24 12:08:49.376
  running: make helm-test
[BeforeSuite] PASSED [159.885 seconds]
------------------------------
dbc-end2end Creating a DB Creating a DBClaim with invalid version
/home/drew/src/github.com/infobloxopen/db-controller/test/e2e/dbc_end2end_test.go:71
  2024-10-30T12:10:30-05:00     INFO    configLog       loading config
  STEP: Checking if dbinstance exists: box-3-dwells-db-1-416e183c @ 10/30/24 12:10:30.515
  STEP: dbc status.error messages it not empty: dwells/dwells-db-1 @ 10/30/24 12:10:31.146
  [FAILED] in [It] - /home/drew/src/github.com/infobloxopen/db-controller/test/e2e/dbc_end2end_test.go:144 @ 10/30/24 12:13:31.147
• [FAILED] [180.633 seconds]
dbc-end2end Creating a DB [It] Creating a DBClaim with invalid version
/home/drew/src/github.com/infobloxopen/db-controller/test/e2e/dbc_end2end_test.go:71

  [FAILED] Timed out after 180.001s.
  Expected
      <string>: can not create Cloud DB instance box-3-dwells-db-1-1ec9b27c it is being deleted
  to contain substring
      <string>: invalid database version
  In [It] at: /home/drew/src/github.com/infobloxopen/db-controller/test/e2e/dbc_end2end_test.go:144 @ 10/30/24 12:13:31.147
------------------------------
S [SKIPPED] [0.000 seconds]
dbc-end2end Creating a DB [It] Updating a databaseclaim
/home/drew/src/github.com/infobloxopen/db-controller/test/e2e/dbc_end2end_test.go:148

[drewwells]

DEMO
active connections before code change

k get dbc -A --no-headers | awk '{print $1"/"$2}' | xargs -I{} -n1 ./scripts/psql-open.sh {} -c 'SELECT sum(numbackends) from pg_stat_database;'
Error: Unable to retrieve DSN from secret scheduler-dbclaim in namespace atlas-jobs
Error: Unable to retrieve DSN from secret tasks-dbclaim in namespace atlas-jobs
DatabaseClaim: auditlog/auditlog-dbapi
 sum
-----
  30
(1 row)

DatabaseClaim: authz/authz-dbapi
 sum
-----
  73
(1 row)

Error from server (NotFound): secrets "sample-secret" not found
Error: Unable to retrieve DSN from secret sample-secret in namespace default
DatabaseClaim: dwells/dwells-dbproxy-test
 sum
-----
   3
(1 row)

DatabaseClaim: entitlement/entitlement-dbapi
 sum
-----
  25
(1 row)

DatabaseClaim: identity/identity-api
 sum
-----
  27
(1 row)

DatabaseClaim: ncs/network-catalog-service-dbapi
 sum
-----
  25
(1 row)

Error: Unable to retrieve DSN from secret news-postgresql-config in namespace news
DatabaseClaim: ngp-cp/atlas-bootstrap-app-dbapi
 sum
-----
  70
(1 row)

DatabaseClaim: ngp-cp/config-generator-dbapi
 sum
-----
  69
(1 row)

DatabaseClaim: ngp-cp/host-activation-dbapi
 sum
-----
  25
(1 row)

DatabaseClaim: qa/qa-local-pg
 sum
-----
   5
(1 row)

Error: Unable to retrieve DSN from secret atlas-realm-manager-pgsql-config in namespace realm-manager
Error: Unable to retrieve DSN from secret recyclebin-postgresql-config in namespace recyclebin
DatabaseClaim: spire-server/spire
 sum
-----
  26
(1 row)

[drewwells]

Significant reduction in open database connections

 k get dbc -A --no-headers | awk '{print $1"/"$2}' | xargs -I{} -n1 ./scripts/psql-open.sh {} -c 'SELECT sum(numbackends) from pg_stat_database;'
Error: Unable to retrieve DSN from secret scheduler-dbclaim in namespace atlas-jobs
Error: Unable to retrieve DSN from secret tasks-dbclaim in namespace atlas-jobs
DatabaseClaim: auditlog/auditlog-dbapi
 sum
-----
   9
(1 row)

DatabaseClaim: authz/authz-dbapi
 sum
-----
   6
(1 row)

Error from server (NotFound): secrets "sample-secret" not found
Error: Unable to retrieve DSN from secret sample-secret in namespace default
DatabaseClaim: dwells/dwells-dbproxy-test
 sum
-----
   1
(1 row)

DatabaseClaim: entitlement/entitlement-dbapi
 sum
-----
   5
(1 row)

DatabaseClaim: identity/identity-api
 sum
-----
   8
(1 row)

DatabaseClaim: ncs/network-catalog-service-dbapi
 sum
-----
   6
(1 row)

Error: Unable to retrieve DSN from secret news-postgresql-config in namespace news
DatabaseClaim: ngp-cp/atlas-bootstrap-app-dbapi
 sum
-----
   6
(1 row)

DatabaseClaim: ngp-cp/config-generator-dbapi
 sum
-----
   5
(1 row)
DatabaseClaim: spire-server/spire
 sum
-----
   7
(1 row)