Error "Connection terminated due to connection timeout" on GCP Cloud Run functions Gen 2

[Matheus-Ribeiro-ITA]

Context

• I have some Google cloud run functions, similar to AWS lambdas, that access a postgres DB with node pg;
• The functions are requested multiple times for the same instance. This means that global variable are shared between requests;
• I created a global Pool for reutilizing it between function requests, but I create a PoolClient for each request
• The PoolClient is used for queries in the function and released at the end;
• The Pool is never ended;
• The Pool config is:

    max: 50,
    idleTimeoutMillis: 30 * 1000, 
    connectionTimeoutMillis: 10 * 1000,

Code example

The code would look something like this:

import { onRequest } from 'firebase-functions/v2/https';
import { getBdConfig } from '../../submodules/for-backend/postgres';
import { Pool } from 'pg';

const pool: Pool | undefined;

export const exampleCloudFunction = onRequest(
    async (
      //...
    ) => {
        // initialize global pool if not initialized
        if (!pool) {
            pool = new Pool(getBdConfig(postgresSecrets));
        }
        const client = await pool!.connect();
        
        try {
            //...
            // Run business logic and queries
        } finally {
            client.release();
        }
    }

Problem

After some successful requests I'm getting connection timeout when running pool!.connect();, even when I add to retry 3 times. Any idea of what might be causing it?

For example in one case, it ran for 6 requests successfully than it failed on the 7th and kept working for the others 3 request. It took 2 min from the first request to the failed one. Another example, it ran successfully for 194 requests and failed on 3 requests.

I added some logs and there is only 1 connection, 0 idle and 0 waiting in the pool after the connection timeout.

Similar problems:

I read these other issues, but I'm still not sure what it is. Some about Lambda closing TCP connections after some time.

• Connection timing out on aws lambda #1180
• Bug: node postgres forever hangs when the network service is turned off on server. #2973
• 7.18.2: "Connection terminated unexpectedly" when using client.query with a pool when pool has been idle for 10 minutes (running in AWS Lambda) #2112

[gajus]

Experiencing similar issues.

For what it is worth, pinning older version of pg didn't help.

But the fact that we are also using GCP makes me wonder if something else is going on.

Our issues started roughly at around Monday 28th, and still trying to track down the source.

[Matheus-Ribeiro-ITA]

@gajus I figure out what was happening on my side. It was a problem with the VPC connector from GCP and not related to node-postgres. As soon as I removed the VPC connector, the connection timeout was reduced from 1 every 6 attempts to 1 in every 600 attempts. Then a simple retry worked fine.

I will close this issue. I hope this helps you and future developers.

Edit: Feel free to reopen the issue if needed

[gajus]

I think in our case the problem might be a high CPU usage on the client, as opposed to the database.