minIdleConns redis client option is causing connections exhaustion

[mkosta]

Noticed an issue in new redis client Options (k6 v50)

Using executor with settings like this

    exec: "measureRedisPerformance",
    startRate:  10000,
    timeUnit: '10m',
    preAllocatedVUs: 1350,
    stages: [
      { target: 10000, duration: '1m' },
      { target: 10000, duration: '60m' },
    ],    

Connection string to open Client connection
const redisClient = new redis.Client({socket:{host:"localhost",port:6379,minIdleConns:1500,maxConnAge:10000,poolSize:20}});

If i provide any value in minIdleConns (default value not mentioned in Docs), but say 1600 more than defacto used 1353
connections are consumed non stop, and in a split second like (ok in a minute) are reaching maxClients =10k and erros are thrown obviously

However if minIdleConns not specified, connected_clients value reaches around 1353 (looks like around preallocated vus+ some) and is properly reused, not going all the way to maxClients=10k. and despite tps rate growing 1353+ connections are reused

It seems this paramater if provided causes mishandling of idle connections, if ommited from Options everything is ok

[olegbespalov]

Hi @mkosta!

I'm sorry for not getting back to you sooner.

If the minIdleConns isn't specified, we do fall to a default Redis client, so connections are not closed.

// Minimum number of idle connections, which is useful when establishing
// new connection is slow.
/Defaultt is 0. the idle connections are not closed by Defaultt.

You're right that this is not mentioned in the docs and I'm going to open the PR for fixing it.

Generally, xk6-redis just proxies this value to the redis Golang's client, so the connections are managed inside the Redis client.

However, how do you initiate the client? Is it per VU, per iteration? 🤔

[mkosta]

@olegbespalov sorry i skipped the executor line)
here is an example of the script

import redis from 'k6/experimental/redis';

export const options = {
  scenarios: {
    redisPerformance: {
    executor: 'ramping-arrival-rate',
    exec: "measureRedisPerformance",
    startRate: __ENV['ITR'] !== undefined ? __ENV.ITR : 10000,
    timeUnit: '10m',

    preAllocatedVUs: 1350,

    stages: [
      { target: __ENV['ITR'] !== undefined ? __ENV.ITR : 10000, duration: '1m' },
      { target: __ENV['ITR'] !== undefined ? __ENV.ITR : 10000, duration: '60m' },
    ],    
  }
  }
};

const redisClient = new redis.Client({socket:{host:"localhost",port:6379,minIdleConns:1500,maxConnAge:0,poolSize:20}});

export async function measureRedisPerformance() {
  let key = "k2";
  await redisClient.set(key,1,100);
  await redisClient.incrBy(key, 10);
  const value = await redisClient.get(key);
  console.log(value);
  if (value !== '11') {
    throw new Error('foo should have been incremented to 11');
  }
}

export async function setup() {
}

export async function teardown() {
  
}

[mkosta]

and you see how they look like. as if pool is exploding but connections were not used. NULL commands sent

[mkosta]

then you change minIdleConns to 0 (as you mentioned to default value) or dont provide at all
and 1357 clients are opened. small pool is used, but each opened connection was used at least once to get key

[olegbespalov]

@mkosta, thanks for the input!

Unfortunately, since we proxy these parameters downstream, the issue still sounds more related to the Redis Go client that is used underneath the xk6-redis.

After checking the client repo, I see that sometimes there are presented issue with this parameter, e.g.:

• Strange behavior when new connect with MinIdleConns redis/go-redis#1681
• connection pool size  redis/go-redis#2439

So, it could be one of the corner cases 😢

I'm still wondering if not setting (or setting to 0) minIdleConns somehow blocks you?

[mkosta]

@olegbespalov thanks. if it's a Go issue it's a Go issue)
as long as you dont use/change this parameter works fine) it is when you turn on "I am an expert i can set up connections properties" mode then things do not work as expected. to be honest this parameter is least important in most cases. so if the problem is that far then lets just hope go-redis will fix it one day.