Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with Redlock Stuck in Redis Cluster Setup #295

Open
zura-japoshvili opened this issue May 21, 2024 · 0 comments
Open

Issue with Redlock Stuck in Redis Cluster Setup #295

zura-japoshvili opened this issue May 21, 2024 · 0 comments

Comments

@zura-japoshvili
Copy link

zura-japoshvili commented May 21, 2024

I am experiencing an issue where Redlock appears to get stuck when trying to acquire a lock in a Redis cluster environment. This problem does not occur when using a single independent Redis node.

Environment:

Redlock Version: 3.1.2 (also tested with 5.0.0-beta.2)
Redis Version: 7.0.4
ioredis Version: 5.3.2
Redis Setup: Cluster with 6 nodes, created using Docker Compose

Issue Details:

When using a Redis cluster setup, the Redlock implementation gets stuck during the lock acquisition process.
This issue does not occur when using a single standalone Redis node.

Code Snippet for Redis Provider:

`export const redisProvider: Provider = {
provide: 'REDIS_CLIENT',
useFactory: (configService: ConfigService): Redis | Cluster => {
const clusterNodes = configService.get<string[]>('redis.clusterNodes');

if (clusterNodes && clusterNodes.length > 0) {
  const nodes = clusterNodes.map((uri) => {
    const url = new URL(uri);
    return {
      host: url.hostname,
      port: parseInt(url.port, 10),
    };
  });

  const clusters = new Redis.Cluster(nodes, {
    clusterRetryStrategy: (times) => Math.min(times * 100, 2000),
    slotsRefreshTimeout: 2000,
    dnsLookup: (address, callback) => callback(null, address),
    redisOptions: {
      keyPrefix: configService.get<string>('redis.prefix'),
    },
  });

  return clusters;
} else {
  const redis = new Redis(configService.get<string>('redis.host'), {
    keyPrefix: configService.get<string>('redis.prefix'),
  });

  return redis;
}

},
inject: [ConfigService],
};`

Code Snippet for Redlock Usage:

`@Injectable()
export class OptioRedlockService implements OnModuleInit {
protected readonly logger = new Logger(this.constructor.name);
private redlock: Redlock;

constructor(
@Inject(redisConfig.KEY)
private config: ConfigType,
@InjectRedisClient() private client: Cluster,
) {}

onModuleInit() {
const nodes = this.config.clusterNodes.map((uri) => {
const url = new URL(uri);
return new Client({ port: parseInt(url.port, 10), host: url.hostname });
});

this.redlock = new Redlock([this.client], {
  driftFactor: 0.01,
  retryCount: 300,
  retryDelay: 200,
  retryJitter: 200,
});

this.redlock.on('clientError', (err) => {
  console.error('A Redlock client error occurred:', err);
});

}

async executeWithLock(
key: string,
operation: () => Promise,
delay: number = 0,
lockTtl: number = 60_000,
): Promise {
const lock = await this.acquireLock([lock:${key}], lockTtl);

try {
  await sleep(delay);
  this.logger.debug(`Operation with key: ${key} is being locked`);
  return await operation();
} finally {
  await this.releaseLock(lock, key).catch((error) => {
    throw new Error(`Failed to release lock: ${error.message}`);
  });
}

}

async acquireLock(
resources: string[],
duration: number,
retries: number = 3,
): Promise {
let attempt = 0;

while (attempt < retries) {
  try {
    const lock = await this.redlock.acquire(resources, duration);
    return lock;
  } catch (error) {
    const errorMessage = (error as Error)?.message;
    this.logger.error(`Attempt ${attempt + 1} failed: ${errorMessage}`);
    if (attempt >= retries - 1)
      throw new Error(`Failed to acquire lock after ${retries} attempts`);
    await sleep(1000);
    attempt++;
  }
}

throw new Error('Failed to acquire lock');

}

async releaseLock(lock: Lock, key: string) {
let attempts = 0;
const maxAttempts = 3;
let released = false;

while (!released && attempts < maxAttempts) {
  try {
    await this.redlock.release(lock);
    this.logger.log(`Lock with key: ${key} released`);
    released = true;
  } catch (error) {
    attempts++;
    const errorMessage = (error as Error)?.message;
    this.logger.error(`Attempt ${attempts} failed to release lock: ${errorMessage}`);
    if (attempts < maxAttempts) {
      await sleep(100 * attempts);
    }
  }
}

if (!released) {
  this.logger.error('Failed to release lock after multiple attempts');
}

}
}
`
Questions:

  1. Compatibility: Can Redlock work seamlessly with Redis clusters, or is it intended to be used only with independent Redis nodes?
  2. Configuration Issue: If Redlock does support Redis clusters, where might my configuration be incorrect or suboptimal?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant