Add addScript, addBulkScript, and Related Functions for existing Lua Script Atomic Integration

[dillonstreator]

Description:

I would like to propose a feature that enhances BullMQ's integration with Redis Lua scripting. Specifically, it would be extremely beneficial if the Queue class exposed methods such as addScript, addBulkScript, and other variants of existing queue functions (e.g., pauseScript, resumeScript, etc.) that return the Lua script strings used internally by BullMQ.

Motivation:

There are scenarios where atomicity is crucial, such as when custom Redis operations need to be performed alongside job queuing. While Lua scripting in Redis is a powerful tool for atomic operations, it is currently challenging to enqueue jobs as part of a custom Lua script because:

1. Developers must manually replicate BullMQ's internal job queuing logic in their scripts, which is brittle and prone to breaking with future changes to BullMQ's internals.
2. There is no native way to integrate BullMQ's job-enqueuing logic directly into an existing Lua script.

This feature would enable developers to safely enqueue jobs as part of larger, atomic Lua scripts without needing to understand or replicate BullMQ's internals.

Proposed Solution:

Introduce methods to the Queue class (or relevant classes) that generate and return the Lua script string for existing functions, such as:

• addScript(jobName: string, data: object, options: JobOptions): string
• addBulkScript(jobs: Array<{ name: string; data: object; options?: JobOptions }>): string
• pauseScript(): string
• resumeScript(): string
• ...Script(): string

These methods would return the Lua script string that BullMQ uses internally for the corresponding operations. Developers could then inject these script strings into their existing Lua scripts, ensuring atomicity without sacrificing maintainability.

Example Usage:

import { Queue } from 'bullmq';

const queue = new Queue('my-queue');

// Get the Lua script for adding a job
const addJobScript = queue.addScript('testJob', { foo: 'bar' }, { attempts: 3 });

// Combine it with custom Lua logic
const customLuaScript = `
  -- Custom logic here
  local customResult = redis.call('SET', 'custom:key', 'value')

  -- BullMQ job enqueue logic
  ${addJobScript}

  return customResult
`;

// Execute the combined Lua script
const redisClient = queue.client;
const scriptSha = await redisClient.scriptLoad(customLuaScript);
await redisClient.evalSha(scriptSha, { keys: [], arguments: [] });

Benefits:

1. Atomicity: Ensures that job queuing and custom Redis operations occur together atomically.
2. Maintainability: Shields developers from BullMQ's internal implementation, reducing the risk of breaking changes.
3. Flexibility: Empowers advanced use cases where custom Lua scripts are required.

Backward Compatibility:

This feature is non-breaking as it introduces new methods without altering existing behavior.

Alternatives:

The current alternative is to manually replicate BullMQ's internal logic for job queuing in custom Lua scripts, which is error-prone and fragile.

Closing Thoughts:

This feature would greatly enhance BullMQ's usability for developers who rely on Redis Lua scripting for atomic operations. I hope the team considers this addition as it aligns well with BullMQ's mission of providing robust and flexible job queueing capabilities.

Thank you for your consideration!

[manast]

Thank you for the feature request. There are few things that I think are important to consider about this proposal. First of all, I find the requirement of combining other operations with BullMQ operations into atomic transactions something that could potentially be an anti pattern. Yes, BullMQ uses Redis, but I do not think that from the application perspective we should be aware of that, the same way as if we use an external service accessible via a HTTP REST api we should not care how that service is implemented, just care about the interface it provides and what guarantees it provides.

If you are in the need of having atomic transactions that combine user code with BullMQ code, maybe there are other ways to achieve the same results. For instance, I think that if you create your jobs such as they are always idempotent, as well as exploit the fact that you cannot have 2 jobs with the same IDs in a given queue, or you use the deduplicationId, then you can design solutions that do not require atomicity but still give you the same guarantees, or guarantees that are acceptable for your use case.

If we despite my reluctancy expressed above were to provide the possibility of doing what you ask, it would not work the way you described, as our scripts are static, and only loaded once, so we have no way to generate a version of the add script for example, for a given set of arguments, as you propose above. I think a better approach would be to have it the other way around, like you define "hooks" in some scripts, for example, let's say we had a "before-add" hook in the add job script, then by passing your custom lua script to the BullMQ's add script we could call your code with LUA's load string (https://www.lua.org/manual/5.1/manual.html#pdf-loadstring) and using pcall to avoid any user code exceptions break BullMQ's internals.

Providing this functionality could open a can of worms of new bugs and issues produced by users mis-using the feature, which I would not love too much...