feat(job): allow passing debounce as option #2666

roggervalf · 2024-07-20T05:22:16Z

ref #240 #2086 #1318 #1799

manast · 2024-07-23T09:37:22Z

src/classes/job.ts

@@ -200,6 +205,8 @@ export class Job<
      ? { id: opts.parent.id, queueKey: opts.parent.queue }
      : undefined;

+    this.debounceId = opts.debouncing ? opts.debouncing.id : undefined;


I think the name of the option should be "debounce" instead of "debouncing".

manast · 2024-07-23T09:38:43Z

src/classes/job.ts

@@ -323,6 +330,10 @@ export class Job<
      job.repeatJobKey = json.rjk;
    }

+    if (json.deid) {
+      job.debounceId = json.deid;


wouldn't it be better to use the same opts structure instead of having a new debounceId field in the job class? This will keep the class cleaner, specially if we add more options to the debounce option.

I didn't save all the option because ttl is not reuse as debounceId, similar to repeatable jobs, we only save repeatJobKey instead of saving all repeat options in job instance.

manast · 2024-07-23T09:40:41Z

docs/gitbook/guide/jobs/debouncing.md

@@ -0,0 +1,39 @@
+# Debouncing
+
+Debouncing a job implies delaying and deduplicating it.


-> Debouncing in BullMQ is a process where job execution is delayed and deduplicated based on specific identifiers. It ensures that within a specified period, or until a specific job is completed or failed, no new jobs with the same identifier will be added to the queue. Instead, these attempts will trigger a debounced event.

manast · 2024-07-23T09:41:34Z

docs/gitbook/guide/jobs/debouncing.md

+Debouncing a job implies delaying and deduplicating it.
+
+## Fixed Mode
+


-> In the Fixed Mode, debouncing works by assigning a delay (Time to Live, TTL) to a job upon its creation. If a similar job (identified by a unique debouncer ID) is added during this delay period, it is ignored. This prevents the queue from being overwhelmed with multiple instances of the same task, thus optimizing the processing time and resource utilization.

manast · 2024-07-23T09:51:58Z

docs/gitbook/guide/jobs/debouncing.md

+);
+```
+
+For the next 5 seconds, after adding this job, next jobs added with same **debounce id** will be ignored and a _debounced_ event will be triggered by our QueueEvent class.


-> In this example, after adding the house painting job with the debouncing parameters (id and ttl), any subsequent job with the same debouncing ID customValue added within 5 seconds will be ignored. This is useful for scenarios where rapid, repetitive requests are made, such as multiple users or processes attempting to trigger the same job.

manast · 2024-07-23T09:53:52Z

docs/gitbook/guide/jobs/debouncing.md

+```
+
+While this job is not moved to completed or failed state, next jobs added with same **debounce id** will be ignored and a _debounced_ event will be triggered by our QueueEvent class.
+


This mode is particularly useful for jobs that have a long running time or those that must not be duplicated until they are resolved, such as processing a file upload or performing a critical update that should not be repeated if the initial attempt is still in progress.

manast · 2024-07-23T09:59:09Z

src/classes/queue.ts

+  async removeDebounceKey(id: string): Promise<number> {
+    const client = await this.client;
+
+    return client.del(this.toKey('debounce:') + id);


The postfix for this key could be just "de:" to keep it as small as possible.

manast · 2024-07-23T10:04:57Z

src/commands/addDelayedJob-6.lua

@@ -86,10 +88,20 @@ else
    end
 end

+local debounceId = opts['debo'] and opts['debo']['id']


This code should be refactored as it is exactly the same on 4 different places.

manast · 2024-07-23T10:05:47Z

src/commands/addDelayedJob-6.lua

+    return debouncedJobKey
+  end
+end
+
 -- Store the job.
 local delay, priority = storeJob(eventsKey, jobIdKey, jobId, args[3], ARGV[2],


as we are storing the opts with the debounce options, why do we need to pass the debounceId again?

yeah I can refactor it

manast · 2024-07-23T10:06:52Z

src/commands/includes/debounceJob.lua

+  Function to debounce a job.
+]]
+
+local function debounceJob(prefixKey, debounceId, ttl, jobId, eventsKey, maxEvents)


I think it is more Redis friendly to pass the debounceKey as a command key instead of building it inside lua, as all the components of the key are known before calling this script.

manast

Just a thought. What happens if you have a job with debounce id and some TTL, and the job fails but you have retries? should the TTL reset in this case?

manast · 2024-07-24T07:50:56Z

src/commands/includes/debounceJob.lua

+  local debounceId = debounceOpts and debounceOpts['id']
+  if debounceId then
+    local ttl = debounceOpts['ttl']
+    local isFirstSet


In fact "isFirstSet" is not a correct naming for this variable, as for example, you could have many jobs using the same debounceId, and when the debounceId expires a new job will be added and processed, so basically this variable should just be named "keyExists" or something like that.

manast · 2024-07-24T08:01:50Z

src/commands/includes/removeJob.lua

 --- @include "removeJobKeys"
 --- @include "removeParentDependencyKey"

-local function removeJob(jobId, hard, baseKey)
+local function removeJob(jobId, hard, baseKey, shouldRemoveDebounceKey)


In general, I think passing booleans as flags is worse than passing an object with optional flags in it, because the code is more legible, however, it would also be slower... so we can keep it like this I guess. But we should add a comment when used, for example:

removeJob(key, hard, baseKey, true --[[remove debounce key]]--)

roggervalf · 2024-07-24T12:16:37Z

Just a thought. What happens if you have a job with debounce id and some TTL, and the job fails but you have retries? should the TTL reset in this case?

No in case of fixed I think, as It's name implies to keep it for that periodo of time. Better to use extended mode if they want to keep it in retries

manast

LGTM

# [5.11.0](v5.10.4...v5.11.0) (2024-07-29) ### Features * **job:** allow passing a debounce as option ([#2666](#2666)) ([163ccea](163ccea))

jamesholcomb · 2024-08-01T12:22:18Z

👏 👏 👏 wow almost 6 years from the original OptimalBits/bull#1034 but happy to finally see this!

roggervalf force-pushed the debounced branch from fb64d03 to d9f1c1d Compare July 22, 2024 02:09

manast reviewed Jul 23, 2024

View reviewed changes

manast reviewed Jul 24, 2024

View reviewed changes

roggervalf force-pushed the debounced branch from 5c54545 to af3ccb0 Compare July 24, 2024 12:37

roggervalf changed the title ~~feat(job): allow passing a debounceId as option~~ feat(job): allow passing a debounce as option Jul 24, 2024

roggervalf added 12 commits July 25, 2024 21:39

feat(job): allow passing a debounceId as option

a175b55

fix: separate get flag in set call bc not supported in redis v6

d9c0abf

docs(jobs): add debouncing section

87399a9

feat: extend debounce logic

d916f47

chore: update job attributes

dc3837a

refactor: address comments

bb16307

chore: change debo to de

0755ced

chore: change export references

a851681

chore: fix debo ref

a28cc43

chore: remove debounceKey when manual removal

34f1fed

test: add test case when removing debounced job

375641a

chore: address comments

bbfa2e2

roggervalf force-pushed the debounced branch from af3ccb0 to bbfa2e2 Compare July 26, 2024 03:39

manast approved these changes Jul 29, 2024

View reviewed changes

roggervalf merged commit 163ccea into master Jul 29, 2024
10 checks passed

roggervalf deleted the debounced branch July 29, 2024 12:14

github-actions bot pushed a commit that referenced this pull request Jul 29, 2024

chore(release): 5.11.0 [skip ci]

1b31a6c

# [5.11.0](v5.10.4...v5.11.0) (2024-07-29) ### Features * **job:** allow passing a debounce as option ([#2666](#2666)) ([163ccea](163ccea))

roggervalf changed the title ~~feat(job): allow passing a debounce as option~~ feat(job): allow passing debounce as option Aug 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(job): allow passing debounce as option #2666

feat(job): allow passing debounce as option #2666

roggervalf commented Jul 20, 2024 •

edited

Loading

manast Jul 23, 2024

manast Jul 23, 2024

roggervalf Jul 23, 2024

manast Jul 23, 2024

manast Jul 23, 2024

manast Jul 23, 2024

manast Jul 23, 2024

manast Jul 23, 2024

manast Jul 23, 2024

manast Jul 23, 2024

roggervalf Jul 23, 2024

manast Jul 23, 2024

manast left a comment

manast Jul 24, 2024

manast Jul 24, 2024

roggervalf commented Jul 24, 2024

manast left a comment

jamesholcomb commented Aug 1, 2024

		@@ -0,0 +1,39 @@
		# Debouncing

		Debouncing a job implies delaying and deduplicating it.

		Debouncing a job implies delaying and deduplicating it.

		## Fixed Mode

		```

		While this job is not moved to completed or failed state, next jobs added with same debounce id will be ignored and a _debounced_ event will be triggered by our QueueEvent class.

feat(job): allow passing debounce as option #2666

feat(job): allow passing debounce as option #2666

Conversation

roggervalf commented Jul 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roggervalf commented Jul 24, 2024

manast left a comment

Choose a reason for hiding this comment

jamesholcomb commented Aug 1, 2024

roggervalf commented Jul 20, 2024 •

edited

Loading