Get job return value

[kimmobrunfeldt]

I have the following setup, where I use bull as the job/message queue using this pattern which you have documented and recommended in other issues. I have one problem which I haven't been able to solve.

Data flow:

• HTTP request comes in
• A job based on the HTTP request is sent to a Queue('worker') e.g. jobId 2
• Worker processes the job, and sends the result back to Queue('server'). The response "job" is just a message. It has a different jobId, e.g. 3
• Problem: How to forward the job result to the correct HTTP response? We have multiple web instances so that they are running in different machines.

I wasn't able to implement the described scenario with the recommended two queues approach. I would need something more from the API. A few possibilities:

1. Provide a job.on('completed') API. Then the instance which is creating the job, could listen the completion event and respond the result back as a HTTP response.

2. Provide a way to "wait or get" a completed job with a data filter. For example:

   // server is the queue where worker responds a result from a job.
   // When the worker job was created, it was given a jobId. For example `2`.
   // There needs to be a way to link a result to a started job. I have used the pattern
   // to save the worker job id inside the data of result.
   // So the worker responds data: server.add({jobId: workerJob.jobId, result: /* .. */); 
   var server = Queue('server', /* ... redis details ... */);
   
   // The server/web application should have some mechanism like this 
   // which could be used to get the correct result back
   server.getOrWait(function(jobData) {
       return jobData.jobId === workerJob.jobId;
   })
   .then(function(job) {
       // Here we could get the correct result from job.data
       // ... and respond the data back as a http response
   });

Other relevant issues which directly didn't answer my problem: #11, #178, #209

If you think the newest 0.7.0 release resolves this problem, could you explain how the API works? I can also help to document the usage to readme.

[kimmobrunfeldt]

Here is the request/data flow in a picture to make it more clear:

[manast]

I think I understand your use case. In my opinion you should not use bull for a case like this. The pattern described in the README is decoupling the service that puts a job into the queue from the service that handles the completion of the job. I don't know the details of what you are trying to do, but in the case you want to notify a job status to the client that requested the job you could also use redis pubsub mechanism, and websocket communication to update progress or status of a job. I dont think it is a good idea to keep a http request idling while bull is processig a job.

[kimmobrunfeldt]

@manast Ok I understand, thanks for the response. In my use case the processing takes about 0.5 - 5 seconds and I would like to make the API for the user as easy as possible. So implementing it as a "blocking" HTTP call, should be fine in this case. I have separate workers so I'll be able to scale the workers independently from the web instances.

[VitorBrangioni]

Same issue, any solution?

[rinogo]

For those who land here from Google: Bull 3.0.0. RC 4 added support for the glorious await job.finished(). Enjoy! :)

[Jiale-Li]

对于那些从谷歌登陆这里的人：Bull 3.0.0。RC 4 添加了对光荣的 await job.finished（） 的支持。享受！:)

It took me four hours to find here.
It's really great！

[ajhurliman]

To be a little bit more clear (since it looks like it's missing in the bull docs), job.finished resolves to the job result so it's more useful than just giving the ability to await the job completing, e.g.

const pdfJob = await pdfQueue.add({ ...data })
const pdfJobResult = await pdfJob.finished();
console.log(pdfJobResult) // results are here