Question: Multiple Worker Instances (e.g. Multiple Docker Containers Running Video Transcoding Process)

[tommyc38]

I am having a tough time finding any guidance on running multiple app instances all connected the he same distributed RedisDB. Below is how I was planning on setting up BullJs to run a transcoding service and simply want to know if my understanding is correct. Forgive me in advance if this question is stupid, I don't have a lot of experience in backend architecture. The only examples I really found only seem to demonstrate a scenario of one consumer instance running at a time.

Thought process / Scenario:

1.) Front end is notified video uploaded successfully and calls a micro service to add the video transcoding job to the Reddis queue (e.g. bulljs producer) and sends back the job object containing the job ID so the status can be tracked on the front end.

2.). I have five Nodejs consumer instances (e.g. docker containers) up and listening on the same RedisDB for jobs. Assume all five instances aren't processing anything when the job is queued, concurrency is set 1, and rate limit is 10. The first instance to respond to Reddis is awarded the job and begins processing but keeps listening since it's rate limit is 10 and only has one job. Is that accurate? My big concern is the same job will be processed by all five instances, instead of one. Moreover, if 11 more jobs come in and the same instance keeps "beating" the others to process the jobs, will that instance stop listening after the 10th job (one currently processing and 10 in that instance's queue), thus allowing another instance to pickup the job? <----anymore clarification in this step would be super helpful.

3.) My front end polls Reddis using the job ID to update loading indicator in the UI.

4.) The job finishes and the consumer instance begins listening for more jobs. Front end sees this and continues on...

Any help would be greatly appreciated!!!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[BigMilk-cmd]

The only thing that I have seen that is close to what you're looking for is "Distributed Transcoding". It's not much talked about but none the less there are still people working on such things for things like Plex Server Transcoding tasks being distributed to a Raspberry Pi 4 Cluster network working with the Master:Slave Structure. You may be able to find something in researching that.

[BigMilk-cmd]

To just get this straight, you're talking about getting a queue of jobs from 0-9 and distributing the work for example job in queue index 0 to machine 1 and job in queue index 1 to machine 2? Does that sound on par with what you're discussing?

[tommyc38]

Yes, something like that. The documentation only gives a simple example where both the queue and the worker are basically on the same machine. I wanted more clarification for a more practical scenario. I haven't played with it yet but will be soon. I'll spin up a bunch of workers, connect them to the same Reddis cloud instance, and then hammer the queue with jobs and see what happens. I'll report back to this thread in case someone else is looking for more details.

[BigMilk-cmd]

NOT Elegant or Refined

As much as I can tell I, a way you can do this is by creating an array, list or object storing the port information, or any information needed to construct a new redis instance. You can set a counter globally that resets every time the max iteration is met and use that number to equally distribute the jobs to the proper redis container. For example:

//Check if there is a job in your wait list

int NEXT_RDB = '0' ;                      //Sets data base count for limiting
string CURR_RDB = ""                   //Contains the current Port of the RedisDB

function Redis_Constructor(NEXT_RDB){      //You **can** implement another for loop to set your max limit ("10")
    for(NEXT_RDB; NEXT_RDB < 5; NEXT_RDB++){
          return NEXT_RDB == '0' ? CURR_RDB = "6379"
                   : NEXT_RDB == '1' ? CURR_RDB = "6380"
                   : NEXT_RDB == '2' ? CURR_RDB = "6381"
                   : NEXT_RDB == '3' ? CURR_RDB = "6382"
                   : NEXT_RDB == '4' ? CURR_RDB = "6383";

           var redis = new Redis(CURR_RDB, "localhost",[options]);
                                    ^           ^           ^
                                    |           |           |  //Then however you insert or pass your jobs into Redis
    }                          //[port]     [host]     [options]
NEXT_RDB = '0';
}

Sources

1. https://github.com/luin/ioredis/blob/master/API.md#new_Redis_new
2. https://github.com/OptimalBits/bull/blob/master/REFERENCE.md

[manast]

@tommyc38 I apologize if the documentation is not clear about it, but the whole point of using Redis as backend is to be able to process jobs from any machine that is connected to that Redis instance, that is how you scale to as many machines as you wish. You do not need to do anything special, just instantiate as many workers as you want each in every machine and they will start consuming the jobs in parallel.

[tommyc38]

@manast Thank you for the response. I figured that's exactly how it worked ...auto magic can always be a bit vague ;). Thank you for the awesome lib!

[hatemjaber]

Is there an example of a multi-server deployment? One producer and n+ workers? I can't seem to find an example to get an idea of what that looks like.

[manast]

@hatemjaber there are many examples and tutorials here: https://blog.taskforce.sh/

[hatemjaber]

I just want to make sure I understand this... Assuming my code in index.ts looked like so:

import { Worker, Job } from 'bullmq'

const worker = new Worker(queueName, async (job: Job) => {
    // Do something with job
    return 'some value';
}, { concurrency: 50 });

worker.on('completed', (job: Job, returnvalue: any) => {
  // Do something with the return value.
});

would this be enough to call it a "worker"? Is it as simple as just running node index.js and this thing would run grabbing jobs from the queue as long as there are jobs left?

what happens when there are no more jobs? does it just sit idle waiting?

Sorry for all the questions. I just learned about this library yesterday and looked around for examples etc... The one thing that I have not seen is a parallel example that has it laid out with multiple instances of workers running.

Thank you for your patience and time in advance!