Multi worker-node setup failing to process when one node is removed/crashes

[magrossi]

Hi,

I have two host VMs running my Kue worker process. If all nodes are online everything works fine, but during some tests if I bring one host down (by killing/suspending the VM - ie. ungraceful worker shutdown) the system gets out of sync.

Similarly with this issue #130 any new jobs are not processed right away, it will process the first inactive job on the queue, leaving the newest ones in inactive forever, until there are new jobs created and so on. Leaving always 2 inactive jobs to be processed.

Due to the nature of my app this breaks completely the user experience. We are calling the stuck job watcher every 1s [queue.watchStuckJobs(1000)] and even then this does not fix the issue.

Is this per design? If one worker crashes unexpectedly the whole system becomes unreliable? Is there anything I can do remedy this?

Thank you!

Kind regards

[behrad]

Can you test your setup with v1 branch @magrossi ?

[magrossi]

I can sure try. Is that branch suited for production use should it pass this small test?
Thank you!

[behrad]

It is actually running in production on my side, however I've not wrapped it up to a release yet.

[magrossi]

Hi @behrad,

The v1 branch seems incompatible with master from the user API perspective. Without changing my client code the new branch does not work. It fails connecting to REDIS I'm guessing.

Any documentation/samples on how to start using this new branch?

Thank you.

[behrad]

It's 90% compatible. have you npm installed Kue's dependencies?

[magrossi]

Yes I did.

[behrad]

aha... another thing is node_redis is replaced with ioredis, so Yes! I think you may need to change to ioredis format of redis options!

[magrossi]

Ah that must be it! I'll have a try and let you know how it goes, thank you.

[jamesjjk]

@behrad facing the same issue, have upgraded to v1 and testing. Just to note your v1 branch does have more then 11 listeners on as single event - which is related to the number of jobs I guess in a kue. Do we really need that many, it exceeds nodes default threshold.

[magrossi]

Hi @behrad,
I have just finished my preliminary tests with version v1 and I can confirm it does work. Do you have any plans for a release? If so, when would that be?
Thank you.

[jamesjjk]

@magrossi What changes did you make to get it work, for me it throws an error almost immediately. I have ioredis installed?

[magrossi]

@jamesjjk I had to change the options I sent out to Kue. You need to add "port" and "host" on the main object, whereas previously it used to sit in a "redis" property.

var options = {
"port": 6379,
"host": "my-redis",
"db": 0,
...
}

Also, I had to remove any calls to "watchStuckJobs" as it is no longer in the API. If you changed your package.json to point to the v1 branch and did an npm install kue afterwards, it should be good to go.

Regards.

[jamesjjk]

@magrossi Thanks worked out, still facing some exceptions with v1 and debugging.

[magrossi]

@jamesjjk hopefully, we'll hear from @behrad soon on when this wonderful revision will have a release version so we can swap it over.
Cheers