Processing Concurrency

[Jellyfrog]

Support for processing multiple jobs at the same time would be nice!

As a workaround, is it safe to run multiple instances with the same database?

[manast]

I prefer to keep the module simple and not introduce a concurrent mechanism builtin, which may not be of everybodies liking. Instead it should be safe to run as many instances of a queue in the same, or different threads and processes. I wrote an example on the README using the cluster module, it works just nice.

[Jellyfrog]

I agree!

[yhpark]

@manast
A process could be in idle state for a long time if async operation is done in job processing. Creating many Queue instances may introduce overhead on both node and redis instance for maintaining connections when there are hundreds of Queue instances. Running several(or many) processes would also create much memory consumption so I think it's also not a good solution. And I'm not sure what you meant about threads.. thread in node?
I hope you reconsider this issue.

[manast]

@yhpark not sure the overhead is so large even with hundreds of redis clients, it should be designed to handle thousands and thousands of concurrent connections. If you can write a test that shows these problems I can take a look at it :).

You can read threads in node: https://github.com/xk/node-threads-a-gogo

[yhpark]

@manast Well even if overhead is not visible immediately it'll definitely consume more cpu/memory. I'm not sure if code complexity issue is more serious than performance or small footprint.
I won't insist further if you're still not convinced, just count me as +1 if other people bring this up.

[manast]

yes, code complexity is a big issue, making it atomic and robust as it is right now is already quite difficult...