Worker concurrency

[dawncold]

Currently, a TaskTiger worker can process only one task concurrently. Is it possible to have one worker prefork multiple pool subprocesses so that it could process multiple tasks at same time?

In our case, we need preload some big modules(long time to load and a significant amount of memory usage) in parent process for performance. Starting multiple workers consumes too much memory, and this problem could be fixed if TaskTiger support worker concurrency.

[thomasst]

Thanks! This is a good suggestion but not currently planned. PRs are welcome though!

Have you considered using batch queues and optimizing your task processing function so a single worker can process tasks quicker?

[dawncold]

Thanks, batch queues should mitigate our case

[kgritesh]

@thomasst would like to work on this, any suggestion on api design you might want for this - i mean how do we mention concurrency on workers. Also i was wondering for workers might also make sense to allow thread level concurrency ie instead of preforking processes, it should be possible to use greenlets/threads/sub process right?

[thomasst]

You'd probably want an option that denotes how many workers you'd want to have and then just fork when starting the main process. I'm not convinced yet we should have an option to provide greenlets/threads.

[piercefreeman]

@kgritesh Also looking for the same worker process concurrency. Did you get started on a fork with the changed logic?