Max worker issue

[jenna-tomkinson]

Consider finding the max between multiprocessing.cpu_count() and num_processes to help avoid this exception case. If there are fewer CPU than commands the pool will handle the start and finish of each one.

Originally posted by @d33bs in #1 (comment)

[jenna-tomkinson]

This issue references the function only calling the exact number of workers as the command or processes as determined by the dictionary (e.g., 3 plates in the dictionary means 3 workers needed).

Per CellProfiler, it is default to run one job at a time, but states:

If you're willing to script some ways to scatter the CellProfiler jobs, though, or don't mind running one job at a time, headless processing can be incredibly helpful!

This potentially opens the door for future work.

[d33bs]

Adding comments from my reply in #1:

But if someone has 3 commands to run based on their dictionary, for example, why would they need to start more than that number of processes?

Yes, great point! This would I think be an edge case that could be covered by another if-condition.

My initial comments mostly pertained to times where someone would want to only provide a limited number of processors (they may have other things running). For example, if there are 5 CP runs to accomplish but the person wants to only use 3 cores to accomplish that work (each additional job beyond 3 would begin as the earlier submitted ones complete).