evaluators.py: Allow MultiprocessingEvaluator to initialize with cpu_count minus N processes #140
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit allows inputting negative integers for n_processes in the MultiprocessingEvaluator defined in evaluators.py. This way the number of logical CPU cores minus that negative number would be used. For example, on a 12 thread processor, -2 results in using 10 threads. There is a guard around the function that always at least one process. The main advantage is that when inputting a negative number, there is always some CPU power available for system and background tasks, which means the system would lag way less. When working in teams or on different devices, you can just leave it to -1 or -2 and always have some computing power to spare.
Closed
|
Another option is was thinking about is allowing a float between 0 and 1, which would indicate the fraction of logical GPU cores that the evaluator is allowed to use. So |
|
I would prefer to stay as close as possible to how this is handled in e.g. concurrent.futures and multiprocessing. The negative indexing is already a deviation, although a convenient one and in line with negative indexing. Giving a special meaning to floats between 0 and 1 would only confuse in my view. |
|
That was what I wanted to check, I agree. |
|
I just pushed the next step for #124. The main question is whether we want to address the visualization also at this stage or leave that for a future release. |
|
I would suggest moving that to 2.3.0, that we also have some time to discuss what the optimal form is. |
EwoutH
added a commit
to EwoutH/EMAworkbench
that referenced
this pull request
Oct 5, 2022
PR quaquel#140 introduced a feature for the MultiprocessingEvaluator to take a negative input for n_processes, which initialize the evaluator with the cpu_count minus that negative integer, leaving that number of cores free. In 7dd1610 this code was refactored, introducing a bug that broke the negative-integer input functionality. When a negative integer is inputted, a AttributeError: 'MultiprocessingEvaluator' object has no attribute 'n_processes' was created. The bug was caused by self.n_processes begin used instead of n_processes, where in earlier code self.n_processes was set equal to n_processes earlier in the process. This commit fixes this behaviour by using n_processes to calculate self.n_processes. Unfortunately this bug landed in the EMAworkbench 2.2.0 release. A cherry pick and new patch release should be considered.
This was referenced Oct 5, 2022
EwoutH
added a commit
that referenced
this pull request
Oct 5, 2022
PR #140 introduced a feature for the MultiprocessingEvaluator to take a negative input for n_processes, which initialize the evaluator with the cpu_count minus that negative integer, leaving that number of cores free. In 7dd1610 this code was refactored, introducing a bug that broke the negative-integer input functionality. When a negative integer is inputted, a AttributeError: 'MultiprocessingEvaluator' object has no attribute 'n_processes' was created. The bug was caused by self.n_processes begin used instead of n_processes, where in earlier code self.n_processes was set equal to n_processes earlier in the process. This commit fixes this behaviour by using n_processes to calculate self.n_processes.
EwoutH
added a commit
that referenced
this pull request
Oct 6, 2022
PR #140 introduced a feature for the MultiprocessingEvaluator to take a negative input for n_processes, which initialize the evaluator with the cpu_count minus that negative integer, leaving that number of cores free. In 7dd1610 this code was refactored, introducing a bug that broke the negative-integer input functionality. When a negative integer is inputted, a AttributeError: 'MultiprocessingEvaluator' object has no attribute 'n_processes' was created. The bug was caused by self.n_processes begin used instead of n_processes, where in earlier code self.n_processes was set equal to n_processes earlier in the process. This commit fixes this behaviour by using n_processes to calculate self.n_processes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit allows inputting negative integers for
n_processesin theMultiprocessingEvaluator. This way the number of logical CPU cores minus that negative number would be used. For example, on a 12 thread processor,-2results in using 10 threads. There is a guard around the function that always at least one process.The main advantage is that when inputting a negative number, there is always some CPU power available for system and background tasks, which means the system would lag way less. When working in teams or on different devices, you can just leave it to
-1or-2and always have some computing power to spare.