CU-86949yar7: Add logged warning for when multiprocessing fails due to being out of memory #415
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The underlying issue manifested in the following way:
When calling
multiprocessing_batch_char_size
the output would come out completely empty. Nothing logged as to what happened or why.This is because the machine in question has 128GB of RAM and the default minimum expected RAM is 10%. And the machine happened to have around 6GB of free memory. Which is less than 5% (and less than the minimum required).
The solution for the user is to specify a smaller
min_free_memory
value when calling the method (e.g0.005
in this case for around 500MB).So what this PR does:
min_free_memory_size
where users can specify a human-readable minimum memory amountThis way, in future cases, we'll know what the issue is and how to fix it.