Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When n_jobs > 128 the Data class crushes #145

Open
diegodoimo opened this issue Sep 17, 2024 · 0 comments
Open

When n_jobs > 128 the Data class crushes #145

diegodoimo opened this issue Sep 17, 2024 · 0 comments

Comments

@diegodoimo
Copy link
Collaborator

The Data class takes all the cores detected by "multiprocessing.cpu_count()" as a default number of jobs. When this number is larger than 128, the Class crushes due to an OPENBLAS memory issue:

OpenBLAS : Program is Terminated. Because you tried to allocate too many memory regions.
This library was built to support a maximum of 128 threads - either rebuild OpenBLAS
with a larger NUM_THREADS value or set the environment variable OPENBLAS_NUM_THREADS to
a sufficiently small number. This error typically occurs when the software that relies on
OpenBLAS calls BLAS functions from many threads in parallel or when your computer has more
cpu cores than what OpenBLAS was configured to handle.

I propose setting the default value to 1 or None, or the value set in the underlying libraries we use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant