KeyError in line 287 of resource_tracker.py with loky backend when setting n_jobs=-1 wtih large dataset?

[PanZiwei]

I am getting Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1 **self._backend_args)when loky backend is placed inside a threading backend task. May I know if it is possible to do so?

Hi, I got a KeyError with joblib==1.1.0 and python==3.8. To be more specific, I am doing random forest modeling building with sklearn with random search to find the best parameters and save the best model.

It's interesting that the script goes smoothly with the small testing dataset, but it crashed with the real large dataset. I am not sure whether it is crashed since the program is still running with the error log message.

An example error is shown below:

Traceback (most recent call last):
  File "/Ziwei/Anaconda3/envs/test/lib/python3.8/site-packages/joblib/externals/loky/backend/resource_tracker.py", line 287, in main
    registry[rtype][name] -= 1
KeyError: '/dev/shm/joblib_memmapping_folder_209071_be605cbda8d24cb8b58e4fb0ede88ff8_3ba79016e9c241a89ea0945d721da241/209071-46914447630592-263c9f9c020242a3a28439d132a26543.pkl'

Not sure if this is the same as #1076 . Since the old issue was raised last year, so I am creating a new issue here. I also found similar issue in #978 and #1201 .

Since the program is still running and the warning message is also raising all the time, should I ignore the error and wait for the program to finish since you mentioned that "These warning messages do not affect the correctness of the results"? Or I should stop the task to find other solutions?

Thank you so much for your help!

[tomMoral]

Hello,

thanks for reporting the issue.

The error message you report should not affect the execution of your script, only the clean up of memmapped files. It is an error in the ressource_tracker.py file that is run in a separate process and responsible for tracking the number of reference to the memmap across processes.

that being said, this behavior is not normal and it would be nice to investigate why it happens. do you think you could come up with a reproducible example so we can investigate further?

[PanZiwei]

Thank you so much for the reply! Do you have an example for the reproducible example? I just use joblib to save the model so I am not sure whether the whole code is in need or not...