BUG: Multiprocessing and caching with mixed versions

[tvwenger]

Describe the issue:

This bug appeared using the SMC sampler in pymc. More details about the issue's discovery are here: pymc-devs/pymc#8022 (comment)

The gist of it is that the SMC sampler dispatches multiple parallel chains using multiprocessing, and sometimes those processes would hang. Switching between versions of pymc/pytensor did not resolve the hanging. At least once, the SMC sampler threw an error that pytensor couldn't access a lock file in ~/.pytensor. That led me to deleting ~/.pytensor, which ultimately solved the problem.

@ricardoV94 and I suspect that there is some caching issue in ~/.pytensor that shows up in multiprocessing.

I'm sorry that I can't help more or provide a MWE, since the bug disappeared after deleting ~/.pytensor.

Reproducable code example:

https://www.pymc.io/projects/examples/en/latest/samplers/SMC2_gaussians.html

Error message:

PyTensor version information:

Latest

Context for the issue:

No response