pyFAI azimuthal integration hangs up in multiprocessing mode

[aleonau]

I am trying to execute a function with azimuthal integration inside via multiprocessing.Pool() and get serious bugs with it. In particular, it persistently hangs up in parallel regime, while doing a single core calculation finishes successfully. Depending on the kernels I used, sometimes it got stuck after doing the single-core execution and executed successfully when I started with multiprocessing avoiding single-core execution. These bugs seem to be python version dependent.

In order to try debugging it, I did the following steps:

1. created a pure python 3.9 environment with only pyFAI (and its requirements) + ipykernel packages installed.
   Result: execution ALWAYS hangs up in parallel mode and finishes successfully via single-core calculation.

2. created a pure python 3.6 environment with only pyFAI (and its requirements) + ipykernel packages installed.
   Result: execution is ALWAYS SUCCESSFUL for both parallel and single-core calculations.

Just in case, I attach the Jupyter notebook and two files with numpy arrays used.
multiproc_issue.zip

P.S. updated the zip-file because check_center_all_in(X, Y) was using a specific filepath related to my folder, everything else remained the same

[kif]

Well, resetting an azimuthal integrator (because the geometry is changed) in a parallel section is not a good idea at all.
There are locks to prevent bad things to occur in muti-threaded mode, but I have no clue what would be the result with multiprocessing.
Thus I tried your code with:
from multiprocessing.pool import ThreadPool as Pool
and it works much better. Forcing a single thread at the OpenMP level also helps to make higher level parallelism more effective.

This does not explain why it breaks with mutliprocessing. With Python 3.14, the default behaviour of the module will change, from fork to spawn also on linux. With spawn it never works while fork almost works.
The issue is likely linked to the serialization of the azimuthal integrator object or of the defined function.

[aleonau]

Well, resetting an azimuthal integrator (because the geometry is changed) in a parallel section is not a good idea at all.
There are locks to prevent bad things to occur in muti-threaded mode, but I have no clue what would be the result with multiprocessing.

This should cause no problems at all because the new instances of azimuthal integrator class are copied to the worker processes (I checked that). Moreover, I defined another function at the end of the notebook called check_center_all_in(X, Y), in which the ai-objects are instantiated inside the function. Thus, there should be no overlapping, the calculations are totally independent.
By the way, commenting out the line with geometry change does not remove the issue. I tested it, too.

The bug itself is totally version dependent. I did similar calculations with multiprocessing.Pool() earlier (maybe with python 3.7), everything worked fine.

Regarding the strategy of calculation, which I want to deploy - I need to do the integration for the SAME image but on the big meshgrid of possible beam center positions and then process the results, finding the optimal position (via doing the pizza-slicing and minimizing the variance, assuming that the image is isotropic). It works fine with single core but I want to increase performance and speed it up. The calculation itself is done on the cluster (Maxwell cluster of DESY).

Anyway, I need parallelization for other means, too. For example, I need to calculate pulse resolved I(q) for ~ 500.000 images in each run/scan. This issue will definitely hang up the calculation in this case as well.

[kif]

I found an issue with your code: apparently, the serialization of the function check_center_all_in(X, Y) does not play nicely with jupyter notebook.
The default serializer, pickle, is known to be safe but picky about user supplied functions. dill is an alternative, much more powerful but be careful: with power comes responsibilities.
I also replaced the Pool from multiprocessing with an alternative from mpire. Other exist like joblib, dask, ... I let you explore the one that best fits your needs. The WorkerPool allows to tune the start_method among threading, fork, forkserver and spawn. Threading works with pickle, all other require dill. fork crashes after a while but this a known side effect of pyFAI using threads internally. It plays nicely with forkserver and spawn, the former being slightly faster.

Here is a snippet of code:

from mpire import WorkerPool
[...]
def check_center_all_in(X, Y):
    "this function is independant of any imports ... helps dill"
    import numpy as np
    import pyFAI
    from pyFAI.azimuthalIntegrator import AzimuthalIntegrator
    print("entering calculation")

    res = []
    
    data2d = np.load('mean_image.npy',allow_pickle = False)
    center2d = np.load('center.npy',allow_pickle = False)

    my_detector = pyFAI.detectors.Detector(pixel1 = 200e-6, pixel2 = 200e-6, max_shape = data2d.shape)
    
    SDD = 7687
    ai = AzimuthalIntegrator(detector = my_detector, dist = SDD/1000, wavelength = 1.24e-10)
    ai.setFit2D(SDD,center2d[1],center2d[0])
    for px, py in zip(X,Y):
        ai.setFit2D(SDD, px, py)
        print(f"integrating...{px},{py}")
        q, I = ai.integrate1d(data2d, 300, unit = "q_A^-1", dummy = np.nan, method = ("no", "histogram", "cython"))
        tmp = np.nanmean(np.nanvar(np.log(I),0),0)
        res.append( [px,py,tmp] )
    print("done")

    return res
[...]
arguments = list(zip(list(X), list(Y)))
with WorkerPool(n_jobs=6, start_method='forkserver', use_dill=True) as pool:
    result_all = pool.map(check_center_all_in, arguments)

print(result_all)