Timeout during generate_cutting_experiments in how_to_generate_exact_sampling_coefficients.ipynb

[garrison]

Now that #514 is merged, we can actually see which notebook hangs intermittently on CI.

=================================== FAILURES ===================================
_ /home/runner/work/circuit-knitting-toolbox/circuit-knitting-toolbox/docs/circuit_cutting/how-tos/how_to_generate_exact_sampling_coefficients.ipynb _
A cell timed out while it was being executed, after 300 seconds.
The message was: Cell execution timed out.
Here is a preview of the cell contents:
-------------------
subexperiments, coefficients = generate_cutting_experiments(
    circuits=subcircuits,
    observables=subobservables,
    num_samples=np.inf,
)
coefficients
-------------------

------------------------------ Captured log call -------------------------------
ERROR    traitlets:client.py:845 Timeout waiting for execute reply (300s).

Full log at:

• https://github.com/Qiskit-Extensions/circuit-knitting-toolbox/actions/runs/8456649627/job/23166889295?pr=471 (ubuntu-latest, 3.8)
• https://github.com/Qiskit-Extensions/circuit-knitting-toolbox/actions/runs/8475888125/job/23224662100#step:6:153 (ubuntu-latest, 3.8)
• https://github.com/Qiskit-Extensions/circuit-knitting-toolbox/actions/runs/8511573980/job/23311448303#step:6:162 (ubuntu-latest, 3.8)
• https://github.com/Qiskit-Extensions/circuit-knitting-toolbox/actions/runs/8513967433 (ubuntu-latest, 3.8)
• https://github.com/Qiskit-Extensions/circuit-knitting-toolbox/actions/runs/8542748098/job/23405055909#step:6:163 (ubuntu-latest, 3.8)
• https://github.com/Qiskit-Extensions/circuit-knitting-toolbox/actions/runs/8585209021/job/23526356875 (ubuntu-latest, 3.8)
• https://github.com/Qiskit-Extensions/circuit-knitting-toolbox/actions/runs/8604410132/job/23578451818?pr=538 (ubuntu-latest, 3.8)

[garrison]

I keep on adding to the list above, but so far this only seems to happen on Python 3.8 on Ubuntu.

[garrison]

I noticed a warning when running the test suite under Python 3.12 that may be a clue.

test/cutting/test_cutting_workflows.py: 48 warnings
test/cutting/test_cutting_roundtrip.py: 420 warnings
  /usr/local/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=17907) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

I was initially confused because we are not using multiprocessing explicitly, but it turns out that Qiskit's transpiler actually uses multiprocessing when a pass manager is asked to transpile multiple circuits in the same call. Indeed, the traceback of the warning is

circuit_knitting/cutting/cutting_experiments.py:188: in generate_cutting_experiments
    def test_generate_cutting_experiments(self):
        with self.subTest("simple circuit and observable"):
            qc = QuantumCircuit(2)
            qc.append(
                TwoQubitQPDGate(QPDBasis.from_instruction(CXGate()), label="cut_cx"),
                qargs=[0, 1],
            )
            comp_coeffs = [
                (0.5, WeightType.EXACT),
                (0.5, WeightType.EXACT),
                (0.5, WeightType.EXACT),
                (-0.5, WeightType.EXACT),
                (0.5, WeightType.EXACT),
                (-0.5, WeightType.EXACT),
            ]
>           subexperiments, coeffs = generate_cutting_experiments(
                qc, PauliList(["ZZ"]), np.inf
            )

test/cutting/test_cutting_experiments.py:52: 
    subexperiments_dict[label] = pass_manager.run(subexperiments)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/py312/lib/python3.12/site-packages/qiskit/transpiler/passmanager.py:420: in wrapper
circuit_knitting/cutting/cutting_experiments.py:188: in generate_cutting_experiments
    return meth(*meth_args, **meth_kwargs)
.tox/py312/lib/python3.12/site-packages/qiskit/transpiler/passmanager.py:182: in run
    subexperiments_dict[label] = pass_manager.run(subexperiments)
.tox/py312/lib/python3.12/site-packages/qiskit/transpiler/passmanager.py:420: in wrapper
    return super().run(
.tox/py312/lib/python3.12/site-packages/qiskit/passmanager/passmanager.py:246: in run
    return meth(*meth_args, **meth_kwargs)
.tox/py312/lib/python3.12/site-packages/qiskit/transpiler/passmanager.py:182: in run
    return parallel_map(
.tox/py312/lib/python3.12/site-packages/qiskit/utils/parallel.py:171: in parallel_map
    return super().run(
.tox/py312/lib/python3.12/site-packages/qiskit/passmanager/passmanager.py:246: in run
    raise error
.tox/py312/lib/python3.12/site-packages/qiskit/utils/parallel.py:161: in parallel_map
    return parallel_map(
.tox/py312/lib/python3.12/site-packages/qiskit/utils/parallel.py:171: in parallel_map
    future = executor.map(_task_wrapper, param)
/usr/local/lib/python3.12/concurrent/futures/process.py:822: in map
    raise error
.tox/py312/lib/python3.12/site-packages/qiskit/utils/parallel.py:161: in parallel_map
    results = super().map(partial(_process_chunk, fn),
/usr/local/lib/python3.12/concurrent/futures/_base.py:608: in map
    future = executor.map(_task_wrapper, param)
/usr/local/lib/python3.12/concurrent/futures/process.py:822: in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
/usr/local/lib/python3.12/concurrent/futures/process.py:794: in submit
    results = super().map(partial(_process_chunk, fn),
/usr/local/lib/python3.12/concurrent/futures/_base.py:608: in map
    self._start_executor_manager_thread()
/usr/local/lib/python3.12/concurrent/futures/process.py:733: in _start_executor_manager_thread
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
/usr/local/lib/python3.12/concurrent/futures/process.py:794: in submit
    self._launch_processes()
/usr/local/lib/python3.12/concurrent/futures/process.py:760: in _launch_processes
    self._start_executor_manager_thread()
/usr/local/lib/python3.12/concurrent/futures/process.py:733: in _start_executor_manager_thread
    self._spawn_process()
/usr/local/lib/python3.12/concurrent/futures/process.py:770: in _spawn_process
    self._launch_processes()
/usr/local/lib/python3.12/concurrent/futures/process.py:760: in _launch_processes
    p.start()
/usr/local/lib/python3.12/multiprocessing/process.py:121: in start
    self._spawn_process()
/usr/local/lib/python3.12/concurrent/futures/process.py:770: in _spawn_process
    self._popen = self._Popen(self)
/usr/local/lib/python3.12/multiprocessing/context.py:282: in _Popen
    p.start()
/usr/local/lib/python3.12/multiprocessing/process.py:121: in start
    return Popen(process_obj)
/usr/local/lib/python3.12/multiprocessing/popen_fork.py:19: in __init__
    self._popen = self._Popen(self)
/usr/local/lib/python3.12/multiprocessing/context.py:282: in _Popen
    self._launch(process_obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <multiprocessing.popen_fork.Popen object at 0x7fafdf377710>
process_obj = <ForkProcess name='ForkProcess-1' parent=6471 unknown>

    return Popen(process_obj)
/usr/local/lib/python3.12/multiprocessing/popen_fork.py:19: in __init__
    def _launch(self, process_obj):
        code = 1
        parent_r, child_w = os.pipe()
        child_r, parent_w = os.pipe()
>       self.pid = os.fork()

which points to exactly the line that is hanging under nbmake in the tests under Python 3.8.

(I wonder if the issue here might be in part due to interaction with pytest and/or nbmake, but it's doesn't appear like nbmake uses any multiprocessing directly.)

So perhaps this is again an instance of https://pythonspeed.com/articles/python-multiprocessing/, in which it could perhaps be fixed by using mp.get_context("spawn") in Qiskit.

Possibly relevant issues

• Python 3.9 on linux hangs when using multiprocessing qiskit#6188 (comment)
• multiprocessing's default posix start method of 'fork' is broken: change to `'forkserver' || 'spawn' python/cpython#84559
• raise a Warning when os.fork() is called and the process has multiple threads python/cpython#100228

[garrison]

Another way we could potentially work around this is by setting QISKIT_NUM_PROCS=1.

[garrison]

The symptom seems to have been fixed by performing the ubuntu tests on Python 3.9 instead of 3.8. With #556, the generate_cutting_experiments function will no longer use the pass manager (and therefore not need multiprocessing), so we should no longer see this even on Python 3.8 once that PR is merged.