Notebook 6.5.x Python Tests CI Job Failing with Jupyter_Client >= 8.0.0

[RRosio]

Describe the bug
Notebook 6.5.x Python Tests CI Job, test_connections of the KernelAPITest is failing. In recent CI environments for the Python Tests, the version of jupyter_client installed is 8.0.3 which which results in the error tornado.simple_httpclient.HTTPTimeoutError: Timeout during request when the run_sync method is called.

To Reproduce
This can be locally reproduced by running the tests in notebook/services/kernels/tests/test_kernels_api.py. Observe that with a version of jupyter_client >= 8.0.0, the test_connections fails. When we install jupyter_client==7.4.9, the test suite passes.

Expected behavior
Expected successful communication with the HTTP client and passing test suite.

Links
Link to one instance of this failure in Python Tests CI Job
https://github.com/jupyter/notebook/actions/runs/4225687193/jobs/7338276859

Desktop

• OS: Apple M1

Additional context
Going back to the changes in jupyter_client in PR jupyter/jupyter_client#835, the first instance of a successfully running test suite where the test failure is present is at jupyter/jupyter_client@00ac661

Linking #6721 as changes in PR jupyter/jupyter_client#835 also showed the first instance of the error described in that issue. Specifically, the commit jupyter/jupyter_client@ab93126 is the first instance where I encountered this error message.

As @Zsailer suggested in the Notebook meeting: some of the improvements for asyncio support may be clashing with the classic notebook server so pinning a version of jupyter_client and pyzmq to quickly mitigate this issue may be the next step.

[smacke]

On my M1 Mac, I'm getting a "production" assertion error when running a classic Jupyter notebook (not nbclassic) with newer versions of Jupyter client (see attached stacktrace.txt), which looks to be identical to the zmq message arrived on closed channel error described by other users in the Jupyter discourse: https://discourse.jupyter.org/t/jupyter-notebook-zmq-message-arrived-on-closed-channel-error/17869/17

Some users suggest that rolling back tornado to 6.1 can fix the issue, but my investigation suggests that this is only because of association with older versions of jupyter_client. I did a git bisect, and narrowed down the issue to a jupyter_client commit in the range from 6fe99ae to 99a800b, which includes the referenced commit 00ac661. (I couldn't get the exact blame because there were other issues I ran into here as well.) This is a range of ~20 or so commits introduced by PR jupyter/jupyter_client#835.

Not sure whether this is helpful but I figured it couldn't hurt to add another data point.

[panlinux]

In Ubuntu and Debian, we are now also starting to see this failure in the test suite that runs at build-time. With jupyter-client 8.6.0, this test fails:

________________________ KernelAPITest.test_connections ________________________

self = <notebook.services.kernels.tests.test_kernels_api.KernelAPITest testMethod=test_connections>

    def test_connections(self):
        kid = self.kern_api.start().json()['id']
        model = self.kern_api.get(kid).json()
        self.assertEqual(model['connections'], 0)

> ws = self.kern_api.websocket(kid)
...
> return future_cell[0].result()
E tornado.simple_httpclient.HTTPTimeoutError: Timeout during request

With jupyter-client 7.4.9, it passes.

In a related project[1], @smacke did some troubleshooting and looks like they found the cause, and jupyter/nbconvert#1972 was merged. Does that translate to what is going on here in notebook?

1. nbconvert ignores ExecuteProcessor.timeout with jupyter_client>=8 nbconvert#1964

[detrout]

I tried fixing some issues in the Debian jupyter-notebook package and I'm definitely getting the time out error in the tests with jupyter-client 8.6.2-3, but I can't find any reference to setting the kernel_manager_class in packages that are installed.

When I downgrade to jupyter-client 7.4.9 in the test container, the test_connections test does pass.

Because the notebook completely broke because of the traitlets warn import, and the I was able to run notebooks even with this test failure I ignored the failing test to get a working package out, though it'd be good to figure what's going on.

Debian unstable current tornado version is 6.4.1-2 and pyzmq is 24.0.1.5+b2. I am also seeing the

WARNING  NotebookApp:zmqhandlers.py:245 zmq message arrived on closed channel
WARNING  NotebookApp:zmqhandlers.py:245 zmq message arrived on closed channel
WARNING  NotebookApp:zmqhandlers.py:245 zmq message arrived on closed channel

during the test failure condition. While looking around for hints of what's going on I did see other bug reports referencing that issue.