Slow and blocking startup of jupyter-ai server extension

[krassowski]

Description

jupyter-ai is the slowest extensions of the ones that I am shipping in various deployments. Even worse, it is blocking the jupyter-server from startup. It would be lovely if we could reduce the startup time.

Reproduce

1. Install jupyter-ai
2. Run jupyter server extension list to get detailed timings, on my laptop it is:

Package jupyter_ai took 2.9546s to import

as compared to all other extensions being ready instantaneously:

 Package jupyterlab took 0.0398s to import
 Package nbclassic took 0.0014s to import
 Package jupyter_server_fileid took 0.0000s to import
 Package jupyter_server_ydoc took 0.0655s to import
 Package jupyter_lsp took 0.0290s to import

3. Run jupyter lab or jupyter server
4. See that jupyter-ai blocks jupyter-server startup for 2-3 seconds
5. See that it prints lots of messages about dependencies being not installed even when we block other providers (so it needlessly attempts some imports)

Expected behavior

1. Async startup sequence is used:
   • either using a bespoke solution as in jupyter-lsp (see Use app's HTTP settings, handle feature settings race condition, CI/build updates, lazy server discovery jupyter-lsp/jupyterlab-lsp#882)
   • using new jupyter-server async hook once merged Add async start hook to ExtensionApp API jupyter-server/jupyter_server#1417
2. No imports are performed for blocked providers

Context

• Operating System and version: Ubuntu
• JupyterLab version: 4.3.1

[dlqqq]

@krassowski Thank you for opening an issue about this! I've also noticed that the startup time has slowly worsened over the last year, and I wish I had the time to fix that.

Could you / another contributor explore adding debug/info logs that trace the time it takes for the extension to initialize? For example, it'd be great to see what % of the time is spent loading the entry points & entry point groups. We need that data to determine where the slowdown is coming from, and to determine the best fix available.

My ability to contribute will be limited, as I need to focus full-time on Jupyter AI v3 very soon. Hope you understand, and I sincerely appreciate your patience as you deal with Jupyter AI issues. 🙏