Add a disconnect button for RayContext in notebooks

[peytondmurray]

Why are these changes needed?

This PR reverts #35426, adding back in a nice button for disconnecting (i.e. calling ray.shutdown()) when ray.init() is called in a notebook.

Testing

The main issue with the previous disconnect PR is that checking for the ipywidgets soft dependency at run time introduces a small performance penalty which bumps some test suites up beyond their timeout upper bound, breaking them. This happens in tests unrelated to where these changes were made previously, making troubleshooting more difficult.

Here, I've introduced some optimizations to avoid this penalty wherever possible:

• In ray.widgets.util.in_notebook, a significant performance penalty was previously being incurred by the try-import/except block upon each call. Now, we just check whether "IPython" in sys.modules is True before attempting an import. This is much faster as IPython is pre-loaded for IPython kernels. I've also added this optimization to repr_fallback_if_colab.
• Secondly, ensure_notebook_deps now does dependency checking at function definition time, rather than on each function call.
• I've also reworked the logic for detecting the current shell, since there's really only one situation where displaying widgets is okay (when the user is running Jupyterlab). In all other cases, we fall back to simple reprs.
• The changes here also solve [Core] Import in Colab produces error #35490.

Here's the artifact from the CI run below loaded onto google colab:

The rendered output has no dashboard URL row (evidently this isn't available when run on colab), but the HTML repr looks okay. The only graphical issue that I see is that colab is doing something weird to the "RAY" text that is supposed to appear by the Ray logo. Here's what this looks like in Jupyterlab:

Related issue number

• Closes [ray and Jupyter] Add a "disconnect" button for the output of ray.init in jupyter #34431
• Closes [Core] Import in Colab produces error #35490
• Reverts Revert "Add a disconnect button to the context widgets in notebooks (… #35426
• See also [CherryPick] Revert "Add a disconnect button to the context widgets in notebooks (#34815)" #35500

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[peytondmurray]

I tested the 3.10 artifact in colab just now. Looks like we successfully detect that we are running in an incompatible shell, and do not display the widget version. Instead we successfully fall back to the _repr_html_ version without the disconnect button.

The only small caveat with this is that google colab messes up jupyterlab styles, but I think that's a secondary concern here.

[xieus]

@rkooo567 @amogkam @bveeramani could you review this PR? Thanks.

[bveeramani]

I'm not sure if I'm the best person to review this. @ericl could you take a look?

[ericl]

Stamping, as the previously broken tests now seem to be passing in this version.

[ericl]

Stamping, as the previously broken tests now seem to be passing in this version.