updating lumino packages

[likewei92]

Allowing more updated versions of lumino packages. This will resolve #4354

Please uncomment this block and take a look at this checklist if your PR is making substantial changes to documentation/impacts files in the doc directory. Check all that apply to your PR, and leave the rest unchecked to discuss with your reviewer! Not all boxes must be checked for every PR :)

If your PR modifies code of the plotly package, we have a different checklist
below :-).

Documentation PR

• I've seen the doc/README.md file
• This change runs in the current version of Plotly on PyPI and targets the doc-prod branch OR it targets the master branch
• If this PR modifies the first example in a page or adds a new one, it is a px example if at all possible
• Every new/modified example has a descriptive title and motivating sentence or paragraph
• Every new/modified example is independently runnable
• Every new/modified example is optimized for short line count and focuses on the Plotly/visualization-related aspects of the example rather than the computation required to produce the data being visualized
• Meaningful/relatable datasets are used for all new examples instead of randomly-generated data where possible
• The random seed is set if using randomly-generated data in new/modified examples
• New/modified remote datasets are loaded from https://plotly.github.io/datasets and added to https://github.com/plotly/datasets
• Large computations are avoided in the new/modified examples in favour of loading remote datasets that represent the output of such computations
• Imports are plotly.graph_objects as go / plotly.express as px / plotly.io as pio
• Data frames are always called df
• fig = <something> call is high up in each new/modified example (either px.<something> or make_subplots or go.Figure)
• Liberal use is made of fig.add_* and fig.update_* rather than go.Figure(data=..., layout=...) in every new/modified example
• Specific adders and updaters like fig.add_shape and fig.update_xaxes are used instead of big fig.update_layout calls in every new/modified example
• fig.show() is at the end of each new/modified example
• plotly.plot() and plotly.iplot() are not used in any new/modified example
• Hex codes for colors are not used in any new/modified example in favour of these nice ones

Code PR

• I have read through the contributing notes and understand the structure of the package. In particular, if my PR modifies code of plotly.graph_objects, my modifications concern the codegen files and not generated files.
• I have added tests (if submitting a new feature or correcting a bug) or
  modified existing tests.
• For a new feature, I have added documentation examples in an existing or
  new tutorial notebook (please see the doc checklist as well).
• I have added a CHANGELOG entry if fixing/changing/adding anything substantial.
• For a new feature or a change in behaviour, I have updated the relevant docstrings in the code to describe the feature or behaviour (please see the doc checklist as well).

[athornton]

Could this please get some attention? I too would like Plotly to be happy with current Jupyterlab, and this is a trivial change.

[marthacryan]

@likewei92 @athornton Hi! Sorry for the delay here. We've been working on our jupyter code a lot and we ended up removing this extension because it isn't necessary for jupyterlab >= 3 and notebook >= 7. Are you able to use plotly in the notebook editor?

[athornton]

...sort of?

Get a cup of coffee. This is going to be a long story.

Installing "plotly" (whether conda or pip) installs 5.24.1. It seems to work fine, except that jupyter labextension list claims it's incompatible (which is what this PR does--sure, the Lumino widgets major versions changed, but the APIs that plotly uses didn't).

In the Rubin Science Platform we also use dash. dash now complains that JupyterLab wants a rebuild. It does this with both 5.24 and 6.0.0rc0. In the RSP production environment, users do not have the power to rebuild the lab, and that is done by design. But even in our build pipeline...we generally assemble our JupyterLab environment (which is complex) via a GitHub Actions pipeline, and rebuilding JupyterLab simply takes too much memory, and the rebuild fails (even with --minimize=False). The rebuild fails for me in GHA and in a 16GB EC2 container, but succeeds on a 32GB MacBook, which gives some indication of how much memory is actually required. But paying for larger GHA runners is not something that's likely to happen here.

So what I really need is for Jupyterlab to be happy enough with the state of its world that it will not claim plotly and dash are incompatible and pop up a dialog prompting the user to try a rebuild.

Since that's Jupyterlab's quite-sensible behavior when it detects incompatibilities, the shortest route to satisfying it goes through modifying package.json for those two packages so the Lab knows that the newer lumino versions it has are just fine for plotly and dash.

The fact that Lab rebuilds are very costly is, of course, one of the primary motivators behind doing JupyterLab prebuilt extensions, and part of why Project Jupyter wants you to build extensions that way now.

So, it looks like Plotly 6 will be compatible (in package.json, rather than just de facto) with Jupyterlab 4 (and hopefully 5+, since 5 should be here soon)...but it's still going to require an updated release of dash to not think it needs a Jupyterlab rebuild. I think I'm fine, at the moment, with removing dash from my build for now and waiting until plotly 6 is released, assuming there's a compatible dash that will accompany it. I can live with Jupyterlab putting a red X next to the plotly extension if I list lab extensions, but I can't live with the user complaints that every time they start a lab they're told that dash requires a jupyterlab rebuild and they get an unhelpful error message if they click the build button, and I don't think many if any users are actually using dash within the Rubin Science Platform environment.

I don't know if further releases (with updated compatibility statements) of the 5.x family and of dash are warranted just to get Jupyterlab to accept that these are working extensions, but a good combing through package.json to ensure compatibility with the JupyterLab versions you want to support is probably something that needs to happen with the next big release of packages for the Python ecosystem.

[marthacryan]

@athornton Thank you for explaining! Could you elaborate on how you're using dash? And what it means for dash to want a jupyterlab rebuild?

You can also go ahead and jupyter labextension uninstall jupyterlab-plotly if you're not in a fresh environment - installation should be as simple as pip install plotly for JupyterLab >=3 (even before my changes). If you run into issues with that, I've actually also seen issues with the jupyter labextension uninstall command not actually removing it so I've had to go and rm manually through the path listed next to jupyterlab-plotly in jupyter labextension list. Those appear to be stored globally so installing jupyter in a new virtual environment won't always remove that extension.

[athornton]

Here's the prompt-for-rebuild:

This dialog pops up on starting a new JupyterLab container and must be dismissed with either "Build" or "Cancel" to proceed. "Build" then generates a 500 error, because (I think) the user cannot write to the filesystems where JupyterLab lives.

Below here there's what's installed. Notably, jupyterlab-plotly is not (separately) one of those things. It is installed via conda installation of the plotly package. The list of what is installed is found at https://github.com/conda-forge/rubinenv-feedstock/blob/main/recipe/meta.yaml and it's the union of the rubin-env and rubin-env-rsp environments (plus a smattering of pip-installed items, cf. https://github.com/lsst-sqre/sciplat-lab/blob/d59fa1ca87046f198a63745048afc75fed391cc8/stage3-py.sh#L48-L53

[adam@adam-nb ~]$ conda env export | grep plotly
  - plotly=5.24.1=pyhd8ed1ab_0
[adam@adam-nb ~]$ conda env export | grep dash
  - dash=2.18.2=pyhd8ed1ab_0
[adam@adam-nb ~]$ jupyter labextension list --verbose
JupyterLab v4.2.6
/opt/lsst/software/stack/conda/envs/lsst-scipipe-9.0.0/share/jupyter/labextensions
        bqplot v0.5.44 enabled  X (python, bqplot)
        ipyevents v2.0.1 enabled OK
        ipyvolume v0.6.3 enabled OK
        jupyter-matplotlib v0.11.4 enabled OK
        jupyter-threejs v2.4.1 enabled OK (python, pythreejs)
        jupyter-vue v1.11.2 enabled OK
        jupyter-vuetify v1.10.0 enabled OK
        jupyter-webrtc v0.6.0 enabled OK
        jupyter_firefly_extensions v4.2.0 enabled OK (python, jupyter_firefly_extensions)
        jupyterlab-datawidgets v7.1.2 enabled OK
        jupyterlab-execute-time v3.2.0 enabled OK (python, jupyterlab_execute_time)
        jupyterlab-plotly v5.24.1 enabled  X
        jupyterlab_iframe v0.5.0 enabled OK
        jupyterlab_pygments v0.3.0 enabled OK (python, jupyterlab_pygments)
        nbdime-jupyterlab v3.0.2 enabled OK
        rsp-jupyter-extensions v0.13.0 enabled OK (python, rsp_jupyter_extensions)
        @jupyter-widgets/jupyterlab-manager v5.0.13 enabled OK (python, jupyterlab_widgets)
        @jupyter-widgets/jupyterlab-sidecar v0.7.0 enabled OK (python, sidecar)
        @bokeh/jupyter_bokeh v4.0.5 enabled OK (python, jupyter_bokeh)
        @jupyter-notebook/lab-extension v7.2.2 enabled OK
        @jupyter-server/resource-usage v1.1.0 enabled OK (python, jupyter-resource-usage)
        @jupyterhub/jupyter-server-proxy v4.4.0 enabled OK
        @lckr/jupyterlab_variableinspector v3.2.4 enabled OK (python, lckr_jupyterlab_variableinspector)
        @pyviz/jupyterlab_pyviz v3.0.3 enabled OK


   The following extensions may be outdated or specify dependencies that are incompatible with the current version of jupyterlab:
        bqplot
        jupyterlab-plotly
        
   If you are a user, check if an update is available for these packages.
   See below for the details.


"bqplot@0.5.44" is not compatible with the current JupyterLab
Conflicting Dependencies:
JupyterLab        Extension              Package
>=2.0.0 <3.0.0    >=1.0.0 <2.0.0         @lumino/messaging
>=2.3.1-alpha.0 <3.0.0>=1.0.0 <2.0.0         @lumino/widgets



"jupyterlab-plotly@5.24.1" is not compatible with the current JupyterLab
Conflicting Dependencies:
JupyterLab        Extension              Package
>=2.0.0 <3.0.0    >=1.2.3 <2.0.0         @lumino/messaging
>=2.3.1-alpha.0 <3.0.0>=1.8.1 <2.0.0         @lumino/widgets


Other labextensions (built into JupyterLab)
   app dir: /opt/lsst/software/stack/conda/envs/lsst-scipipe-9.0.0/share/jupyter/lab
        @plotly/dash-jupyterlab v0.4.3 enabled OK


Disabled extensions:
    @jupyterlab/apputils-extension:announcements
    @jupyterlab/filebrowser-extension:share-file
    @jupyterlab/hub-extension:menu

Build recommended, please run `jupyter lab build`:
    @plotly/dash-jupyterlab needs to be included in build

[athornton]

As to how we're using dash: I don't think we are. I added plotly to our packages a couple years ago (I think) because I wanted 3D scatterplots with easy zooming to work, and I am pretty sure I added dash because the plotly webpage said "you should also install dash, because it's a really easy way to do plots" or something like that. Having standalone apps in the JupyterLab Notebook doesn't seem all that useful to me; I don't know whether any of our users will disagree with me on that.

[marthacryan]

@athornton Ah ok didn't realize you were using conda to install plotly. That may be bundling the jupyterlab extension, although I'm not seeing that in the conda forge recipe for plotly. I'll have to keep looking into this, but thank you for the detailed responses! BTW it does sound like dash won't be a great use-case for your users, so you should feel free to remove it if that unblocks you.

[marthacryan]

I am going to go ahead and close this PR though because the file it changes no longer exists. We're not going to continue releasing an npm package because that is now deprecated in jupyterlab.