ENH: Notebook 3d backend improvements (ipyvtklink)

[GuillaumeFavelier]

I open this issue following #8503 (comment) to keep track of the missing features or anything that could be improved.

Feature status

• Test integration with Google Colab following Using raw.plot in JupyterLab or Colab #8080 (comment)
• HiDPI rendering of 3D and mpl widget (see MRG, ENH: Add a screenshot button to the notebook 3d backend #8708 (comment))
• Screenshot w/time_viewer is off in the vertical direction (https://github.com/mne-tools/mne-python/pull/8708/files/5d34669c5ea051b1c08dfda19cb1ddf0ba727945#diff-a6e2fc2fab82c1377cb29516fc0a7aed813ee4273daad18cf002e7302a331770)
• Update to ipyvtklink (Rename to ipyvtklink Kitware/ipyvtklink#30, MNT: Migrate to ipyvtklink #9341)
• Button/shortcut for autoplay (ENH: Add a playback button to the notebook 3d backend #8741)
• Button/shortcut to take a screenshot (MRG, ENH: Add a screenshot button to the notebook 3d backend #8708)
• Button/shortcut to save a movie (ENH: button to save movie #9190)
• Button/shortcut to display help (ENH: Help button #9123)
• Label time course mode (MNT: notebook conditionals in Brain & _update #9016)

Design/UX

• big square button (asked in MRG: Prototype of notebook viz (ipyvtk) #8503 (comment))

---

Known issues

Resolved

• Does not play well with Firefox (see ENH: Notebook 3d backend improvements (ipyvtklink) #8704 (comment) and ENH: Notebook 3d backend improvements (ipyvtklink) #8704 (comment))

JupyterLab

• matplotlib is not interactive Javascript Error: IPython is not defined
• The mesh is translucent
• A VTK window opens on local instance

Google Colab

• It is not supported yet (asked in MRG: Prototype of notebook viz (ipyvtk) #8503 (comment), standby in ENH: Notebook 3d backend improvements (ipyvtklink) #8704 (comment))

---

A testing notebook is available on gist.github.com.

[GuillaumeFavelier]

About Google Colab, I made some research @agramfort.

Considering that ipyvtk_simple's main viewer class, ViewInteractiveWidget, is based on ipycanvas's Canvas widget, I found that it is a known issue. The short answer is (and I quote Martin Renou):

Custom widgets libraries are not supported by Google colab yet

We could track the progress on this matter on https://github.com/martinRenou/ipycanvas/issues/170 or more directly in googlecolab/colabtools#498.

Anyway, I think those need to be resolved before more can be expected from ipyvtk_simple and finally mne-python.

[larsoner]

For movie and autoplay I think you basically need a timer, e.g. https://stackoverflow.com/questions/57255394/jupyter-and-timer-functions

For a screenshot it seems like it should be fairly easy since the ipyvtk_simple is sending bitmaps, and/or we should be able to get the Renderer directly and get the image from it

For label time course do you need more than https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20List.html#Dropdown ?

For sizing I think it's https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20Styling.html

[GuillaumeFavelier]

Exactly what I was looking for, thank you @larsoner 👍

[GuillaumeFavelier]

The mouse events on Firefox might be fixed by Kitware/ipyvtklink#25. I'll open a PR and give it a try.

[GuillaumeFavelier]

It might not be necessary, I don't have those issues anymore on Firefox 87.0

[GuillaumeFavelier]

For Jupyter Lab integration it might worth trying again once pyvista/pyvista#1116 is fixed.

[GuillaumeFavelier]

Quick update: ipycanvas now works on Google Colab. It's a big milestone for the notebook 3d backend.

https://github.com/martinRenou/ipycanvas/issues/170#issuecomment-1062071346

I'll check ipyvtklink and report back in here.

[agramfort]

this would be a game changer ! Message ID: ***@***.***>

…

[GuillaumeFavelier]

Although now ipycanvas is supported on Google Colab, vtk, even on notebook, still requires a running X server.

My testing workflow is: ipycanvas ✔️ -> ipyvtklink ✔️ -> pyvista (ipyvtklink) ✔️ -> mne-python (notebook)

It's a work in progress but I managed to get it working at low resolution, which is already amazing IMO.
Here is a modified version of ipyvtklink's default example:

output.mp4

---

The important points:

1. setting the custom widget manager

from google.colab import output
output.enable_custom_widget_manager()

2. setting up xvfb

!sudo apt update
!sudo apt-get install libgl1-mesa-dev xvfb
!sudo apt install -yqq libxkbcommon-x11-0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 libxcb-xfixes0 libopengl0

3. start xvfb with the correct DISPLAY variable at the beginning of the cell

!/sbin/start-stop-daemon --start --quiet --background --exec /usr/bin/Xvfb -- :99 -screen 0 800x600x24 -ac +extension GLX +render -noreset

import os
os.environ['DISPLAY'] = ':99'

---

Note:
I tested on Mozilla Firefox 97.0.2.

Known issues:
I encountered a bug where I have to change the size of the viewer explicitly to re-render.

[larsoner]

I think we can close this now that the notebook seems to be working correctly (even nowadays with trame)