ENH: Add a playback button to the notebook 3d backend

[GuillaumeFavelier]

1. This PR adds a set of ipywidgets buttons for playback. Here how it looks like:

output.mp4

Known issue

Does not depend on playback_speed but on the widget's interval value.

2. I also tried backgroundjobsfrom ENH: Notebook 3d backend improvements (ipyvtklink) #8704 (comment) without success so far. I'll give it a try again later this afternoon.

It's an item of #8704.

[larsoner]

Does not depend on playback_speed but on the widget's interval value.

It seems like you should only set "interval" to get the right refresh rate (e.g., 1/60 or 1/30), and the callback should be responsible for picking the correct time points. It seems like this should be the same for the Qt and IPywidgets backends actually -- and then as long as the callback tracks/knows 1) when the last frame was rendered (or if it's the first frame), 2) what the desired time dilation is, and 3) what the current time is (time.time()) and from this can calculate what time index/value it needs to change to, no? Such code would also adapt properly if we tried to get a framerate of 60 Hz but it renders at something slower or variable, which is nice.

[GuillaumeFavelier]

Oh you're right! All this code is in _play() and _advance():

mne-python/mne/viz/_brain/_brain.py

Lines 778 to 802 in fb22fa6

	@safe_event
	def _play(self):
	if self.playback:
	try:
	self._advance()
	except Exception:
	self.toggle_playback(value=False)
	raise
	def _advance(self):
	this_time = time.time()
	delta = this_time - self._last_tick
	self._last_tick = time.time()
	time_data = self._data['time']
	times = np.arange(self._n_times)
	time_shift = delta * self.playback_speed
	max_time = np.max(time_data)
	time_point = min(self._current_time + time_shift, max_time)
	# always use linear here -- this does not determine the data
	# interpolation mode, it just finds where we are (in time) in
	# terms of the time indices
	idx = np.interp(time_point, time_data, times)
	self.callbacks["time"](idx, update_widget=True)
	if time_point == max_time:
	self.toggle_playback(value=False)

[GuillaumeFavelier]

The Play widget is limited to a range of int values. I increased the number of frames depending on the playback_speed but it won't be as smooth as the qt version since it does not reuse _advance() to synchronize the frame rate. Instead, it's a fixed to self.refresh_rate_ms.

[GuillaumeFavelier]

It's ready to go on my end 🚀

[larsoner]

Works great! FWIW on my local Ubuntu-based 3.9 after pip installing the Jupyter widgets I needed to enable them:

jupyter-nbextension enable ipycanvas --py --user
jupyter nbextension enable ipyevents --py --user

Onto the minor issues:

1. when clicking a time point there is a quick hop back to the old one before it continues on from the old one (and this does not happen with the standard Python plotting):

   

   Not a dealbreaker but if it's easy to fix we might as well.

2. Changing the playback speed slider changes the time index even when no playback is ocurring

   

3. Playback speed does not seem correct. The data is 0.25 sec so with speed == 0.1 playback should happen in 2.5 sec. Top is the HTML which goes maybe half as fast as it should, and bottom is the Qt/Python version which goes at the right speed:

   

   

   I suspect this has to do with relying somehow on the refresh rate (which probably does not actually get met by this backend) as opposed to making use of time.time()-based index selection.

[drammock]

One nitpick from the peanut gallery: "toggle visibility" I think would be more clear if it said "toggle controls". Otherwise this is looking really nice.

[GuillaumeFavelier]

This PR requires quite the update. I think the challenge now changed to finding a GUI-agnostic way (Qt/Ipywidget) to playback.

[larsoner]

I think the challenge now changed to finding a GUI-agnostic way (Qt/Ipywidget) to playback.

Instead of making it agnostic I'd use our GUI abstraction layer to deal with the differences, and use a Qt timer and ipywidget object (Play?) as necessary in those backends

[GuillaumeFavelier]

The test fails because Play is stored as an action but is technically not a Button.

[GuillaumeFavelier]

Since I return it as a _Widget anyway, I think it's reasonable to remove it from the list of actions.

[GuillaumeFavelier]

And this is how it looks when integrated in the toolbar:

[GuillaumeFavelier]

I chose the simplest approach which is to connect the playback buttons to the time slider bidirectionally.

The known issues:

1. When the slider reaches the right end, the layout does not fit the dock (but I don't think it's related to this PR)

From:

To:

2. The timeout is fixed (interval=500) so it's not realtime like the Qt solution
3. The timesteps are fixed too (step=1). For this one, I tried implementing something similar to QFloatSlider and use a similar precision of 10000 but I could not manage to connect the slider in this case (or any callback actually).

[GuillaumeFavelier]

In my experiments, the bidirectional connection was not reliable. For example, using the playback to start, changing the slider and coming back to the playback, it will start from the old position:

I can mitigate by duplicating jsdlink():

        jsdlink((play, 'value'), (slider, 'value'))
        jsdlink((slider, 'value'), (play, 'value'))

But it brings lag when the slider is dragged (it snaps the value of the slider).

output.mp4

[GuillaumeFavelier]

TL;DR: The realtime approach may not be possible considering the current limitations so I think the approach based on jsdlink is good enough. What do you think @larsoner, @agramfort ?

I took the time to fix 3) and the solution explained in jupyter-widgets/ipywidgets#1775 really helped. I shared the prototype in be61d64 for reference but I ended up reverting because I am not satisfied with the overall experience. The number of calls to set_time_point increases but the ViewInteractiveWidget just cannot keep up with the updates anyway causing a visible 'lag' (and I tested locally so it might be unusable with a remote connection).

[larsoner]

Some ability to play back is better than none, even if it's effectively a slideshow and the speed slider doesn't really reflect an accurate time dilation. We can always work on improving it later I guess. So if you're happy with how it works now @GuillaumeFavelier let me know and I'll look, test, and merge

[GuillaumeFavelier]

You can have a look @larsoner, I'm ready for reviews.

[larsoner]

Otherwise LGTM

[larsoner]

Should this be a try/except as well?

[GuillaumeFavelier]

Since 0.30.0 pyvista does not use ipyvtk_simple anymore

[larsoner]

Yes but I'm thinking that it should also work for people with 0.29 if possible

[GuillaumeFavelier]

I plan to bump/update those requirements in #9341

[larsoner]

If it's only a line or two to keep it working, it would be nice not to force people to upgrade

[larsoner]

But if they have ipyvtk_simple and pyvista < 0.30 it should work, right? For example:

kwargs = dict()
if LooseVersion(pyvista.__version__) < LooseVersion('0.30'):
    kwargs['jupyter_backend' ] = 'ipyvtk_simple'
else:
    kwargs['jupyter_backend'] = 'ipyvtklink'
viewer = self.plotter.show(return_viewer=True, **kwargs)

[GuillaumeFavelier]

it would be nice not to force people to upgrade

I understand

[GuillaumeFavelier]

I'll rework this part to use LooseVersion instead.

[GuillaumeFavelier]

The API changed too, the following won't work:

self.plotter.show(jupyter_backend="ipyvtk_simple", return_viewer=True)

[larsoner]

Like this block works for >= 0.30 and < 0.30

[GuillaumeFavelier]

But this one does not depend on another package.

If the user does not have ipyvtklink but has pyvista 0.30.0, it just won't work.

[GuillaumeFavelier]

That's why I think bumping is the least painful solution

[larsoner]

Thanks @GuillaumeFavelier !