Kernel is the source of truth: support multiple clients and fixes out of sync bug

[maartenbreddels]

As described in #3111 it can happen that a frontend can get out of sync
due to not echoing back messages from the frontend.

Widgets can opt out of this behaviour if it is too costly and/or does
not make sense, e.g. the data from the file widget.

Fixes #3111

[jasongrout]

In our dev meeting today, we were discussing this, and whether it should be a target for 8.0, or whether it should be tackled in the (hopefully upcoming) explorations in moving the syncing to a real-time collaboration backend.

[jasongrout]

In the dev meeting today, we propose moving this to 9.0 with the (hopefully upcoming) move to an RTC framework for syncing data. Maarten, what do you think?

[jasongrout]

Moving to Future (e.g., 9.0) since there was no follow-up. Maarten, if you'd like to discuss this more, please comment. We can always move it back if that's what everyone decides.

[maartenbreddels]

I think that makes sense!

[SylvainCorlay]

This poses an issue in the following situation:

• A client moves continuously a slider from left to right
• It gets the echos back from past positions of the handles, making the handle jump to lesser values of the slider.

We have seen this bug make slider handles jitter.

[SylvainCorlay]

As per my last comment.

[maartenbreddels]

We can see this from the parent header, if we get an update from the kernel, where the parent header msg_id equals the last msg_id of the update we send to the kernel, (and our value has changed, but that is something backbone should figure out), we should update our back model model. If the msg_id is not equal to our last update message (on a per property/trait basis), it's an old message, and we can ignore it.

[maartenbreddels]

on the client that is changing the slider, there is not jitter at all, actually I also fixed an old but I think that caused more jitter that needed since we set the state of a property, even though it was already planned for sending to the kernel (attr present in _msg_buffer)

[maartenbreddels]

with the env var JUPYTER_WIDGETS_ECHO=0 the echo can be turned off, of simply from the Python module.

[jasongrout]

This protocol change should be documented in https://github.com/jupyter-widgets/ipywidgets/blob/master/packages/schema/messages.md#widget-messaging-protocol-version-2. Since it is backwards compatible, perhaps we bump the protocol version to 2.1?

[maartenbreddels]

Note from @jasongrout:
If we get an update from the frontend, but that trait is marked 'no_echo', we do send it back to the frontend in the 'echo' list, but it does not get the updated value. The frontend code currently handles this, and it's informative, but we should maybe explicitly state this.

[SylvainCorlay]

@maartenbreddels @jasongrout @ibdafna was there a final decision to go ahead and merge after I left the call?

[vidartf]

I haven't tested running of the code, but AFAICT this code now meets all the targets we said we needed for including this in the RC

[vidartf]

On things that were not a prerequisite for inclusion in the RC, but I would consider a prerequisite for release the final version with this on as the default: I think that we should make sure we have tests that properly cover the cases where the previous property lock prevented flickering of values to occur. I'm uncertain if the new code properly handles all of the corner cases there, and more regression tests would go a long way in acting as a proof of that.

[ibdafna]

@SylvainCorlay I believe we agreed to merge, release a new beta tomorrow, and for @maartenbreddels to add documentation about the changes this PR introduces, before releasing a new RC in the new year.

[SylvainCorlay]

On things that were not a prerequisite for inclusion in the RC, but I would consider a prerequisite for release the final version with this on as the default: I think that we should make sure we have tests that properly cover the cases where the previous property lock prevented flickering of values to occur. I'm uncertain if the new code properly handles all of the corner cases there, and more regression tests would go a long way in acting as a proof of that

Yes, we spent an hour this morning trying to break it, both by inspecting the code and trying different scenarios.

[SylvainCorlay]

Merged!

[jasongrout]

I thought we had decided to document this in the widget protocol (i.e., #3195 (review)), then merge. I do think it is important to document this before releasing a release with it, since it is big enough change (especially with default on).

[jasongrout]

I've opened #3330 to track the documentation and protocol version changes.

[ibdafna]

@jasongrout apologies for the mixup. @maartenbreddels will wait for a PR with documentation changes before releasing

[jasongrout]

No problem. We could also just cut the beta from just before the merge commit (i.e., branch off, do the beta release from the branch and tag it there, then merge back into master).

[ibdafna]

@jasongrout will do that!