Enormous widget state

[vidartf]

Steps to repoduce:

• Open a new notebook with the following code (extracted from voxel_painter example notebook):

import pythreejs as three

sizex = sizey = 500

surf_geo = three.SurfaceGeometry(
    z = [0]*sizex*sizey,
    width = 2*sizex,
    height = 2*sizey,
    width_segments = 2*sizex//stepx,
    height_segments = 2*sizey//stepy,
)

• Save notebook with widget state.

Result:
Notebook has more than 500*500 = 250'000 lines, as the z variable is getting one line per entry. This is sufficient to e.g. cause several diff tools to grind to a halt (git gui, github desktop, but not nbdime-web :D).

This might actually be a problem with how Notebook stores lists, but maybe there is a better way to save this?

Related IPyWidgets issue:
The widget state will be kept even after the outputs are cleared. Particularly for the "clear all outputs" action, you would expect all widget state to be cleared as well (I'm not certain how hard it would be to also handle clearing of individual outputs). Now, the only way to clear the widget state is to restart the kernel and click "save notebook with widgets".

[SylvainCorlay]

Related IPyWidgets issue:
The widget state will be kept even after the outputs are cleared. Particularly for the "clear all outputs" action, you would expect all widget state to be cleared as well (I'm not certain how hard it would be to also handle clearing of individual outputs). Now, the only way to clear the widget state is to restart the kernel and click "save notebook with widgets".

On that regard, this will not be true anymore in Jupyterlab and is an artefact of how we display widgets in the classic notebook, for which widgets are not outputs.

[SylvainCorlay]

Result:
Notebook has more than 500*500 = 250'000 lines, as the z variable is getting one line per entry. This is sufficient to e.g. cause several diff tools to grind to a halt (git gui, github desktop, but not nbdime-web :D).
This might actually be a problem with how Notebook stores lists, but maybe there is a better way to save this?

Yeah, I guess that lists other than the list of cells should be stored on a single line...

One way to do this without using a companion file would be to have certain fields be serialized in base64.

[vidartf]

[This] is an artefact of how we display widgets in the classic notebook, for which widgets are not outputs.

Then maybe classic notebook could get a "clear widget state" button? Or is this too specific to this kind of case?

[SylvainCorlay]

Then maybe classic notebook could get a "clear widget state" button? Or is this too specific to this kind of case?

Totally.

[jasongrout]

By the way, I just now implemented saving widget binary state as base64 arrays in jupyter-widgets/ipywidgets#1302.

[vidartf]

With the binary -> base64, and the change in master to use more arrays instead of lists, this is as good as it can get from the pythreejs side. There might be some further tweaks to make, but those should be opened as separate issues/PRs.