ipycanvas not working (binary buffers issue?)

[martinRenou]

Thanks for your work bringing widgets to colab! It has been requested many times by users.

I gave ipycanvas a try, and it seems to render the canvas properly. You can try with

!pip install -q ipycanvas

then

from IPython.display import Javascript
from ipycanvas import Canvas

display(Javascript('''
  google.colab.widgets.installCustomManager('https://ssl.gstatic.com/colaboratory-static/widgets/colab-cdn-widget-manager/6a14374f468a145a/manager.min.js');
'''))

canvas = Canvas(height=900)
canvas.fill_rect(0, 0, 50, 50)
canvas

This results in the Canvas widget properly being created in the output:

Issue

But it's when I try to draw something on the Canvas that things go wrong:

canvas.fill_rect(0, 0, 50, 50)

This results in the following JavaScript error:

Error has occurred while trying to update output. TypeError: Cannot read properties of undefined (reading 'length')
    at bf (output_binary.js?vrz=colab-20210901-060136-RC02_394281090:142)
    at qa.eval [as h] (output_binary.js?vrz=colab-20210901-060136-RC02_394281090:143)
    at sa (output_binary.js?vrz=colab-20210901-060136-RC02_394281090:15)
    at ua.next (output_binary.js?vrz=colab-20210901-060136-RC02_394281090:15)
    at f (output_binary.js?vrz=colab-20210901-060136-RC02_394281090:56)

If I can make a wild guess, I wonder if it could come from around here: https://github.com/googlecolab/colab-cdn-widget-manager/blob/main/src/manager.ts#L237

Context

To give you more context, ipycanvas makes extensive use of comm message for sending "draw calls". So when I do canvas.fill_rect(0, 0, 50, 50) it is sending a comm message (which is mostly a binary buffer) to the front-end. That's why I suspect it's failing in the Comm's on_msg method.

Is there a chance we can access non-minified source of the manager from the cdn? So that I can debug more easily. I tried https://ssl.gstatic.com/colaboratory-static/widgets/colab-cdn-widget-manager/6a14374f468a145a/manager.js but it does not seem to exist.

[blois]

Hi Martin- thanks for the bug report, I'll take a look at it shortly.

The non-minified source can be loaded from https://ssl.gstatic.com/colaboratory-static/widgets/colab-cdn-widget-manager/6a14374f468a145a/manager.dev.js. There are also steps for making changes locally at https://github.com/googlecolab/colab-cdn-widget-manager#development

[martinRenou]

Thanks! That seems so almost too easy to get a dev install 😄 I will give it a try over the week-end

[blois]

This one looks like it'll need a fix in Colab. The issue is that there's a comm message with the data:

{
    "content": {
        "dtype": "uint8",
        "shape": [
            18
        ]
    },
    "method": "custom"
}

Which also includes a single buffer of data. We're assuming elsewhere that every Widget comm message with buffers will also have buffer_paths set. Fixing that assumption appears to address the issue.

[blois]

The fix is now live.

[martinRenou]

Cool! Thanks a lot 😊

[martinRenou]

I gave it a try today. It seems to now fail differently. The Canvas cannot be displayed anymore and the following JS error shows up:

Error has occurred while trying to update output. Error: Widgets module ipycanvas is not supported

[blois]

One change is that we're moving to a different way to opt-into the custom manager- with the new mechanism you can do:

from google.colab import output
output.enable_custom_widget_manager()

Once in the notebook and it will apply to all widgets in later cells. I didn't think this was live quite yet but looks like you may have gotten it.

Next is that it looks like this may have caused a timing change which I'm curious about- the canvas now needs to be displayed before the fill style is set in order for it to take effect:

from ipycanvas import Canvas
canvas = Canvas(height=900)
display(canvas)
canvas.fill_style = 'red'
canvas.fill_rect(0, 0, 50, 50)

Does ipycanvas persist the drawing in the notebook? If so, do you have a pointer as to how? At a glance it looked like the image buffer is sent outside of the widget state and right now we're only persisting the sate.

[martinRenou]

Thank you for you response.

Does ipycanvas persist the drawing in the notebook?

It actually doesn't. The canvas widget is not a really good citizen of the widgets ecosystem as it's kind of stateless and relies too much on custom comms.

I've seen similar issue in the Voila front-end, as the Notebook is executed entirely before displaying the widgets. So the draw commands are sent to the void (no widget in the front-end to receive them) and the canvas is rendered blank afterwards.

This is definitely an issue on the ipycanvas side. I will try to fix it.