From f400a4132017f9008d413d3c23b8dd38c6406be4 Mon Sep 17 00:00:00 2001 From: Jean-Philippe Gravel Date: Wed, 19 Jul 2023 14:30:16 -0400 Subject: [PATCH] Add a layer and filter interface in the 2D canvas This adds new beginLayer and endLayer functions to open and close layers in the canvas. While layers are active, draw calls operate on a separate texture that gets composited to the parent output bitmap when the layer is closed. An optional filter can be specified in beginLayer, allowing effects to be applied to the layer's texture when it's composited its parent. Tests: https://github.com/web-platform-tests/wpt/tree/master/html/canvas/element/layers https://github.com/web-platform-tests/wpt/tree/master/html/canvas/offscreen/layers Fixes #8476 --- source | 628 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 613 insertions(+), 15 deletions(-) diff --git a/source b/source index d641cce9a8b..53bae43b96a 100644 --- a/source +++ b/source @@ -4332,6 +4332,8 @@ a.setAttribute('href', 'https://example.com/'); // change the content attribute
  • SVG title element
  • SVG use element
  • SVG text-rendering property
    • +
  • core attribute
    • +
  • presentation attribute
    • @@ -4344,6 +4346,9 @@ a.setAttribute('href', 'https://example.com/'); // change the content attribute @@ -63246,6 +63251,9 @@ callback BlobCallback = undefined (Blob? blob); when invoked, must run these steps:

      +

    1. If layer-count is not zero, then throw an + "InvalidStateError" DOMException.

      2. +

    2. If this canvas element's bitmap's origin-clean flag is set to false, then throw a "SecurityError" DOMException.

      @@ -63274,6 +63282,9 @@ callback BlobCallback = undefined (Blob? blob); quality) method, when invoked, must run these steps:

        +

      1. If layer-count is not zero, then throw an + "InvalidStateError" DOMException.

        2. +

      2. If this canvas element's bitmap's origin-clean flag is set to false, then throw a "SecurityError" DOMException.

        3. @@ -63373,6 +63384,7 @@ interface CanvasRenderingContext2D { CanvasRenderingContext2DSettings getContextAttributes(); }; CanvasRenderingContext2D includes CanvasState; +CanvasRenderingContext2D includes CanvasLayers; CanvasRenderingContext2D includes CanvasTransform; CanvasRenderingContext2D includes CanvasCompositing; CanvasRenderingContext2D includes CanvasImageSmoothing; @@ -63397,6 +63409,19 @@ interface mixin CanvasState { boolean isContextLost(); // return whether context is lost }; +typedef record<DOMString, any> CanvasFilterPrimitive; +typedef (CanvasFilterPrimitive or sequence<CanvasFilterPrimitive>) CanvasFilterInput; + +dictionary BeginLayerOptions { + CanvasFilterInput? filter = null; +}; + +interface mixin CanvasLayers { + // layers + undefined beginLayer(optional BeginLayerOptions options = {}); + undefined endLayer(); +}; + interface mixin CanvasTransform { // transformations (default transform is the identity matrix) undefined scale(unrestricted double x, unrestricted double y); @@ -63685,6 +63710,16 @@ interface Path2D {

        A CanvasRenderingContext2D object has an output bitmap that is initialized when the object is created.

        + The CanvasRenderingContext2D output bitmap can be replaced and restored + by calls to beginLayer and endLayer, with the drawing state stack + keeping track of all active nested output bitmaps. The + CanvasRenderingContext2D object has a layer-count integer that is initially set to zero, + keeping track of the number of opened nested layers. To access the content of the context's + output bitmap, the steps for reading the context output bitmap must be + used. +

        The output bitmap has an origin-clean flag, which can be set to true or false. Initially, when one of these bitmaps is created, its cases, this will be more memory efficient.

        The bitmap of a canvas element is the one bitmap that's pretty much always going - to be needed in practice. The output bitmap of a rendering context, when it has one, - is always just an alias to a canvas element's bitmap.

        + to be needed in practice. The top level output bitmap of a rendering context, when it + has one, is always just an alias to a canvas element's bitmap. When layers are + opened, implementations must behave as if draw calls operate on a separate output + bitmap that gets composited to the parent output bitmap when the layer is + closed. If the canvas element's bitmap needs to be presented while layers are opened, + layers are automatically closed, so that their content gets drawn to the context's output + bitmap. The drawing state stack is then restored, reopening all pending + layers before the script execution could resume.

        Additional bitmaps are sometimes needed, e.g. to enable fast drawing when the canvas is being painted at a different size than its natural size, @@ -63978,8 +64019,9 @@ context.fillRect(100,0,50,50); // only this square remains

        The canvas state
        -

        Objects that implement the CanvasState interface maintain a stack of drawing - states. Drawing states consist of:

        +

        Objects that implement the CanvasState interface maintain a stack of drawing states. + Drawing states consist of:

        • The current transformation matrix.
          • @@ -64012,10 +64054,12 @@ context.fillRect(100,0,50,50); // only this square remains data-x="dom-context-2d-imageSmoothingEnabled">imageSmoothingEnabled, imageSmoothingQuality.
        • The current dash list.
          • +
        • An optional canvas layer state.
        -

        The rendering context's bitmaps are not part of the drawing state, as they - depend on whether and how the rendering context is bound to a canvas element.

        +

        The rendering context's bitmaps are not part of the drawing state + (with the exception of layer's parentOutputBitmap), as they depend on whether and how + the rendering context is bound to a canvas element.

        Objects that implement the CanvasState mixin have a context lost boolean, that is initialized to false @@ -64030,8 +64074,8 @@ context.fillRect(100,0,50,50); // only this square remains

        Pops the top state on the stack, restoring the context to that state.

        context.reset()
        -

        Resets the rendering context, which includes the backing buffer, the drawing state stack, - path, and styles.

        +

        Resets the rendering context, which includes the backing buffer, the drawing state + stack, path, and styles.

        context.isContextLost()

        Returns true if the rendering context was lost. Context loss can occur due to driver @@ -64042,11 +64086,26 @@ context.fillRect(100,0,50,50); // only this square remains

        The save() method - steps are to push a copy of the current drawing state onto the drawing state stack.

        + steps are to push a copy of the current drawing state onto the + drawing state stack.

        The restore() - method steps are to pop the top entry in the drawing state stack, and reset the drawing state it - describes. If there is no saved state, then the method must do nothing.

        + method steps are:

        + +
          +

        1. If the drawing state stack is empty, + return.

          2. + +

        2. If the canvas layer state at the top of the drawing state stack + is not null, throw an "InvalidStateError" + DOMException.

          3. + +

        3. Let previousDrawingStates be the result of popping the drawing state stack.

          4. + +

        4. Set all current drawing states to the values they have + in previousDrawingStates.

          5. +

        The reset() method steps are to reset the rendering context to its default state.

        @@ -64058,7 +64117,10 @@ context.fillRect(100,0,50,50); // only this square remains

      3. Empty the list of subpaths in context's current default path.

        4. -

      4. Clear the context's drawing state stack.

        5. +

      5. Clear the context's drawing state stack.

        6. + +

      6. Set the context's layer-count to + zero.

      7. Reset everything that drawing state consists of to their initial values.

        8. @@ -64070,6 +64132,521 @@ context.fillRect(100,0,50,50); // only this square remains
        +
        Layers
        + +

        Objects that implement the CanvasLayers mixin have methods (defined in this + section) for managing output bitmap layers.

        + +

        Layers are opened and closed using beginLayer + and endLayer. When a layer is opened, the context + output bitmap is aliased to that layer's output bitmap, such that all + draw calls performed while the layer is active will effectively render onto the layer's + output bitmap. When endLayer is called, + the layer's output bitmap gets rendered to the parent's output bitmap + using the filter specified in beginLayer.

        + +
        + +

        The drawing state stack keeps track of opened layers by holding canvas layer + state structs containing the following items:

        + +
          +
        • parentOutputBitmap, the layer's parent + output bitmap.
          • + +
        • xmlFilterList, the layer's XML filter list.
          • +
        + +
        + +

        The layer rendering states are the subset of the drawing states that are applied to a layer's + output bitmap when it's drawn to its parent output bitmap. The + layer rendering states are:

        +
          +
        • The current global alpha and + compositing and blending + operator.
          • + +
        • The current values of the following attributes: + shadowOffsetX, + shadowOffsetY, + shadowBlur, + shadowColor
          • +
        + +
        +

        Because the layer rendering states are applied on the layer's output, + they cannot also be applied to the layer's content, or else, they would be applied twice. These + states are therefore set to their default value when opening a layer.

        + +

        The transformation matrix and imageSmoothingEnabled are not part of + the layer rendering states because if the layer's output was to be transformed or + smoothed, it would require resampling of the layer's output bitmap, lowering picture + quality for every layer nesting levels. It's better to transform and smooth the innermost layer + content and draw pixels directly to their final coordinate.

        + +

        Filters specified via context.filter are not part of the layer rendering + states and are therefore not applied on the layer's output bitmap. The preferred way to + specify filters is to use beginLayer. Using + context.filter is inefficient because it + requires each individual draw call to be wrapped in a layer. It's better to make this cost + explicit by using beginLayer.

        + +

        The clipping region is the only state that gets applied to both the layer content + and the layer's output bitmap. As such, it's treated as a special case and is not + included in the layer rendering states.

        +
        + +

        CanvasFilterInput is used for describing SVG filters using JavaScript. A + CanvasFilterInput object can be converted to an equivalent XML filter list. An XML filter list is a list of + XML element data for filters that fully describes an SVG filter network.

        + +
        +
        context.beginLayer([ { [ filter: filterInput ] } + ])
        +

        Pushes the current state onto the stack and starts a new layer. While a layer is active, + all draw calls are performed on a separate surface, which will later be drawn as a whole to the + canvas (or parent layer) when the layer is closed.

        + +

        If the filter member is + specified, the filter effect is applied to the layer's resulting texture as it's drawn to the + parent output bitmap.

        + +

        Filters are specified as a single CanvasFilterPrimitive, or a list of + CanvasFilterPrimitive to chain filter effects. CanvasFilterPrimitive + objects have a name property, whose value is one of the supported + filter names, and additional properties corresponding to the settings of the filter. These + latter properties are the same as the XML attribute names when using the corresponding SVG + filter.

        + +
        context.endLayer()
        +

        Pops the top state on the stack, restores the context to that state and draws the layer's + resulting texture to the parent surface (the canvas or the parent layer). The layer's filter, if specified, gets applied, + along with the global rendering states as they were when beginLayer was called.

        +
        + +
        + +

        The beginLayer() method steps are:

        +
          +

        1. Let settings be the result of converting options to the dictionary type + BeginLayerOptions. (This can throw an exception.)

          2. + +

        2. Let currentOutputBitmap be the context's current + output bitmap

          3. + +

        3. Let layerOutputBitmap be a newly created output bitmap, + initialized with the same size and color space + as `currentOutputBitmap` and with an origin-clean flag set to true.

          4. + +

        4. Let xmlFilter be the result of running the steps for building an XML filter list + given settings["filter"].

          5. + +

        5. Let layerState be a new canvas layer state object, initialized + with currentOutputBitmap and xmlFilter.

          6. + +

        6. Run the steps of the save method. + +

        7. Reset the context's layer rendering states to their default value.

          8. + +

        8. Set the canvas layer state at the top of the drawing state stack + to layerState.

          9. + +

        9. Set the context's current output bitmap to + layerOutputBitmap

          10. + +

        10. Increment layer-count by one.

          11. +
        + +

        The endLayer() method steps are:

        +
          +

        1. If the drawing state stack is empty, then + throw an "InvalidStateError" DOMException.

          2. + +

        2. Let layerState be the canvas layer state at the top of the + drawing state stack.

          3. + +

        3. If layerState is null, throw an "InvalidStateError" + DOMException.

          4. + +

        4. Let layerOutputBitmap be the context's current + output bitmap

          5. + +

        5. Let parentOutputBitmap be + layerState["parentOutputBitmap"]

          6. + +

        6. If layerOutputBitmap is marked as not origin-clean, then set the origin-clean flag of parentOutputBitmap + to false.

          7. + +

        7. Set the context's current output bitmap to + parentOutputBitmap

          8. + +

        8. Let filteredLayerOutputBitmap be the result of applying + layerState["xmlFilterList"] to layerOutputBitmap using the + steps to apply an XML filter list.

          9. + +

        9. Let parentDrawingStates be the result of popping the drawing state stack.

          10. + +

        10. Reset all context drawing states to their default + values, then set the current layer rendering states and the clipping + region to the values stored in parentDrawingState.

          11. + +

        11. Draw filteredLayerOutputBitmap onto the context's current + output bitmap using the steps outlined in the drawing model.

          12. + +

        12. Set all current drawing states to the values they have + in parentDrawingStates.

          13. + +

        13. Decrement layer-count by one.

          14. +
        + +
        + +

        For legacy reasons, calling restore() + when the drawing state stack is empty is a no-op. The addition of the layer API + however introduced several new potential pitfalls. For instance, scripts like context.save(); context.endLayer(); or context.beginLayer(); + context.restore(); are problematic. They are symptomatic of web page bugs and user agents + cannot silently fix these bugs on behalf of the page (e.g. did the page intend to call + endLayer() instead of restore(), or is there a missing save()?) For that reason, invalid API sequences involving + layers throw exceptions to make the issue clear to web developers.

        + +
        + +

        The steps for building an XML + filter list require the following definitions:

        + +

        The supported filter names are + "colorMatrix", + "componentTransfer", + "convolveMatrix", + "dropShadow" and + "gaussianBlur". + +

        The XML element data for filters is a struct, with the following items:

        + +
          +

        • A string name

          • + +

        • An ordered map of strings to strings attributes

          • + +

        • A list of XML element data for filters children

          • +
        + +

        To get the IDL type for a canvas filter attribute attrName:

        + +
          +

        1. Let type be the type listed for attrName in Filter + Effects.

          2. + +

        2. If type is "false | true", then return boolean.

          3. + +

        3. If type is "list of <number>s", then return sequence<double>.

          4. + +

        4. If type is "<number>", then return double.

          5. + +

        5. If type is "<integer>", then return long long.

          6. + +

        6. If type is "<number-optional-number>", then + return (double or sequence<double>).

          7. + +

        7. Return DOMString.

          8. +
        + +

        To generate an XML value from a key, + value pair: + +

          +

        1. Let type be the result of getting the IDL type for a canvas filter + attribute for key.

          2. + +

        2. Let idlValue be the result of converting value to type.

          3. + +

        3. If type is (double or sequence<double>), + value is a sequence<double> and value doesn't have two elements, + throw a TypeError exception.

          4. + +

        4. Let xmlValue be the result of converting idlValue to an ECMAScript value, and + then converting that result to a DOMString.

          5. + +

        5. Return xmlValue.

          6. +
        + +

        The steps for building an + XML filter list given + filters are:

        + +
          +

        1. Let xmlFilters be an empty list.

          2. + +

        2. If filters is null, then set filters to « ».

          3. + +

        3. If filters is a CanvasFilterPrimitive, then set filters + to « filters ».

          4. + +
        4. +

          For each filterDict of filters:

          + +
            +

          1. If filterDict["name"] does not exist, then throw a TypeError exception.

            2. + +

          2. Let filterName be the value of filterDict["name"].

            3. + +

          3. If filterName is not one of supported filter names, then + continue.

            4. + +

          4. Let xmlName be the concatenation of "fe", the first + code unit of filterName converted to ASCII uppercase, + and all code units of filterName after the first + one.

            5. + +

          5. Let xmlFilter be a new XML element data for filters whose name is xmlName, whose attributes is an empty ordered map, and whose children is an empty list.

            6. + +

          6. Append xmlFilter to + xmlFilters.

            7. + +
          7. +

            For each key → value of + filterDict:

            + +
              +
            1. +

              If any of the following are true:

              + +
                +

              • key is not the local name of an attribute listed for the filter + primitive given by xmlName

                • + +

              • key is the local name of a core attribute

                • + +

              • key is the local name of a presentation attribute other + than "flood-color" and "flood-opacity"

                • + +

              • key is the local name of a filter primitive + attribute

                • + +

              • key contains U+003A (:)

                • +
              + +

              then continue.

              +
              2. + +
            2. +

              If key is one of "funcR", "funcG", "funcB", "funcA":

              + +
                +

              1. Set value to the result of + converting value to record<DOMString, + any>.

                2. + +

              2. Let xmlTransferName be the concatenation of "fe", + the first code unit of key converted to ASCII + uppercase, and all code units of key + after the first one.

                3. + +

              3. Let transferFunction be a new XML element data for filters + whose name is xmlTransferName, whose + attributes is an empty ordered map, and whose + children is an empty list.

                4. + +
              4. +

                For each transferName → transferValue of value:

                + +
                  +

                1. Let transferFunctionValue be the result of generating an XML value from transferName + and transferValue.

                  2. + +

                2. Set transferFunction's attributes[transferName] to + transferFunctionValue.

                  3. +
                +
                5. + +

              5. Append transferFunction to xmlFilter's children.

                6. +
              +
              3. + +
            3. +

              Otherwise:

              + +
                +

              1. Let attrXMLValue be the result of generating an XML value from key and + value.

                2. + +

              2. Set xmlFilter's attributes[key] to + attrXMLValue.

                3. +
              +
              4. +
            +
            8. +
          +
          5. + +
        5. return xmlFilters
          6. +
        + +

        The steps to apply an XML filter list to an image given an XML filter list filters are:

        +
          +

        1. Let image be the source image

          2. + +
        2. +

          For each filter of filters:

          +
            +

          1. Let svgFilter be an SVG filter, obtained by mapping each attributes of the filter XML filter list to the SVG equivalent.

            2. + +

          2. Render image using svgFilter, creating + filteredImage

            3. + +

          3. Let image be an alias to filteredImage

            4. +
          +
          3. + +

        3. Return image.

          4. +
        + +
        + +
        +

        The following example will create a layer with a colorMatrix + filter that swaps the green and red channels, then blurs the result by 5 pixels:

        + + 
        // canvas is a reference to a <canvas> element
+const context = canvas.getContext('2d');
+context.beginLayer({filter: [
+  {
+    name: 'colorMatrix',
+    type: 'matrix',
+    values: [
+      0, 1, 0, 0, 0,
+      1, 0, 0, 0, 0,
+      0, 0, 1, 0, 0,
+      0, 0, 0, 1, 0
+    ],
+  },
+  {
+    name: 'gaussianBlur',
+    stdDeviation: 5,
+  }
+]});
        +
        + +

        Currently, CanvasFilterInputs can only be linear sequences of + filters. Full filter graphs are a planned expansion of this feature.

        + +
        + +

        Before any operations could access the canvas element's bitmap pixels, the + drawing state stack must be flushed so that all pending layers are closed and + rendered to their parent output bitmap all the way to the canvas + element's bitmap. The steps for flushing the drawing state stack are:

        + +
          +

        1. Save the context's current drawing state in the drawing state + stack by running the save() method + steps.

          2. + +

        2. Let stackBackup be an empty list.

          3. + +

        3. While the drawing state stack is not empty:

          4. + +
            +

          1. Prepend the drawing state at the top of + the drawing state stack to stackBackup.

            2. + +

          2. If the canvas layer state at the top of the drawing state stack + is not empty, run the endLayer() method + steps.

            3. + +

          3. Otherwise, run the restore() method + steps.

            4. +
          + +

        4. return stackBackup.

          5. +
        + +

        Given stackBackup which was returned when flushing the drawing state + stack, the drawing state stack can be restored to the state it was before it + was flushed by running the steps for restoring the drawing state stack:

        + +
          +

        1. For each stateEntry of + stateBackup:

          2. + +
            +

          1. Push stateEntry onto the drawing state + stack.

            2. +
          + +

        2. Restore the context's current drawing state by running the restore() method steps.

          3. +
        + +

        The steps for reading the context output bitmap are:

        + +
          +

        1. Let drawingStateStackBackup be the result of flushing the drawing state + stack.

          2. + +

        2. Let resultBitmap be a reference on the context output + bitmap.

          3. + +

        3. Run the steps for restoring the drawing state stack, given + drawingStateStackBackup.

          4. + +

        4. Return resultBitmap.

          5. +
        + +
        +
        Line styles
        @@ -66424,9 +67001,15 @@ try {
        HTMLCanvasElement
        OffscreenCanvas
        -

        If image has either a horizontal dimension or a vertical dimension - equal to zero, then throw an "InvalidStateError" - DOMException.

        +
        +

        If the rendering context associated with image has a layer-count different than zero, then throw an + "InvalidStateError" DOMException.

        + +

        If image has either a horizontal dimension or a vertical dimension + equal to zero, then throw an "InvalidStateError" + DOMException.

        +
        ImageBitmap
        VideoFrame
        @@ -68038,6 +68621,9 @@ try {

      8. If either the sw or sh arguments are zero, then throw an "IndexSizeError" DOMException.

        9. +

      9. If layer-count is not zero, then throw an + "InvalidStateError" DOMException.

        10. +

      10. If the CanvasRenderingContext2D's origin-clean flag is set to false, then throw a "SecurityError" DOMException.

        11. @@ -68185,6 +68771,9 @@ try {

      11. If IsDetachedBuffer(buffer) is true, then throw an "InvalidStateError" DOMException.

        12. +

      12. If layer-count is not zero, then throw an + "InvalidStateError" DOMException.

        13. +

      13. If dirtyWidth is negative, then let dirtyX be dirtyX+dirtyWidth, and let dirtyWidth be equal @@ -69533,6 +70122,13 @@ interface OffscreenCanvas : EventTarget { internal slot is set to true, then return a promise rejected with an "InvalidStateError" DOMException.

        14. +

      14. If this OffscreenCanvas object's context mode is 2d and the rendering context's layer-count is not zero, then return a promise + rejected with an "InvalidStateError" + DOMException.

        +

      15. If this OffscreenCanvas object's context mode is 2d and the rendering context's OffscreenCanvasRenderingContext2D { }; OffscreenCanvasRenderingContext2D includes CanvasState; +OffscreenCanvasRenderingContext2D includes CanvasLayers; OffscreenCanvasRenderingContext2D includes CanvasTransform; OffscreenCanvasRenderingContext2D includes CanvasCompositing; OffscreenCanvasRenderingContext2D includes CanvasImageSmoothing; @@ -139132,6 +139729,7 @@ INSERT INTERFACES HERE Jasper Bryant-Greene, Jasper St. Pierre, Jatinder Mann, + Jean-Philippe Gravel, Jean-Yves Avenard, Jed Hartman, Jeff Balogh,