Group/Canvas Caching

[ShaMan123]

@asturur
I have an idea how to refactor Group's caching mechanism.
Current Caveats:

1. absolutePositioned clipPath absolutePositioned with objects nested in groups #7142
2. object with shadow Group With ClipPath and Text with Shadow Renders Incorrectly #7454
3. nested selection - rendering without the selected object? feat(V6): Group Rewrite + Nested Selection #7669 (comment)
4. performance - with animating objects
5. performance - with a lot of objects

To handle these caveats I suggest that group should never cache all objects but use a number of cache canvases that will update independently.
To handle caveats 1-4 we simply render these objects directly on group (because it won't cache, we render directly on canvas).
We shall need to handle group's clip path in such a case.
The rest of the static objects will be rendered to dedicated caches. Once an object is marked dirty it will mark the entire group as dirty and it's dedicated cache canvas.
If we add a flag to Object perfWillUpdateFrequently we can filter out animating objects as well.
This should solve 1-4.
Regarding 5 I suggest we expose batchingThreshold or some prop that indicates the amount of static objects we want to batch into a single cache canvas in order to boost perf for huge groups.
This should solve 5.
We can implement the same logic for canvas (for 3-5). I have experimented with this. It is very promising.

Thoughts?

[ShaMan123]

@melchiar ?

[melchiar]

How will you handle group opacity if objects in a group are all cached separately and never cached together? Or am I misunderstanding?

[ShaMan123]

You are right, we will need to handle opacity as well.
We could propagate it down (by default group rendering prevents it from happening using _transformDone)
Or we could simply set opacity when we render the cache onto the main canvas.
Both sound solid.

[melchiar]

the problem is that opacity on grouped objects is supposed to apply to the group as a whole (first example). Propagating the property individually to each object within the group will cause opacity overlapping (second example)

Group dropshadows as well need to render as a single shadow for the whole group, not individual shadows for each object within the group.

[ShaMan123]

Group dropshadows as well need to render as a single shadow for the whole group, not individual shadows for each object within the group.

You mean a shadow set on group, that's not a caveat. I mean a shadow set on an object belonging to a group.

[ShaMan123]

I have another thought regarding the caching mechanism.
I suggest we extract it to a dedicated class so then the dev can override it in light of experiments such as
fabricjs/fabricjs.com#342
http://fabricjs.com//raphael-vs-fabric-simple?n=5000&optimize_caching=1

[ShaMan123]

Another thing I want to discuss is isCacheDirty. Very damaging for performance. Why do we need to check if objects are dirty when invalidating group? Especially when Object._set method does it for us?
We should avoid iterations on _objects

I don't remember why we do that, but let's not make changes again to this branch to address performance issues. With those large changes can come performance regressions. In general large nested groups will be slow.

Eventually there has been something that required us inspect all objects because there could be something that invalidate the cache outside set Example, a lef and top change in a group?

I think it's dead code. You said yourself that Group wasn't touched since 2014.
Investigation shows that you committed 700ee14 in 2016.
It seems to be logic prior to enforcing using set on Object so there wasn't a reliable way of knowing if a nested object was invalidated. This is not the case any longer.
Let's continue this conversation here #7874

Originally posted by @ShaMan123 in #7859 (comment)

[asturur]

so first an assertion.
absolutely positioned clipPath was a mistake.
now let me read everything slowly and see if i can help

[asturur]

So there are a bunch of points here that spin up interesting conversation and performance should be an afterthought in this case.

Sometimes caching is required in order for things to work correctly, the example is the clipPath, if a group has a clipPath we can't avoid caching it, because we won't be able to cut away pixels in isolation if the objects are rendered already on the canvas context.

Having many cache canvases slow down the browser at a certain point and make the library unusable on safari IOS.

Absolute clippath

now absolute clippath was a quick thought. When adding clipPath i said, if i remove the transform before drawing it the clipPath will be applied in the canvas, solving what people were often trying to do, define a visibility area for an object in the canvas ( collages for example ). So taken by the enthusiasm i just thought it was cool and i added it.
If the same can be done adding an image inside a group with a clipPath, i m removing absoluteClipPath immediately.
(i ll open a task/issue for this).
Then removing blurriness for clipPath should be our biggest concern.

Shadows

Shadows are slow. Shadows are different from browser to browser, but people use shadow for some effects.
Now shadow was the reason why we have the function shouldCache that tries to determine if the group can be cached or not, and to do so it needs to run recursively on every object contained and see at what level of the three this breaks.
How do the majority of designs app on the web work? if we stop shadow from rotating with angle, is it ok? It would be fine for maintainability sake to say:
shadow of objecst inside group are not going to rotate, are fixed. Shadow on object on canvas rotate.
Or we can make those fixed too.
The issue would be that then when a group is not cached, the shadow will start to rotate, unless we decide we apply a counter rotation to offsets, that gets complicated when skew is in play.
We can just make the cacheCanvas larger for the object and have the shadow in. With large offset it become enormous and if you don't want to enlarget it on both sides, you need to account for weird offset on rotation.

GlobalCompositeOperation

Is years i want to fix it. It enables effects people are familiar with, caching broke it.
The feature needs to be re-planned with caching in mind, and make it so it works contained at group level.

Trying to answer Shachar points:

1. let's see if we can get rid of absolutePositioned
2. let's see if a strategy to cache shadow simplify things
3. i m not familiaer with 3 i need to set time aside for it.
4. performance when animating things inside a thing that is large and breaks the cache every frame. Seems a bad idea to get in that situation. I don't have a solution for it and i m not sure should be a priority.
5. performance with lot of objects, probably removing render time checks on caches and cacheability would help a bit. I have tried to inspect performances more than once, but the flamegraph get flattened by chrome and is not easy to understand where time is consumed.

[ShaMan123]

• I love absolutePositioned clip path

If the same can be done adding an image inside a group with a clipPath, i m removing absoluteClipPath immediately.

Didn't understand what you meant.

• You didn't address my suggestion for caching. Having a few caches for group... We can limit the amount.

Sometimes caching is required in order for things to work correctly, the example is the clipPath, if a group has a clipPath we can't avoid caching it, because we won't be able to cut away pixels in isolation if the objects are rendered already on the canvas context.

I have worked around this in the eraser. It is possible to propagate the clip path to other caches

[asturur]

i don't want to make caching more complicated thinking of fixes for clipPath only, when we have open PRs for both masks and filters at object levels. When those features are in, we can see how caching can be improved in a way it works for all.

[ShaMan123]

So for now we can suggest the dev to use Layer to boost perf with a demo

[asturur]

If the same can be done adding an image inside a group with a clipPath, i m removing absoluteClipPath immediately.

I wrote it bad.
You can have group, unselectable, and unevented, with a standard clipPath and fixed layout.
You can then have an object moving inside the group.
This should give you the same exact effect of an absolute positioned clipPath. No?

Why do you like the absolute positioned clipPath, what is the use case for which those are irreplaceable?

[ShaMan123]

What you suggest works for 1 level of nesting but not for more when caching.

[asturur]

i m fine removing the deep check on cache invalidation if consistently using the set('dirty') keeps the things working.

[ShaMan123]

seems it does.
I even think that object monitors are redundant because of the dirty bubbling

[ShaMan123]

done 5f3a39f

[asturur]

this could be a no brainer PR.
Can you open it?

[ShaMan123]

Open what? #7882

[ShaMan123]

This is AMAZING because Layer can now be used to boost perf. We can split a canvas/group's objects into 2 sub collections, one for active objects and one for non active ones. We can then use Layer to render all the non actives and have it cached while setting group not to cache.

I am experimenting with this approach
See how promising it is, and this is without implementing render batching

Go to http://fabricjs.com/raphael-vs-fabric-simple?n=5000&optimize_caching=1 to test without layer caching

vs.Raphael.Simple.Shape._.Fabric.js.Benchmarks.-.Google.Chrome.2022-04-07.17-31-28.mp4

Originally posted by @ShaMan123 in #7860 (comment)

[ShaMan123]

It's faster than Raphael

[mattevans-dev]

Hi @ShaMan123 @asturur,
I know the team are hard at work on the Typescript additions, but I was just wondering if there was any progress on this thread? I'm currently trying to apply a globalCompositeOperation on an object within a group and it doesn't get applied. Sounds like the same issue @asturur mentions above.

[ShaMan123]

@mattevans-dev
I have this high in my list. As far I am concerned it is blocking group from maturing out of beta.
Group is my baby so there you go.
If you want, provide a repro and maybe we can help you or it can enrich the conversation.

[ShaMan123]

A client posed a question that made me think of an elegant solution for the following:

Let's say we have a VERY BIG (big in size, not number of objects) object (maybe a group of objects) and we apply a clip path to it.
fabric forces it to cache but we want to disable caching because of the size, we prefer caching the children.
Sure we can propagate the clip path to the children, but truth be told, that's an awful workaround.

How do we support this?

Why don't we use the canvas' cache to render the object in 2 steps?
This way having a clip path doesn't force an object to cache and performance should not drop, I would argue it will boost.

[asturur]

Global composite operation is broken since caching was introduced in 1.7 :(
What it should be done and we can do it in 6.0 is to change the spec of how it works, to respect caching.

In practice global composite operation will work only from an object to its container.
let's take darken for example.

A group with darken will apply darken to all objects on the canvas drawn before it.
an object inside a group with darken will apply darke to all object drawn before it, but only inside the group, then the group will apply its own operation when is drawn.

This is also complicated because the SVG export will probably have an hard time keeping this working, but is one of the few ways we can enable global composite operation with consistency. Basically it will also enforce group caching when activated

[asturur]

Regarding the group and canvas caching, that is a delicate topic.
In my opinion caching doesn't really stop group from maturing, sometimes group with many objects will be slow and you as a developer would dream to have your own caching strategy that optimize that scenario.

I really don't want to embed multiple strategies in the library because different strategies will probably break other features.
Before adding new caching optimization i would like to explore the possibility to draw the transformed object on the top canvas, even if is violating the stack, while weird is still an option.

I really want to avoid feature creep in the library and i would also like to understand more about the cases in which a large group becomes too slow

[ShaMan123]

1. Once you read fix(ActiveSelection): 🚀 multiple nested selection 🗂️ #7882 you will understand why caching blocks group, or at least why group does something silly because of caching
2. I did not suggest fabric should have multiple caching strategies. I suggest caching logic becomes an interface, same as controls, based on an abstract class allowing anyone to write what they want on top of. Fabric will expose the default mechanism, just like there are default controls.
3. 2 step object rendering: as suggested in Group/Canvas Caching #7874 (comment), I wrote VERY BIG group - I meant in dimensions, not necessarily in number of objects. Maybe if we do this we change object caching so it renders cache without the clip path and we clip the object with the 2 step caching.
4. Rendering active object on top canvas: I wrote somewhere (don't remember where) how fabric should use Layer to allow such things. I don't think fabric should do it without layer and it should be easily customizable. I think the rendering logic should be adapted so that the dev can control how many layers there are, the order of the layers and what is drawn on each one. This will allow to draw brushes at the bottom of the stack (new PencilBrush(layerToDrawOn)), I have a demo POC somewhere, etc.. (then for example controlsAboveOverlay is redundant). If this is not clear enough I will elaborate. I want to merge all group PRs before you change this.
5. Rendering active object: you suggested having a ghost for the active object, what about that?
6. globalCompositeOperation: won't the 2 step rendering fix it? Or am I missing something? Could you share some visuals so I can better understand.

[ShaMan123]

Ohh globalCompositeOperation should affect all pixels under what is drawn?
So it poses the same issue as shadow does?

[ShaMan123]

I want to see if I understand the issue with globalCompositeOperation.
Let's say we have a group with a clip path and an object contained in the group that applies a globalCompositeOperation.
Should the operation be applied to all pixels under the object? Or only to pixels in the group?
Regardless if the object is part of a group or not the fact it has a clip path makes it impossible with the current mechanism to apply a globalCompositeOperation for pixels under it.
We could, in such a case, do the following:

1. render what's behind the object onto a separate canvas (let's call it a)
2. render object on a
3. render clip path on a
4. apply the inverted clip path to the main context, clearing the pixels for the object
5. render a on the main context

I will add a POC

[ShaMan123]

A test app

[ShaMan123]

#8271

[ShaMan123]

After adding the tests in #8271 I have to say it makes sense that gco acts only on the layer it is contained in.
We can think of a solution for the other case but it shouldn't be default IMO.
Think of destination-out inside a group. Makes sense that once the group is rendered to the next layer it shouldn't erase everything because it has an object with destination-out gco.

[ShaMan123]

I would like to present my rationale better.
I think objectCaching should be decided by the dev without fabric touching it under the hood.
In cases where shouldCache() === true regardless of objectCaching then we should do a 2 step rendering process.
That way a dev can set objectCaching to false and have a clip path. This is crucial for a group with very big dimensions and a clip path and I am sure for other cases.
Besides it will allow us to set objectCaching to false by default for mobile devices to avoid OOM and perf issues which is great.

In fact it might be good to set objectCaching to false by default entirely and leave it for the dev to decide how to optimize perf

[ShaMan123]

Another suggestion:
We should inform group (recursive upwards) from Object#_set if it should cache or not monitoring specific keys (globalCompositeOperation, shadow etc.)
This will solve the drop in perf due to a walk down the tree every render cycle on every group

[ShaMan123]

Or from the return value of Object#render - it can return a boolean indicating if the object is blocking caching for the container

[ShaMan123]

This diff shows the POC's output compared to the existing state where shadow isn't correct.
We could transform the shadow offset but that doesn't sound viable (worth while only if the object will not rotate a lot......)

The POC can be viewed here refactor-object-rendering...2step-caching
It is based on master...refactor-object-rendering which I want to commit (refactors rendering to make it readable and simpler)

[ShaMan123]

I tried rotating the shadow offset
Works
refactor-object-rendering...fix-shadow-rotation

[ShaMan123]

#8298 takes this discussion into a PR