Add lowLatency flag to CanvasRenderingContext2DSettings

[yell0wd0g]

This is a duplicate of the current state of @kenrussell's #4234 PR (that partially addresses #4087), so I can take it over from here.

Note that this PR is blocking KhronosGroup/WebGL#2753.

---

/canvas.html ( diff )
/references.html ( diff )

[yell0wd0g]

@annevk, @kenrussell, @jrmuizel, @grorg, @travisleithead, @fserb

Gathering a wee bit the status:
(1) - Low Latency as a feature comprises two parts: one is decoupling the input event processing and rendering from the browser main loop, and another is rendering onto the front buffer. (Chrome ToT (M74) has the first part in all platforms, whereas currently only implements front buffer rendering on ChromeOS.)
(2) - The App can use either or both of the features described in (1) to decrease latency from input events to rendering, but of course at the expense of having -maybe- tearing of the rendered contents; this PR is about offering this possibility to application developers which anyway would need to be aware of the implications of what they are doing (see e.g. this article).

With that I'd say the best flag name might be un/synchronized (synchronized: false or unsynchronized: true), conveying the two parts of the desynchronization as described in (1) and doesn't look like something everyone would like to have in their apps. WDYT? (Feel free 👍 this message).

Of course the PR should reflect this tradeoff, if at least in some informatiional non-normative note.

[kenrussell]

It's all personal preference, but I like unsynchronized:true, desynchronized:true, and allowTearing:true.

[domenic]

I'll just note that boolean dictionary values should default to false, so all of @kenrussell's latest suggestions are good in that regard (as is the OP lowLatency: true). But e.g. synchronized: false would be bad.

(Yes, alpha: false breaks this rule. That is a sad historical mistake.)

[yell0wd0g]

unsynchronized: true ping: @annevk, @jrmuizel, @grorg, @travisleithead, @fserb, WDYT?

[kdashg]

I generally do prefer foo: true over unfoo: false, so that !unfoo would just be foo.
If it's a strong recommendation that defaults should be false, I'd prefer desynchronized, which I think better reflects that it's allowing opting out of (the default) synchronization. (unsynchronized is acceptable, but not my preference)

[yell0wd0g]

So, desynchronized: true seems to be leading the pack, @annevk, @travisleithead, does it sounds good to you too?

[othermaciej]

@YellowDoge I wrote in the previous PR for this that I don't really understand what this does, and I feel a bit more confused after your explanation. Some specific questions that came to mind:

• Why would a canvas context flag affect processing of input events? Does setting this flag on any context affect all input events? Also, I didn't see anything in the spec change about processing of input events, did I miss it? Or is this a separate feature that you described solely for context.

• I'm still not totally clear on what "rendering from the front buffer" means even after reading the Wikipedia reference on Multiple Buffering.

• It doesn't seem like the spec has enough info to interoperable implement this, or to write test cases. Maybe that's ok for a performance hint?

• Is this feature intended to be implementable on and/or relevant to platforms other than Chrome OS?

[annevk]

The name seems fine to me, but I'm a bit surprised @othermaciej's feedback on the prior PR went unaddressed. I'll try to be more careful ensuring all loose ends are tied up when we switch the place of discussion.

[yell0wd0g]

@othermaciej's feedback on the prior PR went unaddressed.

I thought it better to agree on a name for the new functionality before even getting to writing the spec patch, since the former will (greatly) affect the latter.

I can reply here to two of the concrete comments from #4360 (comment):

• Why would a canvas context flag affect processing of input events?

Why not? If we want to decrease latency between input event and pixels on screen, this is one of the sources of delay to cut on. Moreover, some elements (like <video>) produce output in a manner essentially decoupled from the event processing loop, so this operational mode would be nothing new to the platform.

• Is this feature intended to be implementable on and/or relevant to platforms other than Chrome OS?

Yes, in my coarse description in #4360 (comment), there's nothing ChromeOS specific, and as a matter of fact we're exploring the implementation on Windows.

[yell0wd0g]

Folks I've uploaded a new version of the patch where hopefully things are clearer. I'm using the term desynchronized since it seems more consensual and nobody has expressed opinions against. I tried to explain in a note what is front buffer rendering, starting from the assumption that readers understand what is rendering, drawing, etc, pretty basic concepts IMHO but please give some feedback from spec-land as to what's the appropriate prerequisites of the reader. Note that front-buffer is just one of the ways to reduce latency (albeit a powerful one), but there's at least another one that is referenced.

[travisleithead]

The name seems fine to me.

• Why would a canvas context flag affect processing of input events?

I was similarly confused, but after @YellowDoge's answer, I understand now that there's no feature work to directly impact how input events are dispatched and processed, rather he meant that from the perspective of elapsed time, if the Javascript code that runs as a result of an input event being dispatched is able to send instructions to a canvas element that is, in turn, rendered to the screen more immediately (due to this feature), then this is a net improvement in the overall time to "process" an input event. If I'm still mistaken, let me know.

[othermaciej]

@travisleithead

I was similarly confused, but after @YellowDoge's answer, I understand now that there's no feature work to directly impact how input events are dispatched and processed[...]

The new spec diff says this (emphasis added)

When a CanvasRenderingContext2D object has its desynchronized flag set to true, then the user agent should optimize the rendering of the canvas to reduce the latency, as measured from input events to rasterization, by either decoupling the input event processing from the normal event processing cycle, bypassing the ordinary user agent/OS rendering algorithm, or both. Insofar as this mode involves bypassing the usual processing mechanisms of events or rasterization or both, it might introduce visible tearing artifacts.

This totally sounds like it would affect input event processing. It describes this as a separate change from the rendering change that is also controlled by the flag. (And it doesn't have enough info to interoperably implement this alternate event processing, if that is what it's supposed to mean).

@YellowDoge can you clarify? Is this flag meant to affect event processing, and not just rendering?

[yell0wd0g]

@YellowDoge can you clarify? Is this flag meant to affect event processing, and not just rendering?

Yes, as it says :

by either decoupling the input event processing from the normal event processing cycle, bypassing the ordinary user agent/OS rendering algorithm, or both.

Reducing the latency can be achieved by reducing any of its components, namely: (a) input event delivery, (b) app logic processing, (c) drawing of the canvas (== execution of the drawing commands), and (d) OS rendering. If any of them is reduced, then the overall latency is reduced, right? Depending on the UA and OS capabilities, reduction of the latency can be done in several ways: reducing (a), reducing (c), reducing (d) or any combination of those.

The flag is a hint -- but if the UA is absolutely unable to improve (== reduce) he latency as defined before, then desynchronized will read back false, meaning it's not supported.

[kdashg]

Can we drop the 'changes input event processing' part? It seems like the primary adjustment we're making here is to desynchronize from the normal DOM update presentation atomicity.

Is there a need to go further than that right now, or can we move forward with just desynchronization from DOM update presentation atomicity?

[othermaciej]

I tend to agree with @jdashg . Changing input event processing is pretty concerning here, and it seems better left out. Reasoning:

• It is surprising for a flag on a canvas graphics context object to change anything about input event handling.
• It's unclear whether the effect is global to all input events or only about events directed to the relevant canvas element somehow.
• While the rendering effects of this patch are not directly observable to web content (they only affect what the user sees on the screen), the event effects sound like they are. In particular, it sounds like timing of input events relative to other tasks or micro tasks could change. So there is interoperability risk if the event handling changes are implemented in slightly different ways.
• At the same time, the specific possible changes to event handling are not described here at all, so it's impossible to implement them interoperably based on the spec, or make test cases that reflect the spec.

An alternative would be to specify what the alternate event handling behavior is, in a way that is precise enough to make test cases (even if it is a MAY-level requirement).

[othermaciej]

Further question about the term "front buffer rendering": could this also be accurately described as "single buffering" or "bypassing double/triple buffering"? I tried to look up resources on it and it sounds the same, but I might be missing a subtle difference. I ask because "single buffering" seems to be a more common term than "front buffer rendering", at least based on search engine frequency.

[yell0wd0g]

Guys PTAL:

• Removed the reference to the input events and left it as ...desynchronizing the canvas paint cycle from the normal DOM cycle, which is something we can control from the Canvas Context. If this is still unclear or off the point, please suggest a wording on the diff (@jdashg, @travisleithead, @othermaciej).
• Added the reference to multiple buffering:

A popular technique for reducing latency is called front buffer rendering, also known as single buffer rendering, where rendering happens in parallel and racily with the scanning out process.

[travisleithead]

One small nit, but otherwise looks good.

[annevk]

Aside: please don't use "guys".