Animation Smoothness Explainer #1003
Conversation
|
I'm genuinely excited to see your proposal regarding the ability to track smoothness in web applications with browser API. In our own product (Excel Online), we would definitely make extensive use of an API like this, and I’m confident many other web applications would benefit from it as well. Over a year ago, I built an internal infrastructure aimed at monitoring animation smoothness, and I’d love to share some insights based on that experience. One key aspect I believe is critical, you did mention in the background section, is the FPS consistency. However, I didn't see it clearly addressed in the specific API options. FPS consistency is crucial because it helps identify the exact points where performance drops occur. For example, you might have a 10-second animation that appears fine in aggregate, but actually experiences complete frame drops for half a second, which wouldn't be reflected in an average FPS calculation over the full duration. I believe the API output should provide enough detail to estimate frame drops more accurately. This could be achieved by either returning the timestamps of received frames, allowing us to calculate the gaps ourselves, or by letting the user specify a time interval (e.g., 100ms) to compute localized FPS values. This is the approach we took internally, and it significantly improved the accuracy of our measurements. Of course, having a native API to rely on, rather than using requestAnimationFrame and its limitations, would be a great advantage. If you'd like, I'd be happy to continue the discussion offline and dive deeper into the topic. |
|
@mmocny did a bunch of work on this. It might be worthwhile for y'all to chat :) |
|
Indeed! We started some chatting at BlinkOn last week, and I've started to review this explainer. Will send comments soon. |
|
@microsoft-github-policy-service agree company="Microsoft" |
| ## Goals | ||
| * Webpage accessible API that captures user-perceived framerate accurately, taking into account both the main and compositor threads. | ||
| * An approach that doesn’t cause webpage performance regressions. | ||
| * Enabling a web developer to control what time interval is considered. |
There was a problem hiding this comment.
In my mind, if an api is intended for a developer to use in real time and actually update the UX, then it might not be a good fit for the performance timeline (which tries to fire observers is a lazy, decoupled fashion).
Perhaps something more akin to Compute Pressure API, or performance.measureUserAgentSpecificMemory(), or navigator.connection.*?
|
|
||
| ### 2. Measuring animation and graphics performance of browsers | ||
|
|
||
| The public benchmark, MotionMark, measures how well different browsers render animation. For each test, MotionMark calculates the most complex animation that the browser can render at certain frame rate. The test starts with a very complex animation that makes the frame rate drop to about half of the expected rate. Then, MotionMark gradually reduces the animation's complexity until the frame rate returns to the expected rate. Through this process, MotionMark can determine the most complex animation that the browser can handle while maintaining an appropriate frame rate and uses this information to give the browser a score. To get an accurate score, it is crucial that MotionMark can measure frame rate precisely. Currently, MotionMark measures frame rate based on rAF calls, which can be impacted by other tasks on the main thread besides animation. It also doesn't take into account animations on the compositor thread. The method using rAF to measure frame rate doesn't reflect the user's actual experience. |
There was a problem hiding this comment.
Agree with these points, but I don't think we need a public web exposed api to address the benchmarking use case.
|
|
||
| Animation frames are rendered on the screen when there is a change that needs to be updated. If they are not updated in a certain amount of time, the browser drops a frame, which may affect animation smoothness. | ||
|
|
||
| The rAF method has the browser call a function (rAF) to update the animation before the screen refreshes. By counting how often rAF is called, you can determine the FPS. If the browser skips calling rAF, it means a frame was dropped. This method helps understand how well the browser handles animations and whether any frames are being dropped. |
There was a problem hiding this comment.
Nit: "it means a frame was dropped" is somewhat of an over-loaded term.
I think "a potential frame rendering opportunity was skipped" might be more appropriate.
I only differentiate because I think there is a difference between an animation frame which the user agent wanted to present not being able to get presented within acceptable latency, vs the user agent choosing to throttle frame rendering opportunities to balance tradeoffs.
A web developer measuring using raf loops is not really able to differentiate.
| The rAF method has the browser call a function (rAF) to update the animation before the screen refreshes. By counting how often rAF is called, you can determine the FPS. If the browser skips calling rAF, it means a frame was dropped. This method helps understand how well the browser handles animations and whether any frames are being dropped. | ||
|
|
||
| #### Limitations | ||
| Using rAF to determine the FPS can be energy intensive and inaccurate. This approach can negatively impact battery life by preventing the skipping of unnecessary steps in the rendering pipeline. While this is not usually the case, using rAF inefficiently can lead to dropped or partially presented frames, making the animation less smooth. It’s not the best method for understanding animation smoothness because it does not take into account factors like compositor offload and offscreen canvas. While rAF can be useful, it isn’t the most accurate and relying on it too heavily can lead to energy waste and suboptimal performance. |
There was a problem hiding this comment.
(The point about compositor frames is valid, but I want to focus on rAF-polling for a second).
In my experience, it's not just the scheduling of rAF task itself that significantly affects performance, but rather the downstream effects of raf-polling on scheduler decision making abilities, can lead to janky moments.
For example, when I keep calling requestAnimationFrame() at 60hz, Chromium will schedule a BeginMainFrame task at the start of each new vsync. No other rendering opportunity will be given to the page until the next vsync.
If your rAF() call does nothing but measure timings, and doesn't update UI, now your page effectively becomes "static" for the next 16ms.
And that means that all "real" UI updates to the page, which are scheduled in the middle of a frame-- such as interactions-- now won't be able to paint until after the next vsync.
In theory, that only delays things a few ms, but in practice what happens is:
- rAF()
- Important UI update (perhaps new UI Event).
- Runs quickly, much less than 16ms.
- Requires a rendering update
- idle time, waiting for next vsync
- ...because of the rAF() at the start of the vsync.
- If not for that, we would be able to schedule a render opportunity nearly ~immediately.
- Idle time allows us to schedule other work, perhaps even idle callbacks...
- potentially long running
- Vsync fires and we schedule BMF, but now main thread is blocked and busy...
- potentially for more than 16ms and we start to "drop" frames.
Chromium is currently experimenting with deferring task scheduling under certain cases (i.e. immediately after interaction UI Event dispatch) in order keep scheduler idle for a little while until next rendering opportunity. But now we are reducing throughput in order to optimize latency. And this new scheduling policy doesn't help with more general case.
Also, nit: Cannot you use rAF for OffscreenCanvas measurement (via the worker itself)? Did you mean desynchronized canvas?
There was a problem hiding this comment.
" it's not just the scheduling of rAF task itself that significantly affects performance, but rather the downstream effects of raf-polling on scheduler decision making abilities, can lead to janky moments."
Thanks for the insight. I had no idea that the very act of calling rAF itself has scheduling implications!
"Chromium is currently experimenting with deferring task scheduling"
Could you please share a design doc that explains this? I'd like to understand the internals of this.
" Cannot you use rAF for OffscreenCanvas measurement (via the worker itself)? Did you mean desynchronized canvas?"
I'm not entirely sure how canvas rendering works but if we use rAF for measurement, won't that have the same problem (i.e downstream effects of raf-polling on scheduler) that we have today? I have this test page where the 2d canvas stutters when I add main thread jank which leads me to believe that the main thread (and hence BMF) is still involved in drawing on the canvas. Please let me know if I misunderstood any part.
There was a problem hiding this comment.
I had no idea that the very act of calling rAF itself has scheduling implications!
Im going to get pedantic here because it matters :P
- Calling
requestAnimationFramemight have small effects on scheduling, in that it might be the first "request for rendering update" and thus request a rendering opportunity.- Although the page might not yet be invalidated, a request for
rAF()callback is expected to update the page, so we have to assume we need a rendering update. - A no-op
rAF()callback will early-exit from rendering, but at least in Chromium, you don't get your rendering opportunity back. You cannot get multiple rAFs per vsync. - There have been requests for a
addEventListener("animationframe")API or something akin to it, which only ask to observe rendering opportunities without themselves requesting one-- but we don't have this.
- Although the page might not yet be invalidated, a request for
- But more importantly, once a "rendering opportunity" is "used up" for a vsync, that is what has the largest implications.
For a page that is already actively animating at 60hz from main thread, adding rAF-polling might have no additional negative effects.
For a page that is mostly idle, adding rAF-polling just for measurement reasons, might have effects on scheduling, and traditionally at least these could be quite negative.
So, some scheduling policies have been added to chromium to help alleviate, at least for the immediate next animation frame after user input.
Could you please share a design doc that explains this?
@shaseley was the primary owner and might have some additional docs to link, but here is the most relevant patch with great description, and here is the CrBug with full patch history and commentary
I'm not entirely sure how canvas rendering works but if we use rAF for measurement,
Main thread canvas with normal rAF will have all the same issue's because the canvas paint operations are synchronized. You can run with desynchronized mode to decouple from raf/rendering, but still all your JS runs on the same thread and will suffer from jank.
But I was referring to OffscreenCanvas in a WebWorker, which now supports requestAnimationFrame on all browsers (part of Baseline)
There was a problem hiding this comment.
Thanks @mmocny, your insight into the implications of rAF scheduling is super helpful!
@jenna-sasson, we should document this somewhere or perhaps add a sentence or two in the "Limitations" part of the explainer (perhaps in a future iteration) about how the very act of using rAF has negative implications (both, on measurement and scheduling).
But I was referring to OffscreenCanvas in a WebWorker, which now supports requestAnimationFrame on all browsers
Gotcha, we'll experiment with webworkers for offscreen canvas' in desynchronized mode. I didn't know that this was an option at all. Learnt something new today :)
| ### Long Animation Frames API | ||
|
|
||
| #### Description | ||
| A long animation frame (LoAF) occurs when a frame takes more than 50ms to render. The Long Animation Frames API allows developers to identify long animation frames by keeping track of the time it takes for frames to complete. If the frame exceeds the threshold, it is flagged. |
There was a problem hiding this comment.
I would add that one key difference is that rAF() is an explicit request for scheduling a rendering opportunity, paired up with a callback (an "event") to observe that rendering opportunity.
LoAF is just a means for observing rendering, it doesn't affect scheduling. I think this makes it a better fit.
(The threshold values can perhaps be easily adjusted for this use case?)
| ### FPS-Emitter | ||
|
|
||
| #### Description | ||
| In the past, Edge had a library called fps-emitter that emits an update event. Once a new instance of FPS-emitter is called, it starts tracking frames per second via the rAF method. When the FPS changes, the result is returned as an EventEmitter. This method builds off the rAF method described above and faces similar limitations. |
There was a problem hiding this comment.
I've built something like this before using rAF() on main thread, and then also rAF() on OffscreenCanvas, and coordinating between the two.
I could then compare in real time when main is lagging / throttled / dropping frames.
I still think this comes with all the baggage of rAF, but have you played with that approach as well?
There was a problem hiding this comment.
We haven't played with that approach yet. However, something else I've played with is measuring the CompositorFrame throughput of the renderer. This is something that is readily available in devtools today (more specifically, in DroppedFrameCounter) and gives a breakdown of complete, partial and dropped frames. I created a prototype that builds on top of that and exposes this to the DOM.
In the experiment below, the main thread is busy, but the cc thread is still able to scroll. "rAF based FPS" shows the incomplete picture whereas the "Renderer FPS" (i.e the new experimental thing) shows partial frames still being pumped out.
There was a problem hiding this comment.
You can find a really old demo of mine here
The bottom left corner has 3 FPS meters:
- Main thread raf visualized on main thread canvas
- Main thread raf visualized on OffscreenCanvas
- Worker raf visualized on OffscreenCanvas
The first two both showcase the low FPS rates due to janky js, but, the second one will "react" to jank immediately while the first is "stuck" showing the old frame rate. This demonstrates the problem of measuring main thread jank from the main thread itself (at least for real-time).
The third is just useful to show what max frame rate you could have. I think the specifics of Worker rAF are potentially underdefined, but I think in Chromium it is synchronized to compisitor frame rate-- and I think it will drop if you have compositor/rasterization related jank.
There was a problem hiding this comment.
Gotcha, thanks for sharing the demo. Ideally, we want something that's native and doesn't rely web workers to measure simple pages. Devs should be able to query the FPS at a given point. Going back to the point you made earlier, do you think we could expose existing data from the CompositorFrameReporter(s) and make it accessible to DOM? We could then add a new API (something similar to Compute Pressure). Is this a feasible path that I should go down?
Also, how can we standardize all of this? Our discussions might be specific to the Chromium architecture. Do other browser vendors also have something like a renderer process with a threads similar to the main and the cc?
| `window.addEventListener("frameratechange", (event) =>{doSomething();})` | ||
|
|
||
| ## Concerns/Open Questions | ||
| 1. The user-perceived smoothness is influenced by both the main thread and the compositor thread. Accurate measurement of frame rates must account for both. Since the compositor thread operates independently of the main thread, it can be difficult to get its frame rate data. However, an accurate frame rate measurements needs to take into account both measurements. |
There was a problem hiding this comment.
I think that this is true for a benchmark / internal browser measurement metric. But for a web exposed API where developers are expected to act and adjust content to fit within constraints, I'm not sure if we really need more than main thread frame rendering signals... If the compositor is struggling to keep up with frame rates, I think it will put back pressure on the main thread scheduling opportunities, anyway.
If you have compositor-driven-animations, then developers typically expected these to be fast and efficient, and not typically able to "adjust down" the quality. It's theoretically possible, but I'm not sure it would actually happen.
Net/net, I think I would focus on:
- A good main-thread "smoothness" metric/signal for web developers to use
- A good overall "smoothness" metric for browser venders to use
Its really only for competitive benchmarking purposes that you sort of want both available in a browser, but perhaps you can just use Worker rAF()?
There was a problem hiding this comment.
"If the compositor is struggling to keep up with frame rates, I think it will put back pressure on the main thread scheduling opportunities, anyway."
That's a good point. So, IIUC, measuring smoothness on the main thread should suffice since the cc updates are presumed to almost always be efficient and if they aren't, main thread updates will slow down due to back pressure. For the main thread smoothness, I believe CompositorFrameReportingController or FrameSequenceMetrics should already have this info. There are also Graphics.Smoothness.* metrics (eg "Graphics.Smoothness.PercentDroppedFrames4.{Thread}{Sequence}") that have granular measurements.
Also, can you please elaborate on what you mean by "Worker rAF"?
| 1. The user-perceived smoothness is influenced by both the main thread and the compositor thread. Accurate measurement of frame rates must account for both. Since the compositor thread operates independently of the main thread, it can be difficult to get its frame rate data. However, an accurate frame rate measurements needs to take into account both measurements. | ||
| 2. Similar to the abandoned [Frame Timing interface](https://wicg.github.io/frame-timing/#introduction). We are currently gathering historical context on how this relates and why it is no longer being pursued. | ||
| 3. Questions to Consider: | ||
| * Should content missing from the compositor frame due to delayed tile rasterization be tracked? |
There was a problem hiding this comment.
Some folks here have started calling this concept frame "contentfulness", to differentiate raw frame "smoothness".
It can include more than just tile raster (such as image decode), including the the actual design of page contents (such as video bitrate / buffering, or real time video game latency, or typical web page loading related layout shifts, images, ads, fonts...). Users seem more sensitive to that stuff than literal graphics pipeline performance (if trying to judge "quality of experience").
|
Merging initial draft to get a static URL that we can share. Will address outstanding feedback in next iterations. |
|
|
||
| Various metrics have been used in the past to try and understand the user’s experience in this space. Some of these were accessible to the webpage, while others were internal browser metrics. Examples of these include: | ||
|
|
||
| * Framerate – the number of frames displayed to the user over time. |
There was a problem hiding this comment.
I think there may be security/privacy implications here, because the presence of a presented frame reveals new information to javascript. For example, imagine a root page (A) with a cross-origin login iframe (B): script on A can listen to the timing of the frames produced by B, and infer information about the password from the timing of the user's keystrokes. Another scenario is spellcheck: a page can toggle the spellcheck field of an input element and learn whether a word is misspelled based on the presence of a presented frame (for showing the spelling underline), and this contains information about the user's spellcheck dictionary. This information is not in requestAnimationFrame today because requesting a frame causes a new frame to be presented.
8 participants
No description provided.