VSYNC API: Support for VSYNC OFF / 120Hz+ / variable refresh displays / etc

EDIT: May 2017 -- COMPLETELY REWRITTEN. Now that I am an Invited Expert to W3C Web Platform Working Group, I've edited this to remove duplication with #785 -- And to add variable refresh rate support. As owner of Blur Busters, I also have an article that refers to this github item

EDIT: June 2017 -- Proposed new API is written here: PHASE 1 and PHASE 2. Comments are encouraged. Priority has been raised because of Apple's new 120Hz iPads

Introduction of VSYNC limitations in current browsers

With the need for a stronger VSYNC to synchronize animations to the display's refresh rate.

HTML API for Uncapped Framerate Support This is also called "VSYNC OFF". More than 80% of users in the now-$1-billion-dollar eSports gaming industry use "VSYNC OFF" because it reduces input lag. Scientific applications also sometimes require this for low-latency tests. Photodiode oscilloscope measurements show that 1000fps @ 60Hz has at ~8ms less input latency than 60fps @ 60Hz. (Almost half a frame-time less lag, because waiting-on-vsync can force a frame to be delayed). In some cases, lag can spike to a full frametime less lag, e.g. 16ms less input lag. Chrome already supports this via a command line option "--disable-gpu-vsync" but is not enabled via JavaScript API

HTML API for Variable Refresh Rate (VRR) Support This includes VESA Adaptive-Sync, HDMI 2.1 VRR, FreeSync, G-SYNC. They all function essentially the same at the application level. During full-screen mode, there needs to be a path of future support of optional JavaScript-controlled decoupling of updatesrates from fixed frequencies. This is to allow useragent apps to support emerging variable refresh rate display technologies for Canvas2D/WebGL/

HTML API for Bypass WDM compositing In full screen mode (not windowed mode, or "fullscreen borderless windowed mode") it is possible for applications of multiple platforms to bypass WDM compositing and decrease input lag by another full frame.

HTML API for Discovery Of Above Even today, there's no reliable way to detect the refresh rate of a monitor in a web browser (except with clever rAF() heuristics in Chrome, FireFox, Opera and Safari). There needs to be JavaScript ability to easily query for refresh rate, VSYNC ON/OFF support, variable refresh rate support, and low-lag full screen mode (bypass WDM compositing).

Historically, end of section 5 of W3C timing standard -- https://www.w3.org/TR/animation-timing/ -- recommends synchronization to refresh rate (VSYNC) -- and work on #785 has fixed some shortcomings of the wording in HTML 5.2. This is great for many applications, but not all.

Competition gamers/eSports like using "VSYNC OFF" in full screen video games. The ability to do a W3C compliant HTML 5.2 Javascript API equivalent of turning on/off "--disable-gpu-vsync" would be highly favoured by full-screen WebGL game developers, to bring browser latencies (buttons-to-pixels) down. Ideally, this should be made possible in both canvas2D and with WebGL.

Many displays support other refresh rates, such as 75Hz. Also, there are many displays other than 60Hz, including gaming displays (120Hz, 144Hz). And VR displays such as Oculus and HTC Vive operate at 90Hz which Chrome supports (360 degree YouTube videos). Also, future TV standards (e.g. 8K 120Hz) may introduce 120Hz as a more standardized refresh rate in coming years.

It is extremely important to standardize how animations behave at refresh rates other than 60Hz, as synchronizing animations exactly to refresh rate is important, regardless of refresh rates. (Even 115fps @ 120Hz results in unsmooth motion), but also still provide an API to optionally decouple update rates from display refresh rate.

Browser comparison on refresh rate support:
www.testufo.com/browser.html

Browser-based motion tests for analysis (test these at refresh rates other than 60Hz):
www.testufo.com
www.vsynctester.com
www.testufo.com/photo
www.testufo.com/eyetracking
www.testufo.com/animation-time-graph

• NOTE1: Don't forget to consider there are multi-monitor desktop computers. Different refresh rates for different monitors is possible. In other words (for multi monitor systems), the refresh rate of the specific display that the browser window is running on. Windows already changes the desktop manager update rate everytime the window moves to a different-refresh-rate display. Some browsers adapt correctly (automatically changes callback rate of requestAnimationFrame() when this happens) when windows are moved, but others mess up (suddenly goes janky / stuttery).*

• NOTE2: Don't forget emerging variable refresh rate displays including G-SYNC, FreeSync, VESA Adaptive-Sync, and HDMI 2.1 VRR. (The next XBox supports variable refresh rate -- Microsoft confirmed) Instead of software waiting for the display refresh, the display waits for the software. The display immediately refreshes the frame delivered by software -- so refresh rate runs at frame rate. E.g. If you get 53fps, the display is 53Hz. Etc. The refresh rate of the display changes dynamically dozens of times per second to exactly match the refresh rate to the framerate, for stutter-free changes to framerate (simulated animation -- test this on Windows/Mac with all other browser tabs closed for the most accurate visual simulation). Eventually, there may need to be a long-term path for browsers becoming "VRR-aware" at least in full-screen mode, very useful for WebGL 3D games.

• NOTE3: WebGL games can use wglSwapIntervalEXT(1) and glSwapIntervalEXT(0) to enable/disable synchronizing to VSYNC. But, they are still hampered by the latency of the window compositing manager. Most operating systems permit a full-screen mode that bypasses windows compositing delays, so this could be a feature (flag). In this particular case, if you both (A) bypass compositing and (B) use glSwapIntervalEXT(0) ... for WebGL applications, you'd get the 'tearing' effects (while gaining useful "eSports-friendly" latency-reducing effects) of running framerates far higher than refresh rate.

For wording fixes to requestAnimationFrame() rate, see #785.
(This is related wording clarifications to existing VSYNC-driven requestAnimationFrame() -- but this is separate from the additional features listed here.)

VSYNC API Requirements For HTML 5.2 or HTML 5.3

Modifications to a VSYNC API needs to solve the following, at least several or all of:

• Ability to detect whether requestAnimationFrame() or canvas (WebGL, etc) is dropping frames
  In other words, frames missing the current refresh cycle of the monitor. Aka dropped/skipped/throttled frames during refresh rate synchronization. Many games and certain kinds of applications need self-awareness to know if frames have missed their refresh rate. Websites created by multiple people including www.testufo.com (my creation) and www.vsynctester.com (@duckware creation) use useragent detection & heuristics. The heuristics is needed to make a guess whether or not a frame was missed. And to make a wild guess whether requestAnimationFrame() is running at a rate different from the display refresh rate. Understandably, this is kind of a witchcraft of clever coding that should be avoided. The existence of this API would also double as confirming that the browser supports VSYNC (currently www.testufo.com/browser.html uses useragent detection to achieve the same thing).
  NOTE: An accurate VSYNC time in requestAnimationFrame() callback parameter is another way to allow useragents to detect dropped frames.

• Ability to enable framerates far higher than refresh rate
  Basically, a JavaScript method of being able to enable/disable the equivalent of Chrome's "--disable-gpu-vsync" flag. At the absolute minimum during full screen mode, maybe as an additional parameter for the existing HTML fullscreen API. This is very important for two emerging major things:

• • REASON 1 -- Letting framerate vary as performance allows, makes things more compatible with future VRR displays. Letting browsers decide to render as fast as possible, and then letting the display automatically decide to raise/lower refresh rate to match the current render-rate / update-rate. Browser vendors can then decide whether or not they want to support the emerging variable refresh rate displays such as G-SYNC / FreeSync / VESA Adaptive-Sync / HDMI 2.1 VRR.

• • REASON 2 -- JavaScript developer access to access an ultra low latency mode for gaming. In eSports gaming (where crazy gamers play CS:GO at 1000fps on NVIDIA GTX Titans) -- the use of 1000fps@60Hz has 8 milliseconds less input lag than 60fps@60Hz. This is consistent with experience by gaming authors. Also, latency were recently done in "--disable-gpu-vsync" and some GPUs such as NVIDIA GTX1080 was able to run the web browser at framerates in excess of 1,000 frames per second on several test pages. This means browser performance is reaching eSports territory. High speed video / input latency benchmarks showed less lag with buttons-to-pixels when framerates far exceed refresh rate. This is because at 1000fps, the frame seen by eyes was rendered only 1ms ago -- the overkill framerate meant that whatever hit the screen was the freshest possible pixel, thus reducing mouse lag & keyboard lag for specialized low latency applications. In fact, in the billion-dollar eSports gaming industry -- Bloomberg News, Forbes, New York Times. I have attended some of these events myself and noticed that more than 90% of paid competition game players use VSYNC OFF (disable GPU vsync) during gaming such as Counterstrike:GO, etc. If we were able to enable Javascript APIs to do this, browsers would more easily (over long term) enter the niche market of olympics-style "milliseconds-matters" races of tight reaction times and simultaneous-draw situations.

• Ability to query whether or not Windows' compositing is bypassed
  Full screen mode can bypass window compositing manager (adds 1 frame of lag -- 16ms). This is important for reducing latency during high-precision applications, eSports gaming applications, Mac can do compositing in sub-frame latency (Safari has 1 frame of lag), while Windows enforces an additional wait (Chrome has 2 frame of lag). Being able to let applications query for this lag, would enhance latency awareness for web applications, and allow "eSports-ready" benchmarking in pure Javascript. If compositing is false then "tearing artifacts" will appear during VSYNC OFF (--disable-gpu-vsync) which is normal. Tearing artifacts look like this (simulated). Generally, the higher the framerate (e.g. 1000fps@60Hz), the less visible tearing artifacts becomes during true compositing-bypassed direct-to-display VSYNC OFF operation (eSports league lowest latency mode).

• Ability to to return the current maximum refresh rate of the display that the browser window is currently on
  While this can be roughly achieved by counting requestAnimationFrame() (except on IE/Edge -- see Strengthen requestAnimationFrame() spec for VSYNC / refresh rate matching in HTML 5.2 #785), there should be an easier mechanism to query. Should ideally be must be multimonitor aware -- different displays can run at diverging refresh rates. (As one example, windows's desktop manager changes update rate to the refresh rate of the monitor that the largest surface area of the window is on). Several browsers have very flawed behaviour on multimonitor systems with divergent refresh rates (including systems with two different 'kinds' of displays: A 60Hz monitor and a 90Hz Oculus VR goggles). Divergent refresh rates are still niche but getting common enough to merit departure away from the "one-refresh-rate" assumption prewritten into earlier standards. Also, if querying the refresh rate of a VRR display (Variable Refresh Rate such as G-SYNC, FreeSync, VESA Adaptive-Sync, and HDMI 2.1 VRR) -- that automatically matches refreshrate to current framerate -- then it should return the maximum refresh rate that the current VRR display is capable of.
  (NOTE: This one might be a reasonable substitute/alternative to missed-frame detection. Knowing the refresh rate means we can count frames per second within a browser page, and compare to refresh rate -- although it's possible to have a smooth 60fps@60Hz and a stuttery 60fps@60Hz when frames don't exactly line-up with display refresh cycles)

There are other APIs that may be needed, but this is the type of "update-rate and latency knowledge" that high-performance JavaScript applications need. This will be required for both WebGL and non-WebGL modes (e.g. HTML5 canvas).

UPDATE: I am now an Invited Expert in W3C Web Platform Working Group. I welcome gaming industry programmers / those in this industry to collaborate on this topic. You can follow up here, but I can also be directly reached at mark[at]blurbusters.com for ideas, suggestions, and technical discussions if you're not a github member. This VSYNC API text is also an article on BlurBusters.

Coincidentially, the Chrome team just targeted 90fps for viewing via Oculus VR goggles;

http://arstechnica.com/gadgets/2016/05/chrome-dev-asserts-browser-is-viable-vr-platform-targets-90-fps-rendering/

Tests on my systems using new modern built-in GPUs (Recent 2015-era and later IGPs ... ordinary builtin GPUs, not even Radeon/NVIDIA highend stuff!) is now able to run Google Earth (angled 3D view) at ~100 frames per second in WebGL in most web browsers. So refute any claims of "there's not enough performance"... On my now-midrange GPU (3-year old GeForce GPU), I'm able to achieve 144fps @ 144Hz with Google Earth at 1920x1080p, in most typical not-too-steep-angled Google Earth 3D views.

Considering some browsers, e.g. EDGE, hardcode to 60fps, even at higher refresh rates, it is critical that browsers SHOULD NOT use hardcoded frame rates, and MUST provide a way to synchronize to VSYNC.

Also, browsing SHOULD try to provide an optional method of reducing latency:
-- Ability to turn off VSYNC if allowed by the display (for low-latency gaming, at exchange of gaining tearing artifacts). This is common in many PC video games.
-- Bypassing display compositing, to reduce scripting-to-display latency. Many full-screen video games use a native full screen mode that has one-frame-less latency (e.g. 1/60th second faster at 60Hz) than running in windowed mode. Low latency mode should be up to browser implementers to implement, but there SHOULD be provisions to cover this in browser standardization. e.g. Going into full screen mode (F11) should optionally try to automatically use the operating system's lowest latency mode if possible where not compromising user experience
-- Alternatively if behaviour differences warrants it: Provide an HTML API to provide an optional flag/argument to JavaScript .requestFullscreen() such as .requestFullscreen("gaming") .... since some OS low latency full screen modes often changes behaviour such as disables overlapped windows (popup windows) when running in this type of full screen mode.

[joeblew99]

Vsync support would be very useful.
Its one of the reasons I can't recommend building a graphics intensive project using pure web technologies at the moment.

---copied from requestAnimationFrame() thread #159 since it is extremely useful here too .... They seem distinct, because it seems to be a decision between revert #159 (keep requestAnimationFrame) or go with #375 (create a new VSYNC API).

For those testing:

Note to those people "assuming 60fps is enough" (in a "640K ought to be enough" fashion):
There is a lot of incorrect assumption that the human eye can only see 60 images per second when it is not really that simple -- lots of /indirect/ artifacts can occur that are still visible at far over >200fps@200Hz --
(1) For example, motion side effects like eye-tracking-based motion blur caused by a non-infinite refresh rate (e.g. demo at www.testufo.com/eyetracking ...). (Note: I'm the creator of all the TestUFO.com motion tests, including this finite-refresh-rate-induced optical illusion) -- it is a great example of an indirect side effect caused by the finiteness of the refresh rate -- and behavior changes at 60Hz, 120Hz, 144Hz 165Hz...
(2) Vision researchers has long found this out, and a great article is written by VR authors -- Oculus' Michael Abrash, "Down the VR Rabbit Hole: Fixing Judder" at http://blogs.valvesoftware.com/abrash/down-the-vr-rabbit-hole-fixing-judder/ -- demonstrating discrete-refreshrate-related artifact still shows up in some situations even at 1000fps@1000Hz ... Trying to perfectly replicate a Holodeck needs an elimination of finite-refreshrate-induced optical illusions, for 100% perfect immersion, and it's a quite difficult thing to achieve...

I frequently reported to FireFox and Chrome about refreshrate/framerate bugs:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=(site:mozilla.org+OR+site:chromium.org)+(rejhon+OR+%22blurbusters%22)+60+120++

A popular animation test pattern used by browser programmers (at least the FireFox and Chrome teams) for requestAnimationFrame() debugging is this one http://www.testufo.com/animation-time-graph --

It uses Javascript High Resolution Time http://www.w3.org/TR/hr-time/ to benchmark the time between VSYNC intervals (by using requestAnimationFrame() as the signal-tick of VSYNC) ...

Two to three years ago, the Chrome programmers obtained a gaming monitor (capable of multiple refresh rates up to 144Hz), and fixed many refresh-rate and fluidity problems, and to this date, Chrome has been the favourite web browser for smooth motion.

Fluidity tests:
Also, here are web browser fluidity comparison benchmarks (how long browsers stay smooth with increasing screen rendertimes):
http://www.blurbusters.com/blur-busters-120hz-web-browser-tests/

These were done by using these tests to enable a hidden "Delay" setting.
http://www.testufo.com/#test=animation-time-graph&easteregg=1
Increasing the delay causes the javascript to do a busywait-loop for the specified number of milliseconds before beginning to paint. It simulates longer-than-expected rendering (e.g. intensive 3D graphics). At the time, Chrome maintained perfect fluidity longer than most of the others.

Hope this helps in motion fluidity testing!

P.S. OpenGL already provide options to synchronize to VSYNC (OpenGL "swap interval", which WebGL also seems to use) but not everyone wants to do animations using WebGL. So what do we do, when requestAnimationFrame() #159 disappears?