Windowing API

[Serentty]

Forgive me if there is already an issue for this. I gave it a quick search before opening this.

So, in the long term, after more core OS functionality has been covered, I think it would make sense to have some sort of API for creating and manipulating windows, so that WASI can be useful for user-facing applications. This would only cover the windows themselves, not widgets inside of them, as that's something that can easily be done at a higher level by something like GTK or Qt, so this wouldn't be a colossal project on the level of Swing. At the very most, this API would offer GDI-like drawing, but perhaps even that would be better left out in favour of a more general graphics API à la #53.

[sbc100]

Seems reasonable in the long term yes. I guess we would want to expose something minimal that things like https://flutter.dev/ can then be built on top of.

[aardappel]

Pretty much every device that is user-facing has a GPU nowadays, so instead of "GDI-like drawing" it should use whatever GPU API(s) end up in WASI (see discussion here: #53), and consequently, any windowing API should be whatever is needed to support that API (similar to e.g. egl, wgl), if not already included in that API. Though that probably doesn't include windowing based input and events, which may need to be considered additionally.

[Serentty]

I think including some sort of immediate-mode fallback might be a good idea. Although GPUs are nearly ubiquitous, it's also a common situation to have access to that GPU gimped in some way, such as inside of a virtual machine. I wouldn't be distraught to see this left out, but it's something that I think would help ensure that it's a well-rounded solution.

Though that probably doesn't include windowing based input and events, which may need to be considered additionally.

Yep. I unfortunately don't have much experience here, but I imagine software would need to know things such as the size of a window, where the cursor is and when it clicks, which virtual desktop the window is on (since even Windows supports this concept now), and stuff like that.

[olanod]

I was thinking that maybe we don't need a windowing API like in the web 🤔 what about keeping it low level and up to the run-time/compiler how to deal with windows if there's need for windows at all.

I'd like to see a frame buffer API some time, for embedded contexts for example a buffer that I fill with some pixel information(e.g. a micro writing B&W pixels of a small e-ink display) is all that would be needed I guess. The frame buffer can be a wasi file descriptor, perhaps the user can request/create more than one buffer or is given one by the host, then with existing APIs something like window resizing could be detected by listening to events on the file descriptor, a __WASI_EVENTTYPE_FD_READ could signal that the buffer changed(window resized?) a __WASI_EVENTTYPE_FD_WRITE could be the requestAnimationFrame of WASI world.
Once the user renders pixels on the buffer by hand or with a graphics API like webGPU then the host decides what to do with that, could be a headless set up where the frame-buffer is dumped to a file or telling the compositor of the platform hey here are some pixels!, put it in the window you gave me!(window was created by the run-time host, not the user?).

[Serentty]

@olanod I think a concept of a window is still necessary. Leaving it up to the runtime isn't sufficient. A lot of programs create multiple windows. The web doesn't have this because it started around the concept of a document, but I don't think we should be imitating the web here. Plus, a big goal of WASI seems to be the ability to recompile existing native software. Without a windowing API, that just isn't possible.

[programmerjake]

Embedded systems can support only one full-screen window and fail to create a new window if it's not fullscreen, the correct size, or there's already a window. alternatively, it can work like Vulkan's VK_KHR_display extension in that you can create surface handles to displays directly and start displaying stuff. maybe some hybrid scheme will work where you can use one of two methods to create windows: creating a standard window or creating a display-as-a-window which has exclusive access to that display. some systems would only support one or the other method:

• highly sandboxed (like on the web): only support creating windows
• embedded: only support display-as-window
• desktop apps: support both (maybe)

[Serentty]

The API is designed to be modular, so I don't there's harm in designing a windowing API that assumes a full desktop, and having a more limited exclusive API also be available for embedded devices (which desktops might as well support too, for the sake of games and the like).

[NotWearingPants]

We will need to have "onfocus" and "onblur" for each window, and maybe "onclose" if we want to allow the user to close individual windows, or not if only the program could close its own windows.

Also be able to provide a name for the window (which implementors might choose to display as a title).

[doug-moen]

I suggest looking at the GLFW library (https://www.glfw.org) as a starting point for a WASI windowing API. It is mature, widely used, cross-platform. It only covers window creation and input event handling. There is very little in GLFW that outside the scope of a WASI windowing API. Drawing inside a window is out of scope, and must be handled by a separate graphics API.

Within that restricted scope, many important platform details and variations are supported using portable abstractions, such as multiple windows, multiple monitors, high DPI; keyboard, mouse, touchpad and gamepad input; window events similar to the onfocus, onblur, onclose suggested by @NotWearingPants. It's a lot of work to figure out a good design for all of this from scratch, so let's begin with a proven design.

GLFW provides portable abstractions for managing windows on a desktop operating system (windows, macos, linux, freebsd). It doesn't support mobile operating systems (android, ios). If we want to support mobile user interfaces, then we need to support touch-screen input events, and sensor input from the accelerometer and gyroscope. The SDL library (which competes with GLFW) has these: https://www.libsdl.org

[aardappel]

@doug-moen I'd second starting with SDL instead. Like you say, it has wider platform support, is more widely used, already comes out of the box with Emscripten (some overlap with other Wasm ecosystems might help users :)

The design of SDL is also more low level (C style) than glfw, which I think is appropriate for WASI.

[MarkMcCaskey]

The interface with hardware (user device and gamepad support) seems quite tricky... I believe it requires a lot of device-specific code. Actually I just found out that there's a W3 working draft about a gamepad API from 2 days ago! https://www.w3.org/TR/gamepad/ that could be relevant here.

In general we should try to stay close to what browser APIs do when we can because they're dealing with a lot of the same portability and sandboxing issues and they have sufficient scale that people care about implementing their standards.

[doug-moen]

@MarkMcCaskey If that gamepad API becomes a web standard, then WASI should support that API, both in browsers and on desktop systems. Which means it should be separated out from the Desktop Windowing API, as I assume that only works on desktop operating systems.

[linclark]

I'm going to close this one out since it won't be actionable until WASI's core APIs are more fleshed out, and we won't want to delve into too much discussion on this topic before that because developments between now and then are likely to change the approach we take.

[yavko]

I just wanted to hop in and say, a implementation of the wayland protocol could work really well with the WASI, and WASM architecture, and it would be cool if this issue could be re-opened.