Hosting Avalonia in WebAssembly

[itsamelambda]

Would it be possible to host Avalonia with WebAssembly and .NET in a browser?

Then we could write the same C# code and run it native in all browser and on all desktops.

[kekekeks]

It would be possible and we are currently considering our options. The major blocker is that HTML5 doesn't have a proper 2D drawing API, so Skia needs to be compiled to wasm and somehow linked to resulting application. The problem with linking is that Mono doesn't currently provide P/Invoke support for wasm target. Another problem is that P/Invoke means indirect calls (i. e. calls via pointer), while WebAssembly doesn't support indirect calls between modules. That means that Skia needs to be linked statically, so we'll need a custom Mono build.

It would be easier with CoreRT, since it supports static linking out of the box, but they are still implementing MSIL instructions for wasm target, so it's not ready even for an experimental port.

[legistek]

Sorry if this is an obviously terrible idea, but why wouldn't you use WebGL?

[danwalmsley]

Maybe this helps? https://github.com/aspnet/Blazor/blob/dev/README.md

[kekekeks]

@pmoorelegistek We will be using WebGL at some point. The problem is that WebGL by itself doesn't support drawing 2D vector graphics. So we need Skia anyway.

[legistek]

Oh I see what you're saying. I didn't realize it lacked basic 2D drawing. I take it the plain HTML5 canvas doesn't have good enough performance either? I've been watching this project for awhile and would love to help how I can. Getting this on the web would be the Holy Grail. (Befitting the name Avalon(ia)) Get Outlook for Android<https://aka.ms/ghei36> From: Nikita Tsukanov Sent: Friday, March 9, 12:30 AM Subject: Re: [AvaloniaUI/Avalonia] Hosting Avalonia in WebAssembly (#1387) To: AvaloniaUI/Avalonia Cc: Peter N. Moore, Mention @pmoorelegistek<https://github.com/pmoorelegistek> We will be using WebGL at some point. The problem is that WebGL by itself doesn't support drawing 2D vector graphics. So we need Skia anyway. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1387 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMw7S1GOycAJX3aLJnlzy5xM6yq_KrTiks5tciGKgaJpZM4SR4pe>.

[legistek]

Hmm. The problem with compiling Skia to WASM is you'll lose any benefits of hardware graphics acceleration on the device. Browser will be doing all the math, painting pixels, etc. I don't know that you'd ever get anything close to 60fps with a complex UI.

Have you tried just using the HTML5 Canvas or SVG?

[kekekeks]

Skia can use OpenGL ES profile for hardware-accelerated rendering. Which is basically what is provided by WebGL.
HTML5 Canvas and SVG are not sufficient to render Avalonia.

[legistek]

You're saying Skia / OpenGL will be able to access hardware acceleration (i.e. the GPU) after compiled to WASM? Are you certain about that? Everything I understand about WASM would lead me to think not.

I would think you'd be better off using three.js - or better yet porting it to C# - wrapped around WebGL.

I'd be happy to look into it some more and experiment. Do you have a working prototype using Mono-to-WASM at the moment with any rendering at all?

[kekekeks]

You're saying Skia / OpenGL will be able to access hardware acceleration (i.e. the GPU) after compiled to WASM

Emscripten toolchain has complete support for WebGL. It might be a bit difficult to make it work with Mono since it has it's own js glue code, but other than that I don't see any issues.

[danwalmsley]

https://github.com/praeclarum/Ooui/wiki/Xamarin.Forms-with-Web-Assembly

No need for Skia now,

[kekekeks]

@danwalmsley it uses HTML. Avalonia can't be rendered with HTML.

[adoris]

Is there are any try to use Skia/WebGL? Any branch for this experiment?

[legistek]

if I understand right, what @kekekeks is saying is that it's not feasible at the moment because Mono can't use P/invoke to call the Skia functions. So while they can compile Skia to WASM, there would be no way for the .NET code to access it.

What I don't understand fully, though, is why it has to be Skia. There are Javascript libraries that provide 2D drawing function wrappers over WebGL (e.g., https://threejs.org/). I'd think it at least worth an experiment to make a DrawingContext that wraps three.js and go from there. If that works, porting threejs to C# and simply including it in the Avalonia build would not be too bad of a project and would probably get you performance on par with Skia.

[kekekeks]

@pmoorelegistek The main problem is the complete lack of a proper text measurement API. We can't get our TextBox to work if we don't have an accurate position of each individual symbol. Even for our TextBlock we need to be able to measure and lay out text lines properly. There is no way we can do that with the APIs available to JavaScript.

[yowl]

I added an issue for pinvoke support on mono so this can be tracked. mono/mono#8007

[TonyHenrique]

See
http://www.noesisengine.com/webgl/Samples.Buttons.html

[legistek]

It looks nice but what are we looking at exactly?

…

________________________________ From: TONY HENRIQUE <notifications@github.com> Sent: Thursday, April 12, 2018 6:38:15 PM To: AvaloniaUI/Avalonia Cc: Peter N. Moore; Mention Subject: Re: [AvaloniaUI/Avalonia] Hosting Avalonia in WebAssembly (#1387) See http://www.noesisengine.com/webgl/Samples.Buttons.html — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1387 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMw7S61hESqCLDWhVkG6jvegNBn7KHWtks5tn-VngaJpZM4SR4pe>.

[kekekeks]

Noesis has it's own rendering engine, if I recall correctly. A proprietary one. We don't have the resources to create a whole new 2D graphics library, so we are using an existing one, which is Skia. Until we have a way of linking it in on wasm target (be it Mono or CoreRT), we won't be able to get Avalonia working in the browser.

[legistek]

Hey Nikita - since this thread keeps coming alive -
I looked deep into three js and the code for text rendering could easily be adapted for text measuring. I'd be willing to make the modifications if you're willing to consider something besides skia.

And beyond that, I don't think a port of three js to C# - or at least enough of it for our purposes - would be at all that difficult.

This is my company, and as you can see from our product we have a vested interest in your project: www.legistek.com

[kekekeks]

Well, you see, the only text measurement API that I've found for HTML5 Canvas is measureText call which returns a TextMetrics object. The problem is that it doesn't support anything but text width:

As you can see, "advanced" properties are only supported on Google Chrome and need to be explicitly enabled in browser settings. Getting text width is unfortunately not enough for implementing our IFormattedText interface.

[kekekeks]

Does three.js have it's own rendering engine that can draw text using TrueType/Freetype fonts with proper subpixel rendering?

[kekekeks]

If it does, it might be worth to create some kind of a wrapper, but currently calling JS code from WASM is very inconvenient.

This is currently the only way of calling JS code from C#, which is basically an eval call.

There is some existing glue code here (JS part here) which uses the internal call from an alternative mono wasm target implementation, which is basically an eval too, so it could be adapted.

[legistek]

https://github.com/mrdoob/three.js/blob/f81506e172571ab106d0164530bbc1a4802fc2d4/src/extras/core/Font.js

They are going through the font glyphs and turning them into path geometries and then feeding them to WebGL. If they can do that then they could have measured them. It looks like nothing more than an oversight that they didn't include measuring functions. (I'm not yet clear on what font format they're expecting but my assumption would be WOFF).

re JS interop, again I would assume the endgame would be porting what we need from three.js to C# so that all the calculating code would be done in WASM and the only time you'd need to interop would be the final render into WebGL. But why not try a POC at least?

[kekekeks]

@pmoorelegistek It seems that Mono WebAssembly SDK is providing libmono and an example C source file with the entry point. That means that we can link additional libraries to the resulting runtime mostly without issues. mono-dl-wasm.c file still needs to be changed for supporting __Internal but at least we could use existing build scripts

[jmacato]

@kekekeks I'd like to bring this lib to our attention: SharpFont; could be useful for text rendering, alongside the PixelFarm renderer that i linked before.

I just noticed that PixelFarm is already using this particular library, so yeah

[Gillibald]

We can measure text etc but we can't apply kerning or any other glyph positioning. Everything else is possible.

[kekekeks]

let the Browser do all the shaping again. This will most likely produce decent results but uses more resources.

The problem is that it is likely to produce different results. Which will break the layout.

[Gillibald]

It should not produce different results in size if the browser follows common standards. The only difference is the quality of rendered glyphs.

[legistek]

It should not produce different results in size if the browser follows common standards
Well, what browsers SHOULD do and what they DO do is not always the same.

Anyway if I'm understanding correctly the challenge is to make sure that text measurements in the layout pass match what the renderer actually produces. But this should be the case if the renderer and the measuring system are part of the same codebase. Thus, if you used HTML+CSS to render, and canvas measureText to measure, you should get agreement between the two. Likewise if you used WebGL + ThreeJS (or even Skia) to both measure and render text, you should get agreement.

The layout pass in C# would still need interop with JS to measure strings in either case, but some clever caching of string and font property combinations along with the fact that full layout passes don't happen terribly often should get you where you need to be.

[legistek]

Also just an FYI, invoking JS from WASM does NOT require an async call in the Mono runtime, even though the current version of Blazor does. Earlier prototypes of Blazor allowed synchronous invocation of JS from C#. (I believe they changed this so that Blazor apps could be transparently switched between running server side - which of course requires an async call - and client side). So you can implement a JS-based text measuring method in your drawing context without having to rewrite everything to be async.

[kekekeks]

JS-based text measuring

The problem is that there are no text measurement APIs on JS side. At least there weren't when I've last checked.

measureText from canvas is simply not sufficient

[legistek]

measureText from canvas is simply not sufficient

Well, there's a lot more you can do in JS to get the text measurements you need. measureText is one of many tools.

The problem is that there are no text measurement APIs on JS side. At least there weren't when I've last checked.

Take a look at three.js's code for rendering text from TTF or WOFF:

https://github.com/mrdoob/three.js/blob/37285df967f0a4d0b1578703e665b35df04361c2/src/extras/core/Font.js

This could easily be expanded to measure rather than just render.

[Gillibald]

We don't gain much from that. We already have the ability to generate svg data for a given string. This will still not produce pixel perfect results.

I think we can work something out with my new text layout that produces GlyphRun objects under the hood. We then just transfer these GlyphRuns to the Javascript side and let it process there again. We can apply same font features etc. but have to rely on the browser's shaping capabilities to produce the same results. In reality this will not be an issue because every browser uses HarfBuzz as well.

https://github.com/Gillibald/Avalonia/blob/080084f14ae6aa367ac825b4199c67a09de86620/src/Avalonia.Visuals/Media/GlyphRun.cs

A lightweight version of this will just contain the characters and info about font properties.

[legistek]

@Gillibald so you'd measure in C#, pass just the text and font info to the browser, and pray it measures the way you assumed it would?

The only way you can guarantee agreement between the measurement and the rendering is for both to be done by the same code.

And since you have to minimize the amount of data moving between C#/wasm and JS, that means the only viable solution is to do both measurement and rendering browser side.

[Gillibald]

Rendering only differs on sub pixel level. Measures are defines by the data that a font defines. The only possible difference in measurement comes from different shaping. So for example our shaping engine selects a ligature for specified text but the browser doesn't. This will measure differently because we use one glyph but the browser would use two glyphs. If we would let the browser measure everything we would need to cross Javascript / Wasm interop two times. Also the amount of data would be huge. Because we would need to transfer measures of each glyph.

[legistek]

Measures are defines by the data that a font defines. The only possible difference in measurement comes from different shaping.

In a perfect world maybe. Work with browsers much? :/

If we would let the browser measure everything we would need to cross Javascript / Wasm interop two times. Also the amount of data would be huge. Because we would need to transfer measures of each glyph.

Cache measurements by font+size+style+string.

[birbilis]

Can't you use a JavaScript-based text rendering engine or something so that you can control what it does? (or a more general 2D rendering engine that also supports text like https://two.js.org/examples/)

[rstm-sf]

mono/SkiaSharp#1219

[aboimpinto]

Uno Platform (https://platform.uno/) is rendering the WPF from Microsoft to WebAssembly. There should not that difficult to port the code to render Avalonia too ...

Maybe, there could be a way to join efforts ...

[Gillibald]

Uno does not render anything as Avalonia does. They use HTML elements.

[yowl]

They also have a Skia backend, not sure it's generally available for Wasm yet, but it is the plan.

[birbilis]

related comment:

On Windows 7, we still have the WebAssembly story, but we also announced a new Skia backend, that can be wrapped in a WPF shell so you can run "natively" and outside of a browser context.

https://developercommunity.visualstudio.com/idea/378455/net-ui-standard.html

also:

When using Uno Platform to get your UWP/WinUI 3.0 and later apps on Windows 7 you now have THREE options for running your app on Windows 7: 1. Run your app as a PWA. Your Windows 7 will need Chrome installed on it. 2. Run your app on Chromium through Electron. 3. Use Uno Platform 3.0 supported Skia backend, that can be wrapped in a WPF shell so you can run natively on Windows 7 and outside of a browser context.

https://platform.uno/windows7/

[aboimpinto]

What about Ubuntu (Linux environment)?

[birbilis]

regarding Uno and Linux see here: https://platform.uno/uno-platform-for-linux/

[aboimpinto]

regarding Uno and Linux see here: https://platform.uno/uno-platform-for-linux/

I was referring Avalonia in Linux environment compiling into WebAssembly

[birbilis]

guess Avalonia could also get ideas from Uno, or maybe even use it as a backend or something?

[Gillibald]

Just to be clear. Text rendering is no longer an issue. The "only" thing we need is a platform implementation for WASM that is setting up WebGL and exposes all input events. The threading implementation might also be tricky.

[congzhangzh]

May be we can learn from google flutter, skia should not be a problem!

[starikcetin]

I don't know if it is of any help, but there is this: lume/glas: WebGL in WebAssembly with AssemblyScript

[mikernet]

Seems skia with p/invoke is possible now:
https://visualstudiomagazine.com/articles/2021/10/15/aspnet-update.aspx

[artemiusgreat]

This, to be exact.
mono/SkiaSharp#1811
https://github.com/mattleibow/SkiaSharpBlazorWebAssembly