Evaluate options for core GUI

[hannobraun]

The current GUI infrastructure infrastructure is very basic, and extending it is probably not desirable. What is desirable, is to integrate an existing solution. I have my sights on egui, since it looks like it could meet all our needs and more, and is specifically designed for being integrated into existing render pipelines.

The objective of this issue is to evaluate egui (and possibly other options), by integrating it into our rendering pipeline, and replacing the existing UI elements (as of this writing, the only UI elements are the 3 lines of text that indicate how to enable/disable the different drawing options). If this evaluation is positive, the code should be merged.

Labeling as https://github.com/hannobraun/Fornjot/labels/good%20first%20issue, as this issue only requires familiarity with egui (or another solution being evaluated) and Fornjot's graphics code (which is wgpu-based). It doesn't require any deep expertise of Fornjot itself.

Further notes

Please note that the scope of this issue specifically is about a "core" GUI, that is integrated with our graphics code and needs to be available wherever that graphics code runs. Example use cases would be the configuration of drawing modes, visualizing the structure of the model, selecting different aspects of the model to render, etc. There might be a need for a less core, platform-specific GUI solution (for example, an "open file" dialog to select a model to load, before Fornjot proper is even launched). This is tracked in #117 .

This issue is a more specific follow-up to #4.

[valpackett]

Looks like egui only has OpenGL backends? (glow and glium in the repo) iced supports wgpu, so you could avoid depending on GL.

But to be honest I'm not a fan of the "core GUI" concept, my ideal dream structure is:

• common backend library — basically CAD kernel but higher level (with sketches, a common project format, etc.)
• headless CLI (with zero GUI dependencies, quickly installable in CI environments) that can at least produce a mesh from a project
• any number of GUIs of various complexity, from a simple viewer that could be embedded on a web page to a full "IDE" similar to OpenSCAD's GUI (made with GTK to feel native on the desktop, etc.)

[hannobraun]

Hey @unrelentingtech, thank you for your comment!

Looks like egui only has OpenGL backends? (glow and glium in the repo) iced supports wgpu, so you could avoid depending on GL.

I am pretty sure I saw a wgpu backend somewhere, but I don't remember where. My impression is that integrating egui into existing backends is a core feature. Depending on GL (or anything else that would somehow work in parallel with wgpu) would definitely be a non-starter.

Thanks for reminding me of iced. I had forgotten about it, and I think in the past I got the wrong impression that it is not suited for this kind of use case. But judging from the feature list, integration with existing systems should be possible, so it's definitely a contender here.

But to be honest I'm not a fan of the "core GUI" concept, my ideal dream structure is:
[...]

I should clarify that this issue is meant to track an actionable task that can be implemented in the near term, and doesn't necessarily reflect the long-term vision. Just a change that makes sense right now. And right now I see the need for a platform-agnostic GUI that is available everywhere the renderer is. Something for implementing sliders to change model parameters, for example, that works regardless of whether you're creating a model on a Mac, or viewing a website that has a model embedded in it.

Whether that concept will endure long-term, I don't know. It might, or it might turn out that authors of those various GUIs want more freedom, and the core GUI is in their way. We'll see.

That dream structure you describe is definitely where things are going. Right now, everything is bundled up in a single host application, for the sake of developer convenience. But I absolutely do intend to split the project into reusable components. I want Fornjot to spawn an ecosystem of related tools.

That reminds me, I should open an issue about splitting the project into smaller modules. This is definitely something that can happen in the near-term, as key components like the kernel and the graphics already sit in their own module trees and have clearly defined interfaces between each other. I'm just a bit worried about doing it before there's a need, as it's going to introduce some friction into the development process.

[hannobraun]

As a follow-up to my previous comment here, I've opened an issue about splitting Fornjot into self-contained components (#141).

[follower]

[One more drive-by comment from me after seeing Fornjot on HN again. :)]

My experience of egui

I've used egui for a handful of projects & with a couple of different integrations, so thought I'd share some comments based on my experience.

Context

For some time lack of an obvious GUI toolkit was a barrier for me to adopting Rust for more projects & I first encountered egui (about a year ago? Seems longer, lol.) after considering a few of the other options which hadn't appealed.

I'd say my current assessment of egui is that I've made GUI-based Rust applications since I started using it & that's what I wanted to do, success! :D

Comparison with other GUI toolkit options

While the immediate mode nature of egui has pros & cons it does lend itself to "hacking something until it works" which is a positive experience for me. :)

(Disjoint capture in closures was a huge win in DX given how egui UI is constructed.)

My overall view is that egui is very much a "first step" in the application UI path & I can see definite potential for other crates with more sophisticated/aesthetically appealing approach but currently it's unclear "who's going to win" amongst the many in-development options out there which makes choosing one challenging. (I also think there's potential for more elegant skinning/themeing which hasn't really been explored yet.)

There are some UI layouts which Immediate Mode isn't capable of with a single-pass approach but that's also still an area of research for egui.

But my impression is that it's certainly suitable for presenting Fornjot's current GUI. :D

Project health

On a wider project basis, from my observations the primary developer is doing a good job of managing things & is pleasant to interact with which is always a plus. :)

As the project is under very active development there are a reasonable number of breaking changes occurring in some releases but this is offset considerably by the excellent changelogs/migration guides included with each release. Seriously, they're amazing, e.g.: https://github.com/emilk/egui/blob/master/CHANGELOG.md

There's also a growing community around the project & it generally seem to be growing in terms of being "a strong choice for an Immediate Mode UI solution for Rust".

(This seems like a helpful overview of how the different egui project crates related to each other.)

Potential for Fornjot to use egui

You're correct that egui (the "library") is designed for integration with existing application systems (as it doesn't render anything by itself) but there are also a number of "official" & 3rd party integrations & a companion eframe crate which is the "framework" option if you just want to slot into an existing application skeleton.

Based on my experience from a recent project I'd say that the egui render/platform backend frameworks are more suited to being the "top" of the application & providing a way to display your rendered output (via "native textures" or the upcoming callback functionality). (As opposed to rendering themselves on a surface/texture provided by you--it's possible but you'll probably have to do more of the setup work that integrations normally do.)

But that's probably not a huge issue given you already have a functioning application anyway which I'm assuming you'd probably rather integrate egui into rather than the other way around.

State of egui & wgpu support

The official integrations currently cover glow, Glium (both using winit) & web (WASM+WebGL).

There's at least 16 3rd party integrations including egui_wgpu_backend for wgpu.

The primary wgpu integration seems to be egui_wgpu_backend but there have been a couple of earlier forks:

• https://crates.io/crates/eww

In addition there are currently in-development projects using it:

• https://github.com/setzer22/blackjack

• https://crates.io/crates/egui_wgpu_backend/reverse_dependencies

(Side note: egui_node_graph was extracted from "Blackjack" & might be of interest.)

In terms of wgpu gaining an official integration, it has been suggested but I did note there was some concern about its incomplete OpenGL support:

• Split out new crate egui-winit from egui_glium emilk/egui#735 (comment)

• Replace Glium emilk/egui#93 (comment)

Maintenance

I did note that the original egui_wgpu_backend developer actually mentioned:

"I don't use egui anymore"

but in an earlier comment also commented:

"I'm not using egui anymore, ... just maintaining the PRs and the release process".

So the crate does appear to still be actively maintained.

Potential next steps

I haven't had much direct experience with wgpu but from a cursory read of the existing Fornjot & egui_gpu_backend example code it seems that there's clearly a bunch of fairly similar setup code that presumably needs to be merged such that the rendering process becomes integrated.

i.e. this:

https://github.com/hannobraun/Fornjot/blob/15294c2ca2fa5ac5016bb29853943b28952f2dae/fj-app/src/graphics/renderer.rs#L38-L52

and this:

    let instance = wgpu::Instance::new(wgpu::Backends::PRIMARY);
    let surface = unsafe { instance.create_surface(&window) };

    // WGPU 0.11+ support force fallback (if HW implementation not supported), set it to true or false (optional).
    let adapter = pollster::block_on(instance.request_adapter(&wgpu::RequestAdapterOptions {
        power_preference: wgpu::PowerPreference::HighPerformance,
        compatible_surface: Some(&surface),
        force_fallback_adapter: false,
    }))
    .unwrap();

    let (device, queue) = pollster::block_on(adapter.request_device(

I am currently attempting to resist the temptation to try to do so. :D

Hope some of this brain dump is helpful! :)

[hannobraun]

Thank you, @follower, this is very helpful!

My overall view is that egui is very much a "first step" in the application UI path & I can see definite potential for other crates with more sophisticated/aesthetically appealing approach but currently it's unclear "who's going to win" amongst the many in-development options out there which makes choosing one challenging. (I also think there's potential for more elegant skinning/themeing which hasn't really been explored yet.)

This is fine by me. What we need is a practical solution for the current and upcoming problems. I think trying to choose the perfect GUI approach at this point is probably a fool's errand. Especially since it doesn't seem unlikely that long-term, things will develop along the lines outlined by @unrelentingtech in his comment above.

Based on my experience from a recent project I'd say that the egui render/platform backend frameworks are more suited to being the "top" of the application & providing a way to display your rendered output (via "native textures" or the upcoming callback functionality). (As opposed to rendering themselves on a surface/texture provided by you--it's possible but you'll probably have to do more of the setup work that integrations normally do.)

But that's probably not a huge issue given you already have a functioning application anyway which I'm assuming you'd probably rather integrate egui into rather than the other way around.

Seems like this is the main issue here, figuring out how to integrate egui and Fornjot in a way that works well, but doesn't bind our whole graphics architecture to egui.

I am currently attempting to resist the temptation to try to do so. :D

Please stop resisting, I'd appreciate the help 😁

But seriously, this is a very important topic (lots of issues are blocked on this one), and for me to handle this would mean taking away time from other very important topics. Tackling the GUI question would certainly be a great way to help the project out, for anyone who's inclined to do so!

[rsaccon]

My first post here, hi all. A while back I tinkered around with integrating egui with truck. Nothing really useful came out of that, but I learnt a lot about egui, CAD kernels and wgpu. I think over time people will integrate fornjot with all kind of GUIs. I started to look into building a Fornjot GUI with rui, which is very, very alpha but I like very much its approach (all GPU, SDF based, no CPU tessellation, but has currently no 3D backend). When I have something to show, I will post it here of course. But it is very specific thing, tailored to my personal interest, and not along the lines of what generally makes most sense, outlined by @unrelentingtech above.

To facilitate external GUIs I think fj-app should be refactored into several modules. I also noticed that the current Fornjot GUI app is only for rendering one model. But for efficiently rendering multiple models, some optimizations at the rendering pipeline are needed, like pooling all models which have been scaled, translated, rotated from the same base model and then apply wgpu instances technique.

Another thing to consider is the quality of the 3d rendering currently implemented in fj-app. Now it is simple and at this stage could probably be ported easily to a non-wgpu platform. But once proper materials are implemented, and Phong shading (or whatever makes more sense), I assume this project is really tied to wgpu (which is fine for me). Or is Fornjot not at ll about the rendering ?

One last thing I think each 3d modeling Gui developer would like to approach differently is the camera. So it should be as pluggable as possible.

[hannobraun]

Thank you for the comment, @rsaccon!

rui sounds interesting. I didn't know there were GUIs based on SDFs. I'm looking forward to see what you come up with, so make sure to post it here somewhere!

To facilitate external GUIs I think fj-app should be refactored into several modules.

That's already happening: #141

I also noticed that the current Fornjot GUI app is only for rendering one model. But for efficiently rendering multiple models, some optimizations at the rendering pipeline are needed, like pooling all models which have been scaled, translated, rotated from the same base model and then apply wgpu instances technique.

Another thing to consider is the quality of the 3d rendering currently implemented in fj-app. Now it is simple and at this stage could probably be ported easily to a non-wgpu platform. But once proper materials are implemented, and Phong shading (or whatever makes more sense), I assume this project is really tied to wgpu (which is fine for me). Or is Fornjot not at ll about the rendering ?

The current graphics code is simply what it needed to be to get us where we are now. It can and should be improved, both for more efficiency and better visuals. Doing so is not a priority for me right now, not because I think it's not important, but because I have so many other things to do.

Tying the graphics code to wgpu is fine by me. I think it's already a great technology, and it will only get better once WebGPU shows up in browsers. Although one thing we might want to consider, is to switch to a higher-level graphics engine at some point. I don't know if that is practical, or even desirable, but if there's something out there that is easier to use, but gives us all the capabilities we need, then that might be worth considering.

One last thing I think each 3d modeling Gui developer would like to approach differently is the camera. So it should be as pluggable as possible.

Good point. I'll keep that in mind, as I keeping working on #141.

[follower]

I am currently attempting to resist the temptation to try to do so. :D

Please stop resisting, I'd appreciate the help

So, you see, I have this fatal flaw: I want to be cool & I want to be helpful...

And, somewhat conveniently for you, for some reason I find CAD-related things pretty cool, so... :D

A demo of egui integrated with Fornjot:

fornjot-egui-poc--simple-screen-recorder-recording-01--2022-05-24_23.41.26.webm.mov

While this demo is very much at the hacky/proof-of-concept level, it does work & I think it is a realistic option for a road to go down.

As shown, egui is essentially being rendered "on top" of whatever fj-app is rendering.

I did attempt to "play nice" with egui & let it take the window events it wanted exclusive access to but that didn't work well with how fj-app currently works & egui also seemed to take events that weren't over its window. So for this demo I feed the events to egui & then also feed them to fj-app.

(Which leads to the unintentionally :) avoided scenario of moving the "title" bar of the egui window causing the model to rotate, which I only just noticed. :D )

(Also, the GUI "overlay" doesn't need to look like a window, that was just the default example I based the demo on. For a rather...different look...this is an example of a egui UI I put together in Bevy: https://rancidbacon.itch.io/darkrun )

Anyway, I need to get some sleep but wanted to at least post this much progress here before I ran out of momentum.

I'll try to come back tomorrow with something a little more code-like for you--although I also discovered along the way that there's probably a better/easier way to achieve this which I only discovered along the way.

Pretty happy to get things this far, eventually. :)

[hannobraun]

So, you see, I have this fatal flaw: I want to be cool & I want to be helpful...

Let me assure you, you are cool and helpful! And I'm lucky to be able to exploit that 😄

But seriously, this is awesome! I hope this will result in a pull request down the line!

Since this is an incremental step based on what we already have (and what I'm already comfortable maintaining), the bar for merging this is super-low. I assume that adding what you already have here would be a good guidepost for anyone (maybe me) who's looking into adding more UI elements.

So don't feel the need to submit a super-polished pull request. I'm happy to merge anything that takes us into the right direction, as long as it doesn't break existing features too badly. (Don't feel discouraged from submitting a super-polished pull request either, if that's what you want and have the time for 😄)

[follower]

Just to update this issue with links to the related PRs:

• Original proof-of-concept based on older versions of Fornjot & egui:
  • Proof of concept egui integration. #642
• Updated proof-of-concept based on Fornjot @ c85c096 and egui ~v0.18.xish including the new core egui-wgpu backend:
  • Updated egui + wgpu + winit proof-of-concept integration #763

[follower]

Screen shot of most recent version:

[hannobraun]

#763 has been merged (thanks again, @follower). Closing.