Stride.UI styling and control structure overhaul proposal

[MechWarrior99]

Overview

The current style and creation of controls in the UI has a number of limitations and issues which this proposal aims to solve.
This is a large proposal, and a long read, sorry in advanced!

Current system

A brief (?) overview of the current systems so we are all sort of up to speed.

UIElements define texture and color properties to use when they are drawn.

There are ElementRenderer classes which handle drawing the elements.
Users can create their own ElementRenderer class and register it with the UIRenderFeature, either for a type, or for a specific element. To register for a type requires making an additional IElementRendererFactory class.

UILibraries allow for configuring UIElements, either individually (like setting up the textures for a Slider), or groups of UIElements (like a group of buttons). Then use these pre-made sets of elements in UIPages. Like with Prefabs, when the source pre-made element changes in a UILibrary, the changes are propagated to all their usage locations in UIPages.

The intended workflow as I understand it is to use UILibraries to set up elements, including the default controls, like Slider, and Button. Which you would then use in your UIPage instead of the default controls directly. And if you needed some special or more advanced rendering of a control. You would create an ElementRenderer for it, and then in code, register it for the element.

Limitations

The current UI has a number of limitations and missing features.

1. It is difficult to significantly change the look of a control.
   1. Requires a C# developer.
   2. Requires manually handling layout.
   3. Requires manually finding and registering the renderer with specific elements.
   4. Slow iteration time.
2. It is prohibitive to make custom controls, requiring a minimum of 3 classes (if it has rendering), and clunky registering. And the API for drawing is clunky.
3. Supporting pseudo states requires defining at least one texture property for each state.
4. Difficult to make and style controls that are composed of other controls, as you have to define them in code.
5. Doesn't support rounded corners, requiring the use textures, slowing down prototyping and iteration, along with development type.
6. Limited 3D support.

Proposed changes

These are my proposed changes that address the listed limitations, and expands the flexibility of the UI system in Stride.

Styling

Add a StyleContainer class, and add a property for it to UIElement.
A StyleContainer is a collection of selector sets, with each selector set having a collection of property name and value pairs associated with it.

Selectors match the UIElement which the StyleContainer is defined on (only supports pseudo states), and elements that are provided by a virtual GetStylableElements() method in UIElement. By default no elements are returned.

Selectors in a set match in order against the ancestors of an element, up to the element that defines the style. Meaning that you can do a TypeSelector followed by a NameSelector, and it will only match elements with the specified name and with an ancestor of the specified type. This is much like CSS selectors if you are familiar.

When a selector set matches a UIElement, it the values from the name-value pairs associated with the selector set will be applied to the matched UIElement.

Reason
Allows the UI to easily support different styles for pseudo states, and controls, while not requiring defining properties for each pseudo state.
Lets controls that have internal elements still be styled in a UILibrary.

Alternatives
An alternative is to create a full CSS like stylesheet system. I decided against this option because creating a performant robust stylesheet is difficult and time consuming. And the main benefit is already covered by UILibraries, as they allow you to create reusable consistently styled elements. Where you can change the style in a central location and it is propagated to all usage locations.
So adding them would duplicate functionality and step on the toes of UILibraries. And UILibrary workflow is smoother and better suited for games than creating stylesheets anyway.

Pseudo states

Add a PseudoStates property to UIElement. A PseudoStates class is a collection of the current pseudo states of the UIElement, defined as strings. Any UIElement derived class can define its own pseudo states.

The main use for pseudo states is for changing styling.

A UIElement adds or removes string pseudo states to and from the PseudoStates class property as the element's state changes. For example, when a ToggleButton is toggled on it would add a :active pseudo state, and remove it when toggled off.

Reason
Allows any UIElement derived class to add any pseudo states to themselves that they need, and easily style them.
Allows the implementation for the pseudo state selector and usage to be very simple and straight forward, along with updating the styling when a pseudo state is added or removed..

Alternatives
An alternative is to allow using a new [PseudoState] attribute on bool and enum properties in a UIElement to denote them as pseudo states. And have have a selector check for the value based on the property name it is assigned.
The downside of this is that the code is significantly more complex, that what is effectively, a list of strings. And requires the use of reflection to get the value of the properties to check them. Though it can be optimized, it still comes with increased code complexity, and has heavy initialization costs.

UIElement Render(..) method

Add Render(..) method to UIElement, which can be overridden to handle the rendering of a UIElement.
As an example usage, Border would override Render to handle drawing the background and border of itself.

Reason
Allows rendering to be self-contained, and would make it easier to create an easier to use API for drawing/rendering.
Required for the other parts of this proposal to function.

Template elements

Add TemplatedUIElement and ElementTemplate classes, both inheriting from UIElement (names subject to change).
TemplatedUIElement would be the new default base class for control elements (Button, Slider, etc.). And it allows for referencing a ElementTemplate element in a UILibrary, and instantiating it as its children when created. If no template is set, it would try to default to a template in a UILibrary set in a new property in the GameSettings.

The ElementTemplate is used to define the elements and configuration of elements used by a TemplatedUIElement and is only meant to be used as a root element inside of UILibrary. It has a type field for specifying the type of TemplatedUIElement it is intended to be used for.

TempaltedUIElement overrides GetStylableElements() and returns all elements from the template it uses. In this way, all child elements can be styled without having to make a new template.

Control elements no longer define texture properties for 'parts' (like the Slider has Thumb, and TrackBackground, etc.). Instead, they are UIElements defined in templates.

Remove the current ElementRenderer system, as it would be redundant with the above systems.

Reason
Allows for easier reuse and combining of controls to create more complex controls.
Allows for deeper customization of the look of a control without requiring the use of C#, making it easier for designers to create the UI.
Allows for adding any type of custom 'part' to the UI without having to hardcode the texture for it.
Allows supporting textures or vector UI.

Brushes

Add a IBrush interface, along with implementing classes SolidColorBrush, TextureBrush, GradientBrush, and MaterialBrush which can be used as properties on UIElements to define how it draws, in place of individual texture and color, properties.

Reason
Is similar to C# WPF style libraries, along with design tools like Figma, so should feel familiar to both programmers and designers.
Allows 'implicitly' supporting colors, and textures, along with any custom shading the user may want, without having to explicitly add properties for them.
Follows common design pattern used in Stride.

Alternatives
Keep defining individual properties for both texture and color, and material. The downside is that it requires defining and manually handling considerably more properties. And isn't extensible if we or a user want to add some additional type of brush in the future.

Initial vector rendering.

Add support for procedural rounded corners through procedural mesh generation.

Reason
Allows the UI to immediately be able to support a range of common UI styles/designs without requiring textures.
Allows nicer UI to be mocked up faster, while not having to leave the editor to create and import textures.
Potentially lower memory footprint due to less textures.
Not difficult to support the Depth property of UIElements.

Alternatives

1. No rounded corners. Without any rounded corners even prototype UIs look pretty ugly, and it is not usable in any game.
2. Use material/shader based method with SDFs instead of mesh generation. This is a valid option and I have not quite looked deep enough in to it to know what the performance tradeoffs would be. Though makes it harder to support 3D UI.

Models

Add a Model property along with a toggle to some UI element that handle rendering (Like Border), to support taking a 3D model and rendering that instead of using a rectangle mesh.

Additional properties like either to receive shadows, receive lighting, and cast shadows, and the rotation of the model, could be added as well.

Reason
Allows the use of 3D models in the UI to create cool and unique UIs.
Allows the creating even more immersive diegetic UIs.
Expands on the existing 3D support of the UI.
Would be fairly easy to add, as the UI rendering system already supports rendering 3D geometry. (Not sure the difficulty of support lighting and shadows though).

Alternatives

1. Dedicated elements for Models. Like a BorderModel element. Downside is that it would require the user to create new templates for every control. But does keep the existing elements 'cleaner'.
2. A ModelBrush containing Model, Material, and other relevant properties. It can just be used with of any IBrush property. Would require handling it as a 'special case' compared to all other brushes. This is a very valid alternative imo, and might be a better option actually?

Conclusion

These changes allow for artists and designers to quickly and iteratively design and style control, without the need for a C# developer. Makes it easier for developers to add new controls as they won't need to define and manage as many properties. And makes it easier to support more dynamic UI. All while working with the general workflows of Stride.
This would fully replace the current ElementRenderer system.

It also doesn't have to be all done at once, it can be done one section at a time for the most part.
And should be noted that some of these are breaking changes, and there just isn't a way around them nicely.
Please share your thoughts and let me know if parts need clarification!

[Doprez]

Templated elements

I love the idea but I dont fully understand how this differs from the current UILibrary? It already allows for template controls that are reusable through the UIPage editor and accessin them through code is relatively easy as well.

Models

This would be super handy to have OOB at least for me. I have been thinking of ways to show inventory items with a 3d view but recently settled on just having the sprite for now.

I cant speak on the rendering classes with my lack of experience but I always love easily overridable functions for customization. Having a Render method the can be overridden sounds as useful as having the Draw method in EntityProcessors.

[MechWarrior99]

I love the idea but I dont fully understand how this differs from the current UILibrary? It already allows for template controls that are reusable through the UIPage editor and accessin them through code is relatively easy as well.

It is mostly barrowed from AvaloniaUI, so I can't take credit. There are a few differences which I think set it apart, though most of them I didn't write out for brevity.

1. With a dedicated class we can introduce a system for specifying required element types. Avalonia does this with an attribute on the class where you specify the name and type that a template most contain.
2. The children are hidden with a template control, meaning only the elements that matter are shown to the designer.
3. Could specify a child in the template as a 'content container' alloing the control to support having additional children by just directly adding them, without having to dig through the hierarchy in the editor to find the element that they should be added to. So if you had a button with some complex hierarchy of other elements, you could still have the current behavior of being able to add child elements directly do it (like a text element).
4. I almost included this point, but didn't seem critical info, and again, brevity. Though maybe was worth it.

I have been thinking of ways to show inventory items with a 3d view

Yeah this would be a handy ue for it as well. Though the main use I had in mind was having the UI elements themselves be 3d. The one thing for that use-case I am not sure on is you would probably want it with a custom directional light, otherwise they would jsut be flat, or use the lighting in the current scene. So not sure how that would work. Might make sense to have another element that can handle it better.

[Basewq]

I'm neither for or against proposal, but would like to throw in some info/speculation into the discussion for some of the topics.

UIElement.Render:
My speculation on ElementRender is that it was done similar to Xamarin.Forms Custom Renderer
The main benefit is the separation of concerns and potential to handle the differences in platforms separately.
I don't know if things like EditText in Stride was planned to 'render' the native controls directly and/or interface with the native controls, I think theoretically it would be done through the renderer(s), though right now the code definitely does not interact with the native controls this way (they interact directly with EditText).

.NET MAUI seems to have doubled down on this approach, though apparently subtly different to Xamarin.Forms (called Handlers now).

I also think the separate Renderer/Handler allows you to override an existing control's Renderer/Handler, so you globally affect all existing controls without having to go through every UI Page (or theoretically 3rd party controls) and changing it to your derived version, which might not always be possible.

Note that I'm not saying Xamarin.Forms or Maui did things right in this aspect, just pointing out the similarity.

If using a Render method, I inevitably expect someone deciding to seal all the Render methods, then you get complaints like this AvaloniaUI/Avalonia#11496 which deter developers from inheriting existing controls, meaning you have to copy a control wholesale anyway.
Of course, the main benefit this way is having access to the control's internal fields, so different pros and cons.

Models:
I feel this is crossing into just a standard entity territory.
I think the main issue with the standard entity object is the lack of easy access to determine if you are mousing over a model, which "UI" specifically does by default.
What happens if someone places a tree model in the UI? I think one problem is the layout system assumes a rectangle shape which can be calculated quickly, but add a model means doing something different (use bounding box?).

[MechWarrior99]

Thanks for the extra info, I didn't realize it was inspired by Xamerin.Forms, though probably should have guessed judging by the rest of the way the UI is built so similar to it.

I don't know if things like EditText in Stride was planned to 'render' the native controls directly and/or interface with the native controls

What do you mean by "'render' the native controls" in this case?

[..] allows you to override an existing control's Renderer/Handler, so you globally affect all existing controls without having to go through every UI Page [..]

For thise proposal, there are two cases that hard handled differently. One is if it is a TemplatedUIElement. In which cases you could simply change the default templated used for the control to update it everywhere (to some extent, you would have to go change any custom styling though). The second case, is for elements like Border, that are really primitive, and for those, you could either use a MaterialBrush most of the time to get a custom look. Or if that doesn't suit your needs, it probably makes sense anyway to make a new custom UIElement anyway.

If using a Render method, I inevitably expect someone deciding to seal all the Render methods

I guess, but that can happen with anything really. And something to catch during a PR review. But yeah, generally I find sealing things to be rather annoying most of the time (though there is good reason for it!).

These are great points to bring up though. Thanks for brining them up and for pointing out the similarities and some of the reason why they are how they are.

Models
Yeah I think it is really tricky, and the part I am the least confident in to be honest. Because for example, Stride supports offsetting elements along the Z axis, giving the UI depth when it is in world space. And allows for depth/thickness of UI elements (I recently opened an issue about it #2489). So, how much depth and 3D-ness should be supported? Because, at some point as you said, it basically starts being the same as entities. But on the other hand, there are valid uses cases for wanting to render 3D in your UI.

Perhapse the solution is to introduce an easier way for entity components to implement some of the input/interactive functionality of UIs. Like what you are saying. And as I'm writing this, I am starting to think this might be a better approach. And add some sort of 'viewport' for displaying entities/models in the UI (like for showing 3D previews of objects and such)?

What happens if someone places a tree model in the UI?

I was just planning to use the bounds of the UI.

[Basewq]

What do you mean by "'render' the native controls" in this case?

Stride has some trickery/hacks with native controls. eg. For Android, when you have an EditText it actually pops up the "real" Android textbox as a separate overlay that pushes the changes back to Stride's EditText.

In Xamarin, you basically have the Renderer which places the native textbox (per each platform) where your control is supposed to be, though this is because Xamarin was quite deep in using the native platform's controls as much as possible. Though thinking more about this, I'm not sure it's practical for Stride since it's essentially a single "canvas" independent of any platform's UI system, and also wouldn't work if the EditText is placed in the 3D world.
I haven't really used the UI in an advanced manner this way so can't really point out any real use cases. Maybe someone wants to add an "AdMob" UI control which differs between Android, iOS? I think you're saying this would be done by your TemplatedUIElement instead of the Renderer if this is changed to your design.

As an aside, I feel there's a bad coupling between the native control and Stride's EditText which should ideally be separated out like a Renderer/Handler or some other way.

For thise proposal, there are two cases that hard handled differently. One is if it is a TemplatedUIElement. In which cases you could simply change the default templated used for the control to update it everywhere (to some extent, you would have to go change any custom styling though). The second case, is for elements like Border, that are really primitive, and for those, you could either use a MaterialBrush most of the time to get a custom look. Or if that doesn't suit your needs, it probably makes sense anyway to make a new custom UIElement anyway.

Ok with me.

What happens if someone places a tree model in the UI?

I was just planning to use the bounds of the UI.

Ok, though with things like align left/right, stretch content, I suspect there would probably be some scaling involved (just pointing out some potential pitfalls).