HITL - Networked UI system

[0mdc]

Motivation and Context

This changeset adds a networked UI system.

How it works

API

The UI system is retained, but the API is written in the style of immediate mode. This allows the system to work with Unity's UI, and potentially any UI framework that could be added to Habitat.

UI controls are contained within "canvases". Each canvas has a vertical layout, meaning that elements are sequentially added from top to bottom.

ui = UIManager(...)
with ui.update_canvas("top_left", dest_mask) as ctx:
    ctx.label(text="Text 1")
    ctx.label(text="Text 2")

Elements can be given a "uid" so that they can be later referred to (e.g. to check whether a button is pressed).

ui = UIManager(...)
with ui.update_canvas("top_left", dest_mask) as ctx:
    ctx.button(uid="button_1", text="Button")

if ui.is_button_pressed("button_1", user_index):
    do_something()

Networking

When the UIContext scope is closed, a UICanvasUpdate is sent to the client.

@dataclass
class UICanvasUpdate:
    clear: bool
    elements: Optional[List[UIElementUpdate]]

• If clear is true, all elements previously present in the canvas are deleted.
• Elements in elements are either updated or created.

The commit_canvas_content() method automatically sets the clear flag if the element uids have changed.

Canvases

Canvases are currently predefined in the implementation. This may change in the future to allow for, among other things, world-space canvas for VR and world-anchored elements.

The following default canvases are created:

"top_left",
"top",
"top_right",
"left",
"center",
"right",
"bottom_left",
"bottom",
"bottom_right",

Implementation

The UI is currently only implemented in Unity.

• Networked UI System 0mdc/siro_hitl_unity_client#40

Example

ui = UIManager(...)
user_index = 0
dest_mask = Mask.from_index(user_index)

with ui.update_canvas("center", dest_mask) as ctx:
    ctx.canvas_properties(
        padding=12, background_color=[0.7, 0.7, 0.7, 0.3]
    )

    ctx.label(text="Title", font_size=32, bold=True, horizontal_alignment=HorizontalAlignment.CENTER)
    ctx.separator()
    ctx.list_item(text_left="Left", text_right="Right")
    ctx.list_item(text_left="Left", text_right="Right")
    ctx.list_item(text_left="Left", text_right="Right")
    ctx.spacer()
    ctx.button(uid="button_1", text="Button")
    ctx.button(uid="button_2", text="Button", enabled=False)
    ctx.spacer()
    ctx.toggle("toggle_1", toggled=False, text_false="False", text_true="True", tooltip="Tooltip 1")
    ctx.toggle("toggle_2", toggled=True, text_false="False", text_true="True", tooltip="Tooltip 2", enabled=False)

if ui.is_button_pressed("button_1", user_index):
    with ui.update_canvas("top_left", dest_mask) as ctx:
        ctx.label(text="button_1 pressed!")

if ui.is_button_pressed("toggle_1", user_index):
    with ui.update_canvas("bottom_left", dest_mask) as ctx:
        ctx.label(text="toggle_1 pressed!")

new_ui-2024-08-26_17.44.27.mp4

Next steps

• The old UI system is deprecated and will gradually be replaced by this one.
  • The old system does not support DPI scaling, VR or advanced formatting.
• There is currently no local fallback.
• Canvases are currently hardcoded. This is fine for now, but may be revisited to enable creation of world-space canvases for VR or anchored UI elements.
• Horizontal and vertical layout groups will be added to allow placing elements side-by-side.
• Some basic elements are missing, such as textboxes. They will be added gradually.
• Viewports may be embedded within UI elements.

How Has This Been Tested

Tested on multiplayer HITL application with Unity clients (Unity implementation).

Types of changes

• [Development]

Checklist

• My code follows the code style of this project.
• I have updated the documentation if required.
• I have read the CONTRIBUTING document.
• I have completed my CLA (see CONTRIBUTING)
• I have added tests to cover my changes if required.

[0mdc]

Added some comments for reviewers.

[0mdc]

There is currently no local fallback.

This is a planned change - the UI system should provide a minimal local implementation.

[0mdc]

Old UI elements will gradually be replaced.

These classes will be removed when all dependencies are gone.

[0mdc]

These are currently hard-coded in the implementation.

This is fine for now. Later, we can make this dynamic to allow for world-space UI (VR or world-anchored elements) or other more advanced scenarios.

[aclegg3]

add a comment here that these are hardcoded in the Unity implementation?

[0mdc]

This is ugly but does the job.

Ideally I'd like to find a better option for serializing/deserializing an union without too much boilerplate or serializer features that depend on a specific library.

A type string could be used, but it would arguably result in more error-prone boilerplate.

[0mdc]

This is not ideal.
See my comment about union serialization.

[aclegg3]

Agreed, this is a bit ugly. Maybe a structure with default Nones and a map based override would be cleaner?

[aclegg3]

Overall looks sound, but hard to comment on the correctness of partial integration too deeply.

My re-phrase, we're moving from abstract objects which update cached metadata to functional calls to update the UI elements immediately. Right?

Nit: I do think the new element dataclasses should be commented to explain the variables and brief docstrings should be added to any function/class.

[aclegg3]

add a comment here that these are hardcoded in the Unity implementation?

[aclegg3]

Agreed, this is a bit ugly. Maybe a structure with default Nones and a map based override would be cleaner?