Multiplayer HITL #1939

0mdc · 2024-05-03T23:17:35Z

Motivation and Context

This changeset introduces multiplayer HITL support.

phase2-2024-05-03_11.56.09.mp4

How It Works

Multiplayer is activated using a new max_client_count config value (up to 32).

habitat_hitl.networking.enable=True
habitat_hitl.networking.max_client_count=2

Overview

Server -> Client Communication

The server communicates to the clients via gfx-replay keyframes.

Keyframes are generated by habitat-sim (Recorder.cpp).
- They are in json format.
- They contain deltas of the world state (loads, instantiations, deletions, updates).
Messages are extra info included to keyframes by habitat-hitl.
- They are user-specific.
- They contain info like text, GUI shapes, camera transforms, etc.

Therefore, keyframes are global. They are consolidated and broadcasted.
Prior to sending keyframes, the user-specific messages are added to them.

Client -> Server Communication

The client communicates to the server via ClientStates and ConnectionRecords.

ClientStates are generated by siro_hitl_unity_client.
- They contain info like user input.
ConnectionRecords are sent once upon connecting.
- They contain session parameters like scene selection and user ID.

Connection

When a new connection occurs:

The new client is assigned a user_slot (pre-allocated based on max_client_count).
- Each user_slot is assigned to a user_index, therefore a GUI-controllable agent.
A on_client_connected event is emitted containing the ConnectionRecord and user_index.

How Has This Been Tested

Tested on local mode, single user mode and multi-user mode.
Tested on AWS at scale.

Types of changes

[Refactoring]
[Development]

Checklist

My code follows the code style of this project.
I have updated the documentation if required.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes if required.

0mdc

Left some comments for reviewers.

0mdc · 2024-05-04T00:49:01Z

habitat-hitl/habitat_hitl/_internal/networking/networking_process.py

+                for user_index, client in self._user_slots.items():
+                    if client.connection_id == connection_id:
+                        client_state["userIndex"] = user_index
+                # TODO: This can happen in some cases (immediately after disconnect?).


I haven't investigated why this happens. This is rare.

0mdc · 2024-05-04T00:50:04Z

habitat-hitl/habitat_hitl/_internal/networking/networking_process.py

+                for user_index in self._user_slots.keys():
+                    if self.is_okay_to_send_keyframes(user_index):
+                        slot = self._user_slots[user_index]
+                        tasks[user_index] = slot.socket.send(


This cannot fail. Exceptions occur when awaiting task.

0mdc · 2024-05-04T00:51:52Z

habitat-hitl/habitat_hitl/_internal/networking/networking_process.py

-            raise ValueError(
-                "isClientReady key not found in initial client message."
-            )
+        # Hack: connection record may be missing. Try to get it from client state.


This reliably occurs on AWS, but not on local network.

I haven't had the time to go to the bottom of this.

0mdc · 2024-05-04T00:54:09Z

habitat-hitl/habitat_hitl/_internal/networking/networking_process.py

-            else networking_config.http_availability_server.code_available
+            networking_config.http_availability_server.code_available
+            if not network_mgr.has_connection()
+            else networking_config.http_availability_server.code_unavailable


This is the code that determines whether a server is available. It is used by a load balancer in single-user and matchmaker in multi-user.

We only consider the server available when there is no connection.

In the multi-user case, the matchmaker dispatches every other user to the previous user's location.

0mdc · 2024-05-04T00:57:04Z

habitat-hitl/habitat_hitl/core/remote_client_state.py

        # Gather all recent keyDown and keyUp events
-        for client_state in client_states:
+        for user_index in range(len(all_client_states)):


The big deltas below are just inclusion into a loop. No other change here.

0mdc

Some more comments.

0mdc · 2024-05-05T15:03:46Z

habitat-hitl/habitat_hitl/core/remote_client_state.py

This file will be refactored into per-user states and a comms manager.

This will be done in a later PR.

0mdc · 2024-05-05T15:04:13Z

habitat-hitl/habitat_hitl/core/client_helper.py

This class will be refactored in a later pass.

aclegg3

Overall looks good. True test is whether it runs, so demo/video is good proof. 🙂

Docstrings are very informative, would love to see more where they are missing.

Thanks for the clarifying comments, overall this follows my expectations. Not enough deep context for meta/design comments. Looks functional and fine to me. 👍

examples/hitl/rearrange_v2/rearrange_v2.py

aclegg3 · 2024-05-06T17:52:10Z

habitat-hitl/habitat_hitl/core/remote_client_state.py

+                # Draw a frustum (forward is flipped (z+))
+                self._gui_drawer.draw_transformed_line(
+                    mn.Vector3(0, 0, 0),
+                    mn.Vector3(size, size, size),
+                    color0,
+                    color1,
+                )
+                self._gui_drawer.draw_transformed_line(
+                    mn.Vector3(0, 0, 0),
+                    mn.Vector3(-size, size, size),
+                    color0,
+                    color1,
+                    destination_mask=server_only,
+                )
                self._gui_drawer.draw_transformed_line(
                    mn.Vector3(0, 0, 0),
-                    mn.Vector3(0, 0, pointer_len),
+                    mn.Vector3(size, -size, size),
                    color0,
                    color1,
                    destination_mask=server_only,
                )
+                self._gui_drawer.draw_transformed_line(
+                    mn.Vector3(0, 0, 0),
+                    mn.Vector3(-size, -size, size),
+                    color0,
+                    color1,
+                    destination_mask=server_only,
+                )


Could be refactored into a for over the size vectors.

* Add max_client_count config. * Add multiplayer support to remote_client_state. * Add multiplayer support to networking process. * Make HitlDriver, ClientState and GuiDrawer fully networked. * Formatting pass. * Add missing include. * Make rearrange_v2 multiplayer. * Fixes for local mode. * Add UserData docstring.

0mdc added 7 commits May 3, 2024 09:06

Add max_client_count config.

02435cc

Add multiplayer support to remote_client_state.

fe53783

Add multiplayer support to networking process.

15c0136

Make HitlDriver, ClientState and GuiDrawer fully networked.

45b5e3f

Formatting pass.

031de75

Add missing include.

de44832

Make rearrange_v2 multiplayer.

c53e993

0mdc requested review from jturner65, aclegg3, zephirefaith and Ram81 May 3, 2024 23:17

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label May 3, 2024

Fixes for local mode.

bff3cfd

0mdc commented May 4, 2024

View reviewed changes

0mdc commented May 5, 2024

View reviewed changes

aclegg3 approved these changes May 6, 2024

View reviewed changes

Add UserData docstring.

15d8a87

0mdc merged commit 5cb0630 into main May 14, 2024
3 of 4 checks passed

0mdc deleted the 0mdc/multiplayer branch May 14, 2024 14:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiplayer HITL #1939

Multiplayer HITL #1939

0mdc commented May 3, 2024 •

edited

Loading

0mdc left a comment

0mdc May 4, 2024

0mdc May 4, 2024

0mdc May 4, 2024

0mdc May 4, 2024

0mdc May 4, 2024

0mdc left a comment

0mdc May 5, 2024

0mdc May 5, 2024

aclegg3 left a comment

aclegg3 May 6, 2024

Multiplayer HITL #1939

Multiplayer HITL #1939

Conversation

0mdc commented May 3, 2024 • edited Loading

Motivation and Context

How It Works

Overview

Server -> Client Communication

Client -> Server Communication

Connection

How Has This Been Tested

Types of changes

Checklist

0mdc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0mdc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aclegg3 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0mdc commented May 3, 2024 •

edited

Loading