I’ve worked in the Mixed Reality lab at UNCW for 3 years, and over the course of the 3 years there has been one single feature that has been requested by developers and users alike that we still have not delivered - a real time stream of what the headset user is seeing. It makes sense, everyone wants to know what’s happening inside the headset, especially when a user is reacting positively or negatively. Likewise, it can be incredibly difficult to communicate changes to remote users or show off demonstrations of new features to individuals without access to the headset. For that reason, I set out with this project to create a solution.
The solution needed to be guided by a few things. It needed to be private, functional on a closed network, and packaged in docker to be usable with one of our biggest clients: Oak Ridge National Labs. It also needed to be able to be retrofitted into an existing project, so we could use it in the education based Virtual Access To Stem Careers project which has been in development for multiple years. As for my own requirements, I wanted the project to be modular. In that sense, if I created a chat service or a video streaming service, I wanted to be able to use those services in other contexts. That led to my decision to chose a microservices architecture for the project.
With all of these aspects in mind, I have delivered a framework for use within the Mixed Reality Lab. It is built primarily in Node.JS and Python for web use and C# for unity scripting. The system utilized native packages where applicable, such as using Unity’s native video streaming library, and I constructed new services to help tie all the pieces together. This system allows developers to add real-time streaming to any existing Unity project as well as synchronous messaging utilizing the Alert service. This project was built with the Mixed Reality labs development style in mind, and as a result will have immediate potential for use in our active projects. All in, it finally answers the single question users and clients ask of the lab during demos: So what are they seeing in there?