add nerf viewer

.gitignore

@@ -508,3 +508,122 @@ doxygen/
 3rdparty
 build_*/
 /wheelhouse
+
+
+# Created by https://www.toptal.com/developers/gitignore/api/pycharm
+# Edit at https://www.toptal.com/developers/gitignore?templates=pycharm
+
+### PyCharm ###
+# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
+# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
+
+# User-specific stuff
+.idea/**/workspace.xml
+.idea/**/tasks.xml
+.idea/**/usage.statistics.xml
+.idea/**/dictionaries
+.idea/**/shelf
+
+# AWS User-specific
+.idea/**/aws.xml
+
+# Generated files
+.idea/**/contentModel.xml
+
+# Sensitive or high-churn files
+.idea/**/dataSources/
+.idea/**/dataSources.ids
+.idea/**/dataSources.local.xml
+.idea/**/sqlDataSources.xml
+.idea/**/dynamic.xml
+.idea/**/uiDesigner.xml
+.idea/**/dbnavigator.xml
+
+# Gradle
+.idea/**/gradle.xml
+.idea/**/libraries
+
+# Gradle and Maven with auto-import
+# When using Gradle or Maven with auto-import, you should exclude module files,
+# since they will be recreated, and may cause churn.  Uncomment if using
+# auto-import.
+# .idea/artifacts
+# .idea/compiler.xml
+# .idea/jarRepositories.xml
+# .idea/modules.xml
+# .idea/*.iml
+# .idea/modules
+# *.iml
+# *.ipr
+
+# CMake
+cmake-build-*/
+
+# Mongo Explorer plugin
+.idea/**/mongoSettings.xml
+
+# File-based project format
+*.iws
+
+# IntelliJ
+out/
+
+# mpeltonen/sbt-idea plugin
+.idea_modules/
+
+# JIRA plugin
+atlassian-ide-plugin.xml
+
+# Cursive Clojure plugin
+.idea/replstate.xml
+
+# SonarLint plugin
+.idea/sonarlint/
+
+# Crashlytics plugin (for Android Studio and IntelliJ)
+com_crashlytics_export_strings.xml
+crashlytics.properties
+crashlytics-build.properties
+fabric.properties
+
+# Editor-based Rest Client
+.idea/httpRequests
+
+# Android studio 3.1+ serialized cache file
+.idea/caches/build_file_checksums.ser
+
+### PyCharm Patch ###
+# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721
+
+# *.iml
+# modules.xml
+# .idea/misc.xml
+# *.ipr
+
+# Sonarlint plugin
+# https://plugins.jetbrains.com/plugin/7973-sonarlint
+.idea/**/sonarlint/
+
+# SonarQube Plugin
+# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin
+.idea/**/sonarIssues.xml
+
+# Markdown Navigator plugin
+# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced
+.idea/**/markdown-navigator.xml
+.idea/**/markdown-navigator-enh.xml
+.idea/**/markdown-navigator/
+
+# Cache file creation bug
+# See https://youtrack.jetbrains.com/issue/JBR-2257
+.idea/$CACHE_FILE$
+
+# CodeStream plugin
+# https://plugins.jetbrains.com/plugin/12206-codestream
+.idea/codestream.xml
+
+# Azure Toolkit for IntelliJ plugin
+# https://plugins.jetbrains.com/plugin/8053-azure-toolkit-for-intellij
+.idea/**/azureSettings.xml
+
+# End of https://www.toptal.com/developers/gitignore/api/pycharm

docs/en/services/xrnerf/README.md

@@ -0,0 +1,29 @@
+# XRNeRF Serving
+
+## 1 Quick Start
+
+### 1.1 Install Dependencies
+
+```shell
+# create a virtual environment
+conda create -n BridgeServer python=3.7
+conda activate BridgeServer
+
+# make sure that your current working directory is xrprimer/requirements/
+cd xrprimer/requirements
+
+# install dependencies
+pip install -r service_xrnerf.txt
+```
+
+### 1.2 Start Server
+
+Once you have dependencies installed, start the server using: 
+
+```shell
+# make sure that your current working directory is xrprimer/python/xrprimer/services/xrnerf
+cd xrprimer/python/xrprimer/services/xrnerf
+
+# start the bridge server
+python run.py
+```

docs/en/web_renderer/xrnerf/README.md

@@ -0,0 +1,53 @@
+# XRNeRF Viewer 
+
+## 1 Overview
+
+The following figure illustrates how the XRNeRF viewer works: 
+
+![alt architecture](/resources/web_renderer/xrnerf/architecture.png)
+
+- ***Web Viewer***. This is where the users can view the XRNeRF render output in an interactive manner. Taking the user desired render config(e.g., camera extrinsic, render resolution) as input, the web viewer send the render config to the bridge server via WebSocket connection. Once the bridge server offers a rendered image, the web viewer receives and displays the image with overlayed scene components(e.g., grid, axis viewer). The web viewer is developed using [Babylon.js](https://www.babylonjs.com/) and packaged into a [React](https://react.dev/) application utilizing [Create React App](https://create-react-app.dev/).
+
+- ***Bridge Server***. The bridge server is designed and implemented to connect the XRNeRF backend and the viewer frontend. Specifically, the bridge server receives the render config from the web viewer and save it to the state. Then, the XRNeRF backend constantly infers images using the state. The render outputs are forwared to the web viewer by the bridge server.
+
+- ***XRNeRF Backend***. The XRNeRF backend is responsible for generating render result using the user-specified settings. It communicates with the bridge server by [ZeroMQ](https://zeromq.org/).
+
+## 2 Quick Start
+
+### 2.1 Start Server
+
+Please follow the instructions in [XRNeRF serving](../../services/xrnerf/README.md) to start the server.
+
+
+### 2.2 Start Client
+
+#### 2.2.1 Install Dependencies
+
+
+The `package.json` contains '' dependicies to build the viewer, which can be installed using Node Package Manager(`npm`). It's suggested to manage `Node.js` installations with Node Version Manager(`nvm`). The installation instructions of `nvm` can be found [here](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
+
+The viewer is built using `Node.js 18.15`. Once you have `Node.js` correctly configured, install node packages:
+
+```shell
+# make sure that your current working directory is 'xrprimer/web_renderer/xrnerf'
+cd xrprimer/web_renderer/xrnerf
+
+# install node packages using configuration in package.json
+npm install
+```
+
+#### 2.2.2 Start Viewer
+
+Build and run the viewer using:
+
+```shell
+# start the viewer
+
+# Windows
+.\start.bat
+
+# TODO: Linux
+# .\start.sh
+```
+
+To open the viewer window, open the browser and visit http://localhost:3000/. If the server has been properly deployed, the viewer will automatically connect to the server and display the render output.

python/xrprimer/services/xrnerf/actions.py

@@ -0,0 +1,15 @@
+from enum import Enum
+
+
+class ViewerActionEnum(str, Enum):
+    UPDATE_CAMERA_TRANSLATION = 'UPDATE_CAMERA_TRANSLATION'
+    UPDATE_CAMERA_ROTATION = 'UPDATE_CAMERA_ROTATION'
+    UPDATE_CAMERA_FOV = 'UPDATE_CAMERA_FOV'
+    UPDATE_RESOLUTION = 'UPDATE_RESOLUTION'
+    UPDATE_RENDER_TYPE = 'UPDATE_RENDER_TYPE'
+
+
+class BackendActionsEnum(str, Enum):
+    UPDATE_STATE = 'UPDATE_STATE'
+    UPDATE_RENDER_RESULT = 'UPDATE_RENDER_RESULT'
+    PING = 'ping'

python/xrprimer/services/xrnerf/bridge_server.py

@@ -0,0 +1,97 @@
+import atexit
+import os
+import signal
+import socket
+import subprocess
+import sys
+import threading
+import time
+from typing import Optional
+
+# TODO: use import paths relative to `xrprimer` rather than the current parent
+# TODO: directory to separate the entrypoint and libraries. Unfortunately, the
+# TODO: xrprimer cannot be installed on a Windows machine. Need to be tested on
+# TODO: MacOS later.
+# import xrprimer.services.xrnerf.server as server
+import server
+from rich import print
+
+
+def is_port_open(port: int):
+    # check whether the given port is open
+    try:
+        sock = socket.socket()
+        sock.bind(('', port))
+        sock.close()
+
+        return True
+    except OSError:
+        return False
+
+
+def bind_port(target_port: Optional[int] = None) -> int:
+    # the user specified a port
+    if target_port:
+        if is_port_open(target_port):
+            return target_port
+
+    # no port specified, returns an available port
+    sock = socket.socket()
+    sock.bind(('', 0))
+    port = sock.getsockname()[1]
+
+    return port
+
+
+def start_bridge_server(
+    websocket_port: int,
+    zmq_port: Optional[int] = None,
+    ip_address: str = '127.0.0.1',
+):
+    # run bridge server as a sub-process
+    args = [sys.executable, '-u', '-m', server.__name__]
+
+    # find an available port for zmq
+    if zmq_port is None:
+        zmq_port = bind_port(zmq_port)
+        print(f'Using ZMQ port: {zmq_port}')
+
+    args.append('--zmq-port')
+    args.append(str(zmq_port))
+    args.append('--websocket-port')
+    args.append(str(websocket_port))
+    args.append('--ip-address')
+    args.append(str(ip_address))
+
+    bridge_server_process = subprocess.Popen(args, start_new_session=True)
+
+    def cleanup(process):
+        process.kill()
+        process.wait()
+
+    def poll_process():
+        """Continually check to see if the viewer bridge server process is
+        still running and has not failed.
+
+        If it fails, alert the user and exit the entire program.
+        """
+        while bridge_server_process.poll() is None:
+            time.sleep(0.5)
+
+        print('[bold red]'
+              'The bridge server subprocess dumped.'
+              '[/bold red]')
+        cleanup(bridge_server_process)
+
+        # windows system do not have signal.SIGKILL
+        # TODO: make sure the kill operation still works on Linux systems
+        # os.kill(os.getpid(), signal.SIGKILL)
+        os.kill(os.getpid(), signal.SIGINT)
+
+    watcher_thread = threading.Thread(target=poll_process)
+    watcher_thread.daemon = True
+    watcher_thread.start()
+    # clean up process when it has shut down
+    atexit.register(cleanup, bridge_server_process)
+
+    return zmq_port

python/xrprimer/services/xrnerf/run_xrnerf_server.py

@@ -0,0 +1,18 @@
+from viewer_state import ViewerState
+
+# TODO: use import paths relative to `xrprimer` rather than the current parent
+# TODO: directory to separate the entrypoint and libraries. Unfortunately, the
+# TODO: xrprimer cannot be installed on a Windows machine. Need to be tested on
+# TODO: MacOS later.
+# from python.xrprimer.services.xrnerf.viewer_state import ViewerState
+
+
+def run_viewer():
+    """start the viewer."""
+    viewer_state = ViewerState(zmq_port=6000, ip_address='127.0.0.1')
+    while True:
+        viewer_state.update_scene()
+
+
+if __name__ == '__main__':
+    run_viewer()