Franka-Teach

A comprehensive teleoperation and data collection system for Franka robots, featuring VR control via Meta Quest, MCAP data recording, and simulation support.

Features

• VR Teleoperation: Full 6DOF control using Meta Quest/Oculus controllers
• MCAP Recording: DROID-compatible data format for robot learning
• Multi-Camera Support: Intel RealSense and ZED camera integration
• Simulation Mode: Test without hardware using FR3 robot simulation
• Real-time Control: 15-30Hz control loop with safety features
• High-Performance Async Architecture: 40Hz data recording with 6.6Hz robot control
• Hot Reload: Automatic server restart on code changes
• Performance Mode: Optimized for high-frequency control and recording
• Data Verification: Automatic validation of recorded trajectories

Table of Contents

• System Requirements
• Installation
  • NUC Setup
  • Lambda Machine Setup
• Quick Start
• VR Teleoperation
• Data Recording
• Camera Setup
• Simulation Mode
• Troubleshooting
• Additional Documentation

System Requirements

Hardware

• Franka Emika Robot (FR3 or Panda)
• NUC computer with real-time kernel
• Lambda workstation or similar
• Meta Quest 2/3 or Oculus headset
• Intel RealSense or ZED cameras (optional)

Software

• Ubuntu 22.04 (NUC) / Ubuntu 20.04+ (Lambda)
• CUDA-capable GPU (for Lambda machine)
• Python 3.10+
• Real-time kernel (NUC only)

Installation

NUC Setup

1. Install Ubuntu 22.04 with real-time kernel

   # Follow instructions at:
   # https://frankaemika.github.io/docs/installation_linux.html#setting-up-the-real-time-kernel

2. Configure network settings

   • Set static IP for robot network interface
   • Ensure NUC can communicate with robot at 172.16.0.4 (right) and 172.16.1.4 (left)

Lambda Machine Setup

1. Clone and setup deoxys_control

   git clone git@github.com:NYU-robot-learning/deoxys_control.git
   cd deoxys_control/deoxys
   
   # Create conda environment
   mamba create -n "franka_teach" python=3.10
   conda activate franka_teach
   
   # Install deoxys (select version 0.13.3 for libfranka when prompted)
   ./InstallPackage
   make -j build_deoxys=1
   pip install -U -r requirements.txt

2. Clone and setup Franka-Teach

   git clone <your-franka-teach-repo>
   cd Franka-Teach
   pip install -r requirements.txt

3. Install ReSkin sensor library (optional)

   git clone git@github.com:NYU-robot-learning/reskin_sensor.git
   cd reskin_sensor
   pip install -e .

4. Setup Oculus Reader for VR control

   cd oculus_reader_app
   # Follow instructions in oculus_reader_app/README.md

5. Configure IP Addresses ⚠️ Important

   You must update the IP addresses in the configuration files to match your network setup:

   # Edit the deoxys configuration files
   cd frankateach/configs
   
   # For right robot:
   vim deoxys_right.yml
   # Update:
   # - PC.IP: Your Lambda machine's IP address
   # - NUC.IP: Your NUC's IP address (same as PC.IP if running on same machine)
   # - ROBOT.IP: Your Franka robot's IP address (e.g., 192.168.1.59)
   
   # For left robot (if applicable):
   vim deoxys_left.yml
   # Update the same fields with appropriate IP addresses

   Note: The franka_server.py will fail to connect with "Waiting for the robot to connect..." if these IP addresses don't match your actual network configuration.

Network Proxy Setup

1. Install FoxyProxy extension in Chrome/Firefox

2. Configure SSH host in ~/.ssh/config:

   Host nuc
       HostName 10.19.248.70
       User robot-lab
       LogLevel ERROR
       DynamicForward 1337
   

3. Configure FoxyProxy to use SOCKS5 proxy on localhost:1337

Quick Start

1. Connect to Robot

# SSH into NUC
ssh nuc

# Access Franka Desk interface via browser (with proxy enabled)
# Right robot: http://172.16.0.4/desk
# Left robot: http://172.16.1.4/desk
# Username: GRAIL
# Password: grail1234

2. Enable Robot

In Franka Desk interface:

1. Click "Unlock joints" (open brakes)
2. Enable FCI mode
3. If gripper issues: Settings → End Effector → Power cycle → Re-initialize

3. Start Deoxys Control (on NUC)

# Terminal 1: Start arm control
cd /home/robot-lab/work/deoxys_control/deoxys
./auto_scripts/auto_arm.sh config/franka_right.yml  # or franka_left.yml

# Terminal 2: Start gripper control (optional)
./auto_scripts/auto_gripper.sh config/franka_right.yml

4. Start Servers (on Lambda)

# Terminal 1: Start Franka server
cd /path/to/Franka-Teach
python franka_server.py

# Terminal 2: Start camera server (optional)
python camera_server.py --config camera_config_example.json

5. Start VR Teleoperation with Recording

# Terminal 3: Start Oculus VR server with MCAP recording
python oculus_vr_server.py

# Or with specific options:
python oculus_vr_server.py --left-controller  # Use left controller
python oculus_vr_server.py --simulation       # Test in simulation
python oculus_vr_server.py --debug           # Debug mode (no robot control)
python oculus_vr_server.py --no-recording    # Disable MCAP recording

VR Teleoperation

Controller Setup

1. Connect Quest headset via USB or WiFi
2. Test connection: python test_oculus_reader.py
3. Calibrate forward direction: Hold joystick + move controller forward

Control Scheme

Right Controller (default)

• Hold Grip: Enable robot movement
• Release Grip: Pause movement (robot stays in place)
• Hold Trigger: Close gripper
• Release Trigger: Open gripper
• Joystick Press: Reset controller orientation

Recording Controls (when enabled)

• A Button: Start new recording / Reset current recording
• B Button: Mark recording as successful and save

See oculus_control_readme.md for detailed VR control instructions.

Data Recording

The system supports MCAP data recording in Labelbox Robotics format:

• Press A button to start/reset recording
• Press B button to mark recording as successful and save
• Recordings are saved to ~/recordings/success/ or ~/recordings/failure/
• See MCAP_RECORDING_README.md for details

Visualizing Robot in Foxglove Studio

The MCAP recordings include the robot URDF model and joint states for visualization:

• See FOXGLOVE_ROBOT_VISUALIZATION.md for setup instructions
• Test with python3 test_foxglove_robot.py to create a sample file

Camera Setup

Configure Cameras

Edit `

Simulation Mode

Simulation Setup

1. Test in simulation: python oculus_vr_server.py --simulation
2. Test without hardware: Use FR3 robot simulation

Troubleshooting

Low Recording Frequency

• Ensure performance mode is enabled (default with run_server.sh)
• Check CPU usage and reduce other processes
• Verify in console output: should show ~39-40Hz

Robot Communication Slow

• The ~149ms latency is hardware limited
• System automatically uses predictive control
• Check network/USB connection quality

VR Controller Not Detected

• Ensure Oculus is connected via USB or network
• Check with adb devices for USB connection
• For network: use --ip <quest-ip-address>

Franka Server Connection Issues

If franka_server.py shows "Waiting for the robot to connect..." indefinitely:

• Check IP configuration: The most common cause is incorrect IP addresses in frankateach/configs/deoxys_right.yml or deoxys_left.yml
• Verify network connectivity: Can you ping the robot from the Lambda machine?
• Check if deoxys is already running: Only one process can connect to the robot at a time

  # Check for running deoxys processes
  ps aux | grep -E "(franka-interface|gripper-interface)"
  # Kill if necessary
  sudo pkill -f "franka-interface|gripper-interface"

• Ensure robot is in FCI mode: Check Franka Desk interface

Additional Documentation

Architecture Overview

The system uses a sophisticated asynchronous architecture to achieve high-frequency data recording (40Hz) while managing slower robot communication (6.6Hz):

VR Thread (50Hz) → Control Thread (40Hz) → Recording Thread (40Hz)
                          ↓
                   Robot Comm Thread → Robot Hardware (6.6Hz)

Key Benefits:

• 6x higher recording rate than traditional synchronous approaches
• Non-blocking operation ensures smooth teleoperation
• Predictive control maintains responsiveness
• Thread-safe data handling with minimal lock contention

For detailed architecture documentation, see ASYNC_ARCHITECTURE_README.md.

Performance Optimization

The system includes several performance optimizations:

Performance Mode (Default)

• Control frequency: 40Hz (2x base rate)
• Position gain: 10.0 (100% higher)
• Rotation gain: 3.0 (50% higher)
• Optimized for tight tracking

Async Features

• Decoupled threads for VR, control, recording, and robot I/O
• Predictive control when robot feedback is delayed
• Non-blocking queues for data flow
• Lock-free design where possible

See ASYNC_ARCHITECTURE_README.md for detailed performance documentation.

Hot Reload

Enable automatic server restart on code changes:

./run_server.sh --hot-reload

This monitors Python files and configs, restarting the server when changes are detected. Perfect for rapid development and testing.

See HOT_RELOAD_README.md for details.

Data Format

Recordings are saved in MCAP format with the following structure:

• /robot_state: Joint positions, cartesian pose, gripper state (40Hz)
• /action: Velocity commands sent to robot (40Hz)
• /vr_controller: Raw VR controller data (40Hz)
• /tf: Transform tree for visualization
• /joint_states: ROS-compatible joint states

Documentation

• Async Architecture & Performance
• Hot Reload Feature
• MCAP Recording Format
• Foxglove Visualization
• VR Robot Mapping

Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes and test with hot reload
4. Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Based on the DROID VRPolicy implementation
• Uses Deoxys control framework for Franka robots
• MCAP format for efficient data storage