🥚 EGG - Experimental Generalized Gateway

Telepresence + Inference at the edge

Quick Start

Install and run the experimental Orchestrator

To install and run the Orchestrator, simply copy and paste the following command into your terminal:

curl -O https://raw.githubusercontent.com/robit-man/EGG/main/install_and_run_orchestrator.sh && chmod +x install_and_run_orchestrator.sh && ./install_and_run_orchestrator.sh

Bonus: For a quick look at a generic peripheral component that has baseline functionality and can be adapted for your particular usecase in the ecosystem, look at the Plug!

Overview

EGG is a multi-modal data acquisition, inference, storage, processing, and training/fine-tuning system deployed at the edge. Built on the NVIDIA Jetson AGX Orin platform, EGG integrates various peripherals to deliver comprehensive AI capabilities.

Currently undergoing testing and refactoring on JetPack 6, with a focus on the Orchestrator system and the integration of additional peripherals. For detailed instructions, refer to the Installation Guide.

Tested Systems

Orchestrator Scripts

• Hardware: Jetson AGX ORIN 32GB DEVKIT
• Software Versions:
  • L4T: 36.3.0
  • JetPack: 6.0 (rev 2)
  • CUDA: 12.2
  • cuDNN: 8.9.4.25
  • Python: 3.10.12

Agent Interface

• Version: 5.1.2
• Hardware: Jetson AGX ORIN 32GB DEVKIT
• Software Versions:
  • L4T: 35.2.1
  • JetPack: 5.1.2
  • CUDA: 11.4.315
  • cuDNN: 8.6.*
  • Python: 3.8.10

---

Table of Contents

1. Acquire Parts and Construct
2. Setup and Remote Access
3. Install Dependencies
4. Experiment with the Orchestrator and Peripherals
5. Experiment with TTS / ASR LLM Services
6. Collaborations
7. Additional Resources
8. Conclusion

---

1. Acquire Parts and Construct

To build the EGG system, follow these steps:

1. Order Components: Refer to the Bill of Materials to order all necessary parts.
2. 3D Print Parts: Print all components available in the egg-parts.stp file.
3. Assembly: Construct the system following the Assembly Guide (Work in Progress).

---

2. Setup and Remote Access

2.1. Determine JetPack Compatibility

• Consult the Compatibility Matrix to select the appropriate JetPack version based on your hardware and host machine.

2.2. Install Ubuntu

1. Download Ubuntu:

   • Obtain Ubuntu 24.04 LTS.

2. Install on Host Machine:

   • Install Ubuntu on a compatible host machine that will run the Jetson SDK Manager.

2.3. Install JetPack

1. Download JetPack:

   • Access the JetPack 6 Download Page and download the SDK Manager suitable for your host Linux machine.

2. Install JetPack:

   • Launch the SDK Manager:

     sudo ./sdkmanager

   • Follow the on-screen instructions to install JetPack 6.
   • Important: Ensure that you save and build all firmware on the same drive to avoid installation issues.

2.4. Set Up Remote Access with NoMachine

1. Download NoMachine:

   • Obtain the ARM64 version of NoMachine from the official website.

2. Install NoMachine:

   sudo dpkg -i nomachine_8.14.2_1_arm64.deb

3. Configure NoMachine for Audio:

   • Open NoMachine settings and navigate to Server Settings > Devices.
   • Disable the NX audio option to enable audio streaming and microphone forwarding.

4. Access the Device Remotely:

   • Use machine credentials to access the Jetson AGX Orin on the LAN or via port forwarding.

---

3. Install Dependencies

EGG relies on various software dependencies to function correctly. Follow these steps to install them:

3.1. Install NVIDIA RIVA

1. Download RIVA Quickstart:

   • Access the Riva Quickstart ARM64 and download version 2.16.0.

2. Alternatively, Download via CLI:

   ngc registry resource download-version nvidia/riva/riva_quickstart:2.16.0

3. Organize Riva Files:

   mkdir -p /home/$(whoami)/RIVA

   • Extract and place all Riva files in the RIVA folder, resulting in paths like /home/$(whoami)/RIVA/riva_start.sh.

4. Initialize Riva:

   sudo bash /home/$(whoami)/RIVA/riva_init.sh

5. Configure Docker Daemon:

   • Edit the Docker daemon configuration to recognize NVIDIA as the default runtime:

     sudo gedit /etc/docker/daemon.json

   • Add the following content:

     {
         "default-runtime": "nvidia",
         "runtimes": {
             "nvidia": {
                 "path": "nvidia-container-runtime",
                 "args": []
             }
         }
     }

   • Restart Docker:

     sudo systemctl restart docker

6. Start Riva:

   sudo bash /home/$(whoami)/RIVA/riva_start.sh

3.2. Install Custom Voice Models

• For example, install GLaDOS TTS.

3.3. Install Jetson Containers and NANOLLM

1. Install Jetson Containers:

   git clone https://github.com/dusty-nv/jetson-containers
   bash jetson-containers/install.sh

2. Install NANOLLM:

   • Follow the installation instructions provided by dusty-nv.

3.4. Install Additional Dependencies

• Install any missing dependencies as needed.
• If issues arise, provide console output to improve this documentation.
• Refer to JetsonHacks for helpful resources.

---

4. Experiment with the Orchestrator and Peripherals

Explore the Orchestrator System to manage and integrate peripherals.

4.1. Overview of Components

• Orchestrator: Central hub managing interactions between peripherals.
• ASR Engine: Automatic Speech Recognition for converting spoken language into text.
• TTS Engine: Text-to-Speech synthesis for generating spoken language from text.
• SLM Engine: Speech Language Model for processing and routing data between ASR and TTS Engines.

4.2. Setting Up the Orchestrator

1. Navigate to Orchestrator Directory:

   cd /path/to/orchestrator

   Note: Replace /path/to/orchestrator with the actual path where the Orchestrator folder is located.

2. Set Up Each Component: For each subfolder (orch, slm, asr, tts), execute the corresponding Python script.

   • Orchestrator:

     cd orch
     python3 orch.py

   • SLM Engine:

     cd ../slm
     python3 slm.py

   • ASR Engine:

     cd ../asr
     python3 asr.py

   • TTS Engine:

     cd ../tts
     python3 tts.py

3. Automate Startup (Optional)

   To ensure that the orchestrator and its components start automatically on system boot, create a systemd service or add scripts to the startup applications.

   Example using systemd:

   • Create Service File:

     sudo nano /etc/systemd/system/egg-orchestrator.service

     • Add the Following Content:

       [Unit]
       Description=EGG Orchestrator Service
       After=network.target
       
       [Service]
       Type=simple
       ExecStart=/usr/bin/python3 /path/to/orchestrator/orch/orch.py
       ExecStartPost=/usr/bin/python3 /path/to/orchestrator/slm/slm.py
       ExecStartPost=/usr/bin/python3 /path/to/orchestrator/asr/asr.py
       ExecStartPost=/usr/bin/python3 /path/to/orchestrator/tts/tts.py
       Restart=on-failure
       User=your_username
       
       [Install]
       WantedBy=multi-user.target

       Note: Replace /path/to/orchestrator with the actual path and your_username with your actual username.

   • Enable and Start the Service:

     sudo systemctl enable egg-orchestrator.service
     sudo systemctl start egg-orchestrator.service

4. Verify Operation

   Ensure that all components are running correctly by checking their logs or status.

   sudo systemctl status egg-orchestrator.service

   Additionally, monitor the terminal outputs of each script to verify successful initialization and communication between components.

---

5. Experiment with TTS / ASR LLM Services

Utilize the Agent Interface to automate Python scripts for TTS and ASR services.

5.1. Overview

The Agent Interface facilitates automated interactions between the TTS and ASR services, streamlining the workflow for speech recognition and synthesis tasks.

5.2. Running Agent Interface Scripts

1. Navigate to Agent Interface Directory:

   cd /path/to/agent_interface

   Note: Replace /path/to/agent_interface with the actual path.

2. Execute Automation Scripts:

   python3 agent_interface.py

   • Ensure that all necessary dependencies and environment variables are set before running the scripts.

5.3. Configuration

• Refer to the Agent Interface Documentation for detailed configuration and usage instructions.

---

6. Collaborations

EGG is a collaborative effort involving hardware and software partners to enhance its capabilities.

• Fractional Robots
• Hyperspawn: GitHub
• RokoNetwork: RokoNetwork on X
• Fractional Robots Telegram Group: Join the Group

---

7. Additional Resources

• EGG Orchestrator System Overview: For a comprehensive understanding of the EGG Orchestrator system, including detailed descriptions of each component and their interactions, refer to the EGG Orchestrator Documentation.
• NVIDIA Developer Forums: Engage with the community and seek help at the NVIDIA Developer Forums.
• GitHub Repositories:
  • Jetson Containers
  • EGG Project
• Helpful Tutorials:
  • JetsonHacks Docker Containers Tutorial