End-to-end autonomy (HRI, Perception, Planning, Navigation, Manipulation, Controls) on a Stretch RE1 robot

• This was my Master's capstone project at CMU Robotics. It involved developing and integrating an end-to-end autonomy stack for an indoor assistive robot.
• At the end of our project (Dec-2023), we successfully deployed our robot at an Assisted Living Facility in Pittsburgh and Carnegie Mellon University featured our work.

Demo @ CMU Robotics Institute

Demo @ Vincentian Senior Living, Pittsburgh

Important Links: Project YouTube, Project Website, Project Github

Use-Case:

1. A user provides a voice command to Alfred (Stretch RE1 robot) for fetching an object.
2. Alfred understands this command and sets out to fetch the object.
   • It first plans a path to the object's approximate location in the environment and then autonomously navigates to that location.
   • Upon reaching the object location, Alfred searches for the object.
   • Once it finds the object, it aligns with the object so it can successfully grasp it.
   • Alfred then plans a path to grasping the object with its arm and goes on to grasp it.
   • Once the object has been grasped, Alfred returns back to the user with the desired object.
3. The user is happy! :)

More Videos

Navigation	Manipulation

System Architectures

Functional Architecture

Cyber-Physical Architecture

Software Architectures

Overall Deployment Systen

Mission Planner Node

Database Structure

Overall Structure

Tech Stack

• For HRI, we are using Google Cloud Speech-to-Text API and the ChatGPT API.
• For Navigation, we are using ROS Movebase, STVL for 3D Obstacle Avoidance, GMapping for 2D Lidar SLAM, AMCL for 2D Localization, A* for global planning and DWA for local planning.
• For Perception and Manipulation, we are using Yolo v8 for object detection, Graspnet for grasp point detection and have implemented multiple finite state machines for different stages of our pipeline.
• For overall system integration, we have a system-level finite state machine.
• For low-level joint control, we are leveraging the Stretch RE1's open-source APIs.

My primary role in this project:

• Use ROS / C++ / Python to set up and optimize the navigation stack (localization, planning & 3D obstacle avoidance)
• Design & implement a robust software architecture which integrates a cloud-server, HRI interfaces, navigation / manipulation stacks and a high-level task planner.
• Through the course of the project, I also contributed to various perception, manipulation and system-integration aspects of the project.

Project Poster