Reval

Reval is an open-source framework to evaluate the performance of Robotics platforms. Currently it supports Husky platform, Turtblebot3. The useres can evalute the performance of a mission for a given gazebo envirnoment (or on their own gazebo envirnment) for different configurations in an automated fashion and log the results. In addition, Reval supports the following metrics to evaluate the quality of a mission:

Evaluation metrics

Metrics	Description
DWA F	# of failed produced path by DWA planner
DWA NP	# of re-planning by DWA planner
DWA IT	# of DWA invalid trajectory
RR	# of rotate recovery excuted
RCU	# of ClearCostMaps recovery executed for unstuck robot
RCL	# of ClearCostMaps layer recovery executed
IRC	# of invalid rotation cmd
ERG	# of error rotating on the goal
DRMS	The square root of the average of the squared horizontal position errors, ; where standard deviation of the delta x and y, . Probability of 65%
2DRMS	Twice the Distance Root Mean Squared (DRMS) of the horizontal position error, . Probability of 95%
CPE	The radius of circle centered at the true position, containing the position estimate with probability of 50%.
ED	Euclidean distance between actual goal location and the location the robot reached.
RNS	robustness in narrow spaces, ; where Ns is the total narrow spaces in the gazebo environment, and passed_Ns is the narrow spaces that the robot successfully crossed.
DT	total distance traveled during a mission
BP	battery percentage. For more details: Gazebo-ROS battery plugin
Col	number of collisions in a mission
MT	time taken to complete a mission
MS	mission success. Example: if the robot successfully reached point A to B

Reval supports both the Husky simulator and Hysky physical robot. The instructions provided below are for Husky simulator. To run Reval on the physical Husky, first setup your husky using Husky UGV Tutorial then follow the below instructions.

Build status

Build Type	Status
ROS melodic
ROS noetic

Platform	Status
Husky UGV
TurtleBot3
OceanWATERS

Requirements

• Ubuntu 18 or Ubuntu 20
• ROS Melodic or ROS Noetic
• Python 2.7 (for ROS Melodic), Python 3.6+ (for ROS Noetic)

Installations

Installing the husky simulator

For ROS Melodic

sudo apt-get install ros-melodic-husky-simulator
sudo apt-get install ros-melodic-husky-navigation
sudo apt-get install ros-melodic-husky-desktop

For ROS Noetic

sudo apt-get install ros-noetic-husky-simulator
sudo apt-get install ros-noetic-husky-navigation
sudo apt-get install ros-noetic-husky-desktop

Installing rosbag for Python

Download and install ros_readbag.py using these commands:

Download the file

wget https://raw.githubusercontent.com/ElectricRCAircraftGuy/eRCaGuy_dotfiles/master/useful_scripts/ros_readbagfile.py

Make it executable

chmod +x ros_readbagfile.py

Ensure you have the ~/bin directory for personal binaries

mkdir -p ~/bin

Move this executable script into that directory as ros_readbagfile, so that it will be available as that command

mv ros_readbagfile.py ~/bin/ros_readbagfile

Create a symlink in ~/bin to this script so you can run it from anywhere:

ln -si "${PWD}/ros_readbagfile.py" ~/bin/ros_readbagfile

If this is the first time ever creating the "~/bin" dir, then log out and log back in to your Ubuntu user account to cause Ubuntu to automatically add your ~/bin dir to your executable PATH.

Re-source your ~/.bahsrc file

source ~/.bashrc

Install dependencies

Clone the repo

git clone https://github.com/softsys4ai/Reval.git

Installing the dependencies

sudo apt install ripgrep
pip install pandas
pip install tqdm
pip install tabulate 

Or cd Reval/ run ./requirements.sh. If you face permission denied, first run chmod +x requirements.sh

Building Reval

Source your ROS setup.sh file

source /opt/ros/<ros distro>/setup.bash

Run catkin build on the Reval root directory

cd Reval/

catkin build

N.b. If you face Catkin command not found, install sudo apt-get install python3-catkin-tools OR you can use catkin_make

If everything is correct, you should see something similar to the following output

Running Reval

cd Reval/

source your new setup.sh file. You need source this setup.sh file everytime you open a new Terminal

source devel/setup.bash

To evaluate the mission run

python reval.py

optional arguments:
  -h, --help    show this help message and exit
  -v , -viz     turn on/off visualization of gazebo and rviz (default: On)
  -e , -epoch   number of data-points to be recorded (default: 1)

examaple: python reval.py -v off -e 10

Demo

Visualization off:

Visualization on:

Customizations

• To define your custom configuration options: Set Configuration
• To use your own custom gazebo environment: Use custom gazebo environment
• To change the goal locations: Update Goal location

Contacts

Please feel free to contact via email if you have any feedbacks. Thank you for using Reval!

Name	Email
Md Abir Hossen	mhossen@email.sc.edu
Pooyan Jamshidi	pjamshid@cse.sc.edu