https://github.com/arthurfenderbucker/LaTTe-Language-Trajectory-TransformEr
Main improvements:
- Trajectory modifications in the 3D space + speed
- Interactive modification in realtime with direct image input
- Multi-platform evaluation
IROS2022: Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers
example of multiple iterative interactions over the initial erroneous trajectory (red)
tested on Ubuntu 18.04 and 20.04
Environment setup
conda create --name py38 --file spec-file.txt python=3.8
conda activate py38
Install CLIP + opencv
pip install ftfy regex tqdm
pip install git+https://github.com/openai/CLIP.git
pip install opencv-python
Download models
conda install -c conda-forge gdown
gdown --folder https://drive.google.com/drive/folders/1HQNwHlQUOPMnbPE-3wKpIb6GMBz5eqDg?usp=sharing -O models/.
Download syntetic dataset
gdown --folder https://drive.google.com/drive/folders/1_bhWWa9upUWwUs7ln8jaWG_bYxtxuOCt?usp=sharing -O data/.
cd src
python modify_draw.py
How to use:
- press 'o' to load the original trajectory
- press 'm' to modify the trajectory using our model for the given input on top.
- press 't' to set a different interaction text.
- press 'u' to update the trajctory setting the modified traj as the original one
IMPORTANT: make sure that conda isn't initialized in your .bashrc file, otherwise, you might face conflicts between the python versions
terminal 1
roscore
terminal 2
roscd NL_trajectory_reshaper/src
source $HOME/cvbridge_build_ws/install/setup.bash --extend
python modify_draw.py --ros true
overview of the project model_overview.ipynb
plots and ablation studies plots_and_ablasion_exp.ipynb
generate syntetic dataset data_generator/data_generator.py