E-RAFT: Dense Optical Flow from Event Cameras

This is the code for the paper E-RAFT: Dense Optical Flow from Event Cameras by Mathias Gehrig, Mario Millhäusler, Daniel Gehrig and Davide Scaramuzza.

We also introduce DSEC-Flow (download here), the optical flow extension of the DSEC dataset. We are also hosting an automatic evaluation server and a public benchmark!

Visit our project webpage or download the paper directly here for more details. If you use any of this code, please cite the following publication:

@InProceedings{Gehrig3dv2021,
  author = {Mathias Gehrig and Mario Millh\"ausler and Daniel Gehrig and Davide Scaramuzza},
  title = {E-RAFT: Dense Optical Flow from Event Cameras},
  booktitle = {International Conference on 3D Vision (3DV)},
  year = {2021}
}

Download

Download the network checkpoints and place them in the folder checkpoints/:

Checkpoint trained on DSEC

Checkpoint trained on MVSEC 20 Hz

Checkpoint trained on MVSEC 45 Hz

Installation

Please install conda. Then, create new conda environment with python3.7 and all dependencies by running

conda env create --file environment.yml

Datasets

DSEC

The DSEC dataset for optical flow can be downloaded here. We prepared a script download_dsec_test.py for your convenience. It downloads the dataset directly into the OUTPUT_DIRECTORY with the expected directory structure.

download_dsec_test.py OUTPUT_DIRECTORY

MVSEC

To use the MVSEC dataset for our approach, it needs to be pre-processed into the right format. For your convenience, we provide the pre-processed dataset here:

MVSEC Outdoor Day 1 for 20 Hz evaluation

MVSEC Outdoor Day 1 for 45 Hz evaluation

Experiments

DSEC Dataset

For the evaluation of our method with warm-starting, execute the following command:

python3 main.py --path <path_to_dataset>

For the evaluation of our method without warm-starting, execute the following command:

python3 main.py --path <path_to_dataset> --type standard

MVSEC Dataset

For the evaluation of our method with warm-starting, trained on 20Hz MVSEC data, execute the following command:

python3 main.py --path <path_to_dataset> --dataset mvsec --frequency 20

For the evaluation of our method with warm-starting, trained on 45Hz MVSEC data, execute the following command:

python3 main.py --path <path_to_dataset> --dataset mvsec --frequency 45

Arguments

--path : Path where you stored the dataset

--dataset : Which dataset to use: ([dsec]/mvsec)

--type : Evaluation type ([warm_start]/standard)

--frequency : Evaluation frequency of MVSEC dataset ([20]/45) Hz

--visualize : Provide this argument s.t. DSEC results are visualized. MVSEC experiments are always visualized.

--num_workers : How many sub-processes to use for data loading (default=0)