LiteFlowNet

The network structure of LiteFlowNet. For the ease of representation, only a 3-level design is shown.

A cascaded flow inference module M:S in NetE.

This repository (https://github.com/twhui/LiteFlowNet) is the offical release of LiteFlowNet for my paper LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation in CVPR 2018 (Spotlight paper, 6.6%). The up-to-date version of the paper is available on arXiv.

LiteFlowNet is a lightweight, fast, and accurate opitcal flow CNN. We develop several specialized modules including (1) pyramidal features, (2) cascaded flow inference (cost volume + sub-pixel refinement), (3) feature warping (f-warp) layer, and (4) flow regularization by feature-driven local convolution (f-lconv) layer. LiteFlowNet outperforms PWC-Net (CVPR 2018) on KITTI and has a smaller model size (less than PWC-Net by ~40%). For more details about LiteFlowNet, you may visit my project page.

Oral presentation at CVPR 2018 is also available on YouTube.

	KITTI12 Testing Set (Out-Noc)	KITTI15 Testing Set (Fl-all)	Model Size (M)
FlowNet2 (CVPR17)	4.82%	10.41%	162.49
PWC-Net (CVPR18)	4.22%	9.60%	8.75
LiteFlowNet (CVPR18)	3.27%	9.38%	5.37

LiteFlowNet2

NEW! Our extended work (LiteFlowNet2, TPAMI 2020) is now available at https://github.com/twhui/LiteFlowNet2.

LiteFlowNet2 in TPAMI 2020, another lightweight convolutional network, is evolved from LiteFlowNet (CVPR 2018) to better address the problem of optical flow estimation by improving flow accuracy and computation time. Comparing to our earlier work, LiteFlowNet2 improves the optical flow accuracy on Sintel clean pass by 23.3%, Sintel final pass by 12.8%, KITTI 2012 by 19.6%, and KITTI 2015 by 18.8%. Its runtime is 2.2 times faster!

	Sintel Clean Testing Set	Sintel Final Testing Set	KITTI12 Testing Set (Out-Noc)	KITTI15 Testing Set (Fl-all)	Model Size (M)	Runtime* (ms) GTX 1080
FlowNet2 (CVPR17)	4.16	5.74	4.82%	10.41%	162	121
PWC-Net+	3.45	4.60	3.36%	7.72%	8.75	40
LiteFlowNet2	3.48	4.69	2.63%	7.62%	6.42	40

Note: *Runtime is averaged over 100 runs for a Sintel's image pair of size 1024 × 436.

LiteFlowNet3

NEW! Our extended work (LiteFlowNet3, ECCV 2020) is now available at https://github.com/twhui/LiteFlowNet3.

We ameliorate the issue of outliers in the cost volume by amending each cost vector through an adaptive modulation prior to the flow decoding. We further improve the flow accuracy by exploring local flow consistency. To this end, each inaccurate optical flow is replaced with an accurate one from a nearby position through a novel warping of the flow field. LiteFlowNet3 not only achieves promising results on public benchmarks but also has a small model size and a fast runtime.

	Sintel Clean Testing Set	Sintel Final Testing Set	KITTI12 Testing Set (Avg-All)	KITTI15 Testing Set (Fl-fg)	Model Size (M)	Runtime* (ms) GTX 1080
LiteFlowNet (CVPR18)	4.54	5.38	1.6	7.99%	5.4	88
LiteFlowNet2 (TPAMI20)	3.48	4.69	1.4	7.64%	6.4	40
HD3 (CVPR19)	4.79	4.67	1.4	9.02%	39.9	128
IRR-PWC (CVPR19)	3.84	4.58	1.6	7.52%	6.4	180
LiteFlowNet3 (ECCV20)	3.03	4.53	1.3	6.96%	5.2	59

Note: *Runtime is averaged over 100 runs for a Sintel's image pair of size 1024 × 436.

License and Citation

This software and associated documentation files (the "Software"), and the research paper (LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation) including but not limited to the figures, and tables (the "Paper") are provided for academic research purposes only and without any warranty. Any commercial use requires my consent. When using any parts of the Software or the Paper in your work, please cite the following paper:

@InProceedings{hui18liteflownet,    
 author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},    
 title = {{LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation}},    
 booktitle = {{Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},    
 year = {2018},  
 pages = {8981--8989},
 url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} 
}

Datasets

1. FlyingChairs dataset (31GB) and train-validation split.
2. RGB image pairs (clean pass) (37GB) and flow fields (311GB) for Things3D dataset.
3. Sintel dataset (clean + final passes) (5.3GB).
4. KITTI12 dataset (2GB) and KITTI15 dataset (2GB) (Simple registration is required).

	FlyingChairs	FlyingThings3D	Sintel	KITTI
Crop size	448 x 320	768 x 384	768 x 384	896 x 320
Batch size	8	4	4	4

Prerequisite

The code package comes as the modified Caffe from DispFlowNet and FlowNet2 with our new layers, scripts, and trained models.

Reimplementations in Pytorch and TensorFlow are also available.

Installation was tested under Ubuntu 14.04.5/16.04.2 with CUDA 8.0, cuDNN 5.1 and openCV 2.4.8/3.1.0.

Edit Makefile.config (and Makefile) if necessary in order to fit your machine's settings.

For openCV 3+, you may need to change opencv2/gpu/gpu.hpp to opencv2/cudaarithm.hpp in /src/caffe/layers/resample_layer.cu.

If your machine installed a newer version of cuDNN, you do not need to downgrade it. You can do the following trick:

1. Download cudnn-8.0-linux-x64-v5.1.tgz and untar it to a temp folder, say cuda-8-cudnn-5.1

2. Rename cudnn.h to cudnn-5.1.h in the folder /cuda-8-cudnn-5.1/include

3. $ sudo cp cuda-8-cudnn-5.1/include/cudnn-5.1.h /usr/local/cuda/include/	
   $ sudo cp cuda-8-cudnn-5.1/lib64/lib* /usr/local/cuda/lib64/

4. Replace #include <cudnn.h> to #include <cudnn-5.1.h> in /include/caffe/util/cudnn.hpp.

Compiling

$ cd LiteFlowNet
$ make -j 8 tools pycaffe

Feature warping (f-warp) layer

The source files include /src/caffe/layers/warp_layer.cpp, /src/caffe/layers/warp_layer.cu, and /include/caffe/layers/warp_layer.hpp.

The grid pattern that is used by f-warp layer is generated by a grid layer. The source files include /src/caffe/layers/grid_layer.cpp and /include/caffe/layers/grid_layer.hpp.

Feature-driven local convolution (f-lconv) layer

It is implemented using off-the-shelf components. More details can be found in /models/testing/depoly.prototxt or /models/training_template/train.prototxt.template by locating the code segment NetE-R.

Other layers

Two custom layers (ExpMax and NegSquare) are optimized in speed for forward-pass. Generalized Charbonnier loss is implemented in l1loss_layer. The power factor (alpha) can be adjusted in l1_loss_param { power: alpha l2_per_location: true }.

Training

1. Prepare the training set. In /data/make-lmdbs-train.sh, change YOUR_TRAINING_SET and YOUR_TESTING_SET to your favourite dataset.

$ cd LiteFlowNet/data
$ ./make-lmdbs-train.sh

2. Copy files from /models/training_template to a new model folder (e.g. NEW). Edit all the files and make sure the settings are correct for your application. Model for the complete network is provided. LiteFlowNet uses stage-wise training to boost the performance. Please refer to my paper for more details.

$ mkdir LiteFlowNet/models/NEW
$ cd LiteFlowNet/models/NEW
$ cp ../training_template/solver.prototxt.template solver.prototxt	
$ cp ../training_template/train.prototxt.template train.prototxt
$ cp ../training_template/train.py.template train.py

3. Create a soft link in your new model folder

$ ln -s ../../build/tools bin

4. Run the training script

$ ./train.py -gpu 0 2>&1 | tee ./log.txt

Trained models

The trained models (liteflownet, liteflownet-ft-sintel, liteflownet-ft-kitti) are available in the folder /models/trained. Untar the files to the same folder before you use it.

liteflownet: Trained on Chairs and then fine-tuned on Things3D.

liteflownet-ft-sintel: Model used for Sintel benchmark.

liteflownet-ft-kitti: Model used for KITTI benchmark.

Testing

1. Open the testing folder

$ cd LiteFlowNet/models/testing

2. Create a soft link in the folder /testing

$ ln -s ../../build/tools bin

3. Replace MODE in ./test_MODE.py to batch if all the images has the same resolution (e.g. Sintel dataset), otherwise replace it to iter (e.g. KITTI dataset).

4. Replace MODEL in lines 9 and 10 of test_MODE.py to one of the trained models (e.g. liteflownet-ft-sintel).

5. Run the testing script. Flow fields (MODEL-0000000.flo, MODEL-0000001.flo, ... etc) are stored in the folder /testing/results having the same order as the image pair sequence.

$ test_MODE.py img1_pathList.txt img2_pathList.txt results

Evaluation

Average end-point error can be computed using the provided script /models/testing/util/endPointErr.m

Reimplementations in PyTorch and TensorFlow

1. A PyTorch-based reimplementation of LiteFlowNet is available at https://github.com/sniklaus/pytorch-liteflownet.
2. A TensorFlow-based reimplementation of LiteFlowNet is also available at https://github.com/keeper121/liteflownet-tf2.

Declaration

The early version of LiteFlowNet was submitted to ICCV 2017 for review in March 2017. The improved work was published in CVPR 2018.