FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
The FlowNet demonstrated that optical flow estimation can be cast as a learning problem. However, the state of the art with regard to the quality of the flow has still been defined by traditional methods. Particularly on small displacements and real-world data, FlowNet cannot compete with variational methods. In this paper, we advance the concept of end-to-end learning of optical flow and make it work really well. The large improvements in quality and speed are caused by three major contributions: first, we focus on the training data and show that the schedule of presenting data during training is very important. Second, we develop a stacked architecture that includes warping of the second image with intermediate optical flow. Third, we elaborate on small displacements by introducing a sub-network specializing on small motions. FlowNet 2.0 is only marginally slower than the original FlowNet but decreases the estimation error by more than 50%. It performs on par with state-of-the-art methods, while running at interactive frame rates. Moreover, we present faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet.
Models | Training datasets | FlyingChairs | Sintel (training) | KITTI2012 (training) | KITTI2015 (training) | Log | Config | Download | ||
clean | final | EPE | Fl-all | EPE | ||||||
---|---|---|---|---|---|---|---|---|---|---|
FlowNet2CS | FlyingChairs | 1.59 | - | - | - | - | - | log | Config | Model |
FlowNet2CS | Flying Chairs + FlyingThing3d subset | - | 1.96 | 3.69 | 3.50 | 28.28% | 8.23 | log | Config | Model |
FlowNet2CSS | FlyingChairs | 1.55 | - | - | - | - | - | log | Config | Model |
FlowNet2CSS | Flying Chairs + FlyingThing3d subset | - | 1.85 | 3.57 | 3.13 | 25.76% | 7.72 | log | Config | Model |
FlowNet2CSS-sd | Flying Chairs + FlyingThing3d subset + ChairsSDHom | - | 1.81 | 3.69 | 2.98 | 25.66% | 7.99 | log | Config | Model |
FlowNet2 | FlyingThing3d subset | 1.78 | 3.31 | 3.02 | 25.18% | 8.02 | log | Config | Model |
Models | Training datasets | ChairsSDHom | Log | Config | Download |
Flownet2sd | ChairsSDHom | 0.37 | log | Config | Model |
---|
@inproceedings{ilg2017flownet,
title={Flownet 2.0: Evolution of optical flow estimation with deep networks},
author={Ilg, Eddy and Mayer, Nikolaus and Saikia, Tonmoy and Keuper, Margret and Dosovitskiy, Alexey and Brox, Thomas},
booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
pages={2462--2470},
year={2017}
}