rspy

The core implementation of Fast Rolling Shutter Correction in the Wild, TPAMI 2023 and Towards Nonlinear-Motion-Aware and Occlusion-Robust Rolling Shutter Correction, ICCV 2023.

3GS	Gpark

🚀 Features

• A lightweight library for rolling shutter correction which is easy to use.
• Support linear, quadratic, and cubic motion models.
• Support sparse feature correction and can be plugged into 3D vision algorithm pipeline, such as SfM, SLAM, etc.

Installation

pip install git+https://github.com/DelinQu/rspy
# pip install rspy is unavailable now

It's recommended to clone the code and run the demo files in the rspy folder. In our implementation, we use open-mmlab/mmflow to calculate the optical flow. Please refer to the installation for optical flow support, and check the requirements.txt to satisfy the dependency.

Usage

rspy contains linear, quadratic, and cubic models for faster rolling shutter correction. The solver is the core of the rspy, which receives the optical flow fields and returns the correction field. The feats_sampling function warps the RS image back to GS one.

0. Hyparameters

• gamma: the ratio of the exposure time to the frame interval.
• tau: the normalized timestamp warping to.

1. Linear rolling shutter correction

linear_flow receives a optical flow field from $I_{0} \to I_{-1}$ and return correction field $u_{0 \to \tau}$.

F0tau = solver(F0n1, gamma, tau)  # * (1,h,w,2)
rsc_image = feats_sampling(rs_image, -F0tau)

2. Quadratic rolling shutter correction

quadratic_flow receives two optical flow fields from $I_{0} \to I_{-1}$ and $I_{0} \to I_{1}$, and return correction field $u_{0 \to \tau}$.

F0tau = solver(F0n1, F01, gamma, tau)  # * (1,h,w,2)
rsc_image = feats_sampling(rs_image, -F0tau)

3. Cubic rolling shutter correction

cubic_flow receives three optical flow fields from $I_{0} \to I_{-2}$, $I_{0} \to I_{-1}$ and $I_{0} \to I_{1}$, and return correction field $u_{0 \to \tau}$.

F0tau = solver(F0n2, F0n1, F01, gamma, tau)  # * (1,h,w,2)
rsc_image = feats_sampling(rs_image, -F0tau)

🍀 Demo

We provided a demo for rolling shutter correction in rspy， which read the images from the demo folder and save the results in the out folder. Before that, you must download a optical flow model by mim:

mim download mmflow --config raft_8x2_100k_mixed_368x768 

It will be automatically downloaded to ~/.cache/mim. Then, the demo can be run by the following command.

python rspy/demo.py --model=linear

python rspy/demo.py --model=quadratic

python rspy/demo.py --model=cubic

You can also use your own images with a suitable gamma and tau to get a satisfactory result.

📈 Releases

You can see the list of available releases on the GitHub Releases page.

Label	Title in Releases
the foundation of rspy	🐣 Features
relase the pypi package	🚀 Release
To be continue :)	⇨ ▶️

🛡 License

This project is licensed under the terms of the GNU GPL v3.0 license. See LICENSE for more details.

📃 Citation

If you find this project useful for your research, please use the following BibTeX entry:

@ARTICLE{qu2023fast,
  author={Qu, Delin and Liao, Bangyan and Zhang, Huiqing and Ait-Aider, Omar and Lao, Yizhen},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Fast Rolling Shutter Correction in the Wild}, 
  year={2023},
  volume={},
  number={},
  pages={1-18},
  doi={10.1109/TPAMI.2023.3284847}
}

@article{qu2023towards,
  title   = {Towards Nonlinear-Motion-Aware and Occlusion-Robust Rolling Shutter Correction},
  author  = {Delin Qu and Yizhen Lao and Zhigang Wang and Dong Wang and Bin Zhao and Xuelong Li},
  year    = {2023},
  journal = {arXiv preprint arXiv: 2303.18125}
}

Credits

This project was generated with python-package-template