This PyTorch implementation accompanies the paper:
Jochen Gast and Stefan Roth, Deep Video Deblurring: The Devil is in the Details,
ICCV Workshop on Learning for Computational Imaging (ICCVW), Seoul, Korea, November, 2019
Preprint: https://arxiv.org/abs/1909.12196 
Contact: Jochen Gast (jochen.gast@visinf.tu-darmstadt.de)
- Components for Deep Video Deblurring
- Datasets
- Installation, External Packages & Telegram Bot
- Deep Visual Inference Framework
- License
- Citation
Ignoring the framework code, the components for Deep Video Deblurring are contained in the modules
| Modules | Description |
|---|---|
| datasets.gopro_nah | implementation (+ augmentations) for reading Nah et al's GoPro dataset |
| datasets.gopro_su | implementation (+ augmentations) for reading Su et al's GoPro dataset |
| losses.deblurring_error | loss implementation (just plain MSE) |
| models.deblurring.dbn | our baseline implementation for Su et al's DBN network |
| models.deblurring.flowdbn | our FlowDBN implementation |
| models.flow.flownet1s | FlowNet1S optical flow network used in FlowDBN |
| models.flow.pwcnet | PWC optical flow network used in FlowDBN |
| visualizers.gopro_inference | TensorBoard Visualizer for inference (on a given dataset) |
| visualizers.gopro_visualizer | TensorBoard Visualizer for monitoring training progress |
The mirrors of the GoPro datasets can be found here:
| GoPro Dataset | Paper |
|---|---|
| Su et al mirror |
S. Su, M. Delbracio, J. Wang, G. Sapiro, W. Heidrich, and O. Wang. Deep video deblurring for hand-held cameras, CVPR, 2017. |
| Nah et al mirror |
S. Nah, T. H. Kim, and K. M. Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring, CVPR, 2017. |
This code has been tested with Python 3.6, PyTorch 1.2, Cuda 10.0 but is likely to run with newer versions of PyTorch and Cuda. For installing PyTorch, follow the steps here.
Additional dependencies can be installed by running the script
pip install -r requirements.txtThe code for the PWC network relies on a correlation layer implementation provided by Clément Pinard.
To install the correlation layer, run
cd external_packages && . install_spatial_correlation_sampler.shThe framework supports status updates sent via Telegram, i.e. using
logging.telegram(msg, *args)Per default, any finished epoch will send an update with the current losses. Note that the dependencies for the Telegram bot, i.e. filetype jsonpickle telegrambotapiwrapper, are included in requirements.txt, but they are not essential to the framework.
In order to use the Telegram bot, you need to copy dot_machines.json to ~/.machines.json and fill in the required chat_id and tokens. Here you can find out how to obtain these credentials.
The scripts make use of the environment variables DATASETS_HOME pointing to the root folder of the respective dataset,
and EXPERIMENTS_HOME to create output artifacts.
| Scripts | Description |
|---|---|
| scripts.deblurring.gopro_nah | Scripts for GoPro by Nah et al. |
| scripts.deblurring.gopro_su | Scripts for GoPro by Su et al. |
To train the models you need the prewarping networks. Place them in $EXPERIMENTS_HOME/prewarping_networks.
| Models | Description |
|---|---|
| Prewarping Networks | FlowNet1S and PWCNet |
More models will follow soon!
For adding new classes/functions following packages are relevant:
| Modules | Description |
|---|---|
| augmentations | Register (gpu) augmentations |
| contrib | Supposed to be used for PyTorch utilities |
| datasets | Register datasets |
| losses | Register losses |
| models | Register models |
| optim | Register optimizers |
| utils | Any non-PyTorch related utilities |
| visualizers | Register Tensorboard visualizers |
Most things are automated for convenience. To accomodate this automation, there are three essential Python dictionaries being passed around which hold any artifact produced along the way
| Dictionary Name | Description |
|---|---|
| example_dict | The example dictionary. Gets created by a dataset and is supposed to hold all data relevant to a dataset example. Note that an augmentation is supposed to receive and return an example_dict. The example dictionary is received by augmentations, losses, models, and visualizers. |
| model_dict | The model dictionary. Gets created by a model. Supposed to hold any data, i.e. predictions, generated by a model. The example dictionary is received by losses and visualizers. |
| loss_dict | The loss dictionary. Gets created by a loss. Supposed to hold any data, i.e. metrics, generated by a loss. The loss dictionary is received by visualizers. |
Keep this in mind, when adding new functionality.
Again, the framework passes around dictionaries between augmentations, datasets, losses, models, and visualizers. You may want a subset of these tensors to be transfered to Cuda along the way. The are two magic prefixes for dictionary keys to assure that, i.e. input* and target*. Any input or target tensor that should be transfered to Cuda along the way, has to start with one of these prefixes.
E.g. a dataset may return a dictionary with the keys input1 and target1 (both cpu tensors); any augmentation, loss, or model will fetch these as gpu tensors.
This is in particular interesting for the GoPro datasets. The framework allows dataloaders to return multiple -- typically photometric -- augmentations per single example, in order to increase throughput when loading large images. In such a case a tensor returned from a single dataset example should have three dimensions. The pipeline will stack the first dimension into a batch.
Let's say, a dataset returns four examples per load, i.e. input1 and target1 tensors have the dimension 4xHxW. For, e.g. batch_size=8, the resulting tensors (received by the augmentation, loss, model, and visualizers) will be 32xHxW.
There are some automated logging facilities which log any output into the given save directory.
- A
args.txtandargs.jsonfile remembering the passed arguments. - A
logbook.txtgets created remembering all calls tologging.info(msg, *args). - CSV/MAT file generation of any loss (of any epoch) found in the
loss_dict. - For every run, the complete source code gets zipped into the log directory.
- There is always a tensorboard created at the
$SAVE/tbdirectory.
When calling tensorboard summaries in your models for debugging, I suggest to replace any call to torch.utils.tensorboard.SummaryWriter.add_* by utils.summary.*
For instance, instead of
from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter(..)
writer.add_scalar(tensor1)
writer.add_images(tensor2)use
from utils import summary
summary.scalar(tensor1)
summary.images(tensor2)Advantages:
- This will automatically use the created tensorboard in the
savedirectory. - The current epoch gets automatically set by the framework.
- Otherwise, they are identical.
A limited amount of customization is available in constants.py where you can set
- Logging indents, time zones, and colors
- Filename defaults
- Flush intervals
- etc
This code is licensed under the Apache License, Version 2.0, as found in the LICENSE file.
# brief
@inproceedings{Gast:2019:DVD,
Author = {Jochen Gast and Stefan Roth},
Booktitle = {ICCV Workshops},
Title = {Deep Video Deblurring: {T}he Devil is in the Details},
Year = {2019}
}# detailed
@inproceedings{Gast:2019:DVD,
Address = {Seoul, Korea},
Author = {Jochen Gast and Stefan Roth},
Booktitle = {ICCV Workshop on Learning for Computational Imaging (ICCVW)},
Month = nov,
Title = {Deep Video Deblurring: {T}he Devil is in the Details},
Year = {2019}
}