DO-Conv: Depthwise Over-parameterized Convolutional Layer

Created by Jinming Cao, Yangyan Li, Mingchao Sun, Ying Chen, Dani Lischinski, Daniel Cohen-Or, Baoquan Chen, and Changhe Tu. Transactions on Image Processing (TIP) 2022.

Introduction

DO-Conv is a depthwise over-parameterized convolutional layer, which can be used as a replacement of conventional convolutional layer in CNNs in the training phase to achieve higher accuracies. In the inference phase, DO-Conv can be fused into a conventional convolutional layer, resulting in the computation amount that is exactly the same as that of a conventional convolutional layer.

Please see our paper for more details, where we demonstrated the advantages of DO-Conv on various benchmark datasets/tasks.

We Highly Welcome Issues

We highly welcome issues, rather than emails, for DO-Conv related questions.

Moreover, it would be great if a minimal reproduciable example code is provide in the issue.

News

1 . We open source the FUSION code in PyTorch. Run the demo example (see the save_with_fusion function for details) to fold D into W when save model trained with DOConv.

python sample_pt_with_fusion.py

The saved models are in the model folder, and the number of model parameters is the same as that using conventional convolutional layers without introducing extra computation at the inference phase. You can refer to the load_model_with_fusion function for model loading, be noted to use a network structure that is exactly the same as the original model but using conventional convolutional layers.

2 . We provide DOConv for the new pytorch version (pytorch==1.10.2, torchvision==0.11.3).

Replace this line：

from do_conv_pytorch import DOConv2d

with

from do_conv_pytorch_1_10 import DOConv2d

to apply this version of DOConv without any other changes.

ImageNet Classification Performance

We take the model zoo of GluonCV as baselines. The settings in the baselines have been tuned to favor baselines, and they are not touched during the switch to DO-Conv. In other words, DO-Conv is the one and only change over baselines, and no hyper-parameter tuning is conducted to favor DO-Conv. We consider GluonCV highly reproducible, but still, to exclude clutter factors as much as possible, we train the baselines ourselves, and compare DO-Conv versions with them, while reporting the performance provided by GluonCV as reference. The results are summarized in this table where the “DO-Conv” column shows the performance gain over the baselines.

Network	Depth	Reference	Baseline	DO-Conv
Plain	18	-	69.97	+1.01
ResNet-v1	18	70.93	70.87	+0.82
	34	74.37	74.49	+0.49
	50	77.36	77.32	+0.08
	101	78.34	78.16	+0.46
	152	79.22	79.34	+0.07
ResNet-v1b	18	70.94	71.08	+0.71
	34	74.65	74.35	+0.77
	50	77.67	77.56	+0.44
	101	79.20	79.14	+0.25
	152	79.69	79.60	+0.10
ResNet-v2	18	71.00	70.80	+0.64
	34	74.40	74.76	+0.22
	50	77.17	77.17	+0.31
	101	78.53	78.56	+0.11
	152	79.21	79.24	+0.14
ResNext	50_32x4d	79.32	79.21	+0.40
MobileNet-v1	-	73.28	73.30	+0.03
MobileNet-v2	-	72.04	71.89	+0.16
MobileNet-v3	-	75.32	75.16	+0.14

DO-Conv Usage

In thie repo, we provide reference implementation of DO-Conv in Tensorflow (tensorflow-gpu==2.2.0), PyTorch (pytorch==1.4.0, torchvision==0.5.0) and GluonCV (mxnet-cu100==1.5.1.post0, gluoncv==0.6.0), as replacement to tf.keras.layers.Conv2D, torch.nn.Conv2d and mxnet.gluon.nn.Conv2D, respectively. Please see the code for more details.

We highly welcome pull requests for adding support for different versions of Pytorch/Tensorflow/GluonCV.

Example Usage: Tensorflow (tensorflow-gpu==2.2.0)

We show how to use DO-Conv based on the examples provided in the Tutorial of TensorFlow with MNIST dataset.

1 . Run the demo example first to get the accuracy of the baseline.

python sample_tf.py

If there is any wrong at this step, please check whether the tensorflow version meets the requirements.

2 . Replace these lines:

self.conv1 = Conv2D(32, 3, activation='relu')
self.conv2 = Conv2D(8, 3, activation='relu')

with

self.conv1 = DOConv2D(32, 3, activation='relu')
self.conv2 = DOConv2D(8, 3, activation='relu')

to apply DO-Conv without any other changes.

python sample_tf.py

3 . We provide the performance improvement in this demo example as follows. (averaged accuracy (%) of five runs)

	run1	run2	run3	run4	run5	avg	+
Baseline	98.5	98.51	98.54	98.46	98.51	98.504	-
DO-Conv	98.71	98.62	98.67	98.75	98.66	98.682	0.178

4 . Then you can use DO-Conv in your own network in this way.

Example Usage: PyTorch (pytorch==1.4.0, torchvision==0.5.0)

We show how to use DO-Conv based on the examples provided in the Tutorial of PyTorch with MNIST dataset.

1 . Run the demo example first to get the accuracy of the baseline.

python sample_pt.py

If there is any wrong at this step, please check whether the pytorch and torchvision versions meets the requirements.

2 . Replace these lines:

model = nn.Sequential(
    Conv2d(1, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    Conv2d(16, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    Conv2d(16, 10, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.AdaptiveAvgPool2d(1),
    Lambda(lambda x: x.view(x.size(0), -1)),
)

with

model = nn.Sequential(
    DOConv2d(1, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    DOConv2d(16, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    DOConv2d(16, 10, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.AdaptiveAvgPool2d(1),
    Lambda(lambda x: x.view(x.size(0), -1)),
)

to apply DO-Conv without any other changes.

python sample_pt.py

3 . We provide the performance improvement in this demo example as follows. (averaged accuracy (%) of five runs)

	run1	run2	run3	run4	run5	avg	+
Baseline	94.63	95.31	95.23	95.24	95.37	95.156	-
DO-Conv	95.59	95.73	95.68	95.70	95.67	95.674	0.518

4 . Then you can use DO-Conv in your own network in this way.

Example Usage: GluonCV (mxnet-cu100==1.5.1.post0, gluoncv==0.6.0)

We show how to use DO-Conv based on the examples provided in the Tutorial of GluonCV with MNIST dataset.

1 . Run the demo example first to get the accuracy of the baseline.

python sample_gluoncv.py

If there is any wrong at this step, please check whether the mxnet and gluoncv versions meets the requirements.

2 . Replace these lines:

self.conv1 = Conv2D(20, kernel_size=(5,5))
self.conv2 = Conv2D(50, kernel_size=(5,5))

with

self.conv1 = DOConv2D(1, 20, kernel_size=(5, 5))
self.conv2 = DOConv2D(20, 50, kernel_size=(5, 5))

to apply DO-Conv, note that the 'in_channels' in DOConv2D of GluonCV should be set explicitly.

python sample_gluoncv.py

3 . We provide the performance improvement in this demo example as follows. (averaged accuracy (%) of five runs)

	run1	run2	run3	run4	run5	avg	+
Baseline	98.10	98.10	98.10	98.10	98.10	98.10	-
DO-Conv	98.26	98.26	98.26	98.26	98.26	98.26	0.16

4 . Then you can use DO-Conv in your own network in this way.