Codes for Accepted Paper : "MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization" in NeurIPS 2019.
Camera-ready is now released here!
Poster is released here!
Slides is released here!
check meta-quantize-tutorial.ipynb for description.
Please check here.
The following command train a ResNet20 using CIFAR10:
python train_base_model.py -m ResNet20 -d CIFAR10
Or users can use the default pretrained model provided by us. Uploaded in Results/model-dataset/model-dataset-pretrain.pth
The following commands run MetaQuant on ResNet20 using CIFAR10 dataset with dorefa as forward quantization method and SGD as optimization.
The resulting quantized model is quantized using 1 bits: {+1, -1} for all layers (conv, fc).
Initial learning rate is set as 1e-3 and decreases by a factor of 0.1 every 30 epochs: 1e-3->1e-4->1e-5:
CUDA_VISIBLE_DEVICES='0' python meta-quantize.py -m ResNet20 -d CIFAR10 -q dorefa -bw 1 -o SGD -meta MultiFC -hidden 100 -ad 30| Model | Dataset | Forward | Backward | Optimizer | Best Acc | Last 10 Acc | FP Acc |
|---|---|---|---|---|---|---|---|
| ResNet20 | CIFAR10 | dorefa | STE | SGD | 82.704(0.044) | 80.745(2.113) | 91.5 |
| MultiFC | 89.666(0.06) | 88.942(0.466) | |||||
| FC-Grad | 89.618(0.007) | 88.840(0.291) | |||||
| LSTMFC | 89.278(0.09) | 88.305(0.81) | |||||
| STE | Adam | 90.222(0.014) | 89.782(0.172) | ||||
| MultiFC | 90.308(0.012) | 89.941(0.068) | |||||
| FC-Grad | 90.407(0.02) | 89.979(0.103) | |||||
| LSTMFC | 90.310(0.011) | 89.962(0.068) | |||||
| BWN | STE | SGD | 78.550(0.035) | 75.913(3.495) | |||
| FC-Grad | 89.500(0.008) | 88.949(0.231) | |||||
| LSTMFC | 89.890(0.018) | 89.289(0.212) | |||||
| STE | Adam | 90.470(0.004) | 89.896(0.182) | ||||
| FC-Grad | 90.426(0.042) | 90.036(0.109) | |||||
| LSTMFC | 90.494(0.030) | 90.042(0.098) |
Best accuracy refers to the best test accuracy recorded during training, mean and variance (in parentheses) for 5 times experiments are reported.
Last 10 acc refers to the test accuracy in the last 10 epochs for 5 times experiments
Raw experimental data can be found in Results/model-dataset/runs-Quant-Multiple
Shadow around line is the max/min value during the multiple trial experiments.
pytorch > 0.4
For using MetaQuant in your own model, the following steps are required:
- Replace all quantized layer with
meta_utils.meta_quantized_module.MetaConvandmeta_utils.meta_quantized_module.Linear - Construct
layer_name_listas inmodels.quantized_meta_resnet.py, which is used to access layer in MetaQuant. - Write your code as in
meta-quantize.py
Leave an issue if there is any bug and email me if any concerns about paper.
Cite the paper if anything helps you:
@article{chen2019metaquant,
title={MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization},
author={Chen, Shangyu and Wang, Wenya and Pan, Sinno Jialin},
journal={Conference on Neural Information Processing Systems},
year={2019}
}