Merge model compression dev branch to master #1571
Conversation
model compression
* Update Overview.md
* refactor compression sdk * bugfix * bugfix * update ut
* update doc * formatting * bugfix * add import to examples
| |Name|Brief Introduction of Algorithm| | ||
| |---|---| | ||
| | [Level Pruner](./Pruner.md#level-pruner) | Pruning the specified ratio on each weight based on absolute values of weights | | ||
| | [AGP Pruner](./Pruner.md#agp-pruner) | To prune, or not to prune: exploring the efficacy of pruning for model compression. [Reference Paper](https://arxiv.org/abs/1710.01878)| |
There was a problem hiding this comment.
can you explain the AGP pruner in short sentence?
There was a problem hiding this comment.
I dunno how the algorithms works... Maybe @tanglang96 can help?
There was a problem hiding this comment.
AGP means Automated Gradual Pruning, prune with gradually increasing sparsity while training (given initial sparsity and final sparsity) until the final sparsity reached, the smallest magnitude weights are masked with zeros.
There was a problem hiding this comment.
@liuzhe-lz better to add this sentence
There was a problem hiding this comment.
Seems too long. I added "automated gradual pruning".
|
|
||
| ### Usage of user customized compression algorithm | ||
|
|
||
| __[TODO]__ ... |
There was a problem hiding this comment.
In fact we have not considered customized compressor yet...
|
|
||
| The interface for customizing quantization algorithm is similar to that of pruning algorithms. The only difference is that `calc_mask` is replaced with `quantize_weight`. `quantize_weight` directly returns the quantized weights rather than mask, because for quantization the quantized weights cannot be obtained by applying mask. | ||
|
|
||
| ``` |
| ### Other APIs | ||
|
|
||
| Some compression algorithms use epochs to control the progress of compression, and some algorithms need to do something after every minibatch. Therefore, we provide another two APIs for users to invoke. One is `update_epoch`, you can use it as follows: | ||
|
|
There was a problem hiding this comment.
could you explain when the user needs to use 'update_epoch'?
There was a problem hiding this comment.
Added a reference to example compressor using update_epoch. But failed to find an example using step.
| >The binary weight masks are updated every ∆t steps as the network is trained to gradually increase the sparsity of the network while allowing the network training steps to recover from any pruning-induced loss in accuracy. In our experience, varying the pruning frequency ∆t between 100 and 1000 training steps had a negligible impact on the final model quality. Once the model achieves the target sparsity sf , the weight masks are no longer updated. The intuition behind this sparsity function in equation | ||
|
|
||
| ### Usage | ||
| You can prune all weight from %0 to 80% sparsity in 10 epoch with the code below. |
There was a problem hiding this comment.
Yes, seems that's how the algorithm work.
| """ | ||
| def __init__(self, config_list): | ||
| """ | ||
| Configure Args |
There was a problem hiding this comment.
could you follow the docstring format in other tuner...
There was a problem hiding this comment.
I found compressors' docstring format is different from other modules (like tuner).
But I don't know what format we are using exactly.
@SparkSnail Any idea?
|
Updates are going to PR #1590 |
No description provided.