[Compression v2] Auto-Compress Pruner #4280
Conversation
| Auto Compress Pruner | ||
| -------------------- | ||
|
|
||
| For each round, AutoCompressPruner prune the model for the same sparsity to achive the overall sparsity: |
There was a problem hiding this comment.
"for the same ..." -> for a fixed sparsity ratio (e.g., 20%) to achieve the overall sparsity (e.g., 80%).
There was a problem hiding this comment.
change to
For total iteration number :math:N, AutoCompressPruner prune the model that survive the previous iteration for a fixed sparsity ratio (e.g., :math:1-{(1-0.8)}^{(1/N)}) to achieve the overall sparsity (e.g., :math:0.8)
| .. code-block:: bash | ||
|
|
||
| 1. Generate sparsities distribution using SimulatedAnnealingPruner | ||
| 2. Perform ADMM-based structured pruning to generate pruning result for the next round. |
There was a problem hiding this comment.
why "for the next round"?
There was a problem hiding this comment.
change to
for the next iteration
| The origin unwrapped pytorch model to be pruned. | ||
| config_list : List[Dict] | ||
| The origin config list provided by the user. Note that this config_list is directly config the origin model. | ||
| This means the sparsity provided by the origin_masks should also be recorded in the origin_config_list. |
There was a problem hiding this comment.
what do you want to express with sentences?
There was a problem hiding this comment.
if the model already has 0.2 sparsity by origin_masks and the user wants a total 0.8 sparsity, set sparsity=0.8, not 0.6
There was a problem hiding this comment.
how could user provide origin_masks?
There was a problem hiding this comment.
sorry, I forgot I have removed this API in iterative pruner for simplifying, users need to use the scheduler to support origin_masks right now.
I will remove this line.
| keep_intermediate_result : bool | ||
| If keeping the intermediate result, including intermediate model and masks during each iteration. | ||
| finetuner : Optional[Callable[[Module], None]] | ||
| The finetuner handled all finetune logic, use a pytorch module as input, will be called in each iteration. |
There was a problem hiding this comment.
handled -> handles, use -> take
There was a problem hiding this comment.
", will be ..." -> ". It will be called at the end of each iteration, usually for neutralizing the accuracy loss brought by the pruning in this iteration."
| speed_up : bool | ||
| If set True, speed up the model in each iteration. | ||
| dummy_input : Optional[torch.Tensor] | ||
| If `speed_up` is True, `dummy_input` is required for trace the model in speed up. |
There was a problem hiding this comment.
"for trace" -> "for tracing"
| finetuner : Optional[Callable[[Module], None]] | ||
| The finetuner handled all finetune logic, use a pytorch module as input, will be called in each iteration. | ||
| speed_up : bool | ||
| If set True, speed up the model in each iteration. |
There was a problem hiding this comment.
-> If set True, speed up the model at the end of each iteration to make the pruned model compact.
| Supported pruning algorithm ['level', 'l1', 'l2', 'fpgm', 'slim', 'apoz', 'mean_activation', 'taylorfo', 'admm']. | ||
| This iterative pruner will use the chosen corresponding pruner to prune the model in each iteration. | ||
| pruning_params : Dict | ||
| If the pruner corresponding to the chosen pruning_algorithm has extra parameters, put them as a dict to pass in. |
There was a problem hiding this comment.
"the pruner corresponding to the chosen pruning_algorithm" -> "the chosen one-shot pruner"
There was a problem hiding this comment.
-> If the chosen pruning_algorithm has extra parameters,
J-shang
force-pushed
the
compression_v2_autocompress
branch
from
a8ad839 to
d15fa2d
Compare
Need discussion about the interface aboutadmm_paramsinAutoCompressPruner.__init__(), ref to the usage in AutoCompressPruner example, it has too many params to pass in... And ADMM will have more complex interfaces in the future...TODO:
Known Issue:
Loss increases during multi-GPU training.