support fuse layers for ptq #35015
support fuse layers for ptq #35015
Conversation
|
Thanks for your contribution! |
| quant_model = ptq.quantize(model, fuse=True, fuse_list=f_l) | ||
| quant_h = ptq.quantize(model_h, fuse=True, fuse_list=f_l) | ||
| for name, layer in quant_model.named_sublayers(): | ||
| print(name, layer) |
There was a problem hiding this comment.
需要check fuse之后quant_model是否符合预期，比如check layer的类型，而不是只print
There was a problem hiding this comment.
已加assert，用于判断是否还存在bn层
| fuse(bool): Whether fuse layers. | ||
| Default: False. | ||
| fuse_list(list): The layers to fuse. | ||
| Default: None. |
There was a problem hiding this comment.
- 如果设置fuse=True，但fuse_list是None，会发生什么。
- 如果设置，格式是啥
这里的注释不全。
| """ | ||
| Add quant config and hook to the target layer. | ||
|
|
||
| Args: | ||
| model(paddle.nn.Layer): The model to be quantized. | ||
| inplace(bool): Whether apply quantization to the input model. | ||
| Default: False. | ||
| fuse(bool): Whether fuse layers. |
| f_l = [['features.0', 'features.1'], ['features.4', 'features.5']] | ||
| quant_model = self.ptq.quantize(model, fuse=True, fuse_list=f_l) | ||
| for name, layer in quant_model.named_sublayers(): | ||
| print(name, layer) |
| (after_acc_top1, self.eval_acc_top1)) | ||
| self.assertTrue( | ||
| infer_acc_top1 >= after_acc_top1, | ||
| msg='The acc is lower after converting model.') |
There was a problem hiding this comment.
加下注释为什么 after_acc_top1， eval_acc_top1， infer_acc_top1 是这个关系。 和 before_acc_top1是啥关系
There was a problem hiding this comment.
done。after和before主要是convert前后的关系，eval_acc_top1是0.95，用于检查精度是否正确，infer是存储后又load出来
|
|
||
|
|
||
| def fuse_layers(model, layers_to_fuse, inplace=False): | ||
| '''fuse layers in layers_to_fuse''' |
|
|
||
| def _fuse_layers(model, layers_list): | ||
| '''fuse all the layers in layers_list''' | ||
| lay_list = [] |
| setattr(parent_layer, sub_name, new_layers[i]) | ||
|
|
||
|
|
||
| def fuse_func(lay_list): |
There was a problem hiding this comment.
方法命名规则请保持一致，如果是模块内函数，请加上下划线。
| def fuse_func(lay_list): | ||
| '''choose the fuser method and fuse layers''' | ||
| types = tuple(type(m) for m in lay_list) | ||
| fuser_method = layer_list_to_fuse_method.get(types, None) |
There was a problem hiding this comment.
严格来讲，不是layer list to fuse method, 是types_to_fusion_method.
| for i in range(1, len(lay_list)): | ||
| identity = Identity() | ||
| identity.training = lay_list[0].training | ||
| new_layers[i] = identity |
There was a problem hiding this comment.
为什么要加identity呢？不能直接把bn layer删掉么？
There was a problem hiding this comment.
这样方便把bn的post hook改到identity上
| types = tuple(type(m) for m in lay_list) | ||
| fuser_method = layer_list_to_fuse_method.get(types, None) | ||
| new_layers = [None] * len(lay_list) | ||
| fused = fuser_method(*lay_list) |
| def fuse_func(lay_list): | ||
| '''choose the fuser method and fuse layers''' | ||
| types = tuple(type(m) for m in lay_list) | ||
| fuser_method = layer_list_to_fuse_method.get(types, None) |
There was a problem hiding this comment.
请斟酌下"fuser_method"的命名是否合适。
| fused_conv.weight.set_value(fused_weight) | ||
| if fused_conv.bias is None: | ||
| fused_conv.bias = paddle.create_parameter( | ||
| shape=[fused_conv._out_channels], is_bias=True, dtype='float32') |
There was a problem hiding this comment.
dtype设置为bn.bias.dtype?
| fused_linear.weight.set_value(fused_weight) | ||
| if fused_linear.bias is None: | ||
| fused_linear.bias = paddle.create_parameter( | ||
| shape=[fused_linear.weight.shape[1]], is_bias=True, dtype='float32') |
|
|
||
| _logger = get_logger( | ||
| __name__, logging.INFO, fmt='%(asctime)s-%(levelname)s: %(message)s') | ||
|
|
||
|
|
||
| class TestFuseLinearBn(unittest.TestCase): | ||
| """ |
PR types
New features
PR changes
Others
Describe
支持了用于离线量化的层融合。可以将卷积层或全连接层以及其后的bn层融合为一个卷积层，计算在inference下是等价的。在离线量化时，可以指定需要融合的层进行融合。层融合可以进一步加速推理，适配某些不支持bn的硬件。