From 9dc9af95dc5855a3db85bf58263ce0442a6945d4 Mon Sep 17 00:00:00 2001
From: WZMIAOMIAO <605169423@qq.com>
Date: Wed, 29 Dec 2021 14:42:09 +0800
Subject: [PATCH 1/6] add knowledge_distillation_en.md
---
doc/doc_en/knowledge_distillation_en.md | 315 ++++++++++++++++++++++++
1 file changed, 315 insertions(+)
create mode 100755 doc/doc_en/knowledge_distillation_en.md
diff --git a/doc/doc_en/knowledge_distillation_en.md b/doc/doc_en/knowledge_distillation_en.md
new file mode 100755
index 0000000000..9094b66082
--- /dev/null
+++ b/doc/doc_en/knowledge_distillation_en.md
@@ -0,0 +1,315 @@
+# Knowledge Distillation
+
+- [Knowledge Distillation](#Knowledge Distillation)
+ + [1. Introduction](#1)
+ + [1.1 Introduction to Knowledge Distillation](#11)
+ + [1.2 Introduction to PaddleOCR Knowledge Distillation](#12)
+ + [2. Configuration File Analysis](#2)
+ + [2.1 Recognition Model Configuration File Analysis](#21)
+ - [2.1.1 Model Structure](#211)
+ - [2.1.2 Loss Function ](#212)
+ - [2.1.3 Post-processing](#213)
+ - [2.1.4 Metric Calculation](#214)
+ - [2.1.5 Fine-tuning Distillation Model](#215)
+ + [2.2 Detection Configuration File Analysis](#22)
+
+
+
+## 1. Introduction
+
+### 1.1 Introduction to Knowledge Distillation
+
+In recent years, deep neural networks have been proved to be an extremely effective method for solving problems in the fields of computer vision and natural language processing.
+By constructing a suitable neural network and training it, the performance metrics of the final network model will basically exceed the traditional algorithm.
+When the amount of data is large enough, increasing the amount of parameters by constructing a reasonable network model can significantly improve the performance of the model,
+but this brings about the problem of a sharp increase in the complexity of the model. Large models are more expensive to use in actual scenarios.
+Deep neural networks generally have more parameter redundancy. At present, there are several main methods to compress the model and reduce the amount of its parameters.
+Such as pruning, quantification, knowledge distillation, etc., where knowledge distillation refers to the use of teacher models to guide student models to learn specific tasks,
+to ensure that the small model obtains a relatively large performance improvement under the condition of unchanged parameters.
+In addition, in the knowledge distillation task, a mutual learning model training method was also derived.
+The paper [Deep Mutual Learning](https://arxiv.org/abs/1706.00384) pointed out that using two identical models to supervise each other during the training process can achieve better results than a single model training.
+
+
+### 1.2 Introduction to PaddleOCR Knowledge Distillation
+
+Whether it is a large model distilling a small model, or a small model learning from each other and updating parameters,
+they are essentially the output between different models or mutual supervision between feature maps.
+The only difference is (1) whether the model requires fixed parameters. (2) Whether the model needs to be loaded with a pre-trained model.
+For the case where a large model distills a small model, the large model generally needs to load the pre-trained model and fix the parameters.
+For the situation where small models distill each other, the small models generally do not load the pre-trained model, and the parameters are also in a learnable state.
+
+In the task of knowledge distillation, it is not only the distillation between two models, but also the situation where multiple models learn from each other.
+Therefore, in the knowledge distillation code framework, it is also necessary to support this type of distillation method.
+
+The algorithm of knowledge distillation is integrated in PaddleOCR. Specifically, it has the following main features:
+- It supports mutual learning of any network, and does not require the sub-network structure to be completely consistent or to have a pre-trained model. At the same time, there is no limit to the number of sub-networks, just add it in the configuration file.
+- Support arbitrarily configuring the loss function through the configuration file, not only can use a certain loss, but also a combination of multiple losses.
+- Support all model-related environments such as knowledge distillation training, prediction, evaluation, and export, which is convenient for use and deployment.
+
+Through knowledge distillation, in the common Chinese and English text recognition task, without adding any time-consuming prediction,
+the accuracy of the model can be improved by more than 3%. Combining the learning rate adjustment strategy and the model structure fine-tuning strategy,
+the final improvement is more than 5%.
+
+
+## 2. Configuration File Analysis
+
+In the process of knowledge distillation training, there is no change in data preprocessing, optimizer, learning rate, and some global attributes.
+The configuration files of the model structure, loss function, post-processing, metric calculation and other modules need to be fine-tuned.
+
+The following takes the knowledge distillation configuration file for recognition and detection as an example to analyze the training and configuration of knowledge distillation.
+
+
+### 2.1 Recognition Model Configuration File Analysis
+
+The configuration file is in [ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml).
+
+
+#### 2.1.1 Model Structure
+
+In the knowledge distillation task, the model structure configuration is as follows.
+```yaml
+Architecture:
+ model_type: &model_type "rec" # Model category, recognition, detection, etc.
+ name: DistillationModel # Structure name, in the distillation task, it is DistillationModel
+ algorithm: Distillation # Algorithm name
+ Models: # Model, including the configuration information of the subnet
+ Teacher: # The name of the subnet, it must include at least the `pretrained` and `freeze_params` parameters, and the other parameters are the construction parameters of the subnet
+ pretrained: # Does this sub-network need to load pre-training weights
+ freeze_params: false # Do you need fixed parameters
+ return_all_feats: true # Do you need to return all features, if it is False, only the final output is returned
+ model_type: *model_type # Model category
+ algorithm: CRNN # The algorithm name of the sub-network. The remaining parameters of the sub-network are consistent with the general model training configuration
+ Transform:
+ Backbone:
+ name: MobileNetV1Enhance
+ scale: 0.5
+ Neck:
+ name: SequenceEncoder
+ encoder_type: rnn
+ hidden_size: 64
+ Head:
+ name: CTCHead
+ mid_channels: 96
+ fc_decay: 0.00002
+ Student: # Another sub-network, here is a distillation example of DML, the two sub-networks have the same structure, and both need to learn parameters
+ pretrained: # The following parameters are the same as above
+ freeze_params: false
+ return_all_feats: true
+ model_type: *model_type
+ algorithm: CRNN
+ Transform:
+ Backbone:
+ name: MobileNetV1Enhance
+ scale: 0.5
+ Neck:
+ name: SequenceEncoder
+ encoder_type: rnn
+ hidden_size: 64
+ Head:
+ name: CTCHead
+ mid_channels: 96
+ fc_decay: 0.00002
+```
+
+If you want to add more sub-networks for training, you can also add the corresponding fields in the configuration file according to the way of adding `Student` and `Teacher`.
+For example, if you want 3 models to supervise each other and train together, then `Architecture` can be written in the following format.
+
+```yaml
+Architecture:
+ model_type: &model_type "rec"
+ name: DistillationModel
+ algorithm: Distillation
+ Models:
+ Teacher:
+ pretrained:
+ freeze_params: false
+ return_all_feats: true
+ model_type: *model_type
+ algorithm: CRNN
+ Transform:
+ Backbone:
+ name: MobileNetV1Enhance
+ scale: 0.5
+ Neck:
+ name: SequenceEncoder
+ encoder_type: rnn
+ hidden_size: 64
+ Head:
+ name: CTCHead
+ mid_channels: 96
+ fc_decay: 0.00002
+ Student:
+ pretrained:
+ freeze_params: false
+ return_all_feats: true
+ model_type: *model_type
+ algorithm: CRNN
+ Transform:
+ Backbone:
+ name: MobileNetV1Enhance
+ scale: 0.5
+ Neck:
+ name: SequenceEncoder
+ encoder_type: rnn
+ hidden_size: 64
+ Head:
+ name: CTCHead
+ mid_channels: 96
+ fc_decay: 0.00002
+ Student2: # The new sub-network introduced in the knowledge distillation task, the configuration is the same as above
+ pretrained:
+ freeze_params: false
+ return_all_feats: true
+ model_type: *model_type
+ algorithm: CRNN
+ Transform:
+ Backbone:
+ name: MobileNetV1Enhance
+ scale: 0.5
+ Neck:
+ name: SequenceEncoder
+ encoder_type: rnn
+ hidden_size: 64
+ Head:
+ name: CTCHead
+ mid_channels: 96
+ fc_decay: 0.00002
+```
+
+When the model is finally trained, it contains 3 sub-networks: `Teacher`, `Student`, `Student2`.
+
+The specific implementation code of the `DistillationModel` class can refer to [distillation_model.py](../../ppocr/modeling/architectures/distillation_model.py).
+The final model output is a dictionary, the key is the name of all the sub-networks, for example, here are `Student` and `Teacher`, and the value is the output of the corresponding sub-network,
+which can be `Tensor` (only the last layer of the network is returned) and `dict` (also returns the characteristic information in the middle).
+In the recognition task, in order to add more loss functions and ensure the scalability of the distillation method, the output of each sub-network is saved as a `dict`, which contains the sub-module output.
+Take the recognition model as an example. The output result of each sub-network is `dict`, the key contains `backbone_out`, `neck_out`, `head_out`, and `value` is the tensor of the corresponding module. Finally, for the above configuration file, `DistillationModel` The output format is as follows.
+
+```json
+{
+ "Teacher": {
+ "backbone_out": tensor,
+ "neck_out": tensor,
+ "head_out": tensor,
+ },
+ "Student": {
+ "backbone_out": tensor,
+ "neck_out": tensor,
+ "head_out": tensor,
+ }
+}
+```
+
+
+#### 2.1.2 Loss Function
+
+In the knowledge distillation task, the loss function configuration is as follows.
+
+```yaml
+Loss:
+ name: CombinedLoss # Loss function name
+ loss_config_list: # List of loss function configuration files, mandatory functions for CombinedLoss
+ - DistillationCTCLoss: # CTC loss function based on distillation, inherited from standard CTC loss
+ weight: 1.0 # The weight of the loss function. In loss_config_list, each loss function must include this field
+ model_name_list: ["Student", "Teacher"] # For the prediction results of the distillation model, extract the output of these two sub-networks and calculate the CTC loss with gt
+ key: head_out # In the sub-network output dict, take the corresponding tensor
+ - DistillationDMLLoss: # DML loss function, inherited from the standard DMLLoss
+ weight: 1.0
+ act: "softmax" # Activation function, use it to process the input, can be softmax, sigmoid or None, the default is None
+ model_name_pairs: # The subnet name pair used to calculate DML loss. If you want to calculate the DML loss of other subnets, you can continue to add it below the list
+ - ["Student", "Teacher"]
+ key: head_out
+ - DistillationDistanceLoss: # Distilled distance loss function
+ weight: 1.0
+ mode: "l2" # Support l1, l2 or smooth_l1
+ model_name_pairs: # Calculate the distance loss of the subnet name pair
+ - ["Student", "Teacher"]
+ key: backbone_out
+```
+
+Among the above loss functions, all distillation loss functions are inherited from the standard loss function class.
+The main functions are: Analyze the output of the distillation model, find the intermediate node (tensor) used to calculate the loss,
+and then use the standard loss function class to calculate.
+
+Taking the above configuration as an example, the final distillation training loss function contains the following three parts.
+
+- The final output `head_out` of `Student` and `Teacher` calculates the CTC loss with gt (loss weight equals 1.0). Here, because both sub-networks need to update the parameters, both of them need to calculate the loss with gt.
+- DML loss between `Student` and `Teacher`'s final output `head_out` (loss weight equals 1.0).
+- L2 loss between `Student` and `Teacher`'s backbone network output `backbone_out` (loss weight equals 1.0).
+
+For more specific implementation of `CombinedLoss`, please refer to: [combined_loss.py](../../ppocr/losses/combined_loss.py#L23).
+For more specific implementations of distillation loss functions such as `DistillationCTCLoss`, please refer to [distillation_loss.py](../../ppocr/losses/distillation_loss.py)
+
+
+
+#### 2.1.3 Post-processing
+
+In the knowledge distillation task, the post-processing configuration is as follows.
+
+```yaml
+PostProcess:
+ name: DistillationCTCLabelDecode # CTC decoding post-processing of distillation tasks, inherited from the standard CTCLabelDecode class
+ model_name: ["Student", "Teacher"] # For the prediction results of the distillation model, extract the outputs of these two sub-networks and decode them
+ key: head_out # Take the corresponding tensor in the subnet output dict
+```
+
+Taking the above configuration as an example, the CTC decoding output of the two sub-networks `Student` and `Teahcer` will be calculated at the same time.
+Among them, `key` is the name of the subnet, and `value` is the list of subnets.
+
+For more specific implementation of `DistillationCTCLabelDecode`, please refer to: [rec_postprocess.py](../../ppocr/postprocess/rec_postprocess.py#L128)
+
+
+
+#### 2.1.4 Metric Calculation
+
+In the knowledge distillation task, the metric calculation configuration is as follows.
+
+```yaml
+Metric:
+ name: DistillationMetric # CTC decoding post-processing of distillation tasks, inherited from the standard CTCLabelDecode class
+ base_metric_name: RecMetric # The base class of indicator calculation. For the output of the model, the indicator will be calculated based on this class
+ main_indicator: acc # The name of the indicator
+ key: "Student" # Select the main_indicator of this subnet as the criterion for saving the best model
+```
+
+Taking the above configuration as an example, the accuracy metric of the `Student` subnet will be used as the judgment metric for saving the best model.
+At the same time, the accuracy metric of all subnets will be printed out in the log.
+
+For more specific implementation of `DistillationMetric`, please refer to: [distillation_metric.py](../../ppocr/metrics/distillation_metric.py#L24).
+
+
+
+#### 2.1.5 Fine-tuning Distillation Model
+
+There are two ways to fine-tune the distillation recognition task.
+
+1. Fine-tuning based on knowledge distillation: this situation is relatively simple, download the pre-trained model. Then configure the pre-training model path and your own data path in [ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml) to perform fine-tuning training of the model.
+2. Do not use knowledge distillation in fine-tuning: In this case, you need to first extract the student model parameters from the pre-training model. The specific steps are as follows.
+
+- First download the pre-trained model and unzip it.
+```shell
+wget https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_train.tar
+tar -xf ch_PP-OCRv2_rec_train.tar
+```
+
+- Then use python to extract the student model parameters
+
+```python
+import paddle
+# Load the pre-trained model
+all_params = paddle.load("ch_PP-OCRv2_rec_train/best_accuracy.pdparams")
+# View the keys of the weight parameter
+print(all_params.keys())
+# Weight extraction of student model
+s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key}
+# View the keys of the weight parameters of the student model
+print(s_params.keys())
+# Save weight parameters
+paddle.save(s_params, "ch_PP-OCRv2_rec_train/student.pdparams")
+```
+
+After the extraction is complete, use [ch_PP-OCRv2_rec.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml) to modify the path of the pre-trained model (the path of the exported `student.pdparams` model) and your own data path to fine-tune the model.
+
+
+### 2.2 Detection Configuration File Analysis
+
+- coming soon!
From 2c2db64aead732769808e2dda1ce8e21ce57ecfc Mon Sep 17 00:00:00 2001
From: WZMIAOMIAO <605169423@qq.com>
Date: Wed, 29 Dec 2021 14:46:44 +0800
Subject: [PATCH 2/6] update content
---
doc/doc_en/knowledge_distillation_en.md | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/doc/doc_en/knowledge_distillation_en.md b/doc/doc_en/knowledge_distillation_en.md
index 9094b66082..a13e6a6f13 100755
--- a/doc/doc_en/knowledge_distillation_en.md
+++ b/doc/doc_en/knowledge_distillation_en.md
@@ -1,6 +1,7 @@
+
# Knowledge Distillation
-- [Knowledge Distillation](#Knowledge Distillation)
+- [Knowledge Distillation](#0)
+ [1. Introduction](#1)
+ [1.1 Introduction to Knowledge Distillation](#11)
+ [1.2 Introduction to PaddleOCR Knowledge Distillation](#12)
From 0ca94a5f195ef1122425bf5ae23fac84f3a71bfe Mon Sep 17 00:00:00 2001
From: WZMIAOMIAO <605169423@qq.com>
Date: Wed, 29 Dec 2021 14:50:54 +0800
Subject: [PATCH 3/6] update content
---
doc/doc_en/knowledge_distillation_en.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/doc/doc_en/knowledge_distillation_en.md b/doc/doc_en/knowledge_distillation_en.md
index a13e6a6f13..86c8663454 100755
--- a/doc/doc_en/knowledge_distillation_en.md
+++ b/doc/doc_en/knowledge_distillation_en.md
@@ -12,7 +12,7 @@
- [2.1.3 Post-processing](#213)
- [2.1.4 Metric Calculation](#214)
- [2.1.5 Fine-tuning Distillation Model](#215)
- + [2.2 Detection Configuration File Analysis](#22)
+ + [2.2 Detection Model Configuration File Analysis](#22)
@@ -311,6 +311,6 @@ paddle.save(s_params, "ch_PP-OCRv2_rec_train/student.pdparams")
After the extraction is complete, use [ch_PP-OCRv2_rec.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml) to modify the path of the pre-trained model (the path of the exported `student.pdparams` model) and your own data path to fine-tune the model.
-### 2.2 Detection Configuration File Analysis
+### 2.2 Detection Model Configuration File Analysis
- coming soon!
From 84a2d220b47b3ee9719b0c73dd755b8c8af0be6d Mon Sep 17 00:00:00 2001
From: WZMIAOMIAO <605169423@qq.com>
Date: Thu, 30 Dec 2021 14:57:26 +0800
Subject: [PATCH 4/6] update knowledge_distillation_en.md
---
doc/doc_en/knowledge_distillation_en.md | 280 +++++++++++++++++++++++-
1 file changed, 278 insertions(+), 2 deletions(-)
diff --git a/doc/doc_en/knowledge_distillation_en.md b/doc/doc_en/knowledge_distillation_en.md
index 86c8663454..631591a636 100755
--- a/doc/doc_en/knowledge_distillation_en.md
+++ b/doc/doc_en/knowledge_distillation_en.md
@@ -13,6 +13,12 @@
- [2.1.4 Metric Calculation](#214)
- [2.1.5 Fine-tuning Distillation Model](#215)
+ [2.2 Detection Model Configuration File Analysis](#22)
+ - [2.2.1 Model Structure](#221)
+ - [2.2.2 Loss Function](#222)
+ - [2.2.3 Post-processing](#223)
+ - [2.2.4 Metric Calculation](#224)
+ - [2.2.5 Fine-tuning Distillation Model](#225)
+
@@ -281,7 +287,7 @@ For more specific implementation of `DistillationMetric`, please refer to: [dist
#### 2.1.5 Fine-tuning Distillation Model
-There are two ways to fine-tune the distillation recognition task.
+There are two ways to fine-tune the recognition distillation task.
1. Fine-tuning based on knowledge distillation: this situation is relatively simple, download the pre-trained model. Then configure the pre-training model path and your own data path in [ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml) to perform fine-tuning training of the model.
2. Do not use knowledge distillation in fine-tuning: In this case, you need to first extract the student model parameters from the pre-training model. The specific steps are as follows.
@@ -313,4 +319,274 @@ After the extraction is complete, use [ch_PP-OCRv2_rec.yml](../../configs/rec/ch
### 2.2 Detection Model Configuration File Analysis
-- coming soon!
+The configuration file of the detection model distillation is in the ```PaddleOCR/configs/det/ch_PP-OCRv2/``` directory, which contains three distillation configuration files:
+
+- ```ch_PP-OCRv2_det_cml.yml```, Use one large model to distill two small models, and the two small models learn from each other
+- ```ch_PP-OCRv2_det_dml.yml```, Method of mutual distillation of two student models
+- ```ch_PP-OCRv2_det_distill.yml```, The method of using large teacher model to distill small student model
+
+
+#### 2.2.1 Model Structure
+
+In the knowledge distillation task, the model structure configuration is as follows:
+```
+Architecture:
+ name: DistillationModel # Structure name, in the distillation task, it is DistillationModel
+ algorithm: Distillation # Algorithm name
+ Models: # Model, including the configuration information of the subnet
+ Student: # The name of the subnet, it must include at least the `pretrained` and `freeze_params` parameters, and the other parameters are the construction parameters of the subnet
+ pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained # Does this sub-network need to load pre-training weights
+ freeze_params: false # Do you need fixed parameters
+ return_all_feats: false # Do you need to return all features, if it is False, only the final output is returned
+ model_type: det
+ algorithm: DB
+ Backbone:
+ name: MobileNetV3
+ scale: 0.5
+ model_name: large
+ disable_se: True
+ Neck:
+ name: DBFPN
+ out_channels: 96
+ Head:
+ name: DBHead
+ k: 50
+ Teacher: # Another sub-network, here is a distillation example of a large model distill a small model
+ pretrained: ./pretrain_models/ch_ppocr_server_v2.0_det_train/best_accuracy
+ freeze_params: true # The Teacher model is well-trained and does not need to participate in training
+ return_all_feats: false
+ model_type: det
+ algorithm: DB
+ Transform:
+ Backbone:
+ name: ResNet
+ layers: 18
+ Neck:
+ name: DBFPN
+ out_channels: 256
+ Head:
+ name: DBHead
+ k: 50
+
+```
+If DML is used, that is, the method of two small models learning from each other, the Teacher network structure in the above configuration file needs to be set to the same configuration as the Student model.
+Refer to the configuration file for details. [ch_PP-OCRv2_det_dml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.4/configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_dml.yml)
+
+
+The following describes the configuration file parameters [ch_PP-OCRv2_det_cml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.4/configs/det/ch_PP-OCRv2/ch_PP-OCRv2_det_cml.yml):
+
+```
+Architecture:
+ name: DistillationModel
+ algorithm: Distillation
+ model_type: det
+ Models:
+ Teacher: # Teacher model configuration of CML distillation
+ pretrained: ./pretrain_models/ch_ppocr_server_v2.0_det_train/best_accuracy
+ freeze_params: true # Teacher does not train
+ return_all_feats: false
+ model_type: det
+ algorithm: DB
+ Transform:
+ Backbone:
+ name: ResNet
+ layers: 18
+ Neck:
+ name: DBFPN
+ out_channels: 256
+ Head:
+ name: DBHead
+ k: 50
+ Student: # Student model configuration for CML distillation
+ pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained
+ freeze_params: false
+ return_all_feats: false
+ model_type: det
+ algorithm: DB
+ Backbone:
+ name: MobileNetV3
+ scale: 0.5
+ model_name: large
+ disable_se: True
+ Neck:
+ name: DBFPN
+ out_channels: 96
+ Head:
+ name: DBHead
+ k: 50
+ Student2: # Student2 model configuration for CML distillation
+ pretrained: ./pretrain_models/MobileNetV3_large_x0_5_pretrained
+ freeze_params: false
+ return_all_feats: false
+ model_type: det
+ algorithm: DB
+ Transform:
+ Backbone:
+ name: MobileNetV3
+ scale: 0.5
+ model_name: large
+ disable_se: True
+ Neck:
+ name: DBFPN
+ out_channels: 96
+ Head:
+ name: DBHead
+ k: 50
+
+```
+
+The specific implementation code of the distillation model `DistillationModel` class can refer to [distillation_model.py](../../ppocr/modeling/architectures/distillation_model.py).
+
+The final model output is a dictionary, the key is the name of all the sub-networks, for example, here are `Student` and `Teacher`, and the value is the output of the corresponding sub-network,
+which can be `Tensor` (only the last layer of the network is returned) and `dict` (also returns the characteristic information in the middle).
+
+In the distillation task, in order to facilitate the addition of the distillation loss function, the output of each network is saved as a `dict`, which contains the sub-module output.
+The key contains `backbone_out`, `neck_out`, `head_out`, and `value` is the tensor of the corresponding module. Finally, for the above configuration file, the output format of `DistillationModel` is as follows.
+
+```json
+{
+ "Teacher": {
+ "backbone_out": tensor,
+ "neck_out": tensor,
+ "head_out": tensor,
+ },
+ "Student": {
+ "backbone_out": tensor,
+ "neck_out": tensor,
+ "head_out": tensor,
+ }
+}
+```
+
+
+#### 2.2.2 Loss Function
+
+In the task of detection knowledge distillation ```ch_PP-OCRv2_det_distill.yml````, the distillation loss function configuration is as follows.
+```yaml
+Loss:
+ name: CombinedLoss # Loss function name
+ loss_config_list: # List of loss function configuration files, mandatory functions for CombinedLoss
+ - DistillationDilaDBLoss: # DB loss function based on distillation, inherited from standard DBloss
+ weight: 1.0 # The weight of the loss function. In loss_config_list, each loss function must include this field
+ model_name_pairs: # Extract the output of these two sub-networks and calculate the loss between them
+ - ["Student", "Teacher"]
+ key: maps # In the sub-network output dict, take the corresponding tensor
+ balance_loss: true # The following parameters are the configuration parameters of standard DBloss
+ main_loss_type: DiceLoss
+ alpha: 5
+ beta: 10
+ ohem_ratio: 3
+ - DistillationDBLoss: # Used to calculate the loss between Student and GT
+ weight: 1.0
+ model_name_list: ["Student"] # The model name only has Student, which means that the loss between Student and GT is calculated
+ name: DBLoss
+ balance_loss: true
+ main_loss_type: DiceLoss
+ alpha: 5
+ beta: 10
+ ohem_ratio: 3
+```
+
+Similarly, distillation loss function configuration(`ch_PP-OCRv2_det_cml.yml`) is shown below. Compared with the loss function configuration of ch_PP-OCRv2_det_distill.yml, there are three changes:
+```yaml
+Loss:
+ name: CombinedLoss
+ loss_config_list:
+ - DistillationDilaDBLoss:
+ weight: 1.0
+ model_name_pairs:
+ - ["Student", "Teacher"]
+ - ["Student2", "Teacher"] # 1. Calculate the loss of two Student and Teacher
+ key: maps
+ balance_loss: true
+ main_loss_type: DiceLoss
+ alpha: 5
+ beta: 10
+ ohem_ratio: 3
+ - DistillationDMLLoss: # 2. Add to calculate the loss between two students
+ model_name_pairs:
+ - ["Student", "Student2"]
+ maps_name: "thrink_maps"
+ weight: 1.0
+ # act: None
+ key: maps
+ - DistillationDBLoss:
+ weight: 1.0
+ model_name_list: ["Student", "Student2"] # 3. Calculate the loss between two students and GT
+ balance_loss: true
+ main_loss_type: DiceLoss
+ alpha: 5
+ beta: 10
+ ohem_ratio: 3
+```
+
+For more specific implementation of `DistillationDilaDBLoss`, please refer to: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.4/ppocr/losses/distillation_loss.py#L185).
+For more specific implementations of distillation loss functions such as `DistillationDBLoss`, please refer to: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/04c44974b13163450dfb6bd2c327863f8a194b3c/ppocr/losses/distillation_loss.py?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L148)
+
+
+#### 2.2.3 Post-processing
+
+In the task of detecting knowledge distillation, the post-processing configuration of detecting distillation is as follows.
+
+```yaml
+PostProcess:
+ name: DistillationDBPostProcess # The CTC decoding post-processing of the DB detection distillation task, inherited from the standard DBPostProcess class
+ model_name: ["Student", "Student2", "Teacher"] # Extract the output of multiple sub-networks and decode them. The network that does not require post-processing is not set in model_name
+ thresh: 0.3
+ box_thresh: 0.6
+ max_candidates: 1000
+ unclip_ratio: 1.5
+```
+
+Taking the above configuration as an example, the output of the three subnets `Student`, `Student2` and `Teacher` will be calculated at the same time for post-processing calculations.
+Since there are multiple inputs, there are also multiple outputs returned by post-processing.
+For a more specific implementation of `DistillationDBPostProcess`, please refer to: [db_postprocess.py](../../ppocr/postprocess/db_postprocess.py#L195)
+
+
+#### 2.2.4 Metric Calculation
+In the knowledge distillation task, the metric calculation configuration is as follows.
+```yaml
+Metric:
+ name: DistillationMetric
+ base_metric_name: DetMetric
+ main_indicator: hmean
+ key: "Student"
+```
+
+Since distillation needs to include multiple networks, only one network metrics needs to be calculated when calculating the metrics.
+The `key` field is set to `Student`, it means that only the metrics of the `Student` network is calculated.
+Model Structure
+
+
+#### 2.2.5 Fine-tuning Distillation Model
+
+There are three ways to fine-tune the detection distillation task:
+- `ch_PP-OCRv2_det_distill.yml`, The teacher model is set to the model provided by PaddleOCR or the large model you have trained.
+- `ch_PP-OCRv2_det_cml.yml`, Use cml distillation. Similarly, the Teacher model is set to the model provided by PaddleOCR or the large model you have trained.
+- `ch_PP-OCRv2_det_dml.yml`, Distillation using DML. The method of mutual distillation of the two Student models has an accuracy improvement of about 1.7% on the data set used by PaddleOCR.
+
+In fine-tune, you need to set the pre-trained model to be loaded in the `pretrained` parameter of the network structure.
+
+In terms of accuracy improvement, `cml` > `dml` > `distill`. When the amount of data is insufficient or the accuracy of the teacher model is similar to that of the student, this conclusion may change.
+
+In addition, since the distillation pre-training model provided by PaddleOCR contains multiple model parameters, if you want to extract the parameters of the student model, you can refer to the following code:
+```sh
+# Download the parameters of the distillation training model
+wget https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_distill_train.tar
+```
+
+```python
+import paddle
+# Load the pre-trained model
+all_params = paddle.load("ch_PP-OCRv2_det_distill_train/best_accuracy.pdparams")
+# View the keys of the weight parameter
+print(all_params.keys())
+# Extract the weights of the student model
+s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key}
+# View the keys of the weight parameters of the student model
+print(s_params.keys())
+# Save
+paddle.save(s_params, "ch_PP-OCRv2_det_distill_train/student.pdparams")
+```
+
+Finally, the parameters of the student model will be saved in `ch_PP-OCRv2_det_distill_train/student.pdparams` for the fine-tune of the model.
From d3e0ae7cf2501619be1a76cf17394c5ccc562ee0 Mon Sep 17 00:00:00 2001
From: WZMIAOMIAO <605169423@qq.com>
Date: Thu, 30 Dec 2021 15:15:34 +0800
Subject: [PATCH 5/6] update content
---
doc/doc_en/knowledge_distillation_en.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/doc/doc_en/knowledge_distillation_en.md b/doc/doc_en/knowledge_distillation_en.md
index 631591a636..1db9faef5c 100755
--- a/doc/doc_en/knowledge_distillation_en.md
+++ b/doc/doc_en/knowledge_distillation_en.md
@@ -1,10 +1,10 @@
# Knowledge Distillation
-- [Knowledge Distillation](#0)
++ [Knowledge Distillation](#0)
+ [1. Introduction](#1)
- + [1.1 Introduction to Knowledge Distillation](#11)
- + [1.2 Introduction to PaddleOCR Knowledge Distillation](#12)
+ - [1.1 Introduction to Knowledge Distillation](#11)
+ - [1.2 Introduction to PaddleOCR Knowledge Distillation](#12)
+ [2. Configuration File Analysis](#2)
+ [2.1 Recognition Model Configuration File Analysis](#21)
- [2.1.1 Model Structure](#211)
From 9e7601d919c87983efec12e279bdbb36bbb3a572 Mon Sep 17 00:00:00 2001
From: WZMIAOMIAO <605169423@qq.com>
Date: Thu, 30 Dec 2021 15:16:17 +0800
Subject: [PATCH 6/6] add content
---
doc/doc_ch/knowledge_distillation.md | 53 ++++++++++++++++++++--------
1 file changed, 39 insertions(+), 14 deletions(-)
diff --git a/doc/doc_ch/knowledge_distillation.md b/doc/doc_ch/knowledge_distillation.md
index 265703adc9..0923fc9ccc 100644
--- a/doc/doc_ch/knowledge_distillation.md
+++ b/doc/doc_ch/knowledge_distillation.md
@@ -1,8 +1,27 @@
+
# 知识蒸馏
-
++ [知识蒸馏](#0)
+ + [1. 简介](#1)
+ - [1.1 知识蒸馏介绍](#11)
+ - [1.2 PaddleOCR知识蒸馏简介](#12)
+ + [2. 配置文件解析](#2)
+ + [2.1 识别配置文件解析](#21)
+ - [2.1.1 模型结构](#211)
+ - [2.1.2 损失函数](#212)
+ - [2.1.3 后处理](#213)
+ - [2.1.4 指标计算](#214)
+ - [2.1.5 蒸馏模型微调](#215)
+ + [2.2 检测配置文件解析](#22)
+ - [2.2.1 模型结构](#221)
+ - [2.2.2 损失函数](#222)
+ - [2.2.3 后处理](#223)
+ - [2.2.4 蒸馏指标计算](#224)
+ - [2.2.5 检测蒸馏模型Fine-tune](#225)
+
+
## 1. 简介
-
+
### 1.1 知识蒸馏介绍
近年来，深度神经网络在计算机视觉、自然语言处理等领域被验证是一种极其有效的解决问题的方法。通过构建合适的神经网络，加以训练，最终网络模型的性能指标基本上都会超过传统算法。
@@ -13,6 +32,7 @@
此外，在知识蒸馏任务中，也衍生出了互学习的模型训练方法，论文[Deep Mutual Learning](https://arxiv.org/abs/1706.00384)中指出，使用两个完全相同的模型在训练的过程中互相监督，可以达到比单个模型训练更好的效果。
+
### 1.2 PaddleOCR知识蒸馏简介
无论是大模型蒸馏小模型，还是小模型之间互相学习，更新参数，他们本质上是都是不同模型之间输出或者特征图(feature map)之间的相互监督，区别仅在于 (1) 模型是否需要固定参数。(2) 模型是否需要加载预训练模型。
@@ -30,17 +50,19 @@ PaddleOCR中集成了知识蒸馏的算法，具体地，有以下几个主要
通过知识蒸馏，在中英文通用文字识别任务中，不增加任何预测耗时的情况下，可以给模型带来3%以上的精度提升，结合学习率调整策略以及模型结构微调策略，最终提升提升超过5%。
-
+
## 2. 配置文件解析
在知识蒸馏训练的过程中，数据预处理、优化器、学习率、全局的一些属性没有任何变化。模型结构、损失函数、后处理、指标计算等模块的配置文件需要进行微调。
下面以识别与检测的知识蒸馏配置文件为例，对知识蒸馏的训练与配置进行解析。
+
### 2.1 识别配置文件解析
配置文件在[ch_PP-OCRv2_rec_distillation.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec_distillation.yml)。
+
#### 2.1.1 模型结构
知识蒸馏任务中，模型结构配置如下所示。
@@ -176,6 +198,7 @@ Architecture:
}
```
+
#### 2.1.2 损失函数
知识蒸馏任务中，损失函数配置如下所示。
@@ -212,7 +235,7 @@ Loss:
关于`CombinedLoss`更加具体的实现可以参考: [combined_loss.py](../../ppocr/losses/combined_loss.py#L23)。关于`DistillationCTCLoss`等蒸馏损失函数更加具体的实现可以参考[distillation_loss.py](../../ppocr/losses/distillation_loss.py)。
-
+
#### 2.1.3 后处理
知识蒸馏任务中，后处理配置如下所示。
@@ -228,7 +251,7 @@ PostProcess:
关于`DistillationCTCLabelDecode`更加具体的实现可以参考: [rec_postprocess.py](../../ppocr/postprocess/rec_postprocess.py#L128)
-
+
#### 2.1.4 指标计算
知识蒸馏任务中，指标计算配置如下所示。
@@ -245,7 +268,7 @@ Metric:
关于`DistillationMetric`更加具体的实现可以参考: [distillation_metric.py](../../ppocr/metrics/distillation_metric.py#L24)。
-
+
#### 2.1.5 蒸馏模型微调
对蒸馏得到的识别蒸馏进行微调有2种方式。
@@ -279,7 +302,7 @@ paddle.save(s_params, "ch_PP-OCRv2_rec_train/student.pdparams")
转化完成之后，使用[ch_PP-OCRv2_rec.yml](../../configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml)，修改预训练模型的路径（为导出的`student.pdparams`模型路径）以及自己的数据路径，即可进行模型微调。
-
+
### 2.2 检测配置文件解析
检测模型蒸馏的配置文件在PaddleOCR/configs/det/ch_PP-OCRv2/目录下，包含三个蒸馏配置文件：
@@ -287,7 +310,7 @@ paddle.save(s_params, "ch_PP-OCRv2_rec_train/student.pdparams")
- ch_PP-OCRv2_det_dml.yml，采用DML的蒸馏，两个Student模型互蒸馏的方法
- ch_PP-OCRv2_det_distill.yml，采用Teacher大模型蒸馏小模型Student的方法
-
+
#### 2.2.1 模型结构
知识蒸馏任务中，模型结构配置如下所示：
@@ -419,7 +442,8 @@ Architecture:
}
```
-#### 2.1.2 损失函数
+
+#### 2.2.2 损失函数
知识蒸馏任务中，检测ch_PP-OCRv2_det_distill.yml蒸馏损失函数配置如下所示。
@@ -484,8 +508,8 @@ Loss:
关于`DistillationDilaDBLoss`更加具体的实现可以参考: [distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.4/ppocr/losses/distillation_loss.py#L185)。关于`DistillationDBLoss`等蒸馏损失函数更加具体的实现可以参考[distillation_loss.py](https://github.com/PaddlePaddle/PaddleOCR/blob/04c44974b13163450dfb6bd2c327863f8a194b3c/ppocr/losses/distillation_loss.py?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L148)。
-
-#### 2.1.3 后处理
+
+#### 2.2.3 后处理
知识蒸馏任务中，检测蒸馏后处理配置如下所示。
@@ -504,7 +528,8 @@ PostProcess:
关于`DistillationDBPostProcess`更加具体的实现可以参考: [db_postprocess.py](../../ppocr/postprocess/db_postprocess.py#L195)
-#### 2.1.4 蒸馏指标计算
+
+#### 2.2.4 蒸馏指标计算
知识蒸馏任务中，检测蒸馏指标计算配置如下所示。
@@ -518,8 +543,8 @@ Metric:
由于蒸馏需要包含多个网络，甚至多个Student网络，在计算指标的时候只需要计算一个Student网络的指标即可，`key`字段设置为`Student`则表示只计算`Student`网络的精度。
-
-#### 2.1.5 检测蒸馏模型finetune
+
+#### 2.2.5 检测蒸馏模型finetune
检测蒸馏有三种方式：
- 采用ch_PP-OCRv2_det_distill.yml，Teacher模型设置为PaddleOCR提供的模型或者您训练好的大模型