Quantization and scaling

[pooyaww]

@navsuda
In case of DNN, --act_max should have 5 parameters such as --act_max 32 30 30 30 30, while the DNN structure is made of 4 layers

void DNN::run_nn(q7_t* in_data, q7_t* out_data)
{
	// Run all layers
	
	// IP1 
	arm_fully_connected_q7(in_data, ip1_wt, IN_DIM, IP1_OUT_DIM, 1, 7, ip1_bias, ip1_out, vec_buffer);
        // RELU1
	arm_relu_q7(ip1_out, IP1_OUT_DIM);

	// IP2 
	arm_fully_connected_q7(ip1_out, ip2_wt, IP1_OUT_DIM, IP2_OUT_DIM, 2, 8, ip2_bias, ip2_out, vec_buffer);
        // RELU2
	arm_relu_q7(ip2_out, IP2_OUT_DIM);

	// IP3 
	arm_fully_connected_q7(ip2_out, ip3_wt, IP2_OUT_DIM, IP3_OUT_DIM, 2, 9, ip3_bias, ip3_out, vec_buffer);
        // RELU3
	arm_relu_q7(ip3_out, IP3_OUT_DIM);

	// IP4 
	arm_fully_connected_q7(ip3_out, ip4_wt, IP3_OUT_DIM, OUT_DIM, 0, 6, ip4_bias, out_data, vec_buffer);

}

How should I use these 5 --act_max values to scale 4 arm_fully_connected_q7() functions?

[navsuda]

Hi @pooyaww,
The 5 act_max values correspond to the input of first layer and 4 outputs of 4 FC layers.
For quantizing the activations (and finding bias_shift, out_shift parameters), please see Quant_guide. At the end, an example was provided which should help you understand the flow.
Also, --act_max should be powers of 2 only (i.e. 16, 32, 64, ...) in order to do fixed-point quantization (Qm.n).

[gm-jl]

For a 5 layer DS-CNN in the following structure:
CONV1
CONV2_DS
CONV2_PW
CONV3_DS
CONV3_PW
CONV4_DS
CONV4_PW
CONV5_DS
CONV5_PW
FINAL_FC

Why are there 12 act_max values required, and what do they related to?

[navsuda]

12 quantization ranges correspond to the 10 layers that you mentioned + 1 input layer + 1 average pooling layer prior to FINAL_FC.

[ccnankai]

@pooyaww @gm-jl @navsuda
In the quantization process, how to get the value of the quantization ranges -- act_max in quant_test.py .
Where do you get the specific value of the parameter act_max ( x 0 0 0 0 0 0 0 0 0 0 0 ) ?

[pooyaww]

@ccnankai Please don't spam on Issues, instead spend your time on reading materials, codes, articles and other issues. Spamming does not work for open source society.
Currently there is no automated process for finding the best set of maximum quantization values, you should start from the first layer and incrementally quantize each layer to get the optimum performance You need enough data to get the quantization function work properly as the script makes different sections for Training, Validation and Test, with not enough data you don't get to a working solution.

[pooyaww]

@ccnankai Yes, you are right. A greedy approach would lead you to an acceptable result, try levels in the same range you mentioned for the first layer while assigning zeros for other layers then check the accuracy, fix a level for the current layer, then go on until the last layer. It is exactly what has already been mentioned in the quantization tutorial of this repo.

[wayne175]

12 quantization ranges correspond to the 10 layers that you mentioned + 1 input layer + 1 average pooling layer prior to FINAL_FC.

Is this correct?--act_max 32 0 0 0 0 0 0 0 0 0 0 0, ds_cnn useing that --acc_max value.