How to calculate the FLOPs and Params after freezing and pruning a model

[whowho888]

Dear Dr.Jia,

I attempted to run your code and encountered some questions regarding the calculation of the model's params and FLOPs.

dummy_input = torch.randn(1, 3, 32, 32)
flops, params = profile(glb_model, (dummy_input,))

It seems you are using the above statements to compute the model's params and FLOPs. However, after applying operations such as freezing and pruning, the resulting params and FLOPs should theoretically change as described in your paper. Yet, regardless of the pruning ratio, the computed values remain unchanged and reflect the pre-pruning model's params and FLOPs.

I suspect that my understanding of the code might be insufficient, so I would like to ask for your guidance on how to correctly compute the FLOPs and params after the model has undergone freezing and pruning.

Thank you for your time and assistance.

[YongzheJia]

Thanks for your attention to our work!

I'm glad to address your questions. You can try using the "stat" tool to compute the model's params and FLOPs.
Firstly, you can import the tool by
from torchstat import stat
and then use
stat(glb_model, (3, 28, 28)) (after pruning).
This way I think your parameters and FLOPs will be correct.

For the freezing operation, you can calculate the unfrozen part, as the frozen part doesn't need to be trained.

I hope these answers are helpful to you. Once again, thank you for your interest in our work!

[whowho888]

Thank you very much for your patient response!

I had also considered using the
stat(glb_model, (3, 28, 28))
from your code to calculate the model's params and FLOPs. However, after doing so, I encountered errors during subsequent model training. The specific error message is as follows:

Traceback (most recent call last):
File "/root/FedLPS-main/fedml_experiments/standalone/fedlps/main_fedlps.py", line 472, in
fedavgAPI.train()
File "/root/FedLPS-main/fedml_api/standalone/fedavg/fedlps_api.py", line 258, in train
w = client.train(copy.deepcopy(pr_model.state_dict()))
File "/root/FedLPS-main/fedml_api/standalone/fedavg/client.py", line 30, in train
self.model_trainer.train(self.local_training_data, self.device, self.args)
File "/root/FedLPS-main/fedml_api/standalone/fedavg/my_model_trainer_classification.py", line 55, in train
log_probs = model(x)
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 169, in forward
return self.module(*inputs[0], **kwargs[0])
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torchvision/models/resnet.py", line 285, in forward
return self._forward_impl(x)
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torchvision/models/resnet.py", line 268, in _forward_impl
x = self.conv1(x)
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torchstat/model_hook.py", line 47, in wrap_call
itemsize = input[0].detach().numpy().itemsize
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Before reaching out to you, I tried many methods to resolve the issue, but none were successful. Have you encountered this problem before? How did you solve it?

I feel so sorry to bothering you again and look forward to your reply.

[whowho888]

Thank you very much for your patient response!

I had also considered using the stat(glb_model, (3, 28, 28)) from your code to calculate the model's params and FLOPs. However, after doing so, I encountered errors during subsequent model training. The specific error message is as follows:

Traceback (most recent call last):
File "/root/FedLPS-main/fedml_experiments/standalone/fedlps/main_fedlps.py", line 472, in
fedavgAPI.train()
File "/root/FedLPS-main/fedml_api/standalone/fedavg/fedlps_api.py", line 258, in train
w = client.train(copy.deepcopy(pr_model.state_dict()))
File "/root/FedLPS-main/fedml_api/standalone/fedavg/client.py", line 30, in train
self.model_trainer.train(self.local_training_data, self.device, self.args)
File "/root/FedLPS-main/fedml_api/standalone/fedavg/my_model_trainer_classification.py", line 55, in train
log_probs = model(x)
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 169, in forward
return self.module(*inputs[0], **kwargs[0])
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torchvision/models/resnet.py", line 285, in forward
return self._forward_impl(x)
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torchvision/models/resnet.py", line 268, in _forward_impl
x = self.conv1(x)
File "/root/miniconda3/envs/pyt1.9/lib/python3.8/site-packages/torchstat/model_hook.py", line 47, in wrap_call
itemsize = input[0].detach().numpy().itemsize
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Before reaching out to you, I tried many methods to resolve the issue, but none were successful. Have you encountered this problem before? How did you solve it?

I feel so sorry to bothering you again and look forward to your reply.

Although the subsequent training cannot continue, it is possible to calculate the model's parameters and FLOPs after first pruning. Therefore, I attempted to modify the model's pruning rate, but I found that the results calculated by stat still do not change with the pruning rate.

The figure below shows the parameters and FLOPs when the pruning rate is 0.2:

The figure below shows the parameters and FLOPs when the pruning rate is 0.6:

It can be seen that, despite different pruning rates, the calculated values remain exactly the same.