[Features] Add initialization log #1128

MeowZheng · 2021-06-23T07:42:04Z

Motivation

To identify unnoticeable bug when weights initializing, I add initialization log to record this process.

Modification

Call print_log in init_weights in mmcv/runner/base_module.py.
Add init_log function in mmcv/cnn/utils/weight_init.py.
Call init_log in Initializers in mmcv/cnn/utils/weight_init.py.
Call print_log in _initialize_override in mmcv/cnn/utils/weight_init.py.

Use cases

Developers can check whether models' parameters have been initialized as the way they set in init_cfg.
Below is the log msg of RetinaNet initialization.

In [5]: net.init_weights()
2021-06-23 15:36:48,834 - mmcv - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'torchvision://resnet50'}
2021-06-23 15:36:48,834 - mmcv - INFO - load model from: torchvision://resnet50
2021-06-23 15:36:48,834 - mmcv - INFO - Use load_from_torchvision loader
2021-06-23 15:36:49,143 - mmcv - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: fc.weight, fc.bias

2021-06-23 15:36:49,145 - mmcv - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'}
2021-06-23 15:36:49,145 - mmcv - INFO -     initialize weights (torch.Size([256, 512, 1, 1])) of Conv2d in FPN in with XavierInit
2021-06-23 15:36:49,145 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in FPN with 0
2021-06-23 15:36:49,146 - mmcv - INFO -     initialize weights (torch.Size([256, 1024, 1, 1])) of Conv2d in FPN in with XavierInit
2021-06-23 15:36:49,146 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in FPN with 0
2021-06-23 15:36:49,147 - mmcv - INFO -     initialize weights (torch.Size([256, 2048, 1, 1])) of Conv2d in FPN in with XavierInit
2021-06-23 15:36:49,147 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in FPN with 0
2021-06-23 15:36:49,149 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in FPN in with XavierInit
2021-06-23 15:36:49,149 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in FPN with 0
2021-06-23 15:36:49,151 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in FPN in with XavierInit
2021-06-23 15:36:49,151 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in FPN with 0
2021-06-23 15:36:49,154 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in FPN in with XavierInit
2021-06-23 15:36:49,154 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in FPN with 0
2021-06-23 15:36:49,157 - mmcv - INFO -     initialize weights (torch.Size([256, 2048, 3, 3])) of Conv2d in FPN in with XavierInit
2021-06-23 15:36:49,157 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in FPN with 0
2021-06-23 15:36:49,175 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in FPN in with XavierInit
2021-06-23 15:36:49,175 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in FPN with 0
2021-06-23 15:36:49,177 - mmcv - INFO - initialize RetinaHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01, 'override': {'type': 'Normal', 'name': 'retina_cls', 'std': 0.01, 'bias_prob': 0.01}}
2021-06-23 15:36:49,178 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in RetinaHead in with NormalInit
2021-06-23 15:36:49,178 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in RetinaHead with 0
2021-06-23 15:36:49,181 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in RetinaHead in with NormalInit
2021-06-23 15:36:49,181 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in RetinaHead with 0
2021-06-23 15:36:49,185 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in RetinaHead in with NormalInit
2021-06-23 15:36:49,185 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in RetinaHead with 0
2021-06-23 15:36:49,189 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in RetinaHead in with NormalInit
2021-06-23 15:36:49,189 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in RetinaHead with 0
2021-06-23 15:36:49,193 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in RetinaHead in with NormalInit
2021-06-23 15:36:49,193 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in RetinaHead with 0
2021-06-23 15:36:49,197 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in RetinaHead in with NormalInit
2021-06-23 15:36:49,197 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in RetinaHead with 0
2021-06-23 15:36:49,201 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in RetinaHead in with NormalInit
2021-06-23 15:36:49,201 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in RetinaHead with 0
2021-06-23 15:36:49,204 - mmcv - INFO -     initialize weights (torch.Size([256, 256, 3, 3])) of Conv2d in RetinaHead in with NormalInit
2021-06-23 15:36:49,205 - mmcv - INFO -     fill bias (torch.Size([256])) of Conv2d in RetinaHead with 0
2021-06-23 15:36:49,208 - mmcv - INFO -     initialize weights (torch.Size([720, 256, 3, 3])) of Conv2d in RetinaHead in with NormalInit
2021-06-23 15:36:49,208 - mmcv - INFO -     fill bias (torch.Size([720])) of Conv2d in RetinaHead with 0
2021-06-23 15:36:49,219 - mmcv - INFO -     initialize weights (torch.Size([36, 256, 3, 3])) of Conv2d in RetinaHead in with NormalInit
2021-06-23 15:36:49,219 - mmcv - INFO -     fill bias (torch.Size([36])) of Conv2d in RetinaHead with 0
2021-06-23 15:36:49,219 - mmcv - INFO -     initialize weight and bias in retina_cls with {'type': 'Normal', 'std': 0.01, 'bias_prob': 0.01}

codecov · 2021-06-23T08:30:58Z

Codecov Report

Merging #1128 (f34281e) into master (f7caa80) will increase coverage by 0.19%.
The diff coverage is 92.13%.

❗ Current head f34281e differs from pull request most recent head 1a918e9. Consider uploading reports for the commit 1a918e9 to get more accurate results

@@            Coverage Diff             @@
##           master    #1128      +/-   ##
==========================================
+ Coverage   67.64%   67.84%   +0.19%     
==========================================
  Files         159      159              
  Lines       10295    10367      +72     
  Branches     1858     1873      +15     
==========================================
+ Hits         6964     7033      +69     
- Misses       2965     2966       +1     
- Partials      366      368       +2

Flag	Coverage Δ
unittests	`67.84% <92.13%> (+0.19%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcv/image/__init__.py	`100.00% <ø> (ø)`
mmcv/runner/hooks/evaluation.py	`86.70% <77.41%> (-1.46%)`	⬇️
mmcv/cnn/bricks/wrappers.py	`54.20% <100.00%> (ø)`
mmcv/cnn/utils/weight_init.py	`92.18% <100.00%> (+0.84%)`	⬆️
mmcv/image/geometric.py	`94.67% <100.00%> (+0.47%)`	⬆️
mmcv/runner/base_module.py	`78.57% <100.00%> (+2.89%)`	⬆️
mmcv/utils/config.py	`89.88% <100.00%> (+0.02%)`	⬆️
mmcv/utils/misc.py	`95.00% <0.00%> (+1.42%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f7caa80...1a918e9. Read the comment docs.

mmcv/cnn/utils/weight_init.py

ZwwWayne · 2021-06-25T06:09:23Z

LGTM, see if others have any comments.

jshilong · 2021-06-25T11:50:29Z

mmcv/runner/base_module.py

+                loggername = loggernames[0] if len(loggernames) > 0 else 'mmcv'
+                print_log(
+                    f'initialize {modulename} with init_cfg {self.init_cfg}',
+                    logger=loggername)


L52
isinstance(self.init_cfg, (dict, ConfigDict)): change to
isinstance(self.init_cfg, dict): may be clearer
because ConfigDict is subclass of dict

jshilong · 2021-06-25T12:08:58Z

I believe all the keys should only appear once in the log, such as override keys would appear twice, if children's has its own init config, the key would appear several times

jshilong · 2021-06-25T12:15:48Z

Can we maintain a dict to record which type of initialization in the init_cfg is ultimately used

ZwwWayne · 2021-07-02T12:54:36Z

continued in #1150

MeowZheng added 2 commits June 23, 2021 14:37

[Features] Add initialization log

4e5ca58

minors

08aee51

MeowZheng requested review from ZwwWayne and zhouzaida June 23, 2021 07:42

ZwwWayne reviewed Jun 23, 2021

View reviewed changes

mmcv/cnn/utils/weight_init.py Outdated Show resolved Hide resolved

MeowZheng added 2 commits June 24, 2021 21:17

get used logger name

b1f966b

fix bug

1a918e9

MeowZheng requested a review from jshilong June 25, 2021 06:22

jshilong reviewed Jun 25, 2021

View reviewed changes

ZwwWayne mentioned this pull request Jun 27, 2021

Iteration Plan v1.3.9 - July 2021 #1151

Closed

20 tasks

ZwwWayne closed this Jul 2, 2021

zhouzaida mentioned this pull request Jul 10, 2021

Iteration Plan v1.3.10 - July 2021 #1191

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Features] Add initialization log #1128

[Features] Add initialization log #1128

MeowZheng commented Jun 23, 2021

codecov bot commented Jun 23, 2021 •

edited

Loading

ZwwWayne commented Jun 25, 2021

jshilong Jun 25, 2021

jshilong commented Jun 25, 2021 •

edited

Loading

jshilong commented Jun 25, 2021 •

edited

Loading

ZwwWayne commented Jul 2, 2021

[Features] Add initialization log #1128

[Features] Add initialization log #1128

Conversation

MeowZheng commented Jun 23, 2021

Motivation

Modification

Use cases

codecov bot commented Jun 23, 2021 • edited Loading

Codecov Report

ZwwWayne commented Jun 25, 2021

jshilong Jun 25, 2021

Choose a reason for hiding this comment

jshilong commented Jun 25, 2021 • edited Loading

jshilong commented Jun 25, 2021 • edited Loading

ZwwWayne commented Jul 2, 2021

codecov bot commented Jun 23, 2021 •

edited

Loading

jshilong commented Jun 25, 2021 •

edited

Loading

jshilong commented Jun 25, 2021 •

edited

Loading