Some questions about FP16 training

[Shuweis]

Hi! I see that mmediting is support for fp16 training , how can I use it?

[wangruohui]

Some codes are ready, but we haven't fully implemented this feature. you need to modify some source codes and possibly solve some potential bugs.

Currently, mmediting support fp16 based on mmcv.auto_fp16, if some model's forward function is wrapped with @auto_fp16, like this, it potentially supports this feature.

To enable fp16, you need to modify the fp16_enabled variable to True in the __init__, like here. Then you can try to train the model based config files.

Previously, there is a PR #320, but it is some how out-of-dated.

[Shuweis]

I turn fp16_enabled to True，but nothing happened. The training gpu memory did not become small.

[wangruohui]

Oh, my mistake.
I think you also need an FP16OptimzierHook like this.

Depending on whether you are using distributed or non-distributed training, you need to register the hook around line 179 or 304 in mmedit/apis/train.py.

[Shuweis]

https://github.com/open-mmlab/mmdetection/blob/98949809b7179fab9391663ee5a4ab5978425f90/mmdet/apis/train.py#L153
what does fp16_cfg mean? Is it true or false?

[wangruohui]

It's a dictionary like fp16 = dict(loss_scale=512.). The loss_scale variable is used to scale the loss, so that gradient will not underflow in FP16, generally 128 to 512.

At current stage, maybe mmdetection is a good reference. A good trick is to search some keywords in the repo.

[ArchipLab-LinfengZhang]

It's a dictionary like fp16 = dict(loss_scale=512.). The loss_scale variable is used to scale the loss, so that gradient will not underflow in FP16, generally 128 to 512.

At current stage, maybe mmdetection is a good reference. A good trick is to search some keywords in the repo.

Hello. I have tried to use fp16 but find the loss nan problem after around 300~500 iterations. I have tried to use loss_scale=512/128/64/32 but all of them didn't work. I have also tried gradient clip by the way. Do you have any ideas about how to solve this problem?

[zengyh1900]

@LeoXing1996
Please check this issue.

[LeoXing1996]

Hey @Shuweis and @ArchipLab-LinfengZhang , currently MMEdit 1.x has already supported auto-fp16 training, and you are welcome to have a try.
If you still have any problems with FP16 training, please paste your configs and training log, and then we can help you better.