FP16 x gradient clipping

[ZhaofengWu]

According to apex docs, we should have a separate case for gradient clipping under fp16. But it seems not to be the case in pytorch-lightning. Is that correct? https://github.com/PyTorchLightning/pytorch-lightning/blob/732eaee4d735dafd1c90728e5583341d75ff72b5/pytorch_lightning/trainer/training_tricks.py#L26

[github-actions]

Hi! thanks for your contribution!, great first issue!

[Borda]

@neggert @tullie pls ^^

[tullie]

Should we just fix this when we integrate pytorch mixed precision training #1337 (comment)?

[Borda]

depends on what will be faster lol

[williamFalcon]

yeah, let's fix with pytorch mixed precision. We need to remove nvidia apex

[mcarilli]

https://pytorch.org/docs/master/notes/amp_examples.html#gradient-clipping

[samgd]

This is still an issue in 0.7.5 and it's quite critical for training certain recurrent models where mixed precision can cause a 10x reduction in training time.

Can anyone confirm that this is going to be in the 0.7.6 release? We've been hoping it makes 0.7.3, 0.7.4, 0.7.5...

[williamFalcon]

@samgd we already added this! you need to install the latest pytorch from nightly or a specific commit and lightning will use native amp.

it’s not us you’re waiting for haha, it’s pytorch 1.6. We’re already compatible with native amp. @mcarilli

[samgd]

Okay, thanks.

Just to confirm this won't work with 0.7.5 and PyTorch 1.5? What does PyTorch 1.6 add over 1.5, I thought the latter already has native amp?

[williamFalcon]

1.6 does not use nvidia apex. it uses its internal version. 1.5 does not have native amp

[samgd]

Awesome, looking forward to it!

[Borda]

@samgd we already added this! you need to install the latest pytorch from nightly or a specific commit and lightning will use native amp.

it’s not us you’re waiting for haha, it’s pytorch 1.6. We’re already compatible with native amp. @mcarilli

maybe we shall consider to fix it also for lower versions as we still keep back-compatibility...

[mcarilli]

unscale_ before clipping was added in the native amp PR, good. But it might need to be more complicated if there are multiple models and/or optimizers. Does the call to clip_gradients() clip just the gradients owned by the optimizer referred to by optimizer at that point, the gradients owned by a particular model, or all the gradients in the network?

[edenlightning]

@mcarilli mind checking if the issue is still on master?

[edenlightning]

@Borda still relevant?

[teddykoker]

Verified clipping works with native fp16

[williamFalcon]

@teddykoker do we have a test to prevent regressions?

[teddykoker]

Don't believe so, I can add one

[williamFalcon]

it it's quick.