Add an option to disable Trainer.detect_nan_tensors

[quinor]

🚀 Feature

Add an option to disable Trainer.detect_nan_tensors

Motivation

This function tends to be pretty slow when your network has got a lot of parameters, especially in small tensors. For example in my case it took ~0.5s per training iteration.

Pitch

Add an option to the Trainer class that disables calling detect_nan_tensors every epoch.

Alternatives

Remove it all together. Bad idea.

Additional context

[quinor]

I'm currently using a workaround:

trainer.detect_nan_tensors = lambda x: 0

[github-actions]

Hi! thanks for your contribution!, great first issue!

[ethanwharris]

This proposal sounds good :) would maybe go as far as to say that it should be disabled by default, certainly if this speeds up testing :). We should also add detect_nan_tensors to the profiler so that users can see how long it is taking

[Borda]

See profiler results here #1354 (comment)

[williamFalcon]

agreed! let's do this