Add settings to suppress bf16 compile errors in CI on V100 (#8481) (#…

…8482)

* Add settings to suppress bf16 compile errors in CI on V100

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pablo Garay <pagaray@nvidia.com>

examples/nlp/language_modeling/megatron_gpt_pretraining.py

@@ -13,6 +13,8 @@
 # limitations under the License.
 
 
+# To suppress BF16 compile related issue in the CI runs with turing/V100
+import torch._dynamo
 import torch.multiprocessing as mp
 from omegaconf.omegaconf import OmegaConf, open_dict
 
@@ -22,6 +24,8 @@
 from nemo.utils import logging
 from nemo.utils.exp_manager import exp_manager
 
+torch._dynamo.config.suppress_errors = True
+
 mp.set_start_method("spawn", force=True)