Horovod with native 16 precision not working

[mutasem-mattar]

🐛 Bug

To Reproduce

Steps to reproduce the behavior:

1. using precision=16 with distributed_backend=horovod

Traceback (most recent call last):
  File "/workspace/main_lightning.py", line 500, in <module>
    main(hyperparams)
  File "/workspace/main_lightning.py", line 492, in main
    trainer.fit(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 1068, in fit
    results = self.horovod_train(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/distrib_parts.py", line 213, in horovod_train
    model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/core/lightning.py", line 954, in configure_apex
    model, optimizers = amp.initialize(model, optimizers, opt_level=amp_level)

Code sample

trainer = Trainer(
        precision=16,
        gpus=1,
        distributed_backend="horovod")

Environment

• PyTorch Version: 1.6.0+cu101
• How you installed PyTorch: pip

[Borda]

mind have look @tgaddair 🐰

[tgaddair]

Absolutely, let me take a look today and get back to you, @mutasem-mattar.