[fp16] need to automatically convert fp32 inputs to fp16

[stas00]

While this hasn't been an issue with NLP models where inputs are int64 and embeddings convert those into the right float type based on model's weights dtype, for non-NLP models this conversion is lacking. And so the proposal is for Deepspeed to automatically convert float32 inputs to float16 if fp16 is enabled. And do nothing if the inputs aren't fp32.

While the user could convert it themselves this is not the norm since AMP or Apex do it automatically.

Context: huggingface/transformers#11638 (review)

Thank you!

[rosrad]

I think so, DeepSpeed should convert the input to float16 when the fp16 training is enabled. otherwise this is quite different from the AMP/ Apex.

[chrjxj]

probably i got the same case in a cv model training.

with fp16 enabled:

  "fp16": {
    "enabled": true,
    "loss_scale": 0,
    "initial_scale_power": 32,
    "loss_scale_window": 1000,
    "hysteresis": 2,
    "min_loss_scale": 1
  },

the training code got:

    return self._conv_forward(input, self.weight, self.bias)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 395, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same