Understanding MP policy for parameters

[rphlstck]

Hey,

first of all, thank you for the nice work!
I have a question regarding the fixed precision of the parameters to float32 in line 321 of train/train.py:

        # init MixedPrecision
        if args.precision != "fp32":
            cast_dtype = get_mp_policy_dtype(args.precision)
            mp_policy = MixedPrecision(
                param_dtype=torch.float32,
                reduce_dtype=cast_dtype,  # gradient communication
                buffer_dtype=cast_dtype,
            )

Did the training degrade when you used lower precision for the parameters? Or is there something else I am not aware of?
(I am quite new to mixed precision training)