Added customio for seq2seq models and updated input names

[kdulla]

added customio to seq2seq compile and updated input names to match other models

[quic-amitraj]

Apply lint and format @kdulla

[quic-rishinr]

The format changes will go as part of #372. it can be rebased after that.

[quic-xiyushi]

The comment says inputs, but the line below is iterating output_names. We need to add input_features to custom_io.

Otherwise, input_features are still float32, as in the generation code input_features are explicitly converted to float32:
line 1905 inputs["input_features"] = inputs["input_features"].numpy().astype(np.float32) and line 1939 inputs["input_features"] = np.zeros((self.batch_size, self.model.config.num_mel_bins, 1)).astype(np.float32)

[quic-xiyushi]

@kdulla Could you add input_features to custom_io and see if dtype conversion can be removed?

[kdulla]

we are adding inputs to custom_io in this loop, we get the names of inputs by slicing "_RetainedState" off the end of output_names, this is just the most straightforward to get the input_names that will be compatible with future Seq2Seq models added.

input_features are in float32 because that is the output type of WhisperProcessor, the generate is written in a way that the processor inputs are taken directly as input, so we expect float32.

[quic-xiyushi]

input_features are in float32 because that is the output type of WhisperProcessor, the generate is written in a way that the processor inputs are taken directly as input, so we expect float32.

However, for vision models, even though they use AutoProcessor which outputs pixel_values in float32, pixel_values is still part of the custom_io and is set to float16. : https://github.com/quic/efficient-transformers/pull/336/files Additionally, inside the generate function vision_inputs["pixel_values"] = vision_inputs["pixel_values"].astype("float16")

So I don't see any reason that seq2seq models can't do the same. It would be great if the design choices were more consistent, so that in vLLM we don't have to convert multimodal inputs to different data types for different models.

[kdulla]

Have updated seq2seq models to match vision models and use float16

[quic-xiyushi]

Thanks. I have verified the PR in vLLM and it looks good to me.