Some Whisper beam search output (sequences_scores, etc.) is lost in _stack_split_outputs

[drewhouston]

System Info

• transformers version: 4.43.3
• Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.23.2
• Safetensors version: 0.4.3
• Accelerate version: 0.30.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.3.1+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: no
• Using GPU in script?: yes
• GPU type: NVIDIA GeForce RTX 4090 Laptop GPU

Who can help?

@sanchit-gandhi @kamilakesbi

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

In generating short form output (<30 sec):

# inputs is from the processor
gen_kwargs = {
            "max_new_tokens": 400,
            "num_beams": 5,
            "temperature": None,
            "return_timestamps": False,
            "return_dict_in_generate": True,
            "num_return_sequences": 1,
            "output_scores": True,
            "language": "english"
        }
pred_ids = self.model.generate(inputs, **gen_kwargs)
print(pred_ids.__class__)
print(dict((k,type(v)) for k, v in vars(pred_ids).items()))

Expected behavior

GenerateBeamEncoderDecoderOutput seems to lose some fields in a recent version. (Maybe other output forms are also affected, haven't checked.)

Bisecting transformers versions, in 4.42.4 the output looked like:

<class 'transformers.generation.utils.GenerateBeamEncoderDecoderOutput'>
{'sequences': <class 'torch.Tensor'>, 'sequences_scores': <class 'torch.Tensor'>, 'scores': <class 'tuple'>, 'logits': <class 'NoneType'>, 'beam_indices': <class 'torch.Tensor'>, 'encoder_attentions': <class 'NoneType'>, 'encoder_hidden_states': <class 'NoneType'>, 'decoder_attentions': <class 'NoneType'>, 'cross_attentions': <class 'NoneType'>, 'decoder_hidden_states': <class 'NoneType'>, 'past_key_values': <class 'tuple'>}

In 4.43.0 and after sequences_scores and beam_indices became None:

<class 'transformers.generation.utils.GenerateBeamEncoderDecoderOutput'>
{'sequences': <class 'torch.Tensor'>, 'sequences_scores': <class 'NoneType'>, 'scores': <class 'tuple'>, 'logits': <class 'NoneType'>, 'beam_indices': <class 'NoneType'>, 'encoder_attentions': <class 'NoneType'>, 'encoder_hidden_states': <class 'NoneType'>, 'decoder_attentions': <class 'NoneType'>, 'cross_attentions': <class 'NoneType'>, 'decoder_hidden_states': <class 'NoneType'>, 'past_key_values': <class 'tuple'>}

It looks like these get removed in postprocessing, potential culprit in _stack_split_outputs at

transformers/src/transformers/models/whisper/generation_whisper.py

Line 946 in 9451a38

def _stack_split_outputs(self, seek_outputs, model_output_type, device, kwargs):

Which looks like it changed in #30984.

Hacking in if key in ["sequences", "beam_indices", "sequences_scores"]:, for example, fixes it, although I'm not sure what's intended to be handled as tensors vs. tuples, so will defer as to the best way to fix.

[gante]

(cc @sanchit-gandhi / @kamilakesbi , as this is a Whisper-specific question -- I've double-checked that the core generate is returning these additional outputs :) )

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[gante]

cc @eustlb, another issue for your Whisper fixes 🙏

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.