test_encoder_decoder_model_generate for vision_encoder_deocder is flaky

[amyeroberts]

System Info

transformers 4.38.0dev

Who can help?

@gante

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Hard to reproduce, unfortunately.

Running the single test enough times will trigger a failure:

python -m pytest -v tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::ViT2TrOCR::test_encoder_decoder_model_generate

Fails with:

FAILED tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::ViT2TrOCR::test_encoder_decoder_model_generate - AssertionError: torch.Size([13, 8]) != (13, 20)

Reference CI run: https://app.circleci.com/pipelines/github/huggingface/transformers/83611/workflows/666b01c9-1be8-4daa-b85d-189e670fc168/jobs/1078635/tests#failed-test-0

Expected behavior

Non-flaky behaviour for the tests.

[gante]

cc @zucchini-nlp