Use non nested images and batched text Idefics2/3

[yonigozlan]

What does this PR do?

Unlike most other image-text-to-text models, Idefics2 and Idefics3 do not support inputs such as:
images=[image1, image2], text=["text1 <image>", "text2 <image>"]
But needs images to be nested to follow the number of tokens in each prompt:
images=[[image1], [image2]], text=["text1 <image>", "text2 <image>"]

This PR adds support for the former, and also for deducing the nesting of a flat image lists to follow the number of image tokens in each prompt, meaning that this is also supported:
images=[image1, image2, image3], text=["text1 <image>", "text2 <image><image>"]

This is done mostly to have a consistent batching behavior in the image-text-to-text pipeline between Idefics2/3 and other models.

Who can review?

@andimarafioti

[andimarafioti]

Good job! This is a clean way to pass images so it's great that it's supported !

[ArthurZucker]

Thanks, can you -properly test the different error scenario as it is done in:

transformers/tests/models/mllama/test_processor_mllama.py

Line 275 in e4c19d7

def test_process_interleaved_images_prompts_image_error(self):

[ArthurZucker]

indexes can be determined from a cumsum here no?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yonigozlan]

Added tests @ArthurZucker :)

[ArthurZucker]

Thanks, missing one more!

[ArthurZucker]

what about the case when there is 1 image token in 1 prompt, and 1 image:

        text = [
            "This is a test sentence.",
            "In this other sentence we try some good things<image>",
        ]

[yonigozlan]

Hmmm I think this won't raise an error here, I will try

[yonigozlan]

Oh it does raise an error, because it creates an empty list which is not accepted as an image, my bad!
Added the tests and synced them with a copied from