Does Flamingo support NLVR^2 and its paired-image input setting?

[chenxshuo]

Hi there,

thank you for your awesome work!

I wonder whether you have tried OpenFlamingo on NLVR^2. The input in NLVR^2 is always a pair of images along with a question and the output is either true or false.

An example

I am not sure whether the in-context setting in OpenFlamingo supports such paired-image input as demonstration. Shoud I use a pair of <image> to indicate these two images in one example?

Do you have any comments?

Best.

[anas-awadalla]

Hello!

The way I would format the input would be to do <image><|endofchunk|><image>The left image contains twice the number of dogs as the right image, and at least two dogs in total are standing.. One limitation of doing so is that you would get less 'signal' from the first image and the text can only attend to the immediate image. Maybe one thing you can explore is combining the images and passing them in as a single image?

[chenxshuo]

Hi @anas-awadalla many thanks to your reply! Let me try both methods.

[meharbhatia]

Hey @chenxshuo. Do you receive fair results when combining the images as a single image using OpenFlamingo?