Request for the details in VQAv2 evaluation

[Leun9]

In the 2-shot evaluation in the paper, how were the examples selected and how were the tokens (including images, questions, answers, s) concatenated?

[sriniiyer]

Something like this:

<img1>Question: <question1> Answer: <answer1>\n<img2>Question: <question2> Answer: <answer2>\n<img3>Question: <question3> Answer:

And I think the few shot examples are taken from topk examples in the train set. Hope this helps.

[Leun9]

Are the example questions selected from questions of the same category? And do we need "END-OF-TURN"s?

Do you have any plans to release this part of the code?

[Leun9]

And how did you evaluate the performance of image captioning in COCO/Flickr30k? Are there more details about input format and prompts?

[sriniiyer]

1. Not from the same category.
2. If you are using the released model, then yes, you need <eot>. If you are SFTing the model yourself, then you can use whatever format you used. Some of the paper results are SFTed models.
3. Since this type of code is very commonly found online in many repo's, we won't be releasing this at the moment.

[Leun9]

So the prompt that can reproduce the results of 66.0 like this?
<img1>Question: <question1> Answer: <answer1>\n<img2>Question: <question2> Answer: <answer2>\n<img3>Question: <question3> Answer:<eot>

What are the prompts for image captioning evaluations?

[sriniiyer]

What is the actual issue here? Have you tried reproducing and are not able to? Let me know what you have tried so far and what results you're getting.