Corruption JSON files solely have question_type": "robust_qas"

[SM20sam]

In https://huggingface.co/datasets/drive-bench/arena

the json file corresponding to corruptions only have the question_type: robust_qas

This is a serious issue as the eval.py expects these question types:
self.results = {
"perception": {
"MCQ": {"gpt": [], "accuracy": []},
"VQA": {"gpt": [], "language": []}
},
"prediction": {
"VQA": {"gpt": [], "language": []}
},
"planning": {
"VQA": {"gpt": [], "language": []}
},
"behavior": {
"MCQ": {"gpt": [], "accuracy": []}}
}

The json files for corruptions need to be updated.

[Daniel-xsy]

There are two ways to use the corrupted images.

• Corrupted images with original text input. Corresponding to all the other numbers reported in the paper, except Table 5.
• Corrupted image with specific text input for each corruption (the screenshot here), corresponding to Table 5 in the paper.

The current evaluation script only supports the first one. We will provide the script to support the second one soon.

[SM20sam]

For the 1st way, "Corrupted images with original text input. Corresponding to all the other numbers reported in the paper, except Table 5.". In drivebench-test.json, "image_path" point to the uncorrupted images, so do I need to modify drivebench-test.json manually to point to the corrupted images?

[Daniel-xsy]

Yes, since those questions are also used for clean answers evaluated in the paper, we use the original image path by default. The path change is also in our released script.

[SM20sam]

By "The path change is also in our released script.", which script are you referring to? Could you give step-by-step instructions on how to run the 1st way, "Corrupted images with original text input. Corresponding to all the other numbers reported in the paper, except Table 5."

[Daniel-xsy]

Sorry for the confusion. You can refer to the code here. Please let me know if you have further questions!