Add ViLT

[NielsRogge]

What does this PR do?

This PR adds ViLT (Vision and Language Transformer).

It's a very nice, minimal multi-modal model, as it only adds a text embedding layer to an existing ViT.

I've defined the following head models:

• ViltForMaskedLM
• ViltForVisualQuestionAnswering
• ViltForNaturalLanguageVisualReasoning
• ViltForImageRetrievalTextRetrieval (CLIP-like model).

To do:

• add ViltForNaturalLanguageVisualReasoning to the tests. However, I do have a question here: it's the only model that requires config.modality_type_vocab_size = 3 instead of 2. How can I handle this exception in the tests? I could do it like this:

for model_class in self.all_model_classes:
    config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    config.return_dict = True

    if model_class.__name__ == "ViltForNaturalLanguageVisualReasoning":
       config.modality_type_vocab_size = 3

But that's not ideal as it would require overwrite each individual test.

Update: fixed by create a separate ModelTester for this particular model, that overrides the get_config.

[LysandreJik]

This looks good, thanks for working on it @NielsRogge!

I left a few comments, and would love @sgugger's review before this is merged.

[LysandreJik]

AutoModelForMaskeLM should correctly return this, no?

[NielsRogge]

I asked @Narsil about this, but the AutoModelForMaskedLM doesn't currently accept models that take several modalities as input. ViLT can take in pixel_values and input_ids, and you can mask out several input_ids, which the model needs to predict. However, the "fill-mask" pipeline currently only works for models that only take input_ids as input.

[Narsil]

The pipeline fill-mask doesn't work with multi modalities. And it assumes every AutoModelForMaskedLM is for filling text only masks.
As I mentionned too:

• Either we don't make it AutoMaskedLM
• Or we make it AutoMaskedLM but we need an escape hatch of some kind so that the pipeline can know it's not supposed to work (or it works and simply doesn't use the image, or has pure padded image or something along those lines)
• AutoModel should work nonetheless (I assume this discards that)

[sgugger]

I'd like the PR to be green (or mostly green) before reviewing.

[NielsRogge]

@sgugger should be mostly green now.

[sgugger]

Thanks a lot for adding this model!

Regarding your question for the tests, I think the easiest would be to define a new Tester and Test class for ViltForNaturalLanguageVisualReasoning that inehrits from the main Tester and Test class this PR adds, then overrides the method to get the config, so that you don't have to rewrite all the tests.

[sgugger]

I have no idea what Natural Language Visual Reasoning means, so there is probably a better name to find here.

[NielsRogge]

It's because this model was fine-tuned on NLVR: https://lil.nlp.cornell.edu/nlvr/

[sgugger]

It probably deserves a nice introduction in the docstring of that model.

[sgugger]

Same comment as for Beit above.

[sgugger]

Note: with the new build dev job merged, you can preview the doc here :-)

[HuggingFaceDocBuilder]

Great job merging this PR! the documentation will now be removed from the staging environment.