TVLT: Textless Vision-Language Transformer [NeurIPS 2022 bib]
Zineng Tang*, Jaemin Cho*, Yixin Nie*, Mohit Bansal
Learning compact visual-linguistic Transformer representation from low-level continuous visual 👁 and audio👂 perception signal without assuming the prior existence of written texts or tokens
Transformers for Vision-Language (VL) representation learning heavily rely on text-based inputs. (Some works use audio channel only as auxiliary channel)
TVLT takes audio and visual inputs for VL representation learning with minimal modality-specific design and without text-specific modules such as tokenization and automatic speech recognition (ASR).
TVLT is pre-trained with vision-audio mathcing and mask autoencoding (mask and then reconstruct the continuous input of video frames and audio spectrogram), following the previous idea of training scalable vision learners with mask autoencoding on images (the Vision-BERT).
More
TVLT attains performance comparable to its text-based counterpart on various multimodal tasks, such as visual question answering and multimodal sentiment analysis, with 28x faster inference speed and only 1/3 of the parameters.
conda create -n TVLT python=3.8 # You can also use other environment.
The following version have been tested.
torch 1.10.0 1.12.1torchvision 0.11.1 0.12.1torchaudio 0.10.0 0.13.1
You can try other version of pytorch but make sure that it will be compatible with your cuda and cudnn.
pip install -r requirements.txt
Getting familiar with TVLT by trying the following demos.
- Masked Autoecoding on Video Frames and Audio Spectrogram
- Sentiment Analysis on Video and Audio
- Emotional Analysis on Video and Audio
Pretraining (Data + scripts) -> TVLT Pretraining
Download MAE checkpoint here
# Example
bash scripts/pretrain_mae_vam.sh
Finetuning on Downstream (Data + scripts) -> TVLT Finetuning
# Example
bash scripts/finetune_mosei.sh
The model weights are hosted in Huggingface Hub.
If you have tried the demos, some models should have already been downloaded.
The details of each released TVLT models are described in the table below.
| Training | Input Format | Component | Link |
|---|---|---|---|
| Pre-trained on Howto100m + Yttemporal videos | Video 👁+ Audio👂 | Encoder + Decoder | [link] |
| Pre-trained on Howto100m + Yttemporal videos, then finetuned on CMU-MOSEI sentiment analysis | Video 👁+ Audio👂 | Encoder + Classification Head | [link] |
| Pre-trained on Howto100m + Yttemporal videos, then finetuned on CMU-MOSEI emotional analysis | Video 👁+ Audio👂 | Encoder + Classification Head | [link] |
| {re-trained on Howto100m + Yttemporal videos+ASR, then finetuned on CMU-MOSEI emotional analysis | Video 👁+ Text✍️ | Encoder + Classification Head | [link] |
To be contined... (Stay tuned, more pre-trained variants coming soon)
See Folder Structure
- Initial Code Release
- Notebook Demos
- Colab
- Release TTS question audios for VQA (We convert all the textual questions of VQAv2 to audio using Google TTS API.)
...
In our experiment, we pre-train TVLT on HowTo100M and YTtemporal videos. However, we recommend to unlock the power of TVLT by pre-training TVLT on large-scale videos for more generic Vision-Language representation.
The resultant models can be either use to directly process video (with the audio channel) inputs such as audio-image/video retrieval, audio-VQA, TTS-based VQA or to extract visual-acoustic features for other tasks such as speech translation, multimodal content understanding, etc.
@inproceedings{tang2022tvlt,
title = {TVLT: Textless Vision-Language Transformer},
author = {Zineng Tang and Jaemin Cho and Yixin Nie and Mohit Bansal},
booktitle = {NeurIPS},
year = {2022}
}
The idea of this paper is heavily inspired by Masked Autoencoders Are Scalable Vision Learners.
Our codebase is based on ViLT.
We thank the authors for their open-source contributions.
Zineng Tang (zn.tang.terran@gmail.com)