SCN for video captioning

This repo contains the code of using SCN for video captioning, based on the CVPR 2017 paper “Semantic Compositional Networks for Visual Captioning”.

To keep things simple, SCN for image captioning is provided in another separate repo.

Dependencies

This code is written in python. To use it you will need:

• Python 2.7 (do not use Python 3.0)
• Theano 0.7 (you can also use the most recent version)
• A recent version of NumPy and SciPy

Getting started

We provide the code on how to train SCN for video captioning on the Youtube2Text dataset. The SCN used in this experiment is a slightly different version of the original SCN. That is, we feed the video features to each step of the SCN-LSTM, instead of only the first step.

• In order to start, please first download the C3D, ResNet features and tag features for the Youtube2Text dataset we used in the experiments. Put the youtube2text folder inside the ./data folder.

• We also provide our pre-trained model on Youtube2Text. Put the pretrained_model folder into the current directory.

• In order to evaluate the model, please download the standard coco-caption evaluation code. Copy the folder pycocoevalcap into the current directory.

• Now, everything is ready.

How to use the code

1. Run SCN_training.py to start training.

THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python SCN_training.py 

2. Based on our pre-trained model, run SCN_decode.py to generate captions on the Youtube2Text test set. The generated captions are also provided, named youtube2text_scn_test.txt.

3. Now, run SCN_evaluation.py to evaluate the model. The code will output

CIDEr: 0.777, Bleu-4: 0.511, Bleu-3: 0.606, Bleu-2: 0.697, Bleu-1: 0.810, ROUGE_L: 0.706, METEOR: 0.335. 

Train the video tagger

The above results depend on training a good video tagging model. Below, we introduce how we implement this in our paper. In our experiments, in order to train the video tagger, we use data both from Youtube2Text and MSR-VTT. The code is inside the video_tagging folder.

1. In order to start, please first download the ResNet features and captions for the Youtube2Text and MSR-VTT dataset we used in the experiments. They are included in the data folder. Further, inside the preprocess_raw_data folder, we also include how we preprocess the raw data into the format we used in running the code.

2. Run 1_obtain_tags_youtube2text.py to obtain the ground-truth 300 tags for the Youtube2Text dataset.

3. Run 2_obtain_tags_msr_vtt.py to obtain the ground-truth 300 tags for the MSR-VTT dataset. This provides us additional data to train a more robust tagger, compared with using Youtube2Text alone.

4. Run 3_training_video_tagging_model.py to train the video tagger for the Youtube2Text dataset.

THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python 3_training_video_tagging_model.py 

5. The trained model and the resulting predicted tag features are also provided in the above folder, i.e., youtube_tagging_learned_params.npz and tag_feats_pred.mat. The latter is the tag feats we used in running the video captioning code.

6. Now, go back to run SCN_training.py to train the video captioning model.

Citing SCN

Please cite our CVPR paper in your publications if it helps your research:

@inproceedings{SCN_CVPR2017,
  Author = {Gan, Zhe and Gan, Chuang and He, Xiaodong and Pu, Yunchen and Tran, Kenneth and Gao, Jianfeng and Carin, Lawrence and Deng, Li},
  Title = {Semantic Compositional Networks for Visual Captioning},
  booktitle={CVPR},
  Year  = {2017}
}