#Structured Matching for Phrase Localization

Created by Mingzhe Wang at University of Michigan, Ann Arbor.

This is released code for:

Mingzhe Wang, Mahmoud Azab, Noriyuki Kojima, Rada Mihalcea, Jia Deng, Structured Matching for Phrase Localization, ECCV 2016. paper

To run this code, make sure the following are installed:

• Torch7
• hdf5
• Matlab

Getting Started

To test with our pretrained model, run:

cd workspace

Download our pretrained model:

./fetch_model.sh

Download features of test data:

./fetch_test_feat.sh

To evaluate the model of bipartite matching:

cd ../src/lua

th test_matching.lua

The predictions of bounding boxes and corresponding scores are stored in workspace/matching/. Open matlab and run evaluation code from Bryan A. Plummer et al:

p=runEval_arg('../../workspace/matching')

To specify a test file, run with -file test-model. To evaluate the model of structured matching:

cd ../lua

th test_pc.lua

The predictions of bounding boxes and corresponding scores are stored in workspace/matching_pc

p=runEval_arg('../../workspace/matching_pc')

In all experiments, we reported Recall@1 in our ECCV paper.

To train your own models, you need to extract features for both phrases and bounding boxes. You can also download our features (63G) with the following commands:

cd workspace/

./fetch_train_feat.sh

To train a model with bipartite matching, in the src/lua folder, run:

th train_matching.lua

To train a model with structured matching, run:

th train_matching_pc.lua

Output models are stored in workspace/model. Please refer to the comments in train_matching_pc.lua to adjust learning parameters.

To generate your own features, you need to

• Download Flickr30k dataset.
• Download Flickr30kEntities dataset.
• Generate bounding boxes.
• Extract features for bounding boxes.
• Extract features for phrases.

The following tools were used to extract features in our experiments:

• Edgebox (generate bounding boxes)
• Fast-rcnn (image features)
• Fisher vector (textual features)

Please let me know if you have any issues about our code.

Acknowledgements

Thanks to Geoff Leyland for providing an excellent implementation of simplex algorithms in lua.