Grounded semantic parser

This is the code for our paper at EMNLP 2018, Grounding language acquisition by training semantic parsers using captioned videos. Please cite us if you use our code or data and feel free to reach out with any questions.

All code and data are included in this repo except the videos; download here.

Running the Project

Data

There are two main experiment directories:
    visionparser/ (full supervision)
    groundedvisionparser (weak supervision that uses the videos and sentence tracker)

The training and test CCG files are identical except the weakly supervised format does not include the ground truth form.

To download videos, go to Dropbox.

Setting up sentence tracker

This is only necessary when training the model under weak supervision. If you intend to do a fully supervised run as a benchmark, or to use a saved model for either full or weak supervision, you can skip this step.

This tracker is from Seeing what you’re told: Sentence-guided activity recognition in video and A compositional framework for grounding language inference, generation, and acquisition in video.

Running Model

The script pipeline.sh builds the necessary Java code and runs a model from start to finish (this includes training and test). pipeline.sh has many required and optional arguments. Run pipeline.sh -h for a detailed description of arguments.

We write (most of) the experiment files from scratch. The EXPNAME.exp file is written from scratch. We write a unique file for each fold of the cross-validation, defining the training and test sets.

The libraries and model are recompiled.

There are a lot of options for pipeline.sh and they're described in the getopts section of the script.

sample commands:
./pipeline.sh -d visionparser -r RUNNAME -v s -t f (full supervision)
./pipeline.sh -d groundedvision parser -r RUNNAME -v s -o PORT -O 1 -t w (weak supervision)

Overview of directories

• experiments
• st-for-parser: sentence tracker

Editing Project

Definitely the easiest way to edit and run all of the Java code. This project requires Java 8+ (installation help).

Once Eclipse is installed, open and load the project. Add classpath variable

• Window -> Preferences -> Java -> Build Path -> Classpath Variables
  
• new variable named TINY_REPO; set path to lib/spf
  

Switch to workspace that differs from our project folder (e.g. eclipse-workspace).
Import project

• select root directory (e.g. working-VisionParser)
• import dependencies (select lib/spf & select nested directories)

Remove ignore line to run a test case; run as J1test.

We modify the Cornell SPF codebase- (a special thank you for publicly sharing your code!)