Tensorflow implementation of Reasoning about Entailment with Neural Attention, a paper which addresses the problem of Natural Language Inference with an end-to-end Neural Network architecture. The paper was a collaboration from Deepmind, Oxford University, and University College London.
While most approaches to automatically recognizing entailment relations have used classifiers employing hand engineered features derived from complex natural language processing pipelines, in practice their performance has been only slightly better than bag-of-word pair classifiers using only lexical similarity. The only attempt so far to build an end-to-end differentiable neural network for entailment failed to outperform such a simple similarity classifier. In this paper, we propose a neural model that reads two sentences to determine entailment using long short-term memory units. We extend this model with a word-by-word neural attention mechanism that encourages reasoning over entailments of pairs of words and phrases. Furthermore, we present a qualitative analysis of attention weights produced by this model, demonstrating such reasoning capabilities. On a large entailment dataset this model outperforms the previous best neural model and a classifier with engineered features by a substantial margin. It is the first generic end-to-end differentiable system that achieves state-of-the-art accuracy on a textual entailment dataset.
python 2.7(probably works on Python 3 but untested)tensorflow 1.0.0gensim(for loading word embeddings)numpyandpandas
First clone this repository with git clone git@github.com:thundergolfer/reasoning-about-entailment-tensorflow.git OR you can also fork the repository.
Once cloned, install Miniconda with ./install_miniconda.sh. That should succeed outputting Install complete <INSTALL LOCATION> in the terminal.
Now the Miniconda has been installed you can install all packages into a nice conda virtual environment with ./install.sh. [Currently this doesn't support Windows, sorry]
If everything in the install script went OK, you can enter the created virtual environment with source ./run_in_environment.sh.
- Download the SNLI Dataset (info):
./scripts/get_snli_dataset.sh - Download the pre-trained Word2Vec model (info):
./scripts/get_word2vec.sh
There are two other datasets that I thought would be interesting to use. First is Sentences Involving Compositional Knowledge, a precursor to the SNLI dataset.
Second is the Multi-Genre NLI Corpus (MultiNLI), which is a successor to the SNLI dataset and in some ways superior to it. This new corpus can be combined with the SNLI for training. It is also of the same format to the SNLI, so the same loading and processing code can be used.
- To get SICK run
scripts/get_sick_dataset.sh - To get MultiNLI run
scripts/get_multiNLI_dateset.sh
Currently training is done by cd implementation/ && python train.py
Note: These results do not acheive the accuracy results of the paper. Investigating improvements.
coming soon
-
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, Phil Blunsom, Reasoning about Entailment with Neural Attention, 2015.
-
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean, Distributed Representations of Words and Phrases and their Compositionality, 2013.
-
Google, Large-Scale Machine Learning on Heterogeneous Systems, 2015.
-
Samuel R. Bowman, Gabor Angeli, Christopher Potts, Christopher D. Manning, The Stanford Natural Language Processing Group, A large annotated corpus for learning natural language inference, 2015.
The work done in borelien/entailment-neural-attention-lstm-tf and shyamupa/snli-entailment was helpful to me in this project. I hope you find that my work has extended and expanded theirs in interesting and useful ways.
This project is available under the MIT License.
Jonathon Belotti (thundergolfer) - @jonobelotti_IO - jonathonbelotti.com