This is a PyTorch reimplementation of the following paper:
- Hua He, Kevin Gimpel, and Jimmy Lin. Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 1576-1586.
On top of which we performed additional ablation experiments and explored a number of variants, as described in:
- Zhucheng Tu. An Experimental Analysis of Multi-Perspective Convolutional Neural Networks. Master's Thesis, Computer Science, University of Waterloo, 2018.
Please ensure you have followed instructions in the main README doc before running any further commands in this doc. The commands in this doc assume you are under the root directory of the Castor repo.
We provide pre-trained models for SICK, TrecQA, and WikiQA in the Castor-models repository. They are trained using the commands in each of the dataset sections below; evaluation metrics match the reported values in those sections in our environment.
SICK (mp_cnn/mpcnn.sick.model)
$ python -m mp_cnn ../Castor-models/mp_cnn/mpcnn.sick.model --dataset sick \
--skip-training
TrecQA (mp_cnn/mpcnn.trecqa.model)
$ python -m mp_cnn ../Castor-models/mp_cnn/mpcnn.trecqa.model --dataset trecqa \
--holistic-filters 200 --skip-training
WikiQA (mp_cnn/mpcnn.wikiqa.model)
$ python -m mp_cnn ../Castor-models/mp_cnn/mpcnn.wikiqa.model --dataset wikiqa \
--holistic-filters 100 --skip-training
The commands above assume GPU; for running on the CPU, add --device -1.
If you want to train the models yourself, please read on.
To run MP-CNN on the SICK dataset, use the following command. --dropout 0 is for mimicking the original paper, although adding dropout can improve results. If you have any problems running it check the Troubleshooting section below.
python -m mp_cnn mpcnn.sick.model.castor --dataset sick --epochs 19 --dropout 0 --lr 0.0005
| Implementation and config | Pearson's r | Spearman's p | MSE |
|---|---|---|---|
| Paper | 0.8686 | 0.8047 | 0.2606 |
| PyTorch using above config | 0.8738 | 0.8116 | 0.2414 |
To run MP-CNN on the TrecQA dataset, use the following command:
python -m mp_cnn mpcnn.trecqa.model --dataset trecqa --epochs 5 --holistic-filters 200 --lr 0.00018 --regularization 0.0006405 --dropout 0
| Implementation and config | map | mrr |
|---|---|---|
| Paper | 0.764 | 0.827 |
| PyTorch using above config | 0.777 | 0.821 |
This are the TrecQA raw dataset results. The paper results are reported in Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks.
You also need trec_eval for this dataset, similar to TrecQA.
Then, you can run:
python -m mp_cnn mpcnn.wikiqa.model --epochs 10 --dataset wikiqa --epochs 5 --holistic-filters 100 --lr 0.00042 --regularization 0.0001683 --dropout 0
| Implementation and config | map | mrr |
|---|---|---|
| Paper | 0.693 | 0.709 |
| PyTorch using above config | 0.717 | 0.729 |
The paper results are reported in Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks.
To see all options available, use
python -m mp_cnn --help
To optionally visualize the learning curve during training, we make use of https://github.com/lanpa/tensorboard-pytorch to connect to TensorBoard. These projects require TensorFlow as a dependency, so you need to install TensorFlow before running the commands below. After these are installed, just add --tensorboard when running the training commands and open TensorBoard in the browser.
pip install tensorboardX
pip install tensorflow-tensorboard