Rat(ing) Pred(ictor) – A Referenceless NLG Quality Estimation Tool
RatPred predicts NLG quality ratings/ranks using a recurrent neural network. It is trained using a set of human-rated/ranked NLG outputs along with the corresponing source meaning representations (MRs). It can then estimate human ratings/ranks given a MR and an NLG system output only. Unlike most automated metrics used for NLG, such as BLEU or NIST, RatPred does not need human-authored reference texts.
For details on the system architecture and its performance, please refer to our ICML-LGNL paper.
Note that RatPred is highly experimental and only tested, so bugs are inevitable. If you find a bug, feel free to contact me or open an issue.
See USAGE.md. Note that currently RatPred is using Python 2. Python 3 version is in the works.
If you use or refer to RatPred, please cite or INLG 2019 paper:
Ondřej Dušek, Karin Sevegnani, Ioannis Konstas, and Verena Rieser (2019): Automatic Quality Estimation for Natural Language Generation: Ranting (Jointly Rating and Ranking). In INLG 2019, Tokyo, Japan.
You can also refer to our ICML-LGNL paper:
Ondřej Dušek, Jekaterina Novikova, and Verena Rieser (2017): Referenceless Quality Estimation for Natural Language Generation. In Proceedings of the 1st Workshop on Learning to Generate Natural Language, Sydney, Australia.
Author: Ondřej Dušek
Copyright © 2017-2018 Interaction Lab, Heriot-Watt University, Edinburgh.
Copyright © 2019 Institute of Formal and Applied Linguistics, Charles University, Prague.
Licensed under the Apache License, Version 2.0 (see LICENSE.txt).
This research received funding from the EPSRC projects DILiGENt (EP/M005429/1) and MaDrIgAL (EP/N017536/1) and Charles University project PRIMUS/19/SCI/10. The Titan Xp used for this research was donated by the NVIDIA Corporation.