-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement a tool to calculate a BPE vocabulary #78
Comments
Since the algorithm is already defined in the paper, this would be a matter of using the same for Ittoolbox, correct? Or are there some additional factors you would need in the implementation? |
@anjalibhavan well, the first version would be just the algorithm as described in the paper. Later the tool would support weighting lttoolbox transducers according to the vocabulary of the tool. |
The code is implemented in Python by https://github.com/rsennrich/subword-nmt |
A tool should be included in lttoolbox which calculates a BPE vocabulary as defined in this paper: https://arxiv.org/pdf/1508.07909.pdf
The idea is to use BPE to weight our morphological transducers.
The text was updated successfully, but these errors were encountered: