Skip to content

Bilingual lexicon induction (BLI) training and test sets extracted from PanLex - used in the work of Vulić et al. (EMNLP 2019)

License

Notifications You must be signed in to change notification settings

cambridgeltl/panlex-bli

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

34 Commits
 
 
 
 
 
 

Repository files navigation

PanLex-based bilingual lexicons for 210 language pairs (15 languages)

Contact: Ivan Vulić (iv250 AT cam DOT ac DOT uk)

This repo contains bilingual lexicons for 210 language pairs (15 languages in total) used in the empirical comparison paper Do We Really Need Fully Unsupervised Cross-Lingual Embeddings? (Vulić et al., EMNLP 2019)

The bilingual lexicons have been extracted from the PanLex database of translations: more details on the extraction procedure are provided in the paper.

References

If you use these lexicons in your own work, please cite the following paper:

@inproceedings{Vulic:2019clwe,
  author    = {Vuli\'{c}, Ivan and Glava\v{s}, Goran and Reichart, Roi and Korhonen, Anna},
  title     = {Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?},
  booktitle = {Proceedings of the 2019 Conference 
              on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2019},
  }

Please also acknowledge the use of PanLex by citing the following paper:

@inproceedings{Kamholz:2014panlex,
  author    = {David Kamholz and Jonathan Pool and Susan M. Colowick},
  title     = {{PanLex: B}uilding a Resource for Panlingual Lexical Translation},
  booktitle = {Proceedings of the 9th International Conference 
              on Language Resources and Evaluation (LREC)},
  pages     = {3145--3150},
  year      = {2014},
  }

Languages and Lexicons

The following 15 languages are currently covered in the repo. For each language X, we provide a separate zip archive where X is the source language (L1) paired with the remaining 14 target (L2) languages.

If you want to download the entire bundle for all 210 language pairs at once, please click here

Some (Important) Remarks:

  • The format of the lexicons should be self-explanatory. The files are tab-delimited. We provide training lexicons of different size N (N = 5000, 2000, 1000, 500).

  • The provided lexicons are of course not perfect and have not been manually cleaned or verified although the extraction process from PanLex was quite strict in order to focus on high precision. Still, there might be some noise in the lexicons. Therefore, the lexicons should be considered as silver standard.

  • For some language pairs (due to the strict extraction process), the number of pairs in the lexicons is smaller than the desired 5K training pairs or 2K test pairs - please double-check before running any size-related analyses of the lexicons and projection-based methods.

  • For any further questions, please contact Ivan Vulić (iv250 AT cam DOT ac DOT uk)

About

Bilingual lexicon induction (BLI) training and test sets extracted from PanLex - used in the work of Vulić et al. (EMNLP 2019)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published