Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use OSCAR as the 'web' corpus, update tests #47

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

rspeer
Copy link

@rspeer rspeer commented Mar 30, 2021

Some things updated here:

  • The source for 'web' text is now the OSCAR corpus, a cleaned and deduplicated version of Common Crawl. OSCAR replaces our previous Web sources (Leeds Web Corpus and Mokk), which were from the '00s.
  • Text is tokenized using wordfreq 2.4, which changes the tokenization of apostrophes next to some vowels, particularly in French -- this change is already merged in wordfreq but it affects the test data
  • Changed the lists of languages by re-evaluating some Wikipedias on the borderline, excluding languages in corpora that a recent study said were identified with less than 80% accuracy, and along those lines excluding our own Swahili Twitter data that was mostly mis-identified by FTLID

The Snakefile now prints a table of languages and how many sources they have, helping me check the level of support for a given language at a glance.

@frankier
Copy link

This also seems to address a problem where in the current version bad language tags are passed for cleaning up opus counts by introducing the new recount_messy_opus_tokens rule.

@@ -37,6 +37,19 @@ def tokenize_file(
print(' '.join(tokens), file=outfile)


def tokenize_oscar(outfile, language):
import datasets

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datasets should also be added to setup.py

@frankier
Copy link

frankier commented Jul 1, 2021

datasets just added streaming datasets which is useful for OSCAR if you have limited disk. See this commit in my fork: frankier@6fbf8cd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants