Skip to content

Commit

Permalink
Reorganise English tokenizer exceptions (as discussed in #718)
Browse files Browse the repository at this point in the history
Add logic to generate exceptions that follow a consistent pattern (like
verbs and pronouns) and allow certain tokens to be excluded explicitly.
  • Loading branch information
ines committed Jan 3, 2017
1 parent fb9d3bb commit 35b39f5
Showing 1 changed file with 418 additions and 1,783 deletions.
Loading

0 comments on commit 35b39f5

Please sign in to comment.