Pragmatic Tokenizer

Pragmatic Tokenizer is a multilingual tokenizer to split a string into tokens.

Installation

Add this line to your application's Gemfile:

Ruby

gem install pragmatic_tokenizer

Ruby on Rails
Add this line to your application’s Gemfile:

gem 'pragmatic_tokenizer'

Usage

If no language is specified, the library will default to English.
To specify a language use its two character ISO 639-1 code.
Pragmatic Tokenizer will unescape any HTML entities.

Example Usage

text = "\"I said, 'what're you? Crazy?'\" said Sandowsky. \"I can't afford to do that.\""

PragmaticTokenizer::Tokenizer.new.tokenize(text)
# => ["\"", "i", "said", ",", "'", "what're", "you", "?", "crazy", "?", "'", "\"", "said", "sandowsky", ".", "\"", "i", "can't", "afford", "to", "do", "that", ".", "\""]

# You can pass many different options to #initialize:
options = {
  language:            :en, # the language of the string you are tokenizing
  abbreviations:       ['a.b', 'a'], # a user-supplied array of abbreviations (downcased with ending period removed)
  stop_words:          ['is', 'the'], # a user-supplied array of stop words (downcased)
  remove_stop_words:   true, # remove stop words
  contractions:        { "i'm" => "i am" }, # a user-supplied hash of contractions (key is the contracted form; value is the expanded form - both the key and value should be downcased)
  expand_contractions: true, # (i.e. ["isn't"] will change to two tokens ["is", "not"])
  filter_languages:    [:en, :de], # process abbreviations, contractions and stop words for this array of languages
  punctuation:         :none, # see below for more details
  numbers:             :none, # see below for more details
  remove_emoji:        :true, # remove any emoji tokens
  remove_urls:         :true, # remove any urls
  remove_emails:       :true, # remove any emails
  remove_domains:      :true, # remove any domains
  hashtags:            :keep_and_clean, # remove the hastag prefix
  mentions:            :keep_and_clean, # remove the @ prefix
  clean:               true, # remove some special characters
  classic_filter:      true, # removes dots from acronyms and 's from the end of tokens
  downcase:            false, # do not downcase tokens
  minimum_length:      3, # remove any tokens less than 3 characters
  long_word_split:     10 # split tokens longer than 10 characters at hypens or underscores
}

Options

`language`

default = 'en'

To specify a language use its two character ISO 639-1 code as a symbol (i.e. :en) or string (i.e. 'en')

`abbreviations`

default = nil

You can pass an array of abbreviations to overide or compliment the abbreviations that come stored in this gem. Each element of the array should be a downcased String with the ending period removed.

`stop_words`

default = nil

You can pass an array of stop words to overide or compliment the stop words that come stored in this gem. Each element of the array should be a downcased String.

`contractions`

default = nil

You can pass a hash of contractions to overide or compliment the contractions that come stored in this gem. Each key is the contracted form downcased and each value is the expanded form downcased.

`remove_stop_words`

default = false

true
Removes all stop words.
false
Does not remove stop words.

`expand_contractions`

default = false

true
Expands contractions (i.e. i'll -> i will).
false
Leaves contractions as is.

`filter_languages`

default = nil

You can pass an array of languages of which you would like to process abbreviations, stop words and contractions. This language can be indepedent of the language of the string you are tokenizing (for example your text might be German but contain some English stop words that you want to remove). If you supply your own abbreviations, stop words or contractions they will be merged with the abbreviations, stop words and contractions of any languages you add in this option. You can pass an array of symbols or strings (i.e. [:en, :de] or ['en', 'de'])

`punctuation`

default = 'all'

:all
Does not remove any punctuation from the result.
:semi
Removes full stops (i.e. periods) ['。', '．', '.'].
:none
Removes all punctuation from the result.
:only
Removes everything except punctuation. The returned result is an array of only the punctuation.

`numbers`

default = 'all'

:all
Does not remove any numbers from the result
:semi
Removes tokens that include only digits
:none
Removes all tokens that include a number from the result (including Roman numerals)
:only
Removes everything except tokens that include a number

`remove_emoji`

default = false

true
Removes any token that contains an emoji.
false
Leaves tokens as is.

`remove_urls`

default = false

true
Removes any token that contains a URL.
false
Leaves tokens as is.

`remove_domains`

default = false

true
Removes any token that contains a domain.
false
Leaves tokens as is.

`remove_domains`

default = false

true
Removes any token that contains a domain.
false
Leaves tokens as is.

`clean`

default = false

true
Removes tokens consisting of only hypens, underscores, or periods as well as some special characters (®, ©, ™). Also removes long tokens or tokens with a backslash.
false
Leaves tokens as is.

`hashtags`

default = :keep_original

:keep_original
Does not alter the token at all.
:keep_and_clean
Removes the hashtag (#) prefix from the token.
:remove
Removes the token completely.

`mentions`

default = :keep_original

:keep_original
Does not alter the token at all.
:keep_and_clean
Removes the mention (@) prefix from the token.
:remove
Removes the token completely.

`classic_filter`

default = false

true
Removes dots from acronyms and 's from the end of tokens.
false
Leaves tokens as is.

`downcase`

default = true

`minimum_length`

default = 0
The minimum number of characters a token should be.

`long_word_split`

default = nil
The number of characters after which a token should be split at hypens or underscores.

Language Support

The following lists the current level of support for different languages. Pull requests or help for any languages that are not fully supported would be greatly appreciated.

N.B. - contractions might not be applicable for all languages below - in that case the CONTRACTIONS hash should stay empty.

English

Specs: Yes
Abbreviations: Yes
Stop Words: Yes
Contractions: Yes

Arabic

Specs: No
Abbreviations: Yes
Stop Words: Yes
Contractions: No