Code

Separate tokenizer from hasher#162

Code

Merged

Ch4s3 merged 43 commits intojekyll:masterjekyll/classifier-reborn:masterfrom piroor:separate-tokenizer-from-hasherpiroor/classifier-reborn:separate-tokenizer-from-hasherCopy head branch name to clipboardMar 5, 2018

+475-91

Commits

Commits on Jan 16, 2018

• Separate whitespace tokenizer from hasher

  piroor

  committed
  96d2f1a

  Copy full SHA for 96d2f1a

• Separate stopword filter from hasher

  piroor

  committed
  25b112f

  Copy full SHA for 25b112f

• Run tests in deep directories

  piroor

  committed
  010605c

  Copy full SHA for 010605c

• Separate stemmer from hasher

  piroor

  committed
  b66ac69

  Copy full SHA for b66ac69

• Separate tests for stopword and tokenizer from hasher's one

  piroor

  committed
  3188c52

  Copy full SHA for 3188c52

• Reintroduce method to get hash from clean words

  piroor

  committed
  4f60a6b

  Copy full SHA for 4f60a6b

• Fix usage of Stopword filter

  piroor

  committed
  6b433ee

  Copy full SHA for 6b433ee

• Add tests for Tokenizer::Token

  piroor

  committed
  10d3e3a

  Copy full SHA for 10d3e3a

• Add test for TokenFilter::Stemmer

  piroor

  committed
  19d83d9

  Copy full SHA for 19d83d9

• Remove needless conversion

  piroor

  committed
  91523e8

  Copy full SHA for 91523e8

• Unite stemmer and stopword filter to whitespace tokenizer

  piroor

  committed
  d84caa9

  Copy full SHA for d84caa9

• Fix indent

  piroor

  committed
  9df9bfa

  Copy full SHA for 9df9bfa

• Insert seaparator blank lines between meaningful blocks

  piroor

  committed
  164620c

  Copy full SHA for 164620c

• Revert "Insert seaparator blank lines between meaningful blocks"

  Show description for b30c9c5
  piroor

  committed
  b30c9c5

  Copy full SHA for b30c9c5

• Revert "Fix indent"

  Show description for c6c88a5
  piroor

  committed
  c6c88a5

  Copy full SHA for c6c88a5

• Revert "Unite stemmer and stopword filter to whitespace tokenizer"

  Show description for 56fe374
  piroor

  committed
  56fe374

  Copy full SHA for 56fe374

• Fix indent

  piroor

  committed
  a27c4f3

  Copy full SHA for a27c4f3

• Use meaningful variable name

  piroor

  committed
  d7b2519

  Copy full SHA for d7b2519

• Describe new modules and classes

  piroor

  committed
  67caa82

  Copy full SHA for 67caa82

• Give tokenizer and token filters from outside of hasher

  piroor

  committed
  ce7bca0

  Copy full SHA for ce7bca0

• Uniform coding style

  piroor

  committed
  d0bdd5b

  Copy full SHA for d0bdd5b

• Apply enable_stemmer option correctly

  piroor

  committed
  0647ba7

  Copy full SHA for 0647ba7

• Fix invalid URI

  piroor

  committed
  d83fc80

  Copy full SHA for d83fc80

• Don't give needless parameters

  piroor

  committed
  65dc97d

  Copy full SHA for 65dc97d

• Load required modules

  piroor

  committed
  a5d8f4d

  Copy full SHA for a5d8f4d

• Define default token filters for hasher

  piroor

  committed
  1a105ef

  Copy full SHA for 1a105ef

• Fix path to modules

  piroor

  committed
  52e61f2

  Copy full SHA for 52e61f2

• Add description how to use custom tokenizer

  piroor

  committed
  148a150

  Copy full SHA for 148a150

• Define token filter to remove symbol only tokens

  piroor

  committed
  8fe32d3

  Copy full SHA for 8fe32d3

• Fix path to required module

  piroor

  committed
  2477ef1

  Copy full SHA for 2477ef1

• Remove needless parameter

  piroor

  committed
  6880ba5

  Copy full SHA for 6880ba5

• Use langauge option only for stopwords filter

  piroor

  committed
  a9b9639

  Copy full SHA for a9b9639

• Add test for TokenFilter::Symbol

  piroor

  committed
  35c304e

  Copy full SHA for 35c304e

• Remove needless "s"

  piroor

  committed
  829a176

  Copy full SHA for 829a176

• Add how to use custom token filters

  piroor

  committed
  751b15b

  Copy full SHA for 751b15b

• Reject cat token based on regexp

  piroor

  committed
  932a0a1

  Copy full SHA for 932a0a1

• Add tests to custom tokenizer and token filters

  piroor

  committed
  14af4d0

  Copy full SHA for 14af4d0

• Fix usage of custom tokenizer

  piroor

  committed
  3c59f44

  Copy full SHA for 3c59f44

• Add note for custom tokenizer

  piroor

  committed
  d856224

  Copy full SHA for d856224

• Describe spec of custom tokenizer at first

  piroor

  committed
  b82e68d

  Copy full SHA for b82e68d

Commits on Mar 5, 2018

• Accept lambda as custom token filter and tokenizer

  piroor

  committed
  958d3a0

  Copy full SHA for 958d3a0

• Fix mismatched descriptions about method

  piroor

  committed
  7bceef7

  Copy full SHA for 7bceef7

• Add more tests for custom tokenizer and filters

  piroor

  committed
  81824f5

  Copy full SHA for 81824f5