Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate tokenizer from hasher #162

Merged
merged 43 commits into from
Mar 5, 2018
Merged

Separate tokenizer from hasher #162

merged 43 commits into from
Mar 5, 2018

Commits on Jan 16, 2018

  1. Separate whitespace tokenizer from hasher

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    96d2f1a View commit details
    Browse the repository at this point in the history

  2. Separate stopword filter from hasher

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    25b112f View commit details
    Browse the repository at this point in the history

  3. Run tests in deep directories

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    010605c View commit details
    Browse the repository at this point in the history

  4. Separate stemmer from hasher

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    b66ac69 View commit details
    Browse the repository at this point in the history

  5. Separate tests for stopword and tokenizer from hasher's one

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    3188c52 View commit details
    Browse the repository at this point in the history

  6. Reintroduce method to get hash from clean words

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    4f60a6b View commit details
    Browse the repository at this point in the history

  7. Fix usage of Stopword filter

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    6b433ee View commit details
    Browse the repository at this point in the history

  8. Add tests for Tokenizer::Token

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    10d3e3a View commit details
    Browse the repository at this point in the history

  9. Add test for TokenFilter::Stemmer

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    19d83d9 View commit details
    Browse the repository at this point in the history

  10. Remove needless conversion

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    91523e8 View commit details
    Browse the repository at this point in the history

  11. Unite stemmer and stopword filter to whitespace tokenizer

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    d84caa9 View commit details
    Browse the repository at this point in the history

  12. Fix indent

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    9df9bfa View commit details
    Browse the repository at this point in the history

  13. Insert seaparator blank lines between meaningful blocks

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    164620c View commit details
    Browse the repository at this point in the history

  14. Revert "Insert seaparator blank lines between meaningful blocks" 

    This reverts commit 07cf360.
Rollback.
    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    b30c9c5 View commit details
    Browse the repository at this point in the history

  15. Revert "Fix indent" 

    This reverts commit 07e6807.
Rollback.
    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    c6c88a5 View commit details
    Browse the repository at this point in the history

  16. Revert "Unite stemmer and stopword filter to whitespace tokenizer" 

    This reverts commit f256337.
They should be used separatedly.
    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    56fe374 View commit details
    Browse the repository at this point in the history

  17. Fix indent

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    a27c4f3 View commit details
    Browse the repository at this point in the history

  18. Use meaningful variable name

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    d7b2519 View commit details
    Browse the repository at this point in the history

  19. Describe new modules and classes

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    67caa82 View commit details
    Browse the repository at this point in the history

  20. Give tokenizer and token filters from outside of hasher

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    ce7bca0 View commit details
    Browse the repository at this point in the history

  21. Uniform coding style

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    d0bdd5b View commit details
    Browse the repository at this point in the history

  22. Apply enable_stemmer option correctly

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    0647ba7 View commit details
    Browse the repository at this point in the history

  23. Fix invalid URI

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    d83fc80 View commit details
    Browse the repository at this point in the history

  24. Don't give needless parameters

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    65dc97d View commit details
    Browse the repository at this point in the history

  25. Load required modules

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    a5d8f4d View commit details
    Browse the repository at this point in the history

  26. Define default token filters for hasher

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    1a105ef View commit details
    Browse the repository at this point in the history

  27. Fix path to modules

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    52e61f2 View commit details
    Browse the repository at this point in the history

  28. Add description how to use custom tokenizer

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    148a150 View commit details
    Browse the repository at this point in the history

  29. Define token filter to remove symbol only tokens

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    8fe32d3 View commit details
    Browse the repository at this point in the history

  30. Fix path to required module

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    2477ef1 View commit details
    Browse the repository at this point in the history

  31. Remove needless parameter

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    6880ba5 View commit details
    Browse the repository at this point in the history

  32. Use langauge option only for stopwords filter

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    a9b9639 View commit details
    Browse the repository at this point in the history

  33. Add test for TokenFilter::Symbol

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    35c304e View commit details
    Browse the repository at this point in the history

  34. Remove needless "s"

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    829a176 View commit details
    Browse the repository at this point in the history

  35. Add how to use custom token filters

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    751b15b View commit details
    Browse the repository at this point in the history

  36. Reject cat token based on regexp

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    932a0a1 View commit details
    Browse the repository at this point in the history

  37. Add tests to custom tokenizer and token filters

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    14af4d0 View commit details
    Browse the repository at this point in the history

  38. Fix usage of custom tokenizer

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    3c59f44 View commit details
    Browse the repository at this point in the history

  39. Add note for custom tokenizer

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    d856224 View commit details
    Browse the repository at this point in the history

  40. Describe spec of custom tokenizer at first

    @piroor
    piroor committed Jan 16, 2018
    Configuration menu
    Copy the full SHA
    b82e68d View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2018

  1. Accept lambda as custom token filter and tokenizer

    @piroor
    piroor committed Mar 5, 2018
    Configuration menu
    Copy the full SHA
    958d3a0 View commit details
    Browse the repository at this point in the history

  2. Fix mismatched descriptions about method

    @piroor
    piroor committed Mar 5, 2018
    Configuration menu
    Copy the full SHA
    7bceef7 View commit details
    Browse the repository at this point in the history

  3. Add more tests for custom tokenizer and filters

    @piroor
    piroor committed Mar 5, 2018
    Configuration menu
    Copy the full SHA
    81824f5 View commit details
    Browse the repository at this point in the history