[MRG] Refactor phrases

[piskvorky]

Following #2973, I set out to improve the documentation for our phrases module. While at it, I saw many inefficiencies and cumbersome constructs there, so ended up with a larger refactor.

Benefits of this PR:

• significantly less code (790 vs 941 lines – despite adding many comments)
• significantly cleaner code (got rid of the maze of call indirections, weird OOP inheritance)
• significantly faster (42s vs 81s on text8 = almost double!)

Not in this PR:

• Compiled code for performance. Although I did locate all hot-spot code into one function, so cythonizing should be both straightforward and highly effective (a single tight loop).
• Using Bounter to lower RAM footprint, Use Bounter for approx frequency counting #1654.

I also added migration code for load(), so old models continue to work.

[piskvorky]

More detailed timing info, all measured on text8:

action	develop	this PR
create Phrases	36.6s	27.3s
apply Phrases	81s	42.8s
export Phraser	53.4s	10.9s
apply Phraser	57.1s	22s

@gojomo @mpenkov seems better across the board; please review and let's include this in 4.0.0. I'll update the unit tests next.

[piskvorky]

@alexgarel @gojomo do you remember why we score bigrams on word_a + in_between + word_b, instead of just word_a + word_b? What's the point of including the common words in the score calculation?

In other words:

• "bigram" count of the phrase "eye of the beholder" now = #['eye', 'of', 'the', 'beholder']
• why not #['eye', 'beholder'] instead? (phrases "eye of the beholder", "eye beholder", "eye the beholder", "eye of beholder" etc would all share the same bigram count)

[gojomo]

I'd guess that tracks back to feature #1258, with an abortive implementation in #1567, arriving in #1568. It sounded useful to me in some situations, but hard enough to understand, and twisty enough in its effect on existing code, that I suggested a couple times it be a separate class, rather than options on Phrases.

[gojomo]

(And relatedly: my hunch is this has no effect unless that non-default option is activated, because in_between is then always empty.)

[piskvorky]

Your hunch is correct – common_words is empty by default. Although I'm tempted to change the default to "all English articles + prepositions", it seems pretty useful.

I'm leaving the logic of #['eye', 'of', 'the', 'beholder'] vs #['eye', 'beholder'] unchanged. Scoring just the brigram makes more sense IMO, but I don't know if anyone's using common_words and if they are, let's avoid surprises.

The effect of common_words on the code base is now minimal, both in lines-of-code and speed, so separate class not needed.

[gojomo]

Re: defaults

Despite my own perspective as an English monoglot, I'd hate to change a language-agnostic algorithm, that previously had no English-specific initialization, to become English-specific by default – even in a major version bump, with lots of warning. (On the other hand, providing handy switch/presets that make enabling an English-specific mode, and/or other languages, sounds useful.)

[piskvorky]

Implemented in #2979. Not sure which default to use… I agree language-agnostic is nice, but English is the overwhelmingly common use-case.

I eyeballed the exported phrases and common words do make a huge difference in quality.

[piskvorky]

@mpenkov @gojomo Gentlemen, I'll merge this now for the sake of simplicity. Your review comments are still welcome; if any, I'll address them in one of the upcoming PRs.