Bad tokenization of hyphenated words and em dash separated tokens

[rlvoyer]

I just observed a couple of obvious tokenization errors related to handling of hyphens:

In [274]: nlp(u"The well-bred maid instinctively makes little of a guest's accident, and decent--let alone well-bred--people.")
Out[274]: The well-bred maid instinctively makes little of a guest's accident, and decent--let alone well-bred--people.

In [275]: doc = nlp(s.decode("utf-8"))

In [276]: [t for t in doc]
Out[276]:
[The ,
 well,
 -,
 bred ,
 maid ,
 instinctively ,
 makes ,
 little ,
 of ,
 a ,
 guest,
 's ,
 accident,
 , ,
 and ,
 decent--let ,
 alone ,
 well,
 -,
 bred--people,
 .]

Any suggestions for better handling of hyphens and em dashes?

[honnibal]

I've added a rule to the infix.txt file that addresses most of the problem. We still have some difficulty, with the case of well-bred--people. This requires a change to the Tokenizer class, not just the rules being loaded. This has been reported before and we do intend to fix it, but we've not gotten to it yet.

The next data release should include the updated tokenizer rule, allowing you to work with it. If you want to work around the problem currently, you could make the addition to the infix.txt file in your local copy of the data.

[rlvoyer]

I'm happy to send a PR for this if it's fix that doesn't require exhaustive knowledge of the code. A pointer in the right direction would be awesome.

[honnibal]

Sorry for the delay on this. I think I've got it fixed.

I was reluctant to set you loose on it, since the code in the tokenizer is quite tricky, and there's not much documentation.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.