Lemmatization errors when text contains contracted forms of 'be'

[gppatt]

I've noticed some inconsistent behavior here:

nlp = spacy_nlp(u"I'm hungry. You're hungry. He's hungry. It's hungry. We're hungry. They're hungry.")
for tok in nlp:
print tok, tok.lemma_

Gives output:

I i
'm be
hungry hungry
. .
You you
're 're
hungry hungry
. .
He he
's '
hungry hungry
. .
It it
's '
hungry hungry
. .
We we
're 're
hungry hungry
. .
They they
're 're
hungry hungry
. .

A related error is for "won't" (and for the much rarer "shan't"):

nlp = spacy_nlp(u"They won't move.")
for tok in nlp:
print tok, tok.lemma_

They they
wo wo
n't not
move move
. .

I think I once even saw a similar lemmatization error for "can't", but I am not able to recreate this error.

Your Environment

OSX 10.11.6
Spyder 3.0.0
spaCy 1.2.0

[ines]

Thanks for the report! Some of these should probably be handled in the morphological analyser (like "He's", where the lemma is ambiguous). But the others are definitely cases for the TOKENIZER_EXCEPTIONS.

I'm currently in the process of reorganising the language data (see organize-language-data branch). I'll add the missing exceptions, so this should all be fixed in the v2.0 release.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.