to_bytes and from_bytes changes the token lemma

[rajhans]

import spacy
nlp=spacy.load('en')
x1=nlp('I cant do this.')
[t.lemma_ for t in x1]
['i', 'can', 'not', 'do', 'this', '.']
g=x1.to_bytes()
d=spacy.tokens.doc.Doc(nlp.vocab)
d.from_bytes(g)
I cant do this.
[t.lemma_ for t in d]
['i', 'ca', 'nt', 'do', 'this', '.']

[rajhans]

FYI, I am using 1.1.2

[honnibal]

Thanks. I think the bug here is that the serializer tries to get away with not saving the lemmas, because it thinks it can recalculate them given the POS tags. This turns out to be untrue in this case, because the lemma is a special-case. Hmm.

[rajhans]

I see. Maybe as a stop gap for my project, is it possible to know for which words (like cant) can this problem arise? As in if it is a finite knowable set of words, I can just hackishly fix for those.

[honnibal]

Yes, the special-case rules are listed in spacy/en/language_data.py. You want to check whether the "F" value matches the the token.text. If it does, you want to assign the value keyed by "L".

[rajhans]

thanks!

[ines]

Closing this and making #1045 the master issue. Work in progress for spaCy v2.0!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.