Entity recognition is inconsistent across runs

[jaju]

I'm attempting some entity recognition, and the results keep changing for the same sentence when I reload the same model.

Sample run snapshot (verbatim)

>>> import spacy
>>> nlp = spacy.load('en_core_web_sm')
>>> d = nlp('The company that IBM bought had rejected Apple and Google bids')
>>> d.ents
()
>>> nlp = spacy.load('en_core_web_sm')
>>> d = nlp('The company that IBM bought had rejected Apple and Google bids')
>>> d.ents
()
>>> nlp = spacy.load('en_core_web_sm')
>>> d = nlp('The company that IBM bought had rejected Apple and Google bids')
>>> d.ents
(IBM,)
>>> nlp = spacy.load('en_core_web_sm')
>>> d = nlp('The company that IBM bought had rejected Apple and Google bids')
>>> d.ents
(IBM,)
>>> nlp = spacy.load('en_core_web_sm')
>>> d = nlp('The company that IBM bought had rejected Apple and Google bids')
>>> d.ents
(IBM, Apple)

I'm not sure, but this is unexpected given nothing really changes.
Output of spacy info --markdown

Info about spaCy

• spaCy version: 2.0.0a13
• Platform: Darwin-16.7.0-x86_64-i386-64bit
• Python version: 3.6.2
• Models: en, en_core_web_sm, en_vectors_web_lg

[honnibal]

Thanks for the report -- definitely something wrong here.

[jaju]

Please let me know if I can provide any more information, or run additional tests.
Thanks!

[honnibal]

I have it reproduced now, so it shouldn't be long to get the fix sorted :)

[socialglass]

Something similar -

>>> doc = nlp(u"She Was among first investors to get approval for Biotech Fund.")
>>> doc.ents
(first, Biotech Fund)
>>> doc = nlp(u"She Was among the first investors to get approval for Biotech Fund.")
>>> doc.ents
()
>>> doc = nlp(u"I Was among first investors to get approval for Biotech Fund.")
>>> doc.ents
(Biotech Fund,)

[honnibal]

The current version on develop seems to have this fixed already. Hopefully I can get it pushed to spacy-nightly tonight. (The new model also has better parse accuracy, which is nice...)

There are two possible explanations for the inconsistency:

1. Some model preserves its random initialization even after loading (e.g. the model adds the loaded weights, instead of replacing them)

2. Somewhere there's an out-of-bounds read. The eventual calculations would then depend on values from neighbouring memory locations, which would vary between runs.

I think 2) is probably more likely. The good news is that the instability occurs also in the tensor values, not just in the parser or tagger. This means there's only a few places to look. It's probably the maxout or convolution functions.

[jaju]

I updated to the latest nightly build, and the issue has disappeared.
Thanks a lot! That's admirably quick! :)

[honnibal]

No worries!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.