AttributeError: 'KeyedVectors' object has no attribute 'next_index'

[LostInDarkMath]

Problem description

I try to add new vectors to a given model.

Steps/code/corpus to reproduce

Relevant traceback:

  File "d:\projekte\foo\source\backend\venv\lib\site-packages\gensim\models\keyedvectors.py", line 458, in add_vector
    target_index = self.next_index
AttributeError: 'KeyedVectors' object has no attribute 'next_index'

My code:

class Foo:
    def __init__(self) -> None:
        self._model: KeyedVectors = gensim.models.KeyedVectors.load('C:\foo.bar')

   def is_word_known(word: str, language: str) -> bool:
       pass

    def add_word(self, word: str, synonym: str, language: str) -> None:
        if not self.is_word_known(word=synonym, language=language):
            raise ValueError(f'Word "{word}" cannot be added since synonym "{synonym}" is not known.')

        if self.is_word_known(word=word, language=language):
            raise ValueError(''Word "{word}" cannot be added since it is already known by the model.')

        vector = self._model.get_vector(key=word)
        self._model.add_vector(key=word, vector=vector)

Versions

Windows-7-6.1.7601-SP1
Bits 64
NumPy 1.20.2
SciPy 1.6.2
gensim 4.0.1
FAST_VERSION 0

[gojomo]

Thanks for your report! I suspect you're loading an instance of KeyedVectors from an earlier version of Gensim, when adding of vectors was handled differently.

It may be enough of a temporary workaround for you to manually patch self._model.next_index = len(self._model) after it is loaded.

The proper fix for the project is likely to extend the KeyedVectors._load_specials method to detect when next_index is missing and patch it there.

(Separately: I don't see what call to your code, as shown, could really trigger that exception, given the lack of an is_word_known implementation, & the lines before your add_vector call - but maybe something was lost in your transcription of other code, or I'm overlooking something. In any case, from the error & apparent intent, the above may resolve the bug.)

[LostInDarkMath]

Hi @gojomo! Thank you for your fast reply. Yes, I build my model with Gensim 3.8.3 before.

I solved my problem by rebuilding it via

word2vec_keyed_vectors = gensim.models.KeyedVectors.load_word2vec_format('C:\foo.txt', binary=False)
word2vec_keyed_vectors.save('C:\foo.bar')

Luckily, I didn't changed it in the past so rebuilding was the preferred option for me.

[gojomo]

Yes, if you had the same vectors in plain 'word2vec_format', just reloading from there is a good workaround, too.

[gojomo]

PR #3117 may be a minimal fix for the absence of next_index - but there's not yet a test case to prompt this error, & verify expected behavior across whole ops - so there could be other issues with older loads, that only show with load-and-add.