"OverflowError: value too large to convert to int" when training word2vec on a large corpus

[miweru]

Hi,
i am trying to train word2vec on on a large corpus (>500GB) and the newest gensim version (3.8.0) and an error occurs on every Thread:

Exception in thread Thread-10: Traceback (most recent call last): File "/home/michael/miniconda3/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/home/michael/miniconda3/lib/python3.7/threading.py", line 865, in run self._target(*self._args, **self._kwargs) File "/home/michael/miniconda3/lib/python3.7/site-packages/gensim/models/base_any2vec.py", line 175, in _worker_loop_corpusfile total_examples=total_examples, total_words=total_words, **kwargs) File "/home/michael/miniconda3/lib/python3.7/site-packages/gensim/models/word2vec.py", line 794, in _do_train_epoch total_examples, total_words, work, neu1, self.compute_loss) File "gensim/models/word2vec_corpusfile.pyx", line 379, in gensim.models.word2vec_corpusfile.train_epoch_cbow OverflowError: value too large to convert to int

My command for training the Model is:
Word2Vec(corpus_file="encc_tokenized", size = 1024, window = 8, workers = 16)

Thank you for help

[piskvorky]

CC @persiyanov – is this expected? What "value" is that?

[piskvorky]

Looking at the word2vec_corpusfile.pyx code, it's indeed using int for all sort of counters that could be large: expected_examples, effective_words, effective_sentences, total_sentences, indexes… This will need fixing :(

Or was there some reason to using only ints @persiyanov ? I see some vars do use long long, so I presume this was a deliberate choice.

[miweru]

The same problem occurs also with FastText:
Exception in thread Thread-8: Traceback (most recent call last): File "/home/michael/miniconda3/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/home/michael/miniconda3/lib/python3.7/threading.py", line 865, in run self._target(*self._args, **self._kwargs) File "/home/michael/miniconda3/lib/python3.7/site-packages/gensim/models/base_any2vec.py", line 175, in _worker_loop_corpusfile total_examples=total_examples, total_words=total_words, **kwargs) File "/home/michael/miniconda3/lib/python3.7/site-packages/gensim/models/fasttext.py", line 805, in _do_train_epoch total_examples, total_words, work, neu1) File "gensim/models/fasttext_corpusfile.pyx", line 215, in gensim.models.fasttext_corpusfile.train_epoch_cbow OverflowError: value too large to convert to int

[mpenkov]

@persiyanov Ping on this.

[piskvorky]

Another instance of 32 vs 64 bit integers causing trouble for users:

https://groups.google.com/forum/#!topic/gensim/XbH5Sr6RBcI

Not sure if it's the exact same instance as in this ticket… perhaps we need to promote counters from 32 bit to 64 bit more consistently, everywhere.

[persiyanov]

@mpenkov @piskvorky My apologies for the late response.

No, I don't think there was a deliberate choice of using int instead of long long and I believe it's safe to just to replace all int with long long types in all corpus-file-related Cython code.

Do you need a hand on this? I can devote some time next week to fix it.

[piskvorky]

Definitely. If you could grep / review all the *2vec C code for such 32=>64 bit fixes, that would be awesome. We don't want any 32 bit counters. Thanks!

[Xiaoxiong-Liu]

@piskvorky
is this fixed in version 3.8?
I get the same error when trainning when training word2vec on a large corpus.
thx.

[piskvorky]

@Xiaoxiong-Liu yes, a fix should have been a part of 4.0.0beta.

Let me know if this still appears after you upgrade Gensim: pip install --pre --upgrade gensim.