Distributed lda options

[menshikh-iv]

Update distributed LDA support. Now we can run worker/dispatcher in different network segments (not reachable by network broadcast). Broadcast variant also saved.

If you want to use broadcast, reading tutorial https://radimrehurek.com/gensim/dist_lsi.html on official site.

If you want to use new feature, add some arguments when you run a code, for example

1. Execute on all machines
   export PYRO_SERIALIZERS_ACCEPTED=pickle export PYRO_SERIALIZER=pickle'
2. On NS server
   python -m Pyro4.naming --host 0.0.0.0 --port <NS_PORT> -x
3. On workers
   python -m gensim.models.lda_worker --host <NS_HOSTNAME> --port <NS_PORT> --no-broadcast -v
4. On dispatcher
   python -m gensim.models.lda_dispatcher --host <NS_HOSTNAME> --port <NS_PORT> --no-broadcast -v
5. Create LdaModel
   lda = LdaModel(..., ns_conf={"host": NS_HOST, "port": NS_PORT, "broadcast": False})
6. Train it!

[piskvorky]

This is py2.7 only. @tmylk I don't think we can drop support for py2.6 yet... is this import safe?

If it's triggered only on importing lda_dispatcher.py, it's probably fine... but we don't want py2.7+ imports in "core" gensim (at import gensim).

[menshikh-iv]

I checked, this triggered only on importing lda_dispatcher.py or lda_worker.py.
Backport for argparse in setup.py for python < 2.7 (proof)

[piskvorky]

Awesome! This is a great update, and nicely done too.

If you don't mind me asking, how do you use this distributed LDA @menshikh-iv? What is your usecase/goal?

[menshikh-iv]

@piskvorky, I have two usecases:

1. Content classification
2. Similarity search

I need to train LDA on large corpus of 'webpages content' and vectorize all webpages. Train process of LDA are very long. I could use several dedicated servers for training, but they not in local network, therefore I modified distributed LDA for my case.

[piskvorky]

Thanks, interesting! Is this a personal project, academic research or a commercial project? (We keep a list of gensim adopters.)

[menshikh-iv]

@piskvorky personal research for now

[tmylk]

@menshikh-iv Thanks for the PR! Could you add a short notebook-style tutorial for this feature and a note in the changelog?

[menshikh-iv]

@tmylk, unfortunately notebook-style tutorial for this feature is useless, because in notebook I can't demonstrate this feature. Maybe I update this page in documentation with small examples (like this message) ?

About changelog, I should add record to 0.3.12 in CHANGELOG.md ?

And I shoud create new PR for this actions?

[tmylk]

Hi @menshikh-iv, the 0.3.12 is the right version to use. A new small PR would be good.

Updating this page with instructions would be great:
https://github.com/RaRe-Technologies/gensim/blob/develop/docs/src/distributed.rst

[manojpandey]

Documentation changed from rst to markdown here: #859