Word Embeddings

[prise6]

As mentionned in this issue #148 : it would be nice to have a baseline approache of online word embeddings. (card-link)
It seems that few months ago, a PR #189 implemented Glove model in a module name "decomposition". This branch doesn't exist anymore: maybe it's a good start to go on if someone can publish it ?

[MaxHalford]

@raphaelsty do you have the code somewhere?

[raphaelsty]

@MaxHalford @prise6 Hi, sorry for the delay in my response, I lost the notification. I had started to implement Glove with the logistic regression. I can share the draft code I produced.

I'm a bit reserved about the fact that river offers word embeddings. Training time is prohibitive to get decent results without batch-learning or massive amount of text. There are hundreds of resources (pre-trained embeddings) and specialized libraries to convert a sentence into a vector (perfectly compatible with river's pipeline). We could for sur add a method to load pre-trained embeddings but It's a bit like reinventing the wheel in my opinion. Wouldn't it be better to create a tutorial to explain how to use pre-trained word embeddings using river pipeline?

[raphaelsty]

@prise6 @MaxHalford I made a pull request #502 to add an example in the documentation. In this tutorial, I show how to integrate pre-trained embeddings in the river pipeline. 🙂

[prise6]

No worries about the delay @raphaelsty. Thank you for sharing your thoughts about this topic, you're obviousliy right.

My idea, which could be a bad one, is to create a kind of "moving" embedding on top of streaming text data. Online learning, to me, is a real good technique to capture dynamic behaviour. I was wondering if it's possible to learn an evolving embedding, including vocabulary updates and word context.

Actually, i'm interested in your draft you produced if i'ts ok.

And thank you for the documentation you wrote, it's helpfull anyway.