[WIP][HIVEMALL-118] word2vec #116

nzw0301 · 2017-09-21T02:48:47Z

What changes were proposed in this pull request?

Add new algorithm: skip-gram with negative sampling (a.k.a word2vec)

What type of PR is it?

Feature

What is the Jira issue?

https://issues.apache.org/jira/browse/HIVEMALL-118

How was this patch tested?

manual tests on EMR

To train word2vec, I used wikipedia dataset, preprocessed by this perl script.

I evaluated word vector by https://github.com/kudkudak/word-embeddings-benchmarks .

from six import iteritems
from web.datasets.similarity import fetch_MEN, fetch_WS353, fetch_SimLex999, fetch_RW, fetch_RG65, fetch_MTurk
from web.datasets.analogy import fetch_msr_analogy, fetch_google_analogy, fetch_semeval_2012_2, fetch_wordrep
import gensim
from gensim.models.word2vec import Word2Vec, LineSentence

from web.embeddings import load_embedding
from web.evaluate import evaluate_similarity, evaluate_analogy

sim_tasks = {
    "MEN      ": fetch_MEN(),
    "WS353    ": fetch_WS353(),
    "SIMLEX999": fetch_SimLex999(),
    "RW       ": fetch_RW(),
    "RG       ": fetch_RG65(),
    "MTurk    ": fetch_MTurk()
}
analogy_tasks = {
    "google": fetch_google_analogy(),
    "msr   ": fetch_msr_analogy()
}

docs = LineSentence('PATH/TO/PREPROCESSED_DATA')
model = Word2Vec(docs, size=100, window=5, min_count=15, workers=8, negative=15, hs=0, sg=1, iter=1)
model.wv.save_word2vec_format('./gensim_sg.txt')

gensim = load_embedding('./gensim_sg.txt', 'word2vec')

for name, data in iteritems(sim_tasks):
    print("Spearman correlation of scores on {} {}".format(name, evaluate_similarity(gensim, data.X, data.y)))

Spearman correlation of scores on MEN       0.6483416401993833
Spearman correlation of scores on WS353     0.6169418277184877
Spearman correlation of scores on SIMLEX999 0.3070155939988943
Spearman correlation of scores on RW        0.28548732030155277
Spearman correlation of scores on RG        0.6762247194247315
Spearman correlation of scores on MTurk     0.6471504497920156

CBoW model of hivemall

Spearman correlation of scores on MEN       0.6247965194705783
Spearman correlation of scores on WS353     0.6225747519511903
Spearman correlation of scores on SIMLEX999 0.2985588069793148
Spearman correlation of scores on RW        0.27686018664704454
Spearman correlation of scores on RG        0.6528832630934683
Spearman correlation of scores on MTurk     0.6307218892934624

Skip-gram of hivemall

Spearman correlation of scores on MEN       0.6202415358400425
Spearman correlation of scores on WS353     0.6235303875587551
Spearman correlation of scores on SIMLEX999 0.2983910352562464
Spearman correlation of scores on RW        0.2939926699533969
Spearman correlation of scores on RG        0.6782791172666216
Spearman correlation of scores on MTurk     0.6344295663665642

CBoW of hivemall when the number of reducer for training is 4

Spearman correlation of scores on MEN       0.5392243963768429
Spearman correlation of scores on WS353     0.546436543545682
Spearman correlation of scores on SIMLEX999 0.2529742987287988
Spearman correlation of scores on RW        0.28611116074856136
Spearman correlation of scores on RG        0.5040049854449996
Spearman correlation of scores on MTurk     0.5953150581437371

How to use this feature?

please see word2vec.md

Checklist

Did you apply source code formatter, i.e., mvn formatter:format, for your commit?

coveralls · 2017-09-21T03:10:59Z

Coverage decreased (-0.7%) to 40.18% when pulling 7014e85 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-21T06:54:04Z

Coverage decreased (-0.7%) to 40.185% when pulling 7a5fd54 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-21T07:38:21Z

Coverage decreased (-0.7%) to 40.179% when pulling c224912 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-21T12:20:05Z

Coverage decreased (-0.8%) to 40.165% when pulling 8319861 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-21T13:33:33Z

Coverage decreased (-0.8%) to 40.156% when pulling ad2b291 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-21T14:53:22Z

Coverage decreased (-0.8%) to 40.149% when pulling 39d1123 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-22T04:10:22Z

Coverage decreased (-0.8%) to 40.138% when pulling e507561 on nzw0301:skipgram into c2b9578 on apache:master.

myui · 2017-09-22T05:12:24Z

core/src/main/java/hivemall/unsupervised/AbstractWord2vecModel.java

+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package hivemall.unsupervised;


Please move package from hivemall.unsupervised to hivemall.embedding.

myui · 2017-09-22T05:17:03Z

core/src/main/java/hivemall/unsupervised/AbstractWord2vecModel.java

+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+
+public abstract class AbstractWord2vecModel {


Please rename Word2vec to Word2Vec as seen in spark.

coveralls · 2017-09-22T07:02:30Z

Coverage decreased (-0.8%) to 40.137% when pulling bf5d927 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-22T08:53:50Z

Coverage decreased (-0.8%) to 40.144% when pulling a3ccaa8 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-22T09:55:05Z

Coverage decreased (-0.8%) to 40.147% when pulling c7cba82 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-22T11:03:17Z

Coverage decreased (-0.8%) to 40.077% when pulling bbdb561 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-25T03:49:59Z

Coverage decreased (-0.9%) to 40.057% when pulling 7a2f4db on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-25T05:46:32Z

Coverage decreased (-0.9%) to 40.029% when pulling e094552 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-25T06:58:34Z

Coverage decreased (-0.9%) to 40.025% when pulling aede5ec on nzw0301:skipgram into c2b9578 on apache:master.

myui · 2017-09-25T09:18:52Z

core/src/main/java/hivemall/embedding/AbstractWord2VecModel.java

+        }
+    }
+
+    protected static float sigmoid(final float v, final int MAX_SIGMOID,


No need to use Constants for argument: final int MAX_SIGMOID, final int SIGMOID_TABLE_SIZE

myui · 2017-09-25T09:19:36Z

core/src/main/java/hivemall/embedding/AbstractWord2VecModel.java

+
+public abstract class AbstractWord2VecModel {
+    // cached sigmoid function parameters
+    protected final int MAX_SIGMOID = 6;


Constants should be static final.

coveralls · 2017-09-25T11:16:43Z

Coverage decreased (-0.9%) to 40.028% when pulling 4abdb8f on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-25T11:28:25Z

Coverage decreased (-0.9%) to 39.993% when pulling 4abdb8f on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-26T03:32:06Z

Coverage decreased (-0.9%) to 40.05% when pulling 8a42adf on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-26T05:03:50Z

Coverage decreased (-0.9%) to 40.05% when pulling d1b4270 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-26T08:42:13Z

Coverage decreased (-0.9%) to 40.049% when pulling f19d732 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-26T12:14:28Z

Coverage decreased (-0.8%) to 40.149% when pulling 2b66e5e on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-26T15:21:04Z

Coverage decreased (-0.8%) to 40.108% when pulling f0abd4f on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-26T17:24:13Z

Coverage decreased (-0.9%) to 40.068% when pulling c340038 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-27T06:00:43Z

Coverage decreased (-0.9%) to 40.065% when pulling af5b5be on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-27T07:33:11Z

Coverage decreased (-0.9%) to 40.065% when pulling d12ba32 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-27T09:32:15Z

Coverage decreased (-0.9%) to 40.069% when pulling 2415589 on nzw0301:skipgram into c2b9578 on apache:master.

coveralls · 2017-09-27T12:39:19Z

Coverage decreased (-0.9%) to 40.056% when pulling da564b8 on nzw0301:skipgram into c2b9578 on apache:master.

myui · 2017-09-28T07:22:16Z

core/src/main/java/hivemall/embedding/AbstractWord2VecModel.java

+    protected static final int SIGMOID_TABLE_SIZE = 1000;
+    protected float[] sigmoidTable;
+
+


remove unnecessary blank line

myui · 2017-09-28T07:22:52Z

core/src/main/java/hivemall/embedding/AbstractWord2VecModel.java

+
+    @Nonnegative
+    protected int dim;
+    protected int win;


@Nonnegative for each variable (win, neg, iter).

myui · 2017-09-28T07:23:32Z

core/src/main/java/hivemall/embedding/AbstractWord2VecModel.java

+    protected Int2FloatOpenHashTable S;
+    protected int[] aliasWordId;
+
+    protected AbstractWord2VecModel(final int dim, final int win, final int neg, final int iter,


add @Nonnegative for each constructor argument and caller methods.

myui · 2017-09-28T07:24:18Z

core/src/main/java/hivemall/embedding/AbstractWord2VecModel.java

+        }
+    }
+
+    protected static float sigmoid(final float v, final float[] sigmoidTable) {


@Nonnull for sigmoidTable

myui · 2017-09-28T07:24:31Z

core/src/main/java/hivemall/embedding/AbstractWord2VecModel.java

+    }
+
+    protected void updateLearningRate() {
+        // TODO: valid lr?


remove this TODO comment and blank lines.

myui · 2017-09-28T07:46:25Z

core/src/main/java/hivemall/embedding/CBoWModel.java

+import java.util.List;
+
+public final class CBoWModel extends AbstractWord2VecModel {
+    protected CBoWModel(final int dim, final int win, final int neg, final int iter,


add a blank line before constructor.

myui · 2017-09-28T07:46:41Z

core/src/main/java/hivemall/embedding/CBoWModel.java

+
+        updateLearningRate();
+
+        int docLength = doc.length;


final int docLength

myui · 2017-09-28T07:48:42Z

core/src/main/java/hivemall/embedding/Word2VecUDTF.java

+        opts.addOption("win", "window", true, "Context window size [default: 5]");
+        opts.addOption("neg", "negative", true,
+            "The number of negative sampled words per word [default: 5]");
+        opts.addOption("iter", "iteration", true, "The number of iterations [default: 5]");


consistent naming "iters", "iterations" as seen in SLIM.

myui · 2017-09-28T07:49:24Z

core/src/main/java/hivemall/embedding/Word2VecUDTF.java

+        opts.addOption("model", "modelName", true,
+            "The model name of word2vec: skipgram or cbow [default: skipgram]");
+        opts.addOption(
+            "lr",


consistent naming eta0, learningRate for the initial learning rate.

I see.
Does longOpt remain learningRate or remove this field?

remain learningRate for longOpt and use eta0 for initialLearningRate.

myui · 2017-09-28T07:50:21Z

core/src/main/java/hivemall/embedding/Word2VecUDTF.java

+            }
+
+            modelName = cl.getOptionValue("model", modelName);
+            if (!(modelName.equals("skipgram") || modelName.equals("cbow"))) {


"skipgram".equals(modelName) is null safe.

myui · 2017-09-28T07:54:38Z

What type of PR is it? => Improvement should be Feature.

nzw0301 · 2017-09-28T09:16:58Z

@myui I resolved conflicts.

coveralls · 2017-09-28T09:19:42Z

Coverage decreased (-0.6%) to 40.505% when pulling 8696f5f on nzw0301:skipgram into 1e42387 on apache:master.

coveralls · 2017-09-28T09:52:03Z

Coverage decreased (-0.6%) to 40.508% when pulling 0b163fa on nzw0301:skipgram into 1e42387 on apache:master.

myui reviewed Sep 22, 2017

View reviewed changes

myui suggested changes Sep 25, 2017

View reviewed changes

myui reviewed Sep 28, 2017

View reviewed changes

nzw0301 added 23 commits September 28, 2017 18:10

Fix typo

4ab8d25

Rename package name and class name

4e45967

Refactoring: rename variable and cache local variable in nested loop

53e8cd2

Update: use String array instead of Text array

9654057

Update word2vec feature function: cbow

b67dc25

Update to prepare for CBoW model

6361f06

Add CBoW model

acf2e27

Update docs: add CBoW

989d1d7

Fix sigmoid function based on comment

0bff032

Remove unnecessary arguments

98bb72d

Remove split alias sampler

b7503ce

Update query in word2vec document

7dffe1e

Refactor

98d89dc

Change arguments

70d3486

Int support for Negative table

37c5de5

Support int document for train_word2vec to save memory

5b5d021

Fix bug and Update document

c60d438

Apply formatter

cbfc82f

Add reference of alias method

a33bcc5

Add word2vec function to resources

4b741f5

Refactor: update docs and ignore OOV word vector

b6072f3

Fix bug in forward

c8970f3

Update based on github comments

8696f5f

nzw0301 force-pushed the skipgram branch from b711839 to 8696f5f Compare September 28, 2017 09:10

Fix bug based on rebase

0b163fa

asfgit force-pushed the master branch from e2666ec to 2da3f38 Compare December 19, 2017 01:24

asfgit force-pushed the master branch from 300c599 to 20cdb74 Compare January 12, 2018 09:45

		protected static final int SIGMOID_TABLE_SIZE = 1000;
		protected float[] sigmoidTable;

[WIP][HIVEMALL-118] word2vec #116

Are you sure you want to change the base?

[WIP][HIVEMALL-118] word2vec #116

Conversation

nzw0301 commented Sep 21, 2017 • edited Loading

What changes were proposed in this pull request?

What type of PR is it?

What is the Jira issue?

How was this patch tested?

How to use this feature?

Checklist

coveralls commented Sep 21, 2017 • edited Loading

coveralls commented Sep 21, 2017 • edited Loading

coveralls commented Sep 21, 2017 • edited Loading

coveralls commented Sep 21, 2017 • edited Loading

coveralls commented Sep 21, 2017 • edited Loading

coveralls commented Sep 21, 2017 • edited Loading

coveralls commented Sep 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Sep 22, 2017 • edited Loading

coveralls commented Sep 22, 2017 • edited Loading

coveralls commented Sep 22, 2017 • edited Loading

coveralls commented Sep 22, 2017 • edited Loading

coveralls commented Sep 25, 2017 • edited Loading

coveralls commented Sep 25, 2017 • edited Loading

coveralls commented Sep 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Sep 25, 2017 • edited Loading

coveralls commented Sep 25, 2017 • edited Loading

coveralls commented Sep 26, 2017 • edited Loading

coveralls commented Sep 26, 2017 • edited Loading

coveralls commented Sep 26, 2017 • edited Loading

coveralls commented Sep 26, 2017 • edited Loading

coveralls commented Sep 26, 2017 • edited Loading

coveralls commented Sep 26, 2017 • edited Loading

coveralls commented Sep 27, 2017 • edited Loading

coveralls commented Sep 27, 2017 • edited Loading

coveralls commented Sep 27, 2017 • edited Loading

coveralls commented Sep 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

myui commented Sep 28, 2017

nzw0301 commented Sep 28, 2017

coveralls commented Sep 28, 2017

coveralls commented Sep 28, 2017 • edited Loading

nzw0301 commented Sep 21, 2017 •

edited

Loading

coveralls commented Sep 21, 2017 •

edited

Loading

coveralls commented Sep 21, 2017 •

edited

Loading

coveralls commented Sep 21, 2017 •

edited

Loading

coveralls commented Sep 21, 2017 •

edited

Loading

coveralls commented Sep 21, 2017 •

edited

Loading

coveralls commented Sep 21, 2017 •

edited

Loading

coveralls commented Sep 22, 2017 •

edited

Loading

coveralls commented Sep 22, 2017 •

edited

Loading

coveralls commented Sep 22, 2017 •

edited

Loading

coveralls commented Sep 22, 2017 •

edited

Loading

coveralls commented Sep 22, 2017 •

edited

Loading

coveralls commented Sep 25, 2017 •

edited

Loading

coveralls commented Sep 25, 2017 •

edited

Loading

coveralls commented Sep 25, 2017 •

edited

Loading

coveralls commented Sep 25, 2017 •

edited

Loading

coveralls commented Sep 25, 2017 •

edited

Loading

coveralls commented Sep 26, 2017 •

edited

Loading

coveralls commented Sep 26, 2017 •

edited

Loading

coveralls commented Sep 26, 2017 •

edited

Loading

coveralls commented Sep 26, 2017 •

edited

Loading

coveralls commented Sep 26, 2017 •

edited

Loading

coveralls commented Sep 26, 2017 •

edited

Loading

coveralls commented Sep 27, 2017 •

edited

Loading

coveralls commented Sep 27, 2017 •

edited

Loading

coveralls commented Sep 27, 2017 •

edited

Loading

coveralls commented Sep 27, 2017 •

edited

Loading

coveralls commented Sep 28, 2017 •

edited

Loading