LSTM support in CaffeOnSpark #176

mriduljain · 2016-11-02T18:22:38Z

No description provided.

yahoocla · 2016-11-02T18:22:38Z

CLA is valid!

anfeng · 2016-11-02T19:40:01Z

Makefile

@@ -1,40 +1,41 @@
 HOME ?=/home/${USER}
 ifeq ($(shell which spark-submit),)
-     SPARK_HOME ?=/home/y/share/spark
+     SPARK_HOME=~/spark-1.6.0-bin-hadoop2.6


pls undo this change.

anfeng · 2016-11-02T19:42:36Z

Makefile

 else
     SPARK_HOME ?=$(shell which spark-submit 2>&1 | sed 's/\/bin\/spark-submit//g')
 endif
 CAFFE_ON_SPARK ?=$(shell pwd)
-LD_LIBRARY_PATH ?=/home/y/lib64:/home/y/lib64/mkl/intel64
+LD_LIBRARY_PATH ?=/usr/local/cuda/


/home/y/lib64:/home/y/lib64/mkl/intel64:/usr/local/cuda/

anfeng · 2016-11-02T19:42:55Z

Makefile

 LD_LIBRARY_PATH2=${LD_LIBRARY_PATH}:${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib:/usr/lib64:/lib64 
-DYLD_LIBRARY_PATH ?=/home/y/lib64:/home/y/lib64/mkl/intel64
-DYLD_LIBRARY_PATH2=${DYLD_LIBRARY_PATH}:${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib:/usr/lib64:/lib64 
+DYLD_LIBRARY_PATH ?=/usr/local/cuda/lib


/home/y/lib64:/home/y/lib64/mkl/intel64:/usr/local/cuda/

anfeng · 2016-11-02T19:43:10Z

Makefile


 export SPARK_VERSION=$(shell ${SPARK_HOME}/bin/spark-submit --version 2>&1 | grep version | awk '{print $$5}' | cut -d'.' -f1)
 ifeq (${SPARK_VERSION}, 2)
    export MVN_SPARK_FLAG=-Dspark2
 endif

-build:
+build: 


remove extra space

anfeng · 2016-11-02T19:44:55Z

Makefile

-	cd caffe-distri; make clean; cd ..
+clean: 
+	pushd caffe-public; make clean; popd
+	pushd caffe-distri; make clean; popd


why these changes? Let's avoid unwanted changes

anfeng · 2016-11-02T19:54:04Z

caffe-grid/src/main/python/examples/coco/prediction.py

@@ -0,0 +1,8 @@
+from examples.coco.retrieval_experiment import *


anfeng · 2016-11-02T19:58:19Z

caffe-grid/src/main/python/com/yahoo/ml/caffe/Conversions.py

+from pyspark.sql import DataFrame,SQLContext
+from ConversionUtil import getScalaSingleton, toPython
+
+class Conversions:


This file belong to a tools folder. Rename it "DFConversions"

anfeng · 2016-11-02T20:00:45Z

caffe-grid/src/main/python/com/yahoo/ml/caffe/Vocab.py

+from RegisterContext import registerContext
+from pyspark.sql import DataFrame,SQLContext
+
+class Vocab:


move to a subfolder: tools/Vocab.py

anfeng · 2016-11-02T20:09:46Z

data/solver.prototxt

@@ -0,0 +1,14 @@
+net: "/Users/mridul/bigml/CaffeOnSpark/data/train_val.prototxt"


net: "CaffeOnSpark/data/image_train_val.prototxt"

anfeng · 2016-11-02T20:10:06Z

data/train_val.prototxt

@@ -0,0 +1,396 @@
+name: "CaffeNet"


image_train_val.prototxt?

anfeng · 2016-11-02T23:41:49Z

caffe-grid/src/test/scala/com/yahoo/ml/caffe/tools/ToolTest.scala

+    }
+    }.collect()
+
+/*  test("CocoTest") {


Why this large chunk of test code? Should it be uncommented?

Yes...I went OOM on Travis, so doing a check ...will revert back (with less dataset) if this test is the culprit

anfeng · 2016-11-02T23:43:09Z

data/bvlc_reference_net.prototxt

@@ -0,0 +1,396 @@
+name: "CaffeNet"


why "bvlc_reference"? We are using CoS memory layer

reference net is bvlc net

anfeng · 2016-11-02T23:43:39Z

data/bvlc_reference_solver.prototxt

@@ -0,0 +1,14 @@
+net: "CaffeOnSpark/data/bvlc_reference_net.prototxt"


why bvlc_reference?

that's the renamed solver/net file

anfeng · 2016-11-03T04:27:58Z

caffe-grid/src/main/python/com/yahoo/ml/caffe/DisplayUtils.py

@@ -36,6 +40,26 @@ def show_df(df, nrows=10):
    html += "</table>"
    return HTML(html)

+def show_captions(df, nrows=10):


This method is very much caption specific. Let's move it to tools/DFConversions.py or tools/Utils.py.

ok will do shortly. Fixing the dataset for test right now, as the current dataframe with images leads to OOM on Travis

anfeng · 2016-11-03T04:34:45Z

caffe-grid/src/main/python/examples/README-coco.md

+    export DYLD_LIBRARY_PATH=${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib
+    export DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:/usr/local/cuda/lib:/usr/local/mkl/lib/intel64/
+    export LD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}
+    export SPARK_HOME=/Users/mridul/bigml/spark-1.6.0-bin-hadoop2.6


Let's refer to our GetStarted guides (https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_standalone_osx). We should inform our users to set up CAFFE_ON_SPARK and SPARK_HOME etc from our guides. Don't want to expose your own environments to our users.

anfeng · 2016-11-03T04:35:02Z

caffe-grid/src/main/python/examples/README-coco.md

+Steps to run the COCO dataset for Image Captioning
+==================================================
+##### (1) Env setup
+    export CAFFE_ON_SPARK=/Users/mridul/bigml/CaffeOnSpark


see my comment below

hmm...crept in by mistake

anfeng · 2016-11-03T04:37:02Z

caffe-grid/src/main/python/examples/README-coco.md

+    export PYSPARK_PYTHON=Python2.7.10/bin/python
+    export PYTHONPATH=$PYTHONPATH:caffeonsparkpythonapi.zip:caffe_on_grid_archive/lib64:/usr/local/cuda-7.5/lib64
+    export LD_LIBRARY_PATH=Python2.7.10/lib:/usr/local/cuda/lib:caffe_on_grid_archive/lib64/mkl/intel64/:${LD_LIBRARY_PATH}
+    export DYLD_LIBRARY_PATH=Python2.7.10/lib:/usr/local/cuda/lib:caffe_on_grid_archive/lib64/mkl/intel64/:${LD_LIBRARY_PATH}


Why do we have 2 sets of definition for LD_LIBRARY_PATH and DYLD_LIBRARY_PATH. Please consolidate.

anfeng · 2016-11-03T04:42:40Z

caffe-grid/src/main/python/examples/README-coco.md

+    pushd ${CAFFE_ON_SPARK}/data/
+    ln -s ~/Python2.7.10 Python2.7.10
+    unzip ${CAFFE_ON_SPARK}/caffe-grid/target/caffeonsparkpythonapi.zip
+    cat /tmp/coco/parquet/vocab/part* > vocab.txt


We should not need the above command: cat /tmp/coco/parquet/vocab/part* > vocab.txt

inferencing code doesn't understand split vocab files

Please adjust Vocab.save() method to create a single file. You could use coalesce(1, true)

anfeng · 2016-11-03T04:44:26Z

caffe-grid/src/main/python/examples/README-coco.md

+    export IPYTHON_OPTS="notebook --no-browser --ip=127.0.0.1"
+    pushd ${CAFFE_ON_SPARK}/data/
+    ln -s ~/Python2.7.10 Python2.7.10
+    unzip ${CAFFE_ON_SPARK}/caffe-grid/target/caffeonsparkpythonapi.zip


Remove duplicated lines:

ln -s ~/Python2.7.10 Python2.7.10
unzip ${CAFFE_ON_SPARK}/caffe-grid/target/caffeonsparkpythonapi.zip

Stage 6 and 7 are independent of each other. You either run 6 or 7

Then, we should call it (6), and tell user to do either action.

junshi15 · 2016-11-03T19:25:00Z

Please squash your commits.

mriduljain · 2016-11-04T05:42:37Z

Squashed all commit and accommodated all review comments + trimmed testcase. It should pass and be ready for merge

anfeng · 2016-11-04T16:57:17Z

Makefile

 LD_LIBRARY_PATH2=${LD_LIBRARY_PATH}:${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib:/usr/lib64:/lib64 
-DYLD_LIBRARY_PATH ?=/home/y/lib64:/home/y/lib64/mkl/intel64
-DYLD_LIBRARY_PATH2=${DYLD_LIBRARY_PATH}:${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib:/usr/lib64:/lib64 
+DYLD_LIBRARY_PATH ?=/usr/local/cuda/lib


/home/y/lib64:/home/y/lib64/mkl/intel64:/usr/local/cuda/lib

anfeng · 2016-11-04T16:58:25Z

caffe-grid/src/main/python/com/yahoo/ml/caffe/DisplayUtils.py

@@ -4,6 +4,7 @@
 import numpy as np
 from base64 import b64encode
 from google.protobuf import text_format
+import array


please remove the changes in this file

mriduljain · 2016-11-04T16:59:55Z

I am not sure what you mean. We move show_caption out of this so it changed

On Friday, November 4, 2016, anfeng notifications@github.com wrote:

@anfeng commented on this pull request.

In caffe-grid/src/main/python/com/yahoo/ml/caffe/DisplayUtils.py
#176 (review)
:

@@ -4,6 +4,7 @@
import numpy as np
from base64 import b64encode
from google.protobuf import text_format
+import array

please remove the changes in this file

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#176 (review),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACCTVYEHkHFZBeHtUkBEnU6mlAQbzNVLks5q62QzgaJpZM4KnlWU
.

anfeng · 2016-11-04T17:00:51Z

caffe-grid/src/main/python/examples/README-coco.md

+    Initial Setup: https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_standalone
+    export DYLD_LIBRARY_PATH=${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib:/usr/local/cuda/lib:/usr/local/mkl/lib/intel64/:Python2.7.10/lib:/usr/local/cuda/lib:caffe_on_grid_archive/lib64/mkl/intel64/
+    export LD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}
+    export SPARK_HOME=/Users/mridul/bigml/spark-1.6.0-bin-hadoop2.6


User should set up both CAFFE_ON_SPARK and SPARK_HOME per https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_standalone

anfeng · 2016-11-04T17:01:59Z

caffe-grid/src/main/python/examples/README-coco.md

+    export PYSPARK_PYTHON=Python2.7.10/bin/python
+    export PYTHONPATH=$PYTHONPATH:caffeonsparkpythonapi.zip:caffe_on_grid_archive/lib64:/usr/local/cuda-7.5/lib64
+    export IPYTHON_ROOT=~/Python2.7.10
+    unset SPARK_CONF_DIR


Why this unset statement?

anfeng · 2016-11-04T17:04:55Z

caffe-grid/src/main/python/examples/README-coco.md

+    export IPYTHON_OPTS="notebook --no-browser --ip=127.0.0.1"
+    pushd ${CAFFE_ON_SPARK}/data/
+    ln -s ~/Python2.7.10 Python2.7.10
+    unzip ${CAFFE_ON_SPARK}/caffe-grid/target/caffeonsparkpythonapi.zip


Then, we should call it (6), and tell user to do either action.

anfeng · 2016-11-04T17:11:54Z

caffe-grid/src/main/python/examples/README-coco.md

+    pushd ${CAFFE_ON_SPARK}/data/
+    ln -s ~/Python2.7.10 Python2.7.10
+    unzip ${CAFFE_ON_SPARK}/caffe-grid/target/caffeonsparkpythonapi.zip
+    cat /tmp/coco/parquet/vocab/part* > vocab.txt


Please adjust Vocab.save() method to create a single file. You could use coalesce(1, true)

anfeng · 2016-11-04T17:12:48Z

caffe-grid/src/main/scala/com/yahoo/ml/caffe/tools/Vocab.scala

+
+  def save(vocabFilePath: String): Unit = {
+    synchronized {
+      rdd_word.map(word => word.getString(0)).saveAsTextFile(vocabFilePath)


coalesce(1, true).saveAsTextFile(vocabFilePath)

anfeng · 2016-11-04T17:15:32Z

caffe-grid/src/main/python/examples/README-coco.md

+    pushd ${CAFFE_ON_SPARK}/data/
+    ln -s ~/Python2.7.10 Python2.7.10
+    unzip ${CAFFE_ON_SPARK}/caffe-grid/target/caffeonsparkpythonapi.zip
+    cat /tmp/coco/parquet/vocab/part* > vocab.txt


We should remove this "cat" statenment with Vocab.scala change suggested.

anfeng · 2016-11-04T17:15:57Z

caffe-grid/src/main/python/examples/README-coco.md

+    		 --conf spark.task.cpus=${CORES_PER_WORKER} \    
+    		 --conf spark.driver.extraLibraryPath="${DYLD_LIBRARY_PATH}:Python2.7.10/lib" \
+    		 --conf spark.executorEnv.LD_LIBRARY_PATH="${DYLD_LIBRARY_PATH}:Python2.7.10/lib" \
+    		 --conf spark.pythonargs="-model /tmp/coco/parquet/lrcn_coco.model -imagenet lstm_deploy.prototxt -lstmnet lrcn_word_to_preds.deploy.prototxt -vocab vocab.txt -input /tmp/coco/parquet/df_embedded_train2014 -output /tmp/coco/parquet/df_caption_results_train2014" examples/ImageCaption.py


vocab.txt/part-00000

anfeng · 2016-11-04T17:23:02Z

caffe-grid/src/test/scala/com/yahoo/ml/caffe/tools/ToolTest.scala

+
+    FileUtils.deleteQuietly(new File(cocoImageCaptionDF))
+    val df_image_caption = Conversions.Coco2ImageCaptionFile(sqlContext, cocoJson, 4)
+//    val rdd_input_captions = inputDF2PairRDD(df_image_caption)


remove unwanted statement

anfeng · 2016-11-04T20:42:34Z

Makefile

 	jar -xvf caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar META-INF/native/linux64/liblmdbjni.so
 	mv META-INF/native/linux64/liblmdbjni.so ${CAFFE_ON_SPARK}/caffe-distri/distribute/lib
 	${CAFFE_ON_SPARK}/scripts/setup-mnist.sh
-	export LD_LIBRARY_PATH="${LD_LIBRARY_PATH2}"; mvn ${MVN_SPARK_FLAG} -B test
+	export LD_LIBRARY_PATH="${LD_LIBRARY_PATH2}"; GLOG_minloglevel=1 mvn ${MVN_SPARK_FLAG} -B test
 	cd ${CAFFE_ON_SPARK}/caffe-grid/src/main/python/; zip -r caffeonsparkpythonapi  *; cd ${CAFFE_ON_SPARK}/caffe-public/python/; zip -ur ${CAFFE_ON_SPARK}/caffe-grid/src/main/python/caffeonsparkpythonapi.zip *; cd - ; mv caffeonsparkpythonapi.zip ${CAFFE_ON_SPARK}/caffe-grid/target/; cd ${CAFFE_ON_SPARK}
 	export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}; export SPARK_HOME=${SPARK_HOME};${CAFFE_ON_SPARK}/caffe-grid/src/test/python/PythonTest.sh


GLOG_minloglevel=1 ${CAFFE_ON_SPARK}/caffe-grid/src/test/python/PythonTest.sh

anfeng · 2016-11-04T20:42:50Z

Makefile

 	jar -xvf caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar META-INF/native/osx64/liblmdbjni.jnilib
 	mv META-INF/native/osx64/liblmdbjni.jnilib ${CAFFE_ON_SPARK}/caffe-distri/distribute/lib
 	${CAFFE_ON_SPARK}/scripts/setup-mnist.sh
-	export LD_LIBRARY_PATH="${DYLD_LIBRARY_PATH2}"; mvn ${MVN_SPARK_FLAG} -B test
+	export LD_LIBRARY_PATH="${DYLD_LIBRARY_PATH2}"; GLOG_minloglevel=1 mvn ${MVN_SPARK_FLAG} -B test
 	cd ${CAFFE_ON_SPARK}/caffe-grid/src/main/python/; zip -r caffeonsparkpythonapi  *; cd ${CAFFE_ON_SPARK}/caffe-public/python/; zip -ur ${CAFFE_ON_SPARK}/caffe-grid/src/main/python/caffeonsparkpythonapi.zip *; cd -; mv caffeonsparkpythonapi.zip ${CAFFE_ON_SPARK}/caffe-grid/target/; cd ${CAFFE_ON_SPARK}
 	cd ${CAFFE_ON_SPARK}/caffe-grid/src/main/python/; zip -r caffeonsparkpythonapi *; mv caffeonsparkpythonapi.zip ${CAFFE_ON_SPARK}/caffe-grid/target/; cd ${CAFFE_ON_SPARK}
 	export DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}; export SPARK_HOME=${SPARK_HOME};${CAFFE_ON_SPARK}/caffe-grid/src/test/python/PythonTest.sh


GLOG_minloglevel=1 ${CAFFE_ON_SPARK}/caffe-grid/src/test/python/PythonTest.sh

anfeng · 2016-11-04T20:43:26Z

caffe-grid/src/main/python/examples/README-coco.md

@@ -1,15 +1,14 @@
 Steps to run the COCO dataset for Image Captioning
 ==================================================
 ##### (1) Env setup
-    Initial Setup: https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_standalone
+    Set up both CAFFE_ON_SPARK and SPARK_HOME per https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_standalone
    export DYLD_LIBRARY_PATH=${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib:/usr/local/cuda/lib:/usr/local/mkl/lib/intel64/:Python2.7.10/lib:/usr/local/cuda/lib:caffe_on_grid_archive/lib64/mkl/intel64/
    export LD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}
    export SPARK_HOME=/Users/mridul/bigml/spark-1.6.0-bin-hadoop2.6


remove export SPARK_HOME=/Users/mridul/bigml/spark-1.6.0-bin-hadoop2.6

anfeng · 2016-11-04T20:46:06Z

caffe-grid/src/main/python/examples/README-coco.md

    pyspark --master ${MASTER_URL} --deploy-mode client \    
    	    --conf spark.driver.extraLibraryPath="${DYLD_LIBRARY_PATH}:Python2.7.10/lib" \
 	    --conf spark.executorEnv.LD_LIBRARY_PATH="${DYLD_LIBRARY_PATH}:Python2.7.10/lib" \
-	    --files "${CAFFE_ON_SPARK}/data/lstm_deploy.prototxt,${CAFFE_ON_SPARK}/data/vocab.txt,${CAFFE_ON_SPARK}/data/lrcn_word_to_preds.deploy.prototxt,${CAFFE_ON_SPARK}/data/caffe/_caffe.so,${CAFFE_ON_SPARK}/data/bvlc_reference_net.prototxt,${CAFFE_ON_SPARK}/data/bvlc_reference_solver.prototxt,${CAFFE_ON_SPARK}/data/lrcn_cos.prototxt,${CAFFE_ON_SPARK}/data/lrcn_solver.prototxt" \
+	    --files "${CAFFE_ON_SPARK}/data/lstm_deploy.prototxt,${CAFFE_ON_SPARK}/data/vocab.txt/part-00000,${CAFFE_ON_SPARK}/data/lrcn_word_to_preds.deploy.prototxt,${CAFFE_ON_SPARK}/data/caffe/_caffe.so,${CAFFE_ON_SPARK}/data/bvlc_reference_net.prototxt,${CAFFE_ON_SPARK}/data/bvlc_reference_solver.prototxt,${CAFFE_ON_SPARK}/data/lrcn_cos.prototxt,${CAFFE_ON_SPARK}/data/lrcn_solver.prototxt" \


Don't you need to specify an alias for "${CAFFE_ON_SPARK}/data/vocab.txt/part-00000"? Otherwise, you will see a file named part-00000 in home dir.

anfeng · 2016-11-04T22:30:07Z

+1

Mridul Jain added 3 commits October 29, 2016 12:17

Distributed LSTM

33401bd

Distributed LSTM

03d310d

Distributed LSTM

434200f

anfeng reviewed Nov 2, 2016

View reviewed changes

anfeng reviewed Nov 3, 2016

View reviewed changes

anfeng changed the title ~~Lstm inference~~ LSTM support in CaffeOnSpark Nov 3, 2016

mriduljain force-pushed the lstm_inference branch from 1205e32 to 7d907c0 Compare November 4, 2016 05:38

Distributed LSTM

7d907c0

anfeng reviewed Nov 4, 2016

View reviewed changes

All fixed with trimmed testcase

9147590

anfeng reviewed Nov 4, 2016

View reviewed changes

Reduce logging

1c59767

anfeng reviewed Nov 4, 2016

View reviewed changes

mriduljain added the duplicate label Nov 4, 2016

Mridul Jain added 2 commits November 4, 2016 15:03

All changes

b1d0a35

Reduce logs for python test

b5f0a87

mriduljain removed the duplicate label Nov 4, 2016

anfeng merged commit 054aa08 into master Nov 4, 2016

anfeng deleted the lstm_inference branch November 4, 2016 22:30

junshi15 mentioned this pull request Nov 12, 2016

Error while encoding: java.lang.RuntimeException: java.lang.String is not a valid external type for schema of bigint #184

Closed

		@@ -0,0 +1,8 @@
		from examples.coco.retrieval_experiment import *

		@@ -0,0 +1,14 @@
		net: "/Users/mridul/bigml/CaffeOnSpark/data/train_val.prototxt"

		@@ -0,0 +1,14 @@
		net: "CaffeOnSpark/data/bvlc_reference_net.prototxt"

LSTM support in CaffeOnSpark #176

LSTM support in CaffeOnSpark #176

Conversation

mriduljain commented Nov 2, 2016

yahoocla commented Nov 2, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junshi15 commented Nov 3, 2016

mriduljain commented Nov 4, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mriduljain commented Nov 4, 2016

@anfeng commented on this pull request.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anfeng commented Nov 4, 2016