[SPARK-6097][MLLIB] Support tree model save/load in PySpark/MLlib #4854

mengxr · 2015-03-02T20:55:57Z

Similar to MatrixFactorizaionModel, we only need wrappers to support save/load for tree models in Python.

@jkbradley

SparkQA · 2015-03-02T20:58:04Z

Test build #28183 has started for PR 4854 at commit 201b3b9.

This patch merges cleanly.

SparkQA · 2015-03-02T20:59:04Z

Test build #28183 has finished for PR 4854 at commit 201b3b9.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MatrixFactorizationModel(JavaModelWrapper, JavaSaveable, JavaLoader):
- class TreeEnsembleModel(JavaModelWrapper, JavaSaveable):
- class DecisionTreeModel(JavaModelWrapper, JavaSaveable, JavaLoader):
- class RandomForestModel(TreeEnsembleModel, JavaLoader):
- class GradientBoostedTreesModel(TreeEnsembleModel, JavaLoader):
- class Saveable(object):
- class JavaSaveable(Saveable):
- class Loader(object):
- class JavaLoader(Loader):
- java_class = cls._java_loader_class()

AmplabJenkins · 2015-03-02T20:59:05Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28183/
Test FAILed.

jkbradley · 2015-03-02T21:01:12Z

docs/mllib-decision-tree.md

Missing quotation mark (here and in other examples)

SparkQA · 2015-03-02T21:08:04Z

Test build #28185 has started for PR 4854 at commit 8ebcac2.

This patch merges cleanly.

jkbradley · 2015-03-02T21:11:17Z

python/pyspark/mllib/tests.py

jkbradley · 2015-03-02T21:12:33Z

LGTM other than that typo. I'm trying to compile & test now.

SparkQA · 2015-03-02T22:13:52Z

Test build #28185 has finished for PR 4854 at commit 8ebcac2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MatrixFactorizationModel(JavaModelWrapper, JavaSaveable, JavaLoader):
- class TreeEnsembleModel(JavaModelWrapper, JavaSaveable):
- class DecisionTreeModel(JavaModelWrapper, JavaSaveable, JavaLoader):
- class RandomForestModel(TreeEnsembleModel, JavaLoader):
- class GradientBoostedTreesModel(TreeEnsembleModel, JavaLoader):
- class Saveable(object):
- class JavaSaveable(Saveable):
- class Loader(object):
- class JavaLoader(Loader):
- java_class = cls._java_loader_class()

AmplabJenkins · 2015-03-02T22:13:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28185/
Test FAILed.

SparkQA · 2015-03-02T23:02:34Z

Test build #28196 has started for PR 4854 at commit 4586a4d.

This patch merges cleanly.

SparkQA · 2015-03-03T01:02:35Z

Test build #28196 timed out for PR 4854 at commit 4586a4d after a configured wait of 120m.

AmplabJenkins · 2015-03-03T01:02:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28196/
Test FAILed.

mengxr · 2015-03-03T01:13:59Z

test this please

SparkQA · 2015-03-03T01:17:31Z

Test build #28208 has started for PR 4854 at commit 4586a4d.

This patch merges cleanly.

SparkQA · 2015-03-03T02:19:33Z

Test build #28208 has finished for PR 4854 at commit 4586a4d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MatrixFactorizationModel(JavaModelWrapper, JavaSaveable, JavaLoader):
- class TreeEnsembleModel(JavaModelWrapper, JavaSaveable):
- class DecisionTreeModel(JavaModelWrapper, JavaSaveable, JavaLoader):
- class RandomForestModel(TreeEnsembleModel, JavaLoader):
- class GradientBoostedTreesModel(TreeEnsembleModel, JavaLoader):
- class JavaSaveable(Saveable):
- java_class = cls._java_loader_class()

AmplabJenkins · 2015-03-03T02:19:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28208/
Test FAILed.

mengxr · 2015-03-03T02:27:01Z

test this please

SparkQA · 2015-03-03T02:27:37Z

Test build #28215 has started for PR 4854 at commit 4586a4d.

This patch merges cleanly.

SparkQA · 2015-03-03T03:49:42Z

Test build #28215 has finished for PR 4854 at commit 4586a4d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-03T03:49:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28215/
Test PASSed.

Similar to `MatrixFactorizaionModel`, we only need wrappers to support save/load for tree models in Python. jkbradley Author: Xiangrui Meng <meng@databricks.com> Closes #4854 from mengxr/SPARK-6097 and squashes the following commits: 4586a4d [Xiangrui Meng] fix more typos 8ebcac2 [Xiangrui Meng] fix python style 91172d8 [Xiangrui Meng] fix typos 201b3b9 [Xiangrui Meng] update user guide b5158e2 [Xiangrui Meng] support tree model save/load in PySpark/MLlib (cherry picked from commit 7e53a79) Signed-off-by: Xiangrui Meng <meng@databricks.com>

mengxr · 2015-03-03T06:28:17Z

Merged into master and branch-1.3.

catmonkeylee · 2015-03-20T17:40:28Z

When I run the sample code in cluster mode, there is an error.

Traceback (most recent call last):
File "/data1/s/apps/spark-app/app/sample_rf.py", line 25, in
sameModel = RandomForestModel.load(sc, _model_path)
File "/home/s/apps/spark/python/pyspark/mllib/util.py", line 254, in load
java_model = cls._load_java(sc, path)
File "/home/s/apps/spark/python/pyspark/mllib/util.py", line 250, in _load_java
return java_obj.load(sc._jsc.sc(), path)
File "/home/s/apps/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in call
File "/home/s/apps/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.mllib.tree.model.RandomForestModel.load.
: java.lang.UnsupportedOperationException: empty collection
at org.apache.spark.rdd.RDD.first(RDD.scala:1191)
at org.apache.spark.mllib.util.Loader$.loadMetadata(modelSaveLoad.scala:125)
at org.apache.spark.mllib.tree.model.RandomForestModel$.load(treeEnsembleModels.scala:65)
at org.apache.spark.mllib.tree.model.RandomForestModel.load(treeEnsembleModels.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

jkbradley · 2015-03-20T22:45:23Z

Can you please open up a JIRA to report the bug? Also, what is the app you were running, and could you provide the code which created and saved the model originally?

catmonkeylee · 2015-03-21T02:39:42Z

I don't know how to create a new issue, So I 'm sorry to send this mail to you. Please Help me.

I run the code on a spark cluster , spark version is 1.3.0

The test code:

from pyspark import SparkContext, SparkConf
from pyspark.mllib.tree import RandomForest, RandomForestModel
from pyspark.mllib.util import MLUtils

conf = SparkConf().setAppName('LocalTest')
sc = SparkContext(conf=conf)
data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt')
print data.count()
(trainingData, testData) = data.randomSplit([0.7, 0.3])
model = RandomForest.trainClassifier(trainingData, numClasses=2, categoricalFeaturesInfo={},
numTrees=3, featureSubsetStrategy="auto",
impurity='gini', maxDepth=4, maxBins=32)

Evaluate model on test instances and compute test error

predictions = model.predict(testData.map(lambda x: x.features))
labelsAndPredictions = testData.map(lambda lp: lp.label).zip(predictions)
testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())
print('Test Error = ' + str(testErr))
print('Learned classification forest model:')
print(model.toDebugString())

Save and load model

_model_path = "/home/s/apps/spark-app/data/myModelPath"
model.save(sc, _model_path)
sameModel = RandomForestModel.load(sc, _model_path)
sc.stop()

run command:
spark-submit --master spark://t0.q.net:7077 --executor-memory 1G sample_rf.py

Then I get this error :

Traceback (most recent call last):
File "/data1/s/apps/spark-app/app/sample_rf.py", line 25, in
sameModel = RandomForestModel.load(sc, _model_path)
File "/home/s/apps/spark/python/pyspark/mllib/util.py", line 254, in load
java_model = cls._load_java(sc, path)
File "/home/s/apps/spark/python/pyspark/mllib/util.py", line 250, in _load_java
return java_obj.load(sc._jsc.sc(), path)
File "/home/s/apps/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in call
File "/home/s/apps/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.mllib.tree.model.RandomForestModel.load.
: java.lang.UnsupportedOperationException: empty collection
at org.apache.spark.rdd.RDD.first(RDD.scala:1191)
at org.apache.spark.mllib.util.Loader$.loadMetadata(modelSaveLoad.scala:125)
at org.apache.spark.mllib.tree.model.RandomForestModel$.load(treeEnsembleModels.scala:65)
at org.apache.spark.mllib.tree.model.RandomForestModel.load(treeEnsembleModels.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:207)
at java.lang.Thread.run(Thread.java:724)

发自我的 iPhone

在 2015年3月21日，06:45，jkbradley notifications@github.com 写道：

Can you please open up a JIRA to report the bug? Also, what is the app you were running, and could you provide the code which created and saved the model originally?

—
Reply to this email directly or view it on GitHub.

jkbradley · 2015-03-22T17:34:39Z

@catmonkeylee I created a JIRA here: [https://issues.apache.org/jira/browse/SPARK-6457] If you can, please comment on the JIRA; I'll add suggestions there. Thanks!

mengxr added 2 commits March 2, 2015 12:48

support tree model save/load in PySpark/MLlib

b5158e2

update user guide

201b3b9

jkbradley reviewed Mar 2, 2015
View reviewed changes

docs/mllib-decision-tree.md Outdated

Copy link

Member

jkbradley Mar 2, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing quotation mark (here and in other examples)

mengxr added 2 commits March 2, 2015 13:03

fix typos

91172d8

fix python style

8ebcac2

jkbradley reviewed Mar 2, 2015
View reviewed changes

python/pyspark/mllib/tests.py Outdated

Copy link

Member

jkbradley Mar 2, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

fix more typos

4586a4d

asfgit closed this in 7e53a79 Mar 3, 2015

[SPARK-6097][MLLIB] Support tree model save/load in PySpark/MLlib #4854

[SPARK-6097][MLLIB] Support tree model save/load in PySpark/MLlib #4854

Uh oh!

Conversation

mengxr commented Mar 2, 2015

Uh oh!

SparkQA commented Mar 2, 2015

Uh oh!

SparkQA commented Mar 2, 2015

Uh oh!

AmplabJenkins commented Mar 2, 2015

Uh oh!

jkbradley Mar 2, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 2, 2015

Uh oh!

jkbradley Mar 2, 2015

Choose a reason for hiding this comment

Uh oh!

jkbradley commented Mar 2, 2015

Uh oh!

SparkQA commented Mar 2, 2015

Uh oh!

AmplabJenkins commented Mar 2, 2015

Uh oh!

SparkQA commented Mar 2, 2015

Uh oh!

SparkQA commented Mar 3, 2015

Uh oh!

AmplabJenkins commented Mar 3, 2015

Uh oh!

mengxr commented Mar 3, 2015

Uh oh!

SparkQA commented Mar 3, 2015

Uh oh!

SparkQA commented Mar 3, 2015

Uh oh!

AmplabJenkins commented Mar 3, 2015

Uh oh!

mengxr commented Mar 3, 2015

Uh oh!

SparkQA commented Mar 3, 2015

Uh oh!

SparkQA commented Mar 3, 2015

Uh oh!

AmplabJenkins commented Mar 3, 2015

Uh oh!

mengxr commented Mar 3, 2015

Uh oh!

catmonkeylee commented Mar 20, 2015

Uh oh!

jkbradley commented Mar 20, 2015

Uh oh!

catmonkeylee commented Mar 21, 2015

The test code:

Evaluate model on test instances and compute test error

Save and load model

Uh oh!

jkbradley commented Mar 22, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants