[SPARK-9773] [ML] [PySpark] Add Python API for MultilayerPerceptronClassifier

[yanboliang]

Add Python API for MultilayerPerceptronClassifier.

[SparkQA]

Test build #40288 has finished for PR 8067 at commit 70d1da9.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,
  • class MultilayerPerceptronClassifierModel(JavaModel):

[yanboliang]

It looks like we hit SPARK-7379, I try to find some clue.

[SparkQA]

Test build #40433 has finished for PR 8067 at commit 8510817.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,
  • class MultilayerPerceptronClassifierModel(JavaModel):

[mengxr]

@yanboliang Could you describe the bug in the PR description?

[yanboliang]

Jenkins, test this please.

[SparkQA]

Test build #40953 has finished for PR 8067 at commit 8c94570.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,
  • class MultilayerPerceptronClassificationModel(JavaModel):
  • case class FilterNode(condition: Expression, child: LocalNode) extends UnaryLocalNode
  • abstract class LocalNode extends TreeNode[LocalNode]
  • abstract class LeafLocalNode extends LocalNode
  • abstract class UnaryLocalNode extends LocalNode
  • case class ProjectNode(projectList: Seq[NamedExpression], child: LocalNode) extends UnaryLocalNode
  • case class SeqScanNode(output: Seq[Attribute], data: Seq[InternalRow]) extends LeafLocalNode
  • public final class UTF8String implements Comparable<UTF8String>, Externalizable

[SparkQA]

Test build #40954 has finished for PR 8067 at commit 8c94570.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class StringIndexerModel (
  • class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,
  • class MultilayerPerceptronClassificationModel(JavaModel):
  • case class FilterNode(condition: Expression, child: LocalNode) extends UnaryLocalNode
  • abstract class LocalNode extends TreeNode[LocalNode]
  • abstract class LeafLocalNode extends LocalNode
  • abstract class UnaryLocalNode extends LocalNode
  • case class ProjectNode(projectList: Seq[NamedExpression], child: LocalNode) extends UnaryLocalNode
  • case class SeqScanNode(output: Seq[Attribute], data: Seq[InternalRow]) extends LeafLocalNode
  • public final class UTF8String implements Comparable<UTF8String>, Externalizable

[yanboliang]

Jenkins, test this please.

[SparkQA]

Test build #40958 has finished for PR 8067 at commit 8c94570.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,
  • class MultilayerPerceptronClassificationModel(JavaModel):
  • case class FilterNode(condition: Expression, child: LocalNode) extends UnaryLocalNode
  • abstract class LocalNode extends TreeNode[LocalNode]
  • abstract class LeafLocalNode extends LocalNode
  • abstract class UnaryLocalNode extends LocalNode
  • case class ProjectNode(projectList: Seq[NamedExpression], child: LocalNode) extends UnaryLocalNode
  • case class SeqScanNode(output: Seq[Attribute], data: Seq[InternalRow]) extends LeafLocalNode
  • public final class UTF8String implements Comparable<UTF8String>, Externalizable

[mengxr]

@yanboliang Shall we split this PR into two? One makes layers and weights public, which can be merged into 1.5, and the other for the Python API, which will be merged into 1.6.

[yanboliang]

@mengxr Sure, I have submitted #8263 to make layers and weights public, and let this one focus on Python API.

[yanboliang]

Here we also need to make layers as a property function, but if we do that like weights in the following lines, we will hit SPARK-7379. It can work well in Python 2 but raise errors in Python 3.
I propose to make the layers of MultilayerPerceptronClassificationModel as Vector rather than Array[Int] at Scala side. Because PySpark can tackle Vector elegantly. And I found all other interfaces of ML use Vector rather than Array. @mengxr

[mengxr]

Shall we add a package private method to Scala's MPCM that returns a Java list of integers?

[yanboliang]

Agree, done.

[SparkQA]

Test build #42198 has finished for PR 8067 at commit db3c676.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,
  • class MultilayerPerceptronClassificationModel(JavaModel):

[SparkQA]

Test build #42261 has finished for PR 8067 at commit b093862.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,
  • class MultilayerPerceptronClassificationModel(JavaModel):

[mengxr]

sqlContext.createDataFrame([

[SparkQA]

Test build #42315 has finished for PR 8067 at commit 5ac6a70.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,
  • class MultilayerPerceptronClassificationModel(JavaModel):

[yanboliang]

L882 we still keep layers=[1, 1] in doc to tell users the default value.

[mengxr]

LGTM. Merged into master. Thanks!