[SPARK-24557][ML] ClusteringEvaluator support array input

[zhengruifeng]

What changes were proposed in this pull request?

ClusteringEvaluator support array input

How was this patch tested?

added tests

[SparkQA]

Test build #91828 has finished for PR 21563 at commit b126bd4.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #91830 has finished for PR 21563 at commit d6e76e3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mgaido91]

not sure this is the right way. Probably we can face the same issue everywhere we are using DatasetUtils.columnToVector. Probably it is better to fix the problem there.

[zhengruifeng]

@mgaido91 Thanks for your reviewing!
I have considered this, however there exists a problem:
if we want to append metadata into the transformed column (like using method .as(alias: String, metadata: Metadata)) in DatasetUtils.columnToVector, how can we get the name of transformed column?
The only way to do this I know is:

val metadata = ...
val vectorCol = ..
val vectorName = dataset.select(vectorCol) .schema.head.name
vectorCol.as(vectorName, metadata)

[mgaido91]

we have the new column we are returning, so we can get its name with .expr.sql

[zhengruifeng]

@mgaido91 I think it maybe nice to first add a name getter for column

[mgaido91]

we can propose that

[SparkQA]

Test build #91836 has finished for PR 21563 at commit eb6022c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhengruifeng]

@mgaido91 I am sorry to make a force push to update my git username in this PR.
Since I found that my current PRs are not linked to my account and it is troublesome to track them.

and I found that .expr.sql may not be a good way to get the output column name:

val df = Seq((1,2),(3,4),(1,1),(2,-1),(3,1)).toDF("a","b")
val z = (col("a") + col("b")).as("x").as("z")
scala> z.expr.sql
res15: String = (`a` + `b`) AS `x` AS `z`

So I still keep the orignal method.

@MLnick @mengxr @jkbradley Would you please help reveiwing this? Thanks!

[SparkQA]

Test build #93586 has finished for PR 21563 at commit 9064e7b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhengruifeng]

retest this please

[SparkQA]

Test build #93863 has finished for PR 21563 at commit 9064e7b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhengruifeng]

@mengxr I notice that you open a ticket for supporting integer type labels in ClusteringEvalutator, would you like to shepherd this pr too?

[mengxr]

LGTM. Merged into master. Thanks!