[SPARK-30660][ML][PYSPARK] LinearRegression blockify input vectors

[zhengruifeng]

What changes were proposed in this pull request?

1, use blocks instead of vectors for performance improvement
2, use Level-2 BLAS
3, move standardization of input vectors outside of gradient computation

Why are the changes needed?

1, less RAM to persist training data; (save ~40%)
2, faster than existing impl; (30% ~ 102%)

Does this PR introduce any user-facing change?

add a new expert param blockSize

How was this patch tested?

updated testsuites

[zhengruifeng]

testCode:

import org.apache.spark.ml.regression._
import org.apache.spark.storage.StorageLevel


val df = spark.read.format("libsvm").load("/data1/Datasets/a9a/a9a")
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count

new LinearRegression().setMaxIter(10).fit(df)

val lr1 = new LinearRegression().setSolver("l-bfgs").setLoss("squaredError").setMaxIter(100)
val start = System.currentTimeMillis; val model1 = lr1.fit(df); val end = System.currentTimeMillis; end - start


val lr2 = new LinearRegression().setSolver("l-bfgs").setLoss("huber").setMaxIter(100)
val start = System.currentTimeMillis; val model2 = lr2.fit(df); val end = System.currentTimeMillis; end - start

Seq(model1, model2).map(_.summary.totalIterations)
Seq(model1, model2).map(_.summary.objectiveHistory.last)

Result:
This PR:
Duration: 1598, 2473
last objective: 0.30658886019384274, 0.5991272847846535
numIteration: 40, 101

Master:
Duration: 2060, 5015
last objective: 0.306588860193839, 0.5985319305006285
numIteration: 40, 101

[SparkQA]

Test build #117550 has finished for PR 27396 at commit 2da0380.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[zhengruifeng]

retest this please

[SparkQA]

Test build #117558 has finished for PR 27396 at commit 2da0380.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhengruifeng]

cc @srowen

[srowen]

Generally looks OK pending tests, as it's consistent with other similar changes. I assume this doesn't change behavior or API - doesn't look so but just checking? and likewise perf is probably still fine on small data? Also CC @huaxingao

I know code freeze is coming tomorrow. I think we can get this in if there don't turn out to be any issues.

[zhengruifeng]

@srowen The above test is also based on a9a, a small dataset with numFeatures=123, numInstances=32,561

[srowen]

Merged to master