[SPARK-9408] [PySpark] [MLlib] Refactor linalg.py to /linalg

[MechCoder]

I refactored linalg.py to a folder /linalg so that future updates like distributed.py can be made easily.

[MechCoder]

@mengxr This breaks code on my machine, but I can't figure out why. :(

[SparkQA]

Test build #38738 has finished for PR 7731 at commit 200f08f.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

@MechCoder I think you need to add __all__ to __init__.py to import the names, e.g., https://github.com/apache/spark/blob/master/python/pyspark/mllib/stat/__init__.py#L27.

[MechCoder]

That does not work as well. I tried locally (I pushed it still).

from pyspark.mllib.linalg import Vectors
Vectors.dense([0.0, 1.0])

Do you think it's some issue with serialization and deserialization?

[mengxr]

Where is VectorUDT and MatrixUDT?

[MechCoder]

I added those but still it does not work.

I also changed , but it should not matter (but did not push)

pyUDT to pyspark.mllib.linalg.local.MatrixUDT

and @classmethod def module(cls): to pyspark.mllib.local.linalg but it still gives the same error.

[SparkQA]

Test build #38760 has finished for PR 7731 at commit d870a1a.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dusenberrymw]

Yeah, I've been working on this as well, and I've run into the same serialization issues.

[mengxr]

@MechCoder @dusenberrymw Thanks for testing! I will take a look.