Skip to content

[SPARK-6486] [MLlib] [Python] Add BlockMatrix to PySpark. #7761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dusenberrymw wants to merge 7 commits into from
Closed

[SPARK-6486] [MLlib] [Python] Add BlockMatrix to PySpark. #7761

dusenberrymw wants to merge 7 commits into from

Conversation

@dusenberrymw
Copy link
Contributor

@dusenberrymw dusenberrymw commented Jul 29, 2015

@mengxr This adds the BlockMatrix to PySpark. I have the conversions to IndexedRowMatrix and CoordinateMatrix ready as well, so once PR #7554 is completed (which relies on PR #7746), this PR can be finished.

@mengxr
Copy link
Contributor

mengxr commented Jul 29, 2015

@dusenberrymw Did you coordinate with @MechCoder about this? I thought he already started working on it.

@SparkQA
Copy link

SparkQA commented Jul 30, 2015

Test build #38916 has finished for PR 7761 at commit 2d03a0a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class BlockMatrix(object):

@dusenberrymw
Copy link
Contributor Author

dusenberrymw commented Jul 30, 2015

@mengxr We've been discussing things on PR #7554, but I wasn't sure if Manoj was working on this one. It's a pretty natural extension of 7554, so I already had most of it completed.

@MechCoder Apologies if we were both working on this one. I've appreciated the thoughts on #7554, and I'd definitely love any thoughts on this one as well.

@mengxr
Copy link
Contributor

mengxr commented Jul 30, 2015

@dusenberrymw Next time, you should check and ping the JIRA page first before you start coding to avoid duplicate work.

@dusenberrymw
Copy link
Contributor Author

dusenberrymw commented Jul 30, 2015

@mengxr Yes, certainly.

As soon as you merge #7746, I can finish #7554 and this PR. I'd also like to work on SPARK-6488.

@SparkQA
Copy link

SparkQA commented Jul 30, 2015

Test build #38937 has finished for PR 7761 at commit ea46fbe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class BlockMatrix(object):

@dusenberrymw
Copy link
Contributor Author

dusenberrymw commented Jul 30, 2015

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Jul 30, 2015

Test build #38955 has finished for PR 7761 at commit 9a813d3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class BlockMatrix(object):

@SparkQA
Copy link

SparkQA commented Jul 30, 2015

Test build #150 has finished for PR 7761 at commit 9a813d3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class BlockMatrix(object):

@MechCoder
Copy link
Contributor

MechCoder commented Jul 30, 2015

@dusenberrymw You could have pinged me on the JIRA, since I already started work on this. (But no hard feelings)

I can give a pass on this after I get back access to my machine on Saturday.

@dusenberrymw
Copy link
Contributor Author

dusenberrymw commented Jul 30, 2015

@MechCoder Yes, apologies again for not realizing you were also working on this. I look forward to your thoughts!

@mengxr mengxr mentioned this pull request Jul 31, 2015
Closed
@dusenberrymw dusenberrymw force-pushed the SPARK-6486_Add_BlockMatrix_to_PySpark branch from 9a813d3 to b8acc1c Compare August 5, 2015 02:24
@SparkQA
Copy link

SparkQA commented Aug 5, 2015

Test build #39808 has finished for PR 7761 at commit b8acc1c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class BlockMatrix(object):

@SparkQA
Copy link

SparkQA commented Aug 5, 2015

Test build #39834 has finished for PR 7761 at commit 27195c2.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class BlockMatrix(DistributedMatrix):

@asfgit asfgit closed this in 34dcf10 Aug 5, 2015
asfgit pushed a commit that referenced this pull request Aug 5, 2015
@dusenberrymw @mengxr
[SPARK-6486] [MLLIB] [PYTHON] Add BlockMatrix to PySpark.
eedb996 
mengxr This adds the `BlockMatrix` to PySpark.  I have the conversions to `IndexedRowMatrix` and `CoordinateMatrix` ready as well, so once PR #7554 is completed (which relies on PR #7746), this PR can be finished.

Author: Mike Dusenberry <mwdusenb@us.ibm.com>

Closes #7761 from dusenberrymw/SPARK-6486_Add_BlockMatrix_to_PySpark and squashes the following commits:

27195c2 [Mike Dusenberry] Adding one more check to _convert_to_matrix_block_tuple, and a few minor documentation changes.
ae50883 [Mike Dusenberry] Minor update: BlockMatrix should inherit from DistributedMatrix.
b8acc1c [Mike Dusenberry] Moving BlockMatrix to pyspark.mllib.linalg.distributed, updating the logic to match that of the other distributed matrices, adding conversions, and adding documentation.
c014002 [Mike Dusenberry] Using properties for better documentation.
3bda6ab [Mike Dusenberry] Adding documentation.
8fb3095 [Mike Dusenberry] Small cleanup.
e17af2e [Mike Dusenberry] Adding BlockMatrix to PySpark.

(cherry picked from commit 34dcf10)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
@mengxr
Copy link
Contributor

mengxr commented Aug 5, 2015

LGTM. Merged into master and branch-1.5. Thanks!

@dusenberrymw
Copy link
Contributor Author

dusenberrymw commented Aug 5, 2015

Great, thanks @mengxr!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dusenberrymw @mengxr @SparkQA @MechCoder