Misc shuffle join fixes. #1253
Merged
Misc shuffle join fixes. #1253
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test PASSed. |
heuermh
approved these changes
Nov 14, 2016
|
|
|
Not quite ready to go yet, will get to the tests today. |
Resolves bigdatagenomics#1222. ShuffleRegionJoin was importing `org.apache.spark.SparkContext._` instead of `org.apache.spark.rdd.MetricsContext._`.
…in GenomicRDD. Resolves bigdatagenomics#1220. Adds a function called in each of the shuffle join implementations that calculates the sequence dictionary after the join, as well as the partition sizes to request.
fnothaft
force-pushed
the
issues/1220-1222-shuffle-join-fixes
branch
from
c2e6ad8
to
c01b8a7
Compare
|
Test FAILed. Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1253/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 2d7ed52ffbfae554560f53937f3edd9fd72b9b54 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1253/merge^{commit} # timeout=10Checking out Revision 2d7ed52ffbfae554560f53937f3edd9fd72b9b54 (origin/pr/1253/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 2d7ed52ffbfae554560f53937f3edd9fd72b9b54First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
fnothaft
force-pushed
the
issues/1220-1222-shuffle-join-fixes
branch
from
c01b8a7
to
32abf4b
Compare
|
Test PASSed. |
|
Could you squash these commits as you'd like? Then I'll merge using the Rebase and Merge button. |
Added tests to all of the GenomicRDD test suites to run all of the available broadcast and shuffle region join implementations across all of the various GenomicRDD types. This caught several bugs, which were fixed in this commit.
fnothaft
force-pushed
the
issues/1220-1222-shuffle-join-fixes
branch
from
32abf4b
to
85f889a
Compare
|
@heuermh I decided not to squash any of the commits down, but I amended the last commit message to make it more clear about what the last commit does. I've just pushed. |
|
Test PASSed. |
|
Thank you, @fnothaft! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #1220, #1222. Could use some more unit test coverage; I'll make a pass back and get that.