-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ADAM-2044] Update Spark version to 2.4.3, add move to Scala 2.12 script #2056
Conversation
Test FAILed. Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains d43d7a0 # timeout=10Checking out Revision d43d7a0 (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f d43d7a0e351486160c94b3b812a35283c544809dFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.7.3,2.11,2.2.2,ubuntuTriggering ADAM-prb ? 2.6.2,2.11,2.2.2,ubuntuADAM-prb ? 2.7.3,2.11,2.2.2,ubuntu completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Test FAILed. Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 4a64b40 # timeout=10Checking out Revision 4a64b40 (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 4a64b40d237ca102e09b7d3ac46c14036e6de5f3First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,2.2.2,ubuntuTriggering ADAM-prb ? 2.7.3,2.11,2.2.2,ubuntuADAM-prb ? 2.6.2,2.11,2.2.2,ubuntu completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Appears that a new release of
|
Test FAILed. Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 3cecfcb # timeout=10Checking out Revision 3cecfcb (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 3cecfcb376081b31eb7635b9129a10316497c576First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,2.2.2,ubuntuTriggering ADAM-prb ? 2.7.3,2.11,2.2.2,ubuntuADAM-prb ? 2.6.2,2.11,2.2.2,ubuntu completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Test FAILed. Build result: FAILURE[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 85506a7 # timeout=10Checking out Revision 85506a7 (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 85506a75fa0eb7a8c80b613a41248e4e24f0220bFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,2.2.2,ubuntuTriggering ADAM-prb ? 2.7.3,2.11,2.2.2,ubuntuADAM-prb ? 2.6.2,2.11,2.2.2,ubuntu completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Per the most recent failures, it appears that we now have a conflict between parquet 1.8.3 used in adam and 1.10.0 used in Spark 2.4.0. Bumping the parquet version to 1.10.0 hits the error described further in issue #2058.
|
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit 222252e.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse 222252e^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 222252e # timeout=10Checking out Revision 222252e (origin/pr/2056/head) > git config core.sparsecheckout # timeout=10 > git checkout -f 222252e4f5851e0c94b74fac9e008192d4941ebcFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.7.5,2.11,2.2.2,ubuntuADAM-prb ? 2.7.5,2.11,2.2.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit 1262416.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse 1262416^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 1262416 # timeout=10Checking out Revision 1262416 (origin/pr/2056/head) > git config core.sparsecheckout # timeout=10 > git checkout -f 1262416 > git rev-list 222252e # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.2.2,ubuntuADAM-prb ? 2.7.5,2.11,2.2.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit f2cd5b0 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains fe0dafe # timeout=10Checking out Revision fe0dafe (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f fe0dafed5ce81eac3e581d216809a1199998a5c7First time build. Skipping changelog.Triggering ADAM-prb ? 2.7.5,2.11,2.2.3,ubuntuADAM-prb ? 2.7.5,2.11,2.2.3,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Rebased to fix merge; squashed commit from #2107 |
Jenkins, retest this please |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit f2cd5b0 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains fe0dafe # timeout=10Checking out Revision fe0dafe (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f fe0dafe > git rev-list fe0dafe # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.2.3,ubuntuADAM-prb ? 2.7.5,2.11,2.2.3,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit c0fc042 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 153873f # timeout=10Checking out Revision 153873f (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 153873f > git rev-list fe0dafe # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.2.3,ubuntuADAM-prb ? 2.7.5,2.11,2.2.3,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Jenkins, retest this please |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit c0fc042 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 153873f # timeout=10Checking out Revision 153873f (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 153873f > git rev-list 153873f # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.2.3,ubuntuADAM-prb ? 2.7.5,2.11,2.2.3,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Rebased, bumped Spark dependency to 2.4.2. Still fails unit tests |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit bd5b6e0 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 6faa666 # timeout=10Checking out Revision 6faa666 (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 6faa666 > git rev-list 153873f # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.3.2,ubuntuADAM-prb ? 2.7.5,2.11,2.3.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit e1d3cec automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains ddc6f13 # timeout=10Checking out Revision ddc6f13 (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f ddc6f13 > git rev-list 6faa666 # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.3.2,ubuntuADAM-prb ? 2.7.5,2.11,2.3.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Jenkins, retest this please |
Several new compile errors under Scala 2.12
|
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit 30327b0 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 2948c4a # timeout=10Checking out Revision 2948c4a (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 2948c4a > git rev-list ddc6f13 # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.3.2,ubuntuADAM-prb ? 2.7.5,2.11,2.3.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit eba500c automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 0086fa9 # timeout=10Checking out Revision 0086fa9 (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 0086fa9 > git rev-list 2948c4a # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.3.2,ubuntuADAM-prb ? 2.7.5,2.11,2.3.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit 874a4f5 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains c4f4ace # timeout=10Checking out Revision c4f4ace (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f c4f4ace > git rev-list 0086fa9 # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.3.2,ubuntuADAM-prb ? 2.7.5,2.11,2.3.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit 3a771ac automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains e03b030 # timeout=10Checking out Revision e03b030 (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f e03b030 > git rev-list c4f4ace # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.3.2,ubuntuADAM-prb ? 2.7.5,2.11,2.3.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Jenkins, retest this please |
Test PASSed. |
Please review. Note Spark 2.4.2 was released with the binary distribution defaulting to Scala 2.12 in error. Thus we should probably wait to merge this pull request (and related bdg-utils 0.2.15 release) on the release of 2.4.3, which should be coming soon and will fix this error. Otherwise it will be difficult to update the Jenkins build configuration. |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit 488ac7c automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 12a124f # timeout=10Checking out Revision 12a124f (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 12a124f > git rev-list e03b030 # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.3.2,ubuntuADAM-prb ? 2.7.5,2.11,2.3.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit 94e124e automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 869e000 # timeout=10Checking out Revision 869e000 (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 869e000 > git rev-list 12a124f # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.3.2,ubuntuADAM-prb ? 2.7.5,2.11,2.3.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Jenkins, retest this please |
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit 94e124e automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 869e000 # timeout=10Checking out Revision 869e000 (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 869e000 > git rev-list 869e000 # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.3.2,ubuntuADAM-prb ? 2.7.5,2.11,2.3.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Jenkins, retest this please |
Test PASSed. |
@@ -127,7 +130,7 @@ class TransformVariants(val args: TransformVariantsArgs) | |||
stringency = stringency) | |||
|
|||
if (args.outputPath.endsWith(".vcf")) { | |||
maybeSort(maybeCoalesce(variants.toVariantContexts)).saveAsVcf(args, stringency) | |||
maybeSort(maybeCoalesce(variants).toVariantContexts).saveAsVcf(args, stringency) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note this change may have performance implications, I couldn't figure out how to maintain the generic signature for maybeCoalesce
Ping @akmorrow13 @fnothaft @jpdna for review. This now works for me with Scala 2.12 against Spark 2.4.3 in local and clustered mode. I'll squash commits after review. |
<relocations> | ||
<relocation> | ||
<pattern>org.apache.parquet.avro</pattern> | ||
<shadedPattern>org.bdgenomics.adam.shaded.org.apache.parquet.avro</shadedPattern> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whats the shade for again?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly, you don't want to know. 😉
Since parquet-mr
won't include the pull request with the fix we need for some time, I've had to hack up an older version from source to add the fix, build it in adam, then shade parquet-avro
to another package, while excluding the version in the external jar that does not contain the fix, with an extension to maven-shade-plugin
to do so correctly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh my
find . -name "pom.xml" -exec sed -e "s/2.10.6/2.11.12/g" \ | ||
-e "s/2.10/2.11/g" \ | ||
find . -name "pom.xml" -exec sed -e "s/2.12.8/2.11.12/g" \ | ||
-e "s/2.12/2.11/g" \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think these files are being referenced in jenkins-test anymore. Do we want to run jenkins-test on scala 2.11 and 2.12?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Chicken and egg unfortunately, Spark 2.4.x is required for Scala 2.12, and ADAM won't work on Scala 2.12 until this pull request
Test FAILed. Build result: FAILUREGitHub pull request #2056 of commit 9841310 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2056/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 1859f8c # timeout=10Checking out Revision 1859f8c (origin/pr/2056/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 1859f8c > git rev-list 869e000 # timeout=10Triggering ADAM-prb ? 2.7.5,2.11,2.3.2,ubuntuADAM-prb ? 2.7.5,2.11,2.3.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'Test FAILed. |
Jenkins, retest this please |
Test PASSed. |
Fixes #2044, fixes #2146, fixes #2114, fixes #2098, fixes #2091, fixes #2058