misc pom/test/resource improvements #1142

ryan-williams · 2016-08-31T16:16:52Z

these happened while i was doing #1139; individual commits should be reasonably self-explanatory

AmplabJenkins · 2016-08-31T16:28:39Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1440/

Build result: FAILURE

GitHub pull request #1142 of commit 3e02e9b automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1142/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 3722878646d30b56dcaa16c9624bc0467db0486c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1142/merge^{commit} # timeout=10Checking out Revision 3722878646d30b56dcaa16c9624bc0467db0486c (origin/pr/1142/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 3722878646d30b56dcaa16c9624bc0467db0486cFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

ryan-williams · 2016-08-31T16:39:56Z

Some problem with where test-coverage reports are getting written after I changed directory configs from surefire-reports to scalatest-reports; I assumed that was find to just do everywhere but maybe something not in this repo has surefire-reports hard-coded? Looking…

ryan-williams · 2016-08-31T16:49:23Z

The end of the log seems to get through the end of jenkins-test script successfully, so I can't tell what it's trying to do when the failure occurs:

+ echo 'All the tests passed'
All the tests passed
echo
+ echo

Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Publishing Scoverage XML and HTML report...
Setting commit status on GitHub for https://github.com/bigdatagenomics/adam/commit/3722878646d30b56dcaa16c9624bc0467db0486c
Finished: FAILURE

fnothaft · 2016-08-31T16:50:38Z

@ryan-williams IIRC, Jenkins has the test report paths hardcoded. Let me take a look at the config this AM!

heuermh · 2016-08-31T18:10:07Z

Looks reasonable to me.

With scala.version.prefix will need to confirm the move_to... scripts still work correctly.

fnothaft · 2016-08-31T18:49:41Z

Jenkins, test this please.

fnothaft · 2016-08-31T18:51:43Z

src/test/resources/.gitignore

@@ -0,0 +1,3 @@
+# We blanket-ignore .bam files in the top-level of the repo, but want to undo that here.
+!*.bam


Nifty! Didn't know you could do that.

Yea this caused me some problems when I tried to git mv adam-core/* ./ to put all of adam-core in the top directory (after git rming the other modules); sorted.bam* did not come with that move since they were .gitignored (despite having been git added to the repo at some point, maybe before they became .gitignored?), so I was just missing them in my builds, which took me a while to figure out.

In any case, yea .gitignore "whitelists" can be useful in addition to the way it's usually used.

Yeah originally we had a prohibition on adding binary files, but we've relaxed that over time. Thanks for patching this!

fnothaft · 2016-08-31T18:55:08Z

This 99% LGTM! The only nit that I have is that Scala version string changes should be OK but you need to change the scala versions at https://github.com/bigdatagenomics/adam/blob/master/scripts/move_to_scala_2.11.sh#L5 and https://github.com/bigdatagenomics/adam/blob/master/scripts/move_to_scala_2.10.sh#L5. Right now, the Scala 2.11 builds are failing, but I think fixing the two files above will resolve that. Thanks for going through and cleaning these bits up, @ryan-williams!

fnothaft · 2016-08-31T18:56:09Z

Also, I have done the surefire_reports --> scalatest_reports path change in Jenkins, so that should be good to go.

AmplabJenkins · 2016-08-31T19:03:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1441/

Build result: FAILURE

GitHub pull request #1142 of commit 3e02e9b automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1142/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 3722878646d30b56dcaa16c9624bc0467db0486c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1142/merge^{commit} # timeout=10Checking out Revision 3722878646d30b56dcaa16c9624bc0467db0486c (origin/pr/1142/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 3722878646d30b56dcaa16c9624bc0467db0486cFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

ryan-williams · 2016-08-31T20:13:01Z

OK, just pushed an attempt at updating the move_to scripts to use 2.11.8 and 2.10.6

AmplabJenkins · 2016-08-31T20:13:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1442/

Build result: FAILURE

GitHub pull request #1142 of commit 372b573 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1142/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains ee02423 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1142/merge^{commit} # timeout=10Checking out Revision ee02423 (origin/pr/1142/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f ee024234052b2bd21ebcbadd15ff96785320fdccFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2016-08-31T20:50:14Z

pom.xml

@@ -268,7 +268,8 @@
          </execution>
        </executions>
        <configuration>
-          <scalaVersion>2.10.4</scalaVersion>
+          <scalaVersion>${scala.version}</scalaVersion>
+          <recompileMode>incremental</recompileMode>


We'd removed incremental in the past. I'd need to dig through the commit history to find the exact reason, but it has something to do with causing transient build issues. The Jenkins build just failed with an issue related to such. Alternatively, we could do -DrecompileMode=all in scripts/jenkins-test as suggested by davidB/scala-maven-plugin#185. I'm open to either. It'd be nice to have faster Scala compile times in the general case.

Interesting; so incremental compilation in Scala 2.11 either doesn't work correctly at all, or only works in SBT?

Maybe a better option would be to enable a different Maven profile in the presence of a jenkins environment variable?

That way we would get incremental compiles locally, which would be nice, unless it's so broken we don't think we want that.

ryan-williams · 2016-09-01T20:54:14Z

Just following up here, the last build failed on 2.6.0 x 1.5.2 x 2.10 with:

[error] Required file not found: sbt-interface.jar
[error] See zinc -help for information about locating necessary files

and 2.6.0 x 1.5.2 x 2.11 with:

[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.bdgenomics.adam:adam-core_2.11:0.19.1-SNAPSHOT (/home/jenkins/workspace/ADAM-prb/HADOOP_VERSION/2.6.0/SCALAVER/2.11/SPARK_VERSION/1.5.2/label/centos/adam-core/pom.xml) has 1 error
[ERROR]     'dependencies.dependency.version' for org.apache.parquet:parquet-scala_2.10:jar is missing. @ line 155, column 17
[ERROR]

I don't have obvious leads about either one.

heuermh · 2016-09-01T22:37:40Z

The parquet-scala_2.10 dependency is strange as there is no 2.11 version.
http://search.maven.org/#search%7Cga%7C1%7Cparquet-scala

The move_to_scala_2.11.sh script has a workaround, maybe something went wrong
https://github.com/bigdatagenomics/adam/blob/master/scripts/move_to_scala_2.11.sh#L6

AmplabJenkins · 2016-10-04T18:57:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1517/

Build result: FAILURE

GitHub pull request #1142 of commit ef8e35f automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1142/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 711b45729873bba2a021a1feaf618a316d783327 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1142/merge^{commit} # timeout=10Checking out Revision 711b45729873bba2a021a1feaf618a316d783327 (origin/pr/1142/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 711b45729873bba2a021a1feaf618a316d783327First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2016-10-04T19:39:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1518/

Build result: FAILURE

GitHub pull request #1142 of commit 073cf9f automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1142/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 5b75185a8a9b69552fced6383e13828bafe7bc07 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1142/merge^{commit} # timeout=10Checking out Revision 5b75185a8a9b69552fced6383e13828bafe7bc07 (origin/pr/1142/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 5b75185a8a9b69552fced6383e13828bafe7bc07First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

ryan-williams · 2016-10-04T19:45:54Z

I think the missing sbt-interface.jar has to do with zinc not being installed or configured properly on the Jenkins build machine. Not sure what to do about it, maybe I'll try to just turn off incremental-compilation, overall or just for Jenkins.

AmplabJenkins · 2016-10-04T20:15:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1519/

Build result: FAILURE

[...truncated 32 lines...]Triggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

ryan-williams · 2016-10-04T20:19:38Z

All the Spark 2.0.0 builds failed with:

mvn clean -Dhadoop.version=2.3.0 -Dspark.version=2.0.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1g; support was removed in 8.0
[INFO] Scanning for projects...
[ERROR] The build could not read 4 projects -> [Help 1]
[ERROR]   
[ERROR]   The project org.bdgenomics.adam:adam-core-spark2_2.10:0.19.1-SNAPSHOT (/home/jenkins/workspace/ADAM-prb/HADOOP_VERSION/2.3.0/SCALAVER/2.10/SPARK_VERSION/2.0.0/label/centos/adam-core/pom.xml) has 5 errors
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-misc-spark2_2.10:test-jar is missing. @ line 93, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-metrics-spark2_2.10:jar is missing. @ line 99, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-io-spark2_2.10:jar is missing. @ line 103, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-cli-spark2_2.10:jar is missing. @ line 107, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-intervalrdd-spark2_2.10:jar is missing. @ line 111, column 17
[ERROR]   
[ERROR]   The project org.bdgenomics.adam:adam-apis-spark2_2.10:0.19.1-SNAPSHOT (/home/jenkins/workspace/ADAM-prb/HADOOP_VERSION/2.3.0/SCALAVER/2.10/SPARK_VERSION/2.0.0/label/centos/adam-apis/pom.xml) has 3 errors
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-misc-spark2_2.10:test-jar is missing. @ line 105, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.adam:adam-core-spark2_2.10:jar is missing. @ line 115, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.adam:adam-core-spark2_2.10:test-jar is missing. @ line 119, column 17
[ERROR]   
[ERROR]   The project org.bdgenomics.adam:adam-cli-spark2_2.10:0.19.1-SNAPSHOT (/home/jenkins/workspace/ADAM-prb/HADOOP_VERSION/2.3.0/SCALAVER/2.10/SPARK_VERSION/2.0.0/label/centos/adam-cli/pom.xml) has 9 errors
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-misc-spark2_2.10:test-jar is missing. @ line 116, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-misc-spark2_2.10:jar is missing. @ line 122, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-io-spark2_2.10:jar is missing. @ line 126, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-cli-spark2_2.10:jar is missing. @ line 130, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.utils:utils-metrics-spark2_2.10:jar is missing. @ line 134, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.adam:adam-core-spark2_2.10:jar is missing. @ line 142, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.adam:adam-core-spark2_2.10:test-jar is missing. @ line 146, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.adam:adam-apis-spark2_2.10:jar is missing. @ line 152, column 17
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.adam:adam-apis-spark2_2.10:test-jar is missing. @ line 156, column 17
[ERROR]   
[ERROR]   The project org.bdgenomics.adam:adam-assembly-spark2_2.10:0.19.1-SNAPSHOT (/home/jenkins/workspace/ADAM-prb/HADOOP_VERSION/2.3.0/SCALAVER/2.10/SPARK_VERSION/2.0.0/label/centos/adam-assembly/pom.xml) has 1 error
[ERROR]     'dependencies.dependency.version' for org.bdgenomics.adam:adam-cli-spark2_2.10:jar is missing. @ line 115, column 17

Any ideas?

heuermh · 2016-10-04T20:35:02Z

The version for those dependencies should be 0.2.8, which is on Maven Central. Are those variable substitutions coming from a profile? We had trouble with that before.

heuermh · 2016-10-04T19:28:44Z

pom.xml

@@ -476,7 +476,8 @@
      </dependency>
      <dependency>
        <groupId>org.apache.parquet</groupId>
-        <artifactId>parquet-scala_${scala.version.prefix}</artifactId>
+        <!-- This library has no Scala 2.10 version, but using the 2.10 version seems to work. -->


this is correct

heuermh · 2016-10-04T20:39:03Z

pom.xml

@@ -282,9 +282,9 @@
          </jvmArgs>
          <javacArgs>
            <javacArg>-source</javacArg>
-            <javacArg>${java.version}</javacArg>
+            <javacArg>1.7</javacArg>


we require jdk8 because of the transitive dependency on htsjdk through Hadoop-BAM

+1, needs to be changed. Although, is there a reason to move away from the ${java.version} property here, if we're generally moving back towards using the parameters throughout the POM? I would prefer going back to ${java.version} on this line.

Sorry, I got cornered into doing this for a reason that I didn't have paged in and meant to go back and try to reconstruct what that was before responding.

I haven't quite done that yet, but I'm pretty sure it was related to my making the build use zinc; for some reason 1.8 was not acceptable in that setting, and I had to hard-code 1.7 here. I'll try to dig back and get the full state on it for posterity, but in the meantime I was going to give up on zinc in this PR anyway, since I didn't know how to get around the "sbt-interface.jar not found" problem on Jenkins discussed elsewhere on this PR (I believe due to zinc not being installed on Jenkins in some way), so I'll revert this bit momentarily. Thanks!

heuermh · 2016-10-04T20:41:01Z

pom.xml

-    <!-- informative only, used in about template substitution -->
-    <scala.version>2.10</scala.version>
+    <scala.version>2.10.6</scala.version>
+    <scala.version.prefix>2.10</scala.version.prefix>


changing this also requires a change to https://github.com/bigdatagenomics/adam/blob/master/adam-cli/src/main/java-templates/org/bdgenomics/adam/cli/About.java#L28

Interesting; what is the contract with that file? Every Maven property must be declared there? Why?

That file is populated by the templating-maven-plugin and used to generate the cli --version message

$ ./bin/adam-submit --version Using ADAM_MAIN=org.bdgenomics.adam.cli.ADAMMain Using SPARK_SUBMIT=/usr/local/bin/spark-submit e 888~-_ e e e d8b 888 \ d8b d8b d8b /Y88b 888 | /Y88b d888bdY88b / Y88b 888 | / Y88b / Y88Y Y888b /____Y88b 888 / /____Y88b / YY Y888b / Y88b 888_-~ / Y88b / Y888b ADAM version: 0.19.1-SNAPSHOT Commit: ca4d9649bac2523f9951c24f2e184a1b37f9ba06 Build: 2016-10-04 Built for: Scala 2.10 and Hadoop 2.6.0

OK, I looked around this a bit but don't see what change I need to make; help?

The templating-maven-plugin docs say it's useful "if you need for example to have things in that code replaced with some properties values."

${scala.version} is already exposed there (though not used anywhere in the project afaict) and I have no use for exposing ${scala.version.prefix} there, if that's what you mean?

All I meant was that before ${scala.version} was used to populate that field on About.java, so that it could be displayed in the --version string, thus the comment at the previous line 22.

When you used ${scala.version} elsewhere defined as 2.10.6, then the field in About.java should have been updated to use ${scala.version.prefix} instead.

But as mentioned below, I don't think we'll be able to use these properties for substitution in artifact names, so I guess the point is moot.

I think it is OK (preferable, really) to have About print the full Scala version, as opposed to the prefix. I don't think we need to make a change to About.

Ah that makes sense @heuermh, thanks for the explanation. Sounds like in the end we can just leave as is though, lmk if otherwise.

fine with me

heuermh · 2016-10-04T20:42:27Z

pom.xml

      </dependency>
      <dependency>
        <groupId>org.bdgenomics.adam</groupId>
-        <artifactId>adam-core_2.10</artifactId>
+        <artifactId>adam-core_${scala.version.prefix}</artifactId>


I'm fairly certain there's a reason why we can't do this, I know because I tried to do it once. Of course, I don't remember why it didn't work . . .

bigdatagenomics/utils#69

It's OK to use the properties in the artifactId for dependencies, it doesn't work to use properties in the artifactId for the POM itself. I know we've gone back and forth on this a lot, but I think it's actually preferable to do what @ryan-williams is doing here. I'll make a comment at the end with my reasoning.

ryan-williams · 2016-10-04T20:49:47Z

The version for those dependencies should be 0.2.8, which is on Maven Central. Are those variable substitutions coming from a profile? We had trouble with that before.

Those errors are modules' <dependency>s not declaring <version>s (because they should pick them up from the parent POM), but something about that resolution breaking down and Maven thinking they are just missing a <version> specification.

fnothaft

Need the Java version changes in POM and scripts/jenkins-test, otherwise LGTM!

fnothaft · 2016-10-05T15:27:05Z

pom.xml

      </dependency>
      <dependency>
        <groupId>org.bdgenomics.adam</groupId>
-        <artifactId>adam-core_2.10</artifactId>
+        <artifactId>adam-core_${scala.version.prefix}</artifactId>


It's OK to use the properties in the artifactId for dependencies, it doesn't work to use properties in the artifactId for the POM itself. I know we've gone back and forth on this a lot, but I think it's actually preferable to do what @ryan-williams is doing here. I'll make a comment at the end with my reasoning.

fnothaft · 2016-10-05T15:28:02Z

pom.xml

@@ -282,9 +282,9 @@
          </jvmArgs>
          <javacArgs>
            <javacArg>-source</javacArg>
-            <javacArg>${java.version}</javacArg>
+            <javacArg>1.7</javacArg>


+1, needs to be changed. Although, is there a reason to move away from the ${java.version} property here, if we're generally moving back towards using the parameters throughout the POM? I would prefer going back to ${java.version} on this line.

fnothaft · 2016-10-05T15:31:19Z

scripts/jenkins-test

    -e "s:sun.io.serialization.extendedDebugInfo=true:sun.io.serialization.extendedDebugInfo=true -Djava.io.tmpdir=${ADAM_MVN_TMP_DIR}:g" \
    {} \;
 find . -name "*.bak" -exec rm {} \;

 # variable declarations
-export JAVA_HOME=/usr/java/jdk1.8.0_60
+if [ -z "$JAVA_HOME" ]; then


This is a bit ugly, but we always need to overwrite $JAVA_HOME here due to Jenkins doing "shenanigans" with the Java version. The short story is that Jenkins will always set $JAVA_HOME to a bad value, and we are powerless to stop it. There's a much longer and uninteresting backstory. Can you revert this back, or add some other logic to disable overwriting $JAVA_HOME?

Actually, I was too quick on the trigger here. We can more cleanly handle setting the JAVA_HOME value in the Jenkins command that wraps the jenkins-test invocation. Perhaps it'd be cleanest to remove this logic?

Cool! I'll remove it here then altogether, 1m.

and update a few call-sites

ryan-williams · 2016-10-05T16:19:08Z

OK, I reverted the java.version bit and removed the jenkins-test JAVA_HOME-handling; I don't know why the build would avoid the failure above this time around, so am sort of expecting that to happen again, and suspect that it is related to the scala.version.prefix usage throughout the POMs, so let's see what happens there.

Also I'm not sure about Github's code-review flow, but it is telling me I need to make changes to the java.version thing even though I already did (and force-pushed over the old version):

We'll see if that resolves itself given some time? Or maybe force-pushing just breaks everything.

AmplabJenkins · 2016-10-05T16:36:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1521/

Build result: FAILURE

[...truncated 32 lines...]Triggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2016-10-06T00:05:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1522/
Test PASSed.

ryan-williams · 2016-10-06T00:19:46Z

Cool, so it turns out I just hadn't updated the sed regexs in move_to_spark2 to properly handle lines that now have _${scala.version.prefix} instead of _2.10.

I am not sure what to do about this "fnothaft requested changes" error I'm still seeing.

fnothaft · 2016-10-07T06:11:44Z

LGTM! Unless anyone has objections, I will squash this down and merge in the AM.

fnothaft · 2016-10-07T23:16:14Z

Thanks @ryan-williams! I've merged this manually as ad50f7d, e8b0fcd, cd488df, d242a26, a0730f3, and 89942fe. As always, thank you for taking the time to make a nice cleanup pass! It is greatly appreciated.

ryan-williams mentioned this pull request Aug 31, 2016

JFYI: tiny forked adam-core "0.20.0" release #1139

Closed

fnothaft reviewed Aug 31, 2016
View reviewed changes

ryan-williams mentioned this pull request Sep 1, 2016

Upgrade to Scala 2.11 hammerlab/guacamole#438

Open

ryan-williams added 3 commits October 4, 2016 01:08

un-ignore bam/bai in test resources

cc1c623

eof whitespace

547919a

rm unused test resource

31a32d9

ryan-williams force-pushed the misc branch from 372b573 to ef8e35f Compare October 4, 2016 18:43

heuermh reviewed Oct 4, 2016

View reviewed changes

fnothaft requested changes Oct 5, 2016

View reviewed changes

ryan-williams added 4 commits October 5, 2016 16:14

ADAMFunSuite.resourcePath → SparkFunSuite.testFile

7bd49fc

and update a few call-sites

bump scala versions

9db1502

bump resources-plugin version

d6fb341

rename "surefire" (scalatest) reports directory

1679f19

ryan-williams force-pushed the misc branch from f85a815 to 36cd1af Compare October 5, 2016 16:15

ryan-williams added 4 commits October 5, 2016 23:36

make scala.version.prefix a pom property

924e2f9

rm no-op @transient annotations

19bc09e

fix sed "-i" args

af09acb

don't hard-code JAVA_HOME for jenkins

9ae03fe

ryan-williams force-pushed the misc branch from 36cd1af to 9ae03fe Compare October 5, 2016 23:36

fnothaft approved these changes Oct 7, 2016

View reviewed changes

heuermh approved these changes Oct 7, 2016

View reviewed changes

fnothaft closed this Oct 7, 2016

ryan-williams mentioned this pull request Oct 8, 2016

master won't compile #1200

Closed

		@@ -0,0 +1,3 @@
		# We blanket-ignore .bam files in the top-level of the repo, but want to undo that here.
		!*.bam

misc pom/test/resource improvements #1142

misc pom/test/resource improvements #1142

Conversation

ryan-williams commented Aug 31, 2016

AmplabJenkins commented Aug 31, 2016

Build result: FAILURE

ryan-williams commented Aug 31, 2016

ryan-williams commented Aug 31, 2016

fnothaft commented Aug 31, 2016

heuermh commented Aug 31, 2016

fnothaft commented Aug 31, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fnothaft commented Aug 31, 2016

fnothaft commented Aug 31, 2016

AmplabJenkins commented Aug 31, 2016

Build result: FAILURE

ryan-williams commented Aug 31, 2016

AmplabJenkins commented Aug 31, 2016

Build result: FAILURE

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryan-williams commented Sep 1, 2016

heuermh commented Sep 1, 2016

AmplabJenkins commented Oct 4, 2016

Build result: FAILURE

AmplabJenkins commented Oct 4, 2016

Build result: FAILURE

ryan-williams commented Oct 4, 2016

AmplabJenkins commented Oct 4, 2016

Build result: FAILURE

ryan-williams commented Oct 4, 2016

heuermh commented Oct 4, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryan-williams Oct 5, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heuermh Oct 5, 2016 • edited Loading

Choose a reason for hiding this comment

fnothaft Oct 5, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryan-williams commented Oct 4, 2016

fnothaft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryan-williams commented Oct 5, 2016 • edited Loading

AmplabJenkins commented Oct 5, 2016

Build result: FAILURE

AmplabJenkins commented Oct 6, 2016

ryan-williams commented Oct 6, 2016

fnothaft commented Oct 7, 2016

fnothaft commented Oct 7, 2016

ryan-williams Oct 5, 2016 •

edited

Loading

heuermh Oct 5, 2016 •

edited

Loading

fnothaft Oct 5, 2016 •

edited

Loading

ryan-williams commented Oct 5, 2016 •

edited

Loading