Upgrade to Spark 0.9 and Scala 2.10 #78
Conversation
| @@ -55,8 +55,6 @@ class TransformArgs extends Args4jBase with ParquetArgs with SparkArgs { | |||
| class Transform(protected val args: TransformArgs) extends AdamSparkCommand[TransformArgs] with Logging { | |||
| val companion = Transform | |||
|
|
|||
| initLogging() | |||
There was a problem hiding this comment.
Is initLogging no longer needed in Spark 0.9.0?
There was a problem hiding this comment.
I don't think it was required in earlier versions either, but now initLogging() doesn't exist.
|
+1, looks good! |
|
Looks like Jenkins lost this build. Also, did we want to jump to a .0 release from Spark? I know there was some discussion with @massie about waiting until 0.9.1 to upgrade. |
|
Jenkins, test this please. |
|
Actually, the issue is likely that the Jenkins workers don't have Scala 2.10 available. This raises a question about testing. We need to make sure that we're targeting the correct version of Scala. We can't, for example, just upgrade across the board because the old branches for Spark and friends needs 2.9. Agree with @carlyeks that we probably want to move slowly on this; although, it's good to start setting up the test environment to support it. By the time it's polished, the 0.9.1 release will probably be on it's way. :) |
|
All automated tests passed. |
|
It appears that Jenkins [job 75](https://amplab.cs.berkeley.edu/jenkins/view/Pull Request Builders/job/ADAM-prb/75/console) got stuck for 3 days while testing PR #69. I canceled the job and requested a retest of #69, and have configured the ADAM PRB to report a build failure after 3 hours. |
|
Jenkins, test this please. |
|
One or more automated tests failed |
|
One or more automated tests failed |
|
I'm running into the following error when trying to merge this to cut a new RC: [ERROR] /Users/fnothaft/adam-new/adam/adam-core/src/test/scala/edu/berkeley/cs/amplab/adam/util/NormalizationUtilsSuite.scala:69: value samtoolsCigar is not a member of edu.berkeley.cs.amplab.adam.avro.ADAMRecord @jey / @massie Would you know why the implicit conversion is not being picked up? |
|
Not sure, I encountered the same problem. Explicitly taking an argument of RichADAMRecord fixed it. Is maybe an ambiguous conversion? |
|
That's what I've been doing to fix it. I believe that the conversion is unambiguous; it converts properly somewhere else in the class. This may have something to do with it being inside an assert? |
|
This is ready to be merged |
|
All automated tests passed. |
|
All automated tests passed. |
Upgrade to Spark 0.9 and Scala 2.10
|
Thanks, Jey! |
Upgrade to Spark 0.9.0 and Scala 2.10.3. Tests pass under Hadoop 2.2.0 and Hadoop 1.0.4.
NOTE: You will probably also want to upgrade your Scala installation to 2.10.3 to match.