[ADAM-752] Build for many combos of Spark/Hadoop versions. #765
Conversation
|
OK, so a few interesting things.
|
| @@ -26,16 +25,24 @@ export SPARK_DRIVER_MEMORY=8g | |||
|
|
|||
| pushd "$ADAM_TMP_DIR" | |||
|
|
|||
|
|
|||
| if [[ $HADOOP_VERSION == "1.0.4" ]]; then | |||
There was a problem hiding this comment.
Maybe echo $HADOOP_VERSION | egrep '^1.0.' would be more general, in case, we use e.g. 1.0.4 as the hadoop test version?
There was a problem hiding this comment.
or possibly [[ $HADOOP_VERSION =~ ^1\.0 ]], likewise below, if that seems easier
There was a problem hiding this comment.
What @ryan-williams said. :) Much cleaner.
|
That looks pretty neat. I'm guessing there's some config changes you are making in Jenkins that are not depicted here that e.g. populate |
| tar xzvf spark-1.1.0-bin-hadoop1.tgz | ||
| export SPARK_HOME="${ADAM_TMP_DIR}/spark-1.1.0-bin-hadoop1" | ||
| HADOOP=hadoop1 | ||
| elif [[ $HADOOP_VERSION == "2.6.0" ]]; then |
There was a problem hiding this comment.
Maybe echo $HADOOP_VERSION | egrep '^2.6.' would be more general, in case, we use e.g. 2.6.3 as the hadoop test version?
|
This is good stuff, Frank. |
|
Jenkins, retest this please. |
|
Fun times! Guess what doesn't work? If your guess was that running: Would lead to Destruction, Terror, and Mayhem, you win $5! At least, it does for me locally. I will sort this out later, but if anyone has seen this before or has any clues, I would love your thoughts. |
|
Destruction, Terror, and Mayhem and a 45 minute wait to see if your pull request turns green. :) Maybe a middle ground, or set up a quick build in Travis and leave Jenkins 3D Awesome for a nightly integration test? Would a profile work for where the |
fnothaft
force-pushed
the
update-jenkins-script
branch
from
a685df1
to
d687a4f
Compare
|
OK, so what seems to have been the problem is that we had a completely unused (???) dependency on |
|
And the Spark 1.4.1/Hadoop 2.6.0 touchstone builds have passed! Huzzah! Now, for the rest of the builds. |
fnothaft
force-pushed
the
update-jenkins-script
branch
from
d687a4f
to
ec7ee73
Compare
|
Rebased and added a commit to clean up log junk when running 1.4.1 unit tests. |
fnothaft
force-pushed
the
update-jenkins-script
branch
2 times, most recently
from
23b9e41
to
bbdc3ce
Compare
|
Cleaned up RE: the comments above around version checking. |
|
Jenkins, retest this please. |
| @@ -345,12 +345,36 @@ | |||
| <type>test-jar</type> | |||
| <scope>test</scope> | |||
| </dependency> | |||
| <!-- | |||
There was a problem hiding this comment.
any reason for this block?
There was a problem hiding this comment.
I was testing something; this can be removed.
…dependency version mismatch when running unit tests with Spark versions > 1.3.
fnothaft
force-pushed
the
update-jenkins-script
branch
from
bbdc3ce
to
6cade81
Compare
|
Jenkins, retest this please. |
|
IT WORKS! IT WORKS! IT REALLY DOES!!!!!!! |
|
Are all the red Failed - skipped in build 842 supposed to be Not run? Looks like they might be combinations that are not supported, such as Hadoop 1 with Spark 1.4.1. Or maybe I just need my 3D glasses? |
|
@heuermh correct; Spark 1.4.1/Hadoop 1.x is skipped as it won't build with the current way that Spark is packaged as maven artifacts (there was a long discussion of this on a previous ADAM pr). |
|
They show up as greyed out red because the last build of that combo failed. |
|
From what I can see it looks like a 15 minute build time up from 10 minutes, not too bad. +1 |
|
Nice addition, Frank! |
Resolves build comments on #752. I have set Jenkins up as a 3D!!!!!! matrix: