Remove adam-cli jar from classpath during adam-submit #749

fnothaft · 2015-07-31T01:35:49Z

The ./bin/compute-adam-jars.sh script does not remote the adam-cli JAR from the classpath. This exposes SPARK-1921 when running on YARN.

shaneknapp · 2015-07-31T16:34:12Z

bin/compute-adam-jars.sh

@@ -24,6 +24,6 @@ CLASSPATH=$("$ADAM_REPO"/bin/compute-adam-classpath.sh)

 # list of jars to ship with spark; trim off the first and last from the CLASSPATH
 # TODO: brittle? assumes appassembler always puts the $BASE/etc first and the CLI jar last
-ADAM_JARS=$(echo "$CLASSPATH" | tr ":" "," | cut -d "," -f 2-)
+ADAM_JARS=$(echo "$CLASSPATH" | tr ":" "\n" | tail -n +2 | head -n -1 | perl -pe 's/\n/,/ unless eof')


i can't make heads or tails of this statement! rimshot

Up top! makes high five gesture

ryan-williams · 2015-08-03T01:19:07Z

So this undoes the "fix" #669 of #663. Can you say more about how this exposes SPARK-1921 @fnothaft?

fnothaft · 2015-08-03T04:02:03Z

Ah! Good catch @ryan-williams. You all are running Spark-on-YARN on a CDH5 setup at Mt. Sinai, right? Perhaps the behavior is different between our clusters (which wouldn't be surprising, I think the bug is caused by a race condition), but we are in the middle of bringing up a CDH5 cluster at Berkeley and saw the exact behavior described in SPARK-1921 (JAR timestamps differ). By removing the adam-cli jar from the JARS list when calling spark-submit, you don't copy the JAR twice, and thus don't have the timestamp race. It should be easy to make this work for both adam-submit and adam-shell; I'll fix this up.

fnothaft · 2015-08-03T08:10:59Z

@ryan-williams just pushed a new commit to address your comment. Doing things correctly actually makes the adam-submit code a bit cleaner, for certain definitions of clean (specifically, we get rid of the brittle way of determining the CLI jar). Let me know what you think, and I'll squash down and rebase.

ryan-williams · 2015-08-03T14:52:43Z

bin/adam-submit


+ADAM_JARS=$(echo "$ADAM_JARS" | tail -n +1 | perl -pe 's/\n/,/ unless eof')


hm, tail -n +1 doesn't do anything, does it? do you mean tail -n +2 to skip the first line?

Er, fun times. You are correct. This should've been head -n -1 (working too late in the night...) but I swapped that because BSD head doesn't support the -n -1 flag. Let me find a good, portable fix and I'll ping back.

What if we have compute-adam-jars.sh take one cmdline flag, that tells it to strip the CLI jar or not, and then have it do the perl -pe 's/\n/,/ unless eof', since that's what both callers want?

Then compute-adam-jars.sh could do some more structured things to find and remove the CLI jar, and we'd expose a clean interface to that to the two caller scripts.

e.g. put a grep -v <some unique adam cli jar identifier> in compute-adam-jars.sh

Conceptually, I agree with what you are saying. In practice though, I actually think I like this approach a bit better, since it allows us to remove the fragile CLI jar finding code and fall back on the appassembler classpath instead.

Yea, I was missing that you had that grep method in this file previously; it's certainly nice to see it deleted here.

I think there are two orthogonal questions:

find the CLI jar via the assumption that appassembler puts it last on the classpath vs. by greping.

do the above in adam-submit vs. in compute-adam-jars.sh.

I'm ok with either approach to 1), but am somewhat interested in moving 2) to compute-adam-jars.sh, that way adam-submit and adam-shell both save a | perl -pe 's/\n/,/ unless eof', and we can create a cleaner interface to the | head -n -1 that is currently here in adam-submit.

Anyways, the way you have it wfm too, just articulating the options I see.

ryan-williams · 2015-08-03T18:41:04Z

this lgtm btw, unless I've persuaded you here to push CLI-jar detection and removal into compute-adam-jars.sh

massie · 2015-08-04T22:53:02Z

@fnothaft Do you plan to move cli-jar detection tin compute-adam-jars.sh or is this ready to be merged?

@ryan-williams If this is ready to be merged, feel free to hit the button and merge it.

fnothaft · 2015-08-04T22:53:36Z

This isn't ready to merge yet; I still need to make fixes.

fnothaft · 2015-08-05T01:59:40Z

@ryan-williams just pushed a new round of fixes. Let me know what you think of these.

ryan-williams · 2015-08-05T17:47:40Z

Builds failed with

hudson.plugins.git.GitException: Could not checkout null with start point 49a466a4d424b6acd1730c5daafe12f8e40d26b4

idk what that's about. Looking at the code now

fnothaft · 2015-08-05T17:48:35Z

Jenkins, retest this please.

@ryan-williams there's a bug in the Jenkins Git plugin where fetching force-pushes fails occasionally.

ryan-williams · 2015-08-05T17:50:54Z

bin/adam-submit

@@ -22,40 +22,15 @@ SCRIPT_DIR="$(cd `dirname $0`/..; pwd)"

 # Get list of required jars for ADAM
 ADAM_JARS=$("$SCRIPT_DIR"/bin/compute-adam-jars.sh)
+ADAM_CLI_JAR=$(echo "$ADAM_JARS" | rev | cut -d',' -f1 | rev )


huh, isn't $ADAM_JARS already one comma-delimited line at this point? What's the point of the two revs if so?

Maybe I am not so great at cut, but it isn't clear to me that there is an easy way to take just the last item from a comma separated list using cut. However, you can fake this by reversing the list, taking the first item, and reversing the list again.

Blindly copied from http://stackoverflow.com/questions/22727107/how-to-find-the-last-field-using-cut-linux

data=foo,bar,baz,qux last=${data##*,}

huh, I read man rev as reversing the order of lines in a file, not reversing each line's contents, my mistake @fnothaft!

@heuermh's trick above seems maybe a little cleaner, but I'm agnostic at this point and have held up this PR long enough so whatever you want to do sgtm @fnothaft!

fnothaft · 2015-08-07T01:50:10Z

Just pushed a last commit with Michael's suggested change, and with all things squashed down and rebased on master. This should be good to merge.

ryan-williams · 2015-08-07T16:02:58Z

lgtm

massie · 2015-08-07T16:35:31Z

+1

(Feel free to merge @ryan-williams)

heuermh · 2015-08-07T19:10:31Z

the latest diff looks like it adds back the $ADDL_JARS stuff -- was this intentional?
https://github.com/fnothaft/adam/blob/fix-compute-jars/bin/adam-submit#L54

fnothaft · 2015-08-07T19:11:33Z

@heuermh thanks for the catch; not intentional, rather this comes from a rebase gone wrong... I will fix.

fnothaft · 2015-08-07T19:12:45Z

Fixed. Sorry; the rebase on #754 was kind of nasty; lots of messy conflicts.

ryan-williams · 2015-08-07T19:42:15Z

want to retest this @fnothaft ? looks like git force-push problem again

ryan-williams · 2015-08-07T19:42:41Z

good catch @heuermh btw! I looked at it and just assumed it was intentional, woops

fnothaft · 2015-08-07T19:42:49Z

Jenkins, test this please.

fnothaft · 2015-08-07T21:11:45Z

This is passing now; can I get a merge? I will rebase #757 once this is merged.

Remove adam-cli jar from classpath during adam-submit

shaneknapp reviewed Jul 31, 2015
View reviewed changes

ryan-williams reviewed Aug 3, 2015
View reviewed changes

fnothaft force-pushed the fix-compute-jars branch 2 times, most recently from fb37213 to 5787a80 Compare August 5, 2015 01:59

ryan-williams reviewed Aug 5, 2015
View reviewed changes

fnothaft force-pushed the fix-compute-jars branch from 5787a80 to 38952f7 Compare August 7, 2015 01:49

Compute jars does not remote cli jar from classpath.

6a45b6b

fnothaft force-pushed the fix-compute-jars branch from 38952f7 to 6a45b6b Compare August 7, 2015 19:12

ryan-williams added a commit that referenced this pull request Aug 7, 2015

Merge pull request #749 from fnothaft/fix-compute-jars

1cf8ebb

Remove adam-cli jar from classpath during adam-submit

ryan-williams merged commit 1cf8ebb into bigdatagenomics:master Aug 7, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove adam-cli jar from classpath during adam-submit #749

Remove adam-cli jar from classpath during adam-submit #749

fnothaft commented Jul 31, 2015

shaneknapp Jul 31, 2015

fnothaft Jul 31, 2015

ryan-williams commented Aug 3, 2015

fnothaft commented Aug 3, 2015

fnothaft commented Aug 3, 2015

ryan-williams Aug 3, 2015

fnothaft Aug 3, 2015

ryan-williams Aug 3, 2015

ryan-williams Aug 3, 2015

fnothaft Aug 3, 2015

ryan-williams Aug 3, 2015

ryan-williams commented Aug 3, 2015

massie commented Aug 4, 2015

fnothaft commented Aug 4, 2015

fnothaft commented Aug 5, 2015

ryan-williams commented Aug 5, 2015

fnothaft commented Aug 5, 2015

ryan-williams Aug 5, 2015

fnothaft Aug 5, 2015

heuermh Aug 5, 2015

ryan-williams Aug 6, 2015

fnothaft commented Aug 7, 2015

ryan-williams commented Aug 7, 2015

massie commented Aug 7, 2015

heuermh commented Aug 7, 2015

fnothaft commented Aug 7, 2015

fnothaft commented Aug 7, 2015

ryan-williams commented Aug 7, 2015

ryan-williams commented Aug 7, 2015

fnothaft commented Aug 7, 2015

fnothaft commented Aug 7, 2015


		ADAM_JARS=$(echo "$ADAM_JARS" \| tail -n +1 \| perl -pe 's/\n/,/ unless eof')

Remove adam-cli jar from classpath during adam-submit #749

Remove adam-cli jar from classpath during adam-submit #749

Conversation

fnothaft commented Jul 31, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryan-williams commented Aug 3, 2015

fnothaft commented Aug 3, 2015

fnothaft commented Aug 3, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryan-williams commented Aug 3, 2015

massie commented Aug 4, 2015

fnothaft commented Aug 4, 2015

fnothaft commented Aug 5, 2015

ryan-williams commented Aug 5, 2015

fnothaft commented Aug 5, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fnothaft commented Aug 7, 2015

ryan-williams commented Aug 7, 2015

massie commented Aug 7, 2015

heuermh commented Aug 7, 2015

fnothaft commented Aug 7, 2015

fnothaft commented Aug 7, 2015

ryan-williams commented Aug 7, 2015

ryan-williams commented Aug 7, 2015

fnothaft commented Aug 7, 2015

fnothaft commented Aug 7, 2015