[ADAM-132] allowing list of commands to be injected into adam-cli ADAMMain #762

heuermh · 2015-08-07T23:15:42Z

Work in progress towards #132.

Please be brutal in code review; I'm not very confident writing in scala or using scalatest yet.

heuermh · 2015-08-10T21:29:31Z

When working on https://github.com/heuermh/adam-commands, downstream to this pull request, I ran into a build problem

$ mvn package
...
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ adam-commands_2.10 ---
[WARNING] Invalid POM for org.bdgenomics.adam:adam-cli_2.10:jar:0.17.1-SNAPSHOT, transitive dependencies (if any) will not be available, enable debug logging for more details
...
[ERROR] error while loading ADAMMain, Missing dependency 'bad symbolic reference. A
signature in ADAMMain.class refers to term inject in package javax which is not
available. It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling ADAMMain.class.',
required by /Users/xxx/.m2/repository/org/bdgenomics/adam/adam-cli_2.10/0.17.1-SNAPSHOT/adam-cli_2.10-0.17.1-SNAPSHOT.jar(org/bdgenomics/adam/cli/ADAMMain.class)
[ERROR] one error found

massie · 2015-08-11T17:43:45Z

@heuermh Have you figured out the issue with downstream projects yet?

heuermh · 2015-08-11T18:44:59Z

@massie Haven't been able to reproduce today, works fine for me with both JDKs 7 and 8. Must have been some maven snapshot strangeness.

heuermh · 2015-08-11T19:00:38Z

The real question with this pull request is how to actually run the main in the repo above.

Duplicate most of the setup in adam-submit in order to call spark-submit directly? This is how rice and avocado do it.
Accept --main as a parameter in adam-submit? This approach plus the --jars argument is described in adam-commands.
Implement a new ADAMMain that can be extended or delegated to that in turn delegates to SparkLauncher (Investigate SparkLauncher API as a replacement for adam-submit+spark-submit #756)?

Hopefully there's enough there to demonstrate the use case.

heuermh · 2015-08-19T18:36:20Z

This can wait until post-0.17.1 release

AmplabJenkins · 2015-08-24T15:50:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/881/

Build result: FAILURE

[...truncated 16 lines...] > git rev-list 81c1f0d # timeout=10Triggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.2.1,centosTriggering ADAM-prb ? 1.0.4,2.11,1.2.1,centosTriggering ADAM-prb ? 1.0.4,2.10,1.2.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.2.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.3.1,centosADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.2.1,centos completed with result FAILUREADAM-prb ? 1.0.4,2.11,1.2.1,centos completed with result SUCCESSADAM-prb ? 1.0.4,2.10,1.2.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.2.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2015-08-24T17:24:24Z

Jenkins, retest this please

AmplabJenkins · 2015-08-24T17:44:11Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/884/
Test PASSed.

massie · 2015-09-17T17:14:30Z

adam-cli/src/test/scala/org/bdgenomics/adam/cli/ADAMMainSuite.scala

+    Console.withOut(stream) {
+      ADAMMain.main(Array())
+    }
+    assert(stream.toString().contains("flatten"))


Calling the ADAMMain with an empty Array() will cause the help page to output, correct? If so, we should put a comment that effect and change the assert to look for the word "help" or some such.

massie · 2015-09-17T17:22:18Z

With a little cleanup, this looks ready to go. What do you think @heuermh ?

heuermh · 2015-09-17T17:23:23Z

Yep, thank you for the review, @massie

heuermh · 2015-09-17T18:07:39Z

Note even with code fixes as described above, this still won't be usable until this or one of the other options above is addressed

Accept --main as a parameter in adam-submit? This approach plus the --jars argument is described in adam-commands.

…MMain

AmplabJenkins · 2015-09-17T18:29:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/914/

Build result: FAILURE

GitHub pull request #762 of commit e8765b8 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/762/merge^{commit} # timeout=10 > git branch -a --contains 2df3bb2 # timeout=10 > git rev-parse remotes/origin/pr/762/merge^{commit} # timeout=10Checking out Revision 2df3bb2 (origin/pr/762/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 2df3bb2 > git rev-list a9c8d4d8f9e77436900a3080f7e1fbe6a0b53fd0 # timeout=10Triggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

massie · 2015-09-17T18:32:17Z

Jenkins, test this please.

AmplabJenkins · 2015-09-17T18:50:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/915/
Test PASSed.

fnothaft · 2015-10-06T00:28:32Z

@heuermh I want to test this on the cluster. Do you have an example project showing how I could inject a new command?

heuermh · 2015-10-06T03:14:27Z

@fnothaft The example I've been using is here https://github.com/heuermh/adam-commands.

Note the patch regarding the main class argument in the readme, which needs to be updated after the bin scripts have changed. It is soon on my list of things to go to create such a pull request.

See also previous comment #762 (comment)

fnothaft · 2015-10-07T16:18:35Z

Thanks @heuermh! I've verified that this works with #854 and your adam-commands example on the cluster. Merged as 58a4823.

massie reviewed Sep 17, 2015
View reviewed changes

[ADAM-132] allowing list of commands to be injected into adam-cli ADA…

e8765b8

…MMain

heuermh force-pushed the guice branch from 2ba41e5 to e8765b8 Compare September 17, 2015 18:26

fnothaft added this to the 0.18.0 milestone Oct 2, 2015

fnothaft closed this Oct 7, 2015

heuermh deleted the guice branch October 7, 2015 16:20

heuermh mentioned this pull request Oct 7, 2015

Add docs about building downstream apps on top of ADAM #291

Closed

heuermh mentioned this pull request Mar 24, 2016

Allow list of commands to be injected into adam-cli AdamMain #132

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ADAM-132] allowing list of commands to be injected into adam-cli ADAMMain #762

[ADAM-132] allowing list of commands to be injected into adam-cli ADAMMain #762

heuermh commented Aug 7, 2015

heuermh commented Aug 10, 2015

massie commented Aug 11, 2015

heuermh commented Aug 11, 2015

heuermh commented Aug 11, 2015

heuermh commented Aug 19, 2015

AmplabJenkins commented Aug 24, 2015

heuermh commented Aug 24, 2015

AmplabJenkins commented Aug 24, 2015

massie Sep 17, 2015

massie commented Sep 17, 2015

heuermh commented Sep 17, 2015

heuermh commented Sep 17, 2015

AmplabJenkins commented Sep 17, 2015

massie commented Sep 17, 2015

AmplabJenkins commented Sep 17, 2015

fnothaft commented Oct 6, 2015

heuermh commented Oct 6, 2015

fnothaft commented Oct 7, 2015

[ADAM-132] allowing list of commands to be injected into adam-cli ADAMMain #762

[ADAM-132] allowing list of commands to be injected into adam-cli ADAMMain #762

Conversation

heuermh commented Aug 7, 2015

heuermh commented Aug 10, 2015

massie commented Aug 11, 2015

heuermh commented Aug 11, 2015

heuermh commented Aug 11, 2015

heuermh commented Aug 19, 2015

AmplabJenkins commented Aug 24, 2015

Build result: FAILURE

heuermh commented Aug 24, 2015

AmplabJenkins commented Aug 24, 2015

massie Sep 17, 2015

Choose a reason for hiding this comment

massie commented Sep 17, 2015

heuermh commented Sep 17, 2015

heuermh commented Sep 17, 2015

AmplabJenkins commented Sep 17, 2015

Build result: FAILURE

massie commented Sep 17, 2015

AmplabJenkins commented Sep 17, 2015

fnothaft commented Oct 6, 2015

heuermh commented Oct 6, 2015

fnothaft commented Oct 7, 2015