Clean up CLI operation categories and names, and add documentation for CLI #1274

fnothaft · 2016-11-16T19:28:58Z

Builds on #1262.

Removed CalculateDepth command.
Clean up argument doc in TransformFeatures.
Move OUTPUT to an option for Flagstat.
Remove some output saving arguments in the View command that don't exactly make sense.
Remove unused Vcf2ADAM/ADAM2Vcf -dict option and org.bdgenomics.adam.cli.DictionaryCommand trait.
Remove WigFix2ADAM command.
Move away from ParquetLoadSaveArgs.
Move reads2coverage to actions.
Reads2Coverage does not only write to Parquet, so clean up docs.
Clean up fragments<->reads argument names.

AmplabJenkins · 2016-11-16T19:38:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1614/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1274/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains e0e024bb2b997957288d57ae1908e6b9339253b6 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1274/merge^{commit} # timeout=10Checking out Revision e0e024bb2b997957288d57ae1908e6b9339253b6 (origin/pr/1274/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f e0e024bb2b997957288d57ae1908e6b9339253b6First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2016-11-16T19:48:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1615/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1274/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 8103cd4cab0ef23be24b0e4d90989f4400e457ba # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1274/merge^{commit} # timeout=10Checking out Revision 8103cd4cab0ef23be24b0e4d90989f4400e457ba (origin/pr/1274/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 8103cd4cab0ef23be24b0e4d90989f4400e457baFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2016-11-16T20:20:48Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1616/
Test PASSed.

AmplabJenkins · 2016-11-17T00:49:38Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1618/
Test PASSed.

heuermh · 2016-11-17T01:01:59Z

docs/source/50_cli.md

+
+#### Legacy output options {#legacy-output}
+
+Several tools in ADAM support saving back to legacy genomics output formats. Any


legacy genomics output formats, scandalous! :)

heuermh · 2016-11-17T01:07:48Z

docs/source/50_cli.md

+required arguments as `vcf2adam`. `anno2adam` takes the [default
+options](#default-args), and one additional option:
+
+* `-current_db`: A path to an existing Parquet file of VariantAnnotations. These


-current_db is a poorly named argument.

heuermh · 2016-11-17T01:10:14Z

docs/source/50_cli.md

+Dumps a Parquet file to either the console or a text file as
+[JSON](http://www.json.org). Takes one required argument:
+
+1. `FILE(S)`: The file paths to load. This must be a Parquet formatted file.


→ These must be a Parquet formatted files.

…r CLI. * Removed CalculateDepth command. * Clean up argument doc in TransformFeatures. * Move OUTPUT to an option for Flagstat. * Remove some output saving arguments in the View command that don't exactly make sense. * Remove unused Vcf2ADAM/ADAM2Vcf -dict option and `org.bdgenomics.adam.cli.DictionaryCommand` trait. * Remove WigFix2ADAM command. * Move away from ParquetLoadSaveArgs. * Move reads2coverage to actions. * Reads2Coverage does not only write to Parquet, so clean up docs. * Clean up fragments<->reads argument names.

fnothaft · 2016-11-17T04:51:36Z

Addressed review comments.

AmplabJenkins · 2016-11-17T05:16:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1619/
Test PASSed.

heuermh · 2016-11-17T05:27:55Z

Thank you, @fnothaft!

laserson · 2016-12-20T19:09:57Z

Shucks, missed this PR. I use the wigfix2bed CLI command in the Spark book. Out of scope for ADAM?

laserson · 2016-12-20T19:20:17Z

I'll use BEDOPS, which seems to have a pre-written tool for this

heuermh · 2016-12-21T01:32:39Z

@laserson We removed the wigfix2bed because it felt too small for a top-level command. We've also revamped feature support since, so there is no reason to limit export to BED format.

If wigfix is something useful to have, perhaps we should support it as a feature format, and then conversions to/from will be supported by the transformFeatures command. Create an issue if you feel so.

fnothaft · 2016-12-21T02:54:31Z

+1 @heuermh

laserson · 2016-12-21T03:29:20Z

Nah, I've determined the path of maximum laziness is to just use another tool rather than revive the tool in ADAM 😛

fnothaft · 2016-12-21T03:29:58Z

SGTM!

fnothaft mentioned this pull request Nov 16, 2016

Add docs for transform pipeline, ADAM-on-Toil #1262

Merged

fnothaft force-pushed the doc-cli branch from 3f15852 to 403aab8 Compare November 16, 2016 19:41

fnothaft force-pushed the doc-cli branch from 403aab8 to fc1bdab Compare November 16, 2016 19:51

fnothaft force-pushed the doc-cli branch from fc1bdab to 232f141 Compare November 16, 2016 23:40

heuermh reviewed Nov 17, 2016

View reviewed changes

fnothaft force-pushed the doc-cli branch from 232f141 to 875954b Compare November 17, 2016 04:49

heuermh merged commit 8fef9a7 into bigdatagenomics:master Nov 17, 2016

fnothaft deleted the doc-cli branch December 21, 2016 03:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up CLI operation categories and names, and add documentation for CLI #1274

Clean up CLI operation categories and names, and add documentation for CLI #1274

fnothaft commented Nov 16, 2016

AmplabJenkins commented Nov 16, 2016

AmplabJenkins commented Nov 16, 2016

AmplabJenkins commented Nov 16, 2016

AmplabJenkins commented Nov 17, 2016

heuermh Nov 17, 2016

heuermh Nov 17, 2016

heuermh Nov 17, 2016

fnothaft commented Nov 17, 2016

AmplabJenkins commented Nov 17, 2016

heuermh commented Nov 17, 2016

laserson commented Dec 20, 2016

laserson commented Dec 20, 2016

heuermh commented Dec 21, 2016

fnothaft commented Dec 21, 2016

laserson commented Dec 21, 2016

fnothaft commented Dec 21, 2016


		#### Legacy output options {#legacy-output}

		Several tools in ADAM support saving back to legacy genomics output formats. Any

Clean up CLI operation categories and names, and add documentation for CLI #1274

Clean up CLI operation categories and names, and add documentation for CLI #1274

Conversation

fnothaft commented Nov 16, 2016

AmplabJenkins commented Nov 16, 2016

Build result: FAILURE

AmplabJenkins commented Nov 16, 2016

Build result: FAILURE

AmplabJenkins commented Nov 16, 2016

AmplabJenkins commented Nov 17, 2016

heuermh Nov 17, 2016

Choose a reason for hiding this comment

heuermh Nov 17, 2016

Choose a reason for hiding this comment

heuermh Nov 17, 2016

Choose a reason for hiding this comment

fnothaft commented Nov 17, 2016

AmplabJenkins commented Nov 17, 2016

heuermh commented Nov 17, 2016

laserson commented Dec 20, 2016

laserson commented Dec 20, 2016

heuermh commented Dec 21, 2016

fnothaft commented Dec 21, 2016

laserson commented Dec 21, 2016

fnothaft commented Dec 21, 2016