Support locus predicate in Transform #1540

fnothaft · 2017-05-18T23:43:51Z

Resolves #1047, #1539. H/T to @ryan-williams for the really useful genomic-loci library. One pending task:

Update Transform markdown docs.
Add NucleotideContigFragment

Code is ready for review.

coveralls · 2017-05-18T23:56:15Z

Coverage increased (+0.7%) to 82.709% when pulling a50c351 on fnothaft:issues/1047-1539-locus-predicate into fe1cba2 on bigdatagenomics:master.

AmplabJenkins · 2017-05-19T00:00:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2026/
Test PASSed.

heuermh · 2017-05-22T19:29:44Z

@ryan-williams Would adding this dependency complicate your builds too much? E.g. if ADAM depends on genomic-loci that may peg any projects of yours downstream of ADAM to a version you don't like.

Locus/loci as terms aren't used very consistently across our API and docs, might region predicate be better?

AmplabJenkins · 2017-05-24T06:41:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2048/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1540/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains a12689ff51f47565fec54ba02070e64e3970c07d # timeout=10Checking out Revision a12689ff51f47565fec54ba02070e64e3970c07d (origin/pr/1540/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f a12689ff51f47565fec54ba02070e64e3970c07dFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2017-05-24T07:26:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2051/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1540/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 597f25a69ce56cc05f7923b7fbf7e2a1041d3d1d # timeout=10Checking out Revision 597f25a69ce56cc05f7923b7fbf7e2a1041d3d1d (origin/pr/1540/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 597f25a69ce56cc05f7923b7fbf7e2a1041d3d1dFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2017-05-24T08:00:00Z

This is good to go on my end.

coveralls · 2017-05-24T08:26:18Z

Coverage increased (+0.7%) to 82.754% when pulling 3cb7dd1 on fnothaft:issues/1047-1539-locus-predicate into 2820e94 on bigdatagenomics:master.

AmplabJenkins · 2017-05-24T08:30:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2053/
Test PASSed.

heuermh · 2017-06-06T16:18:46Z

Locus/loci as terms aren't used very consistently across our API and docs, might region predicate be better?

@fnothaft @devin-petersohn @akmorrow13 Thoughts on this question?

fnothaft · 2017-06-06T16:21:53Z

Locus/loci as terms aren't used very consistently across our API and docs, might region predicate be better?

Thoughts on this question?

Either way is fine with me.

coveralls · 2017-06-22T08:11:51Z

Changes Unknown when pulling dc74620 on fnothaft:issues/1047-1539-locus-predicate into ** on bigdatagenomics:master**.

AmplabJenkins · 2017-06-22T08:15:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2120/
Test PASSed.

fnothaft · 2017-06-24T03:48:08Z

Ping for review...

heuermh · 2017-07-07T01:02:03Z

I'll vote for region predicate. Haven't heard from @ryan-williams so I assume the circular dependency will be fine. I looked at the dependency tree resulting from adding the dependency and it looks fine to me.

fnothaft · 2017-07-07T04:17:51Z

Rebased, and changed locus predicate to region predicate.

AmplabJenkins · 2017-07-07T04:26:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2178/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1540/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 2cab2548ad207f8ab34c8dcb31688f0bd0696e67 # timeout=10Checking out Revision 2cab2548ad207f8ab34c8dcb31688f0bd0696e67 (origin/pr/1540/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 2cab2548ad207f8ab34c8dcb31688f0bd0696e67First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh

Minor doc nits, then looks good to me

heuermh · 2017-07-07T04:31:18Z

docs/source/50_cli.md

@@ -146,6 +146,14 @@ These options fall into several general categories:
 not load the `attributes` or `origQual` fields of the `AlignmentRecord`.
 * `-aligned_read_predicate`: If loading as Parquet, only loads aligned
 reads.
+ * `-locus_predicate`: A string indicating that reads should be filtered on


missed replace here...

heuermh · 2017-07-07T04:31:29Z

docs/source/50_cli.md

+ separated list, where each element in the list takes the form:
+ * `contig:pos` for a single genomic position, or
+ * `contig:start-end` for a genomic range with closed start and open end
+ E.g., `-locus_predicate 1:100,2:1000-2000` would filter all reads that


...and here

fnothaft · 2017-07-07T06:34:36Z

Fixed the doc nits.

AmplabJenkins · 2017-07-07T06:37:04Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2179/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1540/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 8390f86 # timeout=10Checking out Revision 8390f86 (origin/pr/1540/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 8390f86fd0645ac71582b358189401c6a934887bFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-07-08T02:33:51Z

Jenkins, retest this please

AmplabJenkins · 2017-07-08T02:47:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2181/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1540/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 1aeb4d8db31bc9cc7c33b6625b61164cbd7c016b # timeout=10Checking out Revision 1aeb4d8db31bc9cc7c33b6625b61164cbd7c016b (origin/pr/1540/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 1aeb4d8db31bc9cc7c33b6625b61164cbd7c016bFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-07-08T02:50:45Z

adam-cli/src/test/scala/org/bdgenomics/adam/cli/TransformAlignmentsSuite.scala

@@ -70,4 +70,24 @@ class TransformAlignmentsSuite extends ADAMFunSuite {
 assert(qualityScoreCounts(30) === 92899)
 assert(qualityScoreCounts(10) === 7101)
 }
+
+ sparkTest("run locus predicate") {


sorry, missed these before

heuermh · 2017-07-08T02:51:04Z

adam-cli/src/test/scala/org/bdgenomics/adam/cli/TransformAlignmentsSuite.scala

+ val outputPath2 = tmpFile("predicate.2.adam")
+ val outputPath3 = tmpFile("predicate.3.adam")
+ TransformAlignments(Array(inputPath, outputPath1,
+ "-locus_predicate", "1:0-200,chr2:0-1000",


-locus_predicate → -region_predicate, and below

Resolves bigdatagenomics#1047.

Resolves bigdatagenomics#1539.

fnothaft · 2017-07-08T04:00:43Z

Fixed tests and rebased @heuermh.

coveralls · 2017-07-08T04:23:29Z

Coverage increased (+0.7%) to 84.165% when pulling e10723b on fnothaft:issues/1047-1539-locus-predicate into 0e92cfe on bigdatagenomics:master.

AmplabJenkins · 2017-07-08T04:26:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2183/
Test PASSed.

heuermh · 2017-07-08T05:11:33Z

Thank you, @fnothaft!

fnothaft added this to the 0.23.0 milestone May 18, 2017

fnothaft requested review from heuermh, ryan-williams and akmorrow13 May 18, 2017 23:43

fnothaft mentioned this pull request May 23, 2017

Rename start/end fields in NucleotideContigFragment bigdatagenomics/bdg-formats#138

Closed

fnothaft force-pushed the issues/1047-1539-locus-predicate branch from a50c351 to fb52ef4 Compare May 24, 2017 06:21

fnothaft force-pushed the issues/1047-1539-locus-predicate branch from fb52ef4 to e368695 Compare May 24, 2017 07:17

fnothaft force-pushed the issues/1047-1539-locus-predicate branch from e368695 to 3cb7dd1 Compare May 24, 2017 07:59

fnothaft force-pushed the issues/1047-1539-locus-predicate branch from 3cb7dd1 to dc74620 Compare June 22, 2017 07:56

fnothaft force-pushed the issues/1047-1539-locus-predicate branch from dc74620 to 5e5cf8f Compare July 7, 2017 04:17

heuermh requested changes Jul 7, 2017

View reviewed changes

fnothaft force-pushed the issues/1047-1539-locus-predicate branch from 5e5cf8f to 9f339d7 Compare July 7, 2017 06:25

heuermh approved these changes Jul 8, 2017

View reviewed changes

heuermh requested changes Jul 8, 2017

View reviewed changes

fnothaft added 2 commits July 7, 2017 20:59

[ADAM-1047] Support generating predicates from ReferenceRegions.

5ecf699

Resolves bigdatagenomics#1047.

[ADAM-1539] Support region predicate in TransformAlignments.

e10723b

Resolves bigdatagenomics#1539.

fnothaft force-pushed the issues/1047-1539-locus-predicate branch from 9f339d7 to e10723b Compare July 8, 2017 04:00

heuermh approved these changes Jul 8, 2017

View reviewed changes

heuermh merged commit acb0c07 into bigdatagenomics:master Jul 8, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support locus predicate in Transform #1540

Support locus predicate in Transform #1540

fnothaft commented May 18, 2017 •

edited

Loading

coveralls commented May 18, 2017 •

edited

Loading

AmplabJenkins commented May 19, 2017

heuermh commented May 22, 2017

AmplabJenkins commented May 24, 2017

AmplabJenkins commented May 24, 2017

fnothaft commented May 24, 2017

coveralls commented May 24, 2017 •

edited

Loading

AmplabJenkins commented May 24, 2017

heuermh commented Jun 6, 2017

fnothaft commented Jun 6, 2017

coveralls commented Jun 22, 2017 •

edited

Loading

AmplabJenkins commented Jun 22, 2017

fnothaft commented Jun 24, 2017

heuermh commented Jul 7, 2017

fnothaft commented Jul 7, 2017

AmplabJenkins commented Jul 7, 2017

heuermh left a comment

heuermh Jul 7, 2017

heuermh Jul 7, 2017

fnothaft commented Jul 7, 2017

AmplabJenkins commented Jul 7, 2017

heuermh commented Jul 8, 2017

AmplabJenkins commented Jul 8, 2017

heuermh Jul 8, 2017

heuermh Jul 8, 2017

fnothaft commented Jul 8, 2017

coveralls commented Jul 8, 2017 •

edited

Loading

AmplabJenkins commented Jul 8, 2017

heuermh commented Jul 8, 2017

Support locus predicate in Transform #1540

Support locus predicate in Transform #1540

Conversation

fnothaft commented May 18, 2017 • edited Loading

coveralls commented May 18, 2017 • edited Loading

AmplabJenkins commented May 19, 2017

heuermh commented May 22, 2017

AmplabJenkins commented May 24, 2017

Build result: FAILURE

AmplabJenkins commented May 24, 2017

Build result: FAILURE

fnothaft commented May 24, 2017

coveralls commented May 24, 2017 • edited Loading

AmplabJenkins commented May 24, 2017

heuermh commented Jun 6, 2017

fnothaft commented Jun 6, 2017

coveralls commented Jun 22, 2017 • edited Loading

AmplabJenkins commented Jun 22, 2017

fnothaft commented Jun 24, 2017

heuermh commented Jul 7, 2017

fnothaft commented Jul 7, 2017

AmplabJenkins commented Jul 7, 2017

Build result: FAILURE

heuermh left a comment

Choose a reason for hiding this comment

heuermh Jul 7, 2017

Choose a reason for hiding this comment

heuermh Jul 7, 2017

Choose a reason for hiding this comment

fnothaft commented Jul 7, 2017

AmplabJenkins commented Jul 7, 2017

Build result: FAILURE

heuermh commented Jul 8, 2017

AmplabJenkins commented Jul 8, 2017

Build result: FAILURE

heuermh Jul 8, 2017

Choose a reason for hiding this comment

heuermh Jul 8, 2017

Choose a reason for hiding this comment

fnothaft commented Jul 8, 2017

coveralls commented Jul 8, 2017 • edited Loading

AmplabJenkins commented Jul 8, 2017

heuermh commented Jul 8, 2017

fnothaft commented May 18, 2017 •

edited

Loading

coveralls commented May 18, 2017 •

edited

Loading

coveralls commented May 24, 2017 •

edited

Loading

coveralls commented Jun 22, 2017 •

edited

Loading

coveralls commented Jul 8, 2017 •

edited

Loading