Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A few random Predicate-related cleanups #457

Merged
merged 1 commit into from
Nov 5, 2014

Conversation

ryan-williams
Copy link
Member

I made these while working on adding a predicate for use by adam view, though I haven't gotten that to work yet, thought I'd send these out anyway.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/349/

Build result: FAILURE

GitHub pull request #457 of commit 90de3fe automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-slave-01 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/457/merge^{commit} # timeout=10Checking out Revision 774b1e4 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 774b1e4 > git rev-list a7a2569f68f2981425bb14e2fffc1bdbd8024b8c # timeout=10Triggering ADAM-prb » 2.2.0,centosTriggering ADAM-prb » 1.0.4,centosTriggering ADAM-prb » 2.3.0,centosADAM-prb » 2.2.0,centos completed with result FAILUREADAM-prb » 1.0.4,centos completed with result FAILUREADAM-prb » 2.3.0,centos completed with result FAILURE
Test FAILed.

@ryan-williams
Copy link
Member Author

hrm, the death rattle of the Jenkins job above showed a division by zero error in reads2ref:

+ echo 'Converting read file to pileup format'
Converting read file to pileup format
+ ./bin/adam-submit reads2ref mouse_chrM.bam.reads mouse_chrM.bam.pileup
Spark assembly has been built with Hive, including Datanucleus jars on classpath
2014-11-03 09:24:03 WARN  AbstractLifeCycle:204 - FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:293)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:192)
    at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:202)
    at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:202)
    at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1446)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
    at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1442)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:202)
    at org.apache.spark.ui.WebUI.bind(WebUI.scala:102)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:224)
    at org.bdgenomics.adam.cli.ADAMSparkCommand$class.run(ADAMCommand.scala:55)
    at org.bdgenomics.adam.cli.Reads2Ref.run(Reads2Ref.scala:57)
    at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:96)
    at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2014-11-03 09:24:03 WARN  AbstractLifeCycle:204 - FAILED org.eclipse.jetty.server.Server@7d1fdb25: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:293)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:192)
    at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:202)
    at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:202)
    at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1446)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
    at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1442)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:202)
    at org.apache.spark.ui.WebUI.bind(WebUI.scala:102)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:224)
    at org.bdgenomics.adam.cli.ADAMSparkCommand$class.run(ADAMCommand.scala:55)
    at org.bdgenomics.adam.cli.Reads2Ref.run(Reads2Ref.scala:57)
    at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:96)
    at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2014-11-03 09:24:03 WARN  Utils:71 - Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
2014-11-03 09:24:03 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.ArithmeticException: / by zero
    at org.bdgenomics.adam.cli.Reads2Ref.run(Reads2Ref.scala:69)
    at org.bdgenomics.adam.cli.ADAMSparkCommand$class.run(ADAMCommand.scala:58)
    at org.bdgenomics.adam.cli.Reads2Ref.run(Reads2Ref.scala:57)
    at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:96)
    at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Build step 'Execute shell' marked build as failure
Recording test results
Finished: FAILURE

but I've run scripts/jenkins-test locally and it passed.

Jenkins, retest this please?

@ryan-williams
Copy link
Member Author

nvm, I've repro'd this failure now. investigating

@massie
Copy link
Member

massie commented Nov 3, 2014

Btw, I create PR #458 to fix the division by zero issue.

-Matt

On Mon, Nov 3, 2014 at 2:27 PM, Ryan Williams notifications@github.com
wrote:

nvm, I've repro'd this failure now. investigating


Reply to this email directly or view it on GitHub
#457 (comment).

@ryan-williams
Copy link
Member Author

hm, but readCount should not be 0 here, right? My changes have probably actually broken predicates for that div by 0 to be happening, right?

@ryan-williams
Copy link
Member Author

 Mon 22:49:42  ryan@mbp: adam:predicates$ node -p -e '4 | 256 | 512 | 1024'
1796
 Mon 22:49:52  ryan@mbp: adam:predicates$ samtools view -F 1796 -c mouse_chrM.bam
160512

mouse_chrM.bam actually has 160512 reads that should pass that predicate, AFAICT (and presumably it works fine everywhere but this PR since jenkins is not failing in this way anywhere else)

@massie
Copy link
Member

massie commented Nov 3, 2014

Yes, that seems the most likely explanation.

We need to remove unused code anyway -- especially code that doesn't
protect from divide by zero.

-Matt

On Mon, Nov 3, 2014 at 2:48 PM, Ryan Williams notifications@github.com
wrote:

hm, but readCount should not be 0 here, right? My changes have probably
actually broken predicates for that div by 0 to be happening, right?


Reply to this email directly or view it on GitHub
#457 (comment).

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/354/
Test PASSed.

@ryan-williams
Copy link
Member Author

cool, tests fixed, lmk if you guys have thoughts on these changes

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/365/
Test PASSed.

@fnothaft
Copy link
Member

fnothaft commented Nov 5, 2014

LGTM, can you squash and I'll merge?

- simplify PredicateUtils wrapper

- wrap some long lines

- wrapper for FieldCondition[Boolean] creation

- AlignmentRecordCondition creation conv. wrappers

- add supplementaryAlignment to AlignmentRecordField
@ryan-williams
Copy link
Member Author

squashed

fnothaft added a commit that referenced this pull request Nov 5, 2014
A few random Predicate-related cleanups
@fnothaft fnothaft merged commit e00d71b into bigdatagenomics:master Nov 5, 2014
@fnothaft
Copy link
Member

fnothaft commented Nov 5, 2014

Merged! Thanks @ryan-williams!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants