Randomize ports and turn off Spark UI to reduce bind exceptions in tests

[massie]

No description provided.

[fnothaft]

Nice!

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/556/
Test PASSed.

[fnothaft]

Looking in the logs, it looks like we're still getting bind exceptions:

2015-01-25 11:49:51 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-01-25 11:49:52 WARN  AbstractLifeCycle:204 - FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:293)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:194)
    at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:204)
    at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:204)
    at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1676)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
    at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1667)

Thoughts?

[massie]

The Features2ADAMSuite test class needs to use the SparkFunSuite in to fix the error about multiple spark contexts in a single JVM.

The ./scripts/jenkins-tests script needs to be updated to either (a) have an exclusive lock to prevents multiple scripts from running at once or (b) randomizing the ports that are used for the tests.

[massie]

This PR is step one which fixes the bind exceptions thrown in the SparkFunSuite. I'll update the Jenkins script in a subsequent PR.

[fnothaft]

Sounds good, thanks for the clarification @massie!

The Features2ADAMSuite test class needs to use the SparkFunSuite in to fix the error about multiple spark contexts in a single JVM.

Ah, makes sense. And, this is because the Features2ADAMSuite doesn't create a SparkContext, but Features2ADAM.run() does. I'll open a PR to fix that.

[fnothaft]

Merged! Thanks @massie!