Move non-genomics code

[fnothaft]

This resolves #475 by removing the non-genomics specific code which has been migrated to bdg-utils, and pointing at bdg-utils where necessary. Specifically, we've moved the:

• SparkFunSuite
• Parquet reimplementation
• Metrics
• Instrumentation

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/397/

Build result: FAILURE

GitHub pull request #478 of commit f9812dc automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-slave-01 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/478/merge^{commit} # timeout=10Checking out Revision 342c0ae (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 342c0ae > git rev-list cc683a5 # timeout=10Triggering ADAM-prb » 2.3.0,centosTriggering ADAM-prb » 1.0.4,centosTriggering ADAM-prb » 2.2.0,centosADAM-prb » 2.3.0,centos completed with result FAILUREADAM-prb » 1.0.4,centos completed with result FAILUREADAM-prb » 2.2.0,centos completed with result FAILURE
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/398/

Build result: FAILURE

GitHub pull request #478 of commit d759957 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-slave-01 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/478/merge^{commit} # timeout=10Checking out Revision f98c346 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f f98c346 > git rev-list cc683a5 # timeout=10Triggering ADAM-prb » 2.3.0,centosTriggering ADAM-prb » 1.0.4,centosTriggering ADAM-prb » 2.2.0,centosADAM-prb » 2.3.0,centos completed with result FAILUREADAM-prb » 1.0.4,centos completed with result FAILUREADAM-prb » 2.2.0,centos completed with result FAILURE
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/400/
Test PASSed.

[tdanford]

See the notes in #472, I think merging this should wait on merging that PR first.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/567/

Build result: FAILURE

GitHub pull request #478 of commit 64e2752 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/478/merge^{commit} # timeout=10Checking out Revision 93cb1940a73c7a2b9fb05142628e5c04c2d757c4 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 93cb1940a73c7a2b9fb05142628e5c04c2d757c4 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 2.3.0,centosTriggering ADAM-prb ? 2.2.0,centosTriggering ADAM-prb ? 1.0.4,centosADAM-prb ? 2.3.0,centos completed with result FAILUREADAM-prb ? 2.2.0,centos completed with result FAILUREADAM-prb ? 1.0.4,centos completed with result FAILURE
Test FAILed.

[fnothaft]

I expect this failure because I was building against bigdatagenomics/utils#19. This will pass once bigdatagenomics/utils#19 is merged.

[fnothaft]

Ta-da! And, this now is working with @nfergu's work from #514 on top, and building against bigdatagenomics/utils#19. The build will fail here though, since we haven't merged bigdatagenomics/utils#19 upstream.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/568/

Build result: FAILURE

GitHub pull request #478 of commit 0caf693 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/478/merge^{commit} # timeout=10Checking out Revision 33636cc5c1192c50966bb7fb69ff33cbef865994 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 33636cc5c1192c50966bb7fb69ff33cbef865994 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 2.3.0,centosTriggering ADAM-prb ? 2.2.0,centosTriggering ADAM-prb ? 1.0.4,centosADAM-prb ? 2.3.0,centos completed with result FAILUREADAM-prb ? 2.2.0,centos completed with result FAILUREADAM-prb ? 1.0.4,centos completed with result FAILURE
Test FAILed.

[massie]

Switching to SparkFunSuite in bdg-utils will revert the work I did in ADAM to fix all the test bind errors. We need to port that into bdg-utils.

[fnothaft]

@massie thanks for catching that! I will port that right now.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/569/

Build result: FAILURE

GitHub pull request #478 of commit 71efd46 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/478/merge^{commit} # timeout=10Checking out Revision 6ef8e42 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 6ef8e42 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 2.3.0,centosTriggering ADAM-prb ? 2.2.0,centosTriggering ADAM-prb ? 1.0.4,centosADAM-prb ? 2.3.0,centos completed with result FAILUREADAM-prb ? 2.2.0,centos completed with result FAILUREADAM-prb ? 1.0.4,centos completed with result FAILURE
Test FAILed.

[fnothaft]

Jenkins, retest this please.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/570/

Build result: FAILURE

GitHub pull request #478 of commit 71efd46 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/478/merge^{commit} # timeout=10Checking out Revision 6ef8e42 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 6ef8e42 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 2.2.0,centosTriggering ADAM-prb ? 1.0.4,centosTriggering ADAM-prb ? 2.3.0,centosADAM-prb ? 2.2.0,centos completed with result FAILUREADAM-prb ? 1.0.4,centos completed with result FAILUREADAM-prb ? 2.3.0,centos completed with result FAILURE
Test FAILed.

[fnothaft]

Once bigdatagenomics/utils#22 is merged, this will pass.

[massie]

Jenkins, test this please.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/571/

Build result: FAILURE

GitHub pull request #478 of commit 71efd46 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/478/merge^{commit} # timeout=10Checking out Revision 6ef8e42 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 6ef8e42 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 1.0.4,centosTriggering ADAM-prb ? 2.3.0,centosTriggering ADAM-prb ? 2.2.0,centosADAM-prb ? 1.0.4,centos completed with result FAILUREADAM-prb ? 2.3.0,centos completed with result FAILUREADAM-prb ? 2.2.0,centos completed with result FAILURE
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/572/

Build result: FAILURE

GitHub pull request #478 of commit ea77c10 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/478/merge^{commit} # timeout=10Checking out Revision cc1e819a0f1930f38a1e30509db24aa12b3280e4 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f cc1e819a0f1930f38a1e30509db24aa12b3280e4 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 1.0.4,centosTriggering ADAM-prb ? 2.3.0,centosTriggering ADAM-prb ? 2.2.0,centosADAM-prb ? 1.0.4,centos completed with result SUCCESSADAM-prb ? 2.3.0,centos completed with result FAILUREADAM-prb ? 2.2.0,centos completed with result FAILURE
Test FAILed.

[fnothaft]

Jenkins, retest this please.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/573/

Build result: FAILURE

GitHub pull request #478 of commit ea77c10 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/478/merge^{commit} # timeout=10Checking out Revision cc1e819a0f1930f38a1e30509db24aa12b3280e4 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f cc1e819a0f1930f38a1e30509db24aa12b3280e4 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 1.0.4,centosTriggering ADAM-prb ? 2.3.0,centosTriggering ADAM-prb ? 2.2.0,centosADAM-prb ? 1.0.4,centos completed with result SUCCESSADAM-prb ? 2.3.0,centos completed with result FAILUREADAM-prb ? 2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[fnothaft]

Jenkins, test this please.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/574/
Test PASSed.

[fnothaft]

Now that this really is passing, I've cut a new bdg-utils release. I'll update this PR once that release is present in Maven Central.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/575/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/576/
Test PASSed.

[massie]

Thanks, Frank!

[nfergu]

@fnothaft -- looks great! Just tried the latest version locally and it works well.

Just one comment: I think the Timers object should be in ADAM rather than bdg-utils as it contains ADAM-specific timers. I'll raise separate PRs to move it.

[fnothaft]

@nfergu ah, drat! You are indeed correct! I'd created an earlier version that did that correctly ;). Thanks for catching this, and I'll review and merge your two PRs shortly.