Add basic Documentation for setting up on Google Cloud #340

Georgehe4 · 2017-12-12T05:30:24Z

No description provided.

AmplabJenkins · 2017-12-12T05:43:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/515/
Test PASSed.

akmorrow13 · 2017-12-12T05:55:12Z

docs/cloud/google-cloud.rst

+
+.. code:: bash
+
+    wget https://gist.githubusercontent.com/Georgehe4/6bb1c142a9f68f30f38d80cd9407120a/raw/9b903e3b8746ee8f25911fe98925b53e9777002f/mango_install.sh


Can we put this script in Mango scripts?

akmorrow13 · 2017-12-12T05:56:25Z

docs/cloud/google-cloud.rst

+            gs://mango-initialization-bucket/mango_install.sh
+
+
+Once the above steps are completed, simply ssh into the master node to run Mango.


Do you have the scripts for running docker?

akmorrow13 · 2017-12-12T05:57:02Z

docs/index.rst

@@ -31,6 +31,12 @@ variety of platforms.

   docker/docker-examples

+.. toctree::
+   :caption: Cloud


Google Cloud

We might add an AWS Section later?

AmplabJenkins · 2017-12-13T02:09:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/517/
Test PASSed.

AmplabJenkins · 2017-12-14T08:48:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/523/
Test PASSed.

akmorrow13 · 2017-12-14T18:39:32Z

scripts/google_cloud_docker_install.sh

+# update the apt package index:
+sudo apt-get -y update
+# finally, install docker
+sudo apt-get -y install docker-ce


akmorrow13 · 2017-12-14T18:40:13Z

scripts/google_cloud_docker_run.sh

+       -e HIVE_DIR=${HIVE_DIR} \
+       -e HIVE_CONF_DIR=${HIVE_CONF_DIR} \
+       -e SPARK_DIST_CLASSPATH="/usr/lib/hadoop/etc/hadoop:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*" \
+       --entrypoint=/opt/cgl-docker-lib/mango/bin/mango-notebook \


when you were accessing google storage through docker, did it need any access keys?

Yes I still haven't resolved the access key issue among the spark workers yet for bam files in gs. For now, accessing hdfs works fine but gs still needs to be investigated.

akmorrow13 · 2017-12-14T18:41:15Z

scripts/google_cloud_mango_install.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash


let's put these in bin/GCE/

akmorrow13 · 2017-12-14T18:41:53Z

scripts/google_cloud_mango_install.sh

+fi
+
+if [[ "${ROLE}" == 'Master' ]]; then
+    conda install jupyter


Why are you installing docker here? docker is installed in the container, right?

Docker is not provided on a default gce vm.

AmplabJenkins · 2017-12-15T04:27:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/525/
Test PASSed.

AmplabJenkins · 2017-12-15T04:47:58Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/526/
Test PASSed.

akmorrow13 · 2017-12-15T19:31:50Z

docs/cloud/google-cloud.rst

+
+    hdfs dfs -put /<local machime path> /<hdfs path>
+
+An example docker startup script is available in the Mango `scripts directory <https://github.com/bigdatagenomics/mango/blob/master/bin/gce/google_cloud_docker_run.sh>`__ for running mango notebook [run with root permissions to work with docker].


Can you give the code to copy this from docker so a user can run the script?

AmplabJenkins · 2017-12-18T05:02:48Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/533/
Test PASSed.

akmorrow13 · 2017-12-19T21:58:44Z

bin/gce/google_cloud_docker_run.sh

@@ -0,0 +1,37 @@
+export SPARK_HOME=/usr/lib/spark
+export SPARK_CONF_DIR=/usr/lib/spark/conf


This file is specifically for mango notebook. we can pass a user variable here that states whether to run notebook or browser based on the --entrypoint flag

separate into run-notebook.sh and run-browser.sh

akmorrow13 · 2018-01-03T16:57:23Z

bin/gce/google_cloud_mango_install.sh

+    echo "Jupyter Notebook extensions installed!"
+fi
+
+pip install cigar


shouldn't cigar be installed in the virtual env?

This is so cigar is present on the worker nodes (it is also installed in the venv).

What configurations have you been setting to port python packages over to the worker nodes on the cluster? It looks like the python binary for the worker nodes is not being overwritten by the flags sent to Spark, which leads to "cigar not found" issues if it is not installed this way.

won't this be installed outside of the virtual env? Do you know why cigar is causing an issue and not other imports?

Cigar will be installed in the virtual env, just not through this process.

Since the pyspark jobs depend on cigar in the worker nodes, the python distribution on the worker nodes need to have cigar present. I don't believe we use any other imports in pyspark for mango-notebook.

AmplabJenkins · 2018-01-15T17:05:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/545/
Test PASSed.

…o add_docs

AmplabJenkins · 2018-01-18T05:24:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/547/
Test PASSed.

AmplabJenkins · 2018-01-20T07:48:44Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/555/
Test PASSed.

Georgehe4 · 2018-01-21T22:48:01Z

Hey @akmorrow13, I've added an example file and fixed an issue with block data reads (https://gist.github.com/Georgehe4/043ae9079349ac7b05bee1e0b3b4b4fa) - I believe this should be good to go.

akmorrow13 · 2018-01-22T19:04:57Z

bin/gce/google_cloud_docker_run.sh

+export HADOOP_HDFS=/usr/lib/hadoop-hdfs
+export HADOOP_YARN=/usr/lib/hadoop-yarn
+export HADOOP_MAPREDUCE=/usr/lib/hadoop-mapreduce
+export HIVE_DIR=/usr/lib/hive


Do you need hive export?

Nope - removed

akmorrow13 · 2018-01-22T19:05:10Z

bin/gce/google_cloud_docker_run.sh

+       -e HADOOP_CONF_DIR=${HADOOP_CONF_DIR} \
+       -e HIVE_DIR=${HIVE_DIR} \
+       -e HIVE_CONF_DIR=${HIVE_CONF_DIR} \
+       -e SPARK_DIST_CLASSPATH="/usr/lib/hadoop/etc/hadoop:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*" \


can you comment the use of this line?

Added - this is necessary to specify the hadoop version (and other dependencies) to match the cluster environment in google cloud.

akmorrow13 · 2018-01-22T19:05:44Z

bin/gce/google_cloud_mango_install.sh

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash


can you simplify into 1 install script?

What do you mean? This script will be automatically run upon cluster creation.

akmorrow13 · 2018-01-22T19:06:22Z

docs/cloud/google-cloud.rst

@@ -0,0 +1,84 @@
+Running Mango on Google Cloud
+=============================
+


how do you make a google cloud cluster?

refer to the section below, there's a step to
Create the Cloud Dataproc Cluster

akmorrow13 · 2018-01-22T19:06:46Z

docs/cloud/google-cloud.rst

+
+More information about available public datasets on Google cloud can be found `online <https://cloud.google.com/genomics/v1/public-data>`__
+
+More information on using the dataproc cluster's Spark interface is available through `Google Cloud documentation <https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces>`__


is there an example notebook you can push and reference on a non-1000g datasource?

I will make some changes to mango-google-cloud.ipynb to reference a non-1000g datasource

akmorrow13 · 2018-01-22T19:07:29Z

example-files/mango-google-cloud.ipynb

+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "alignmentFile = \"gs://genomics-public-data/ftp-trace.ncbi.nih.gov/1000genomes/ftp/data/150140/alignment/150140.chrom20.ILLUMINA.bwa.CHM1.20131218.bam\""


what other datasets are hosted besides 1000g? I ask because 1000g is the only datasource on AWS, so it would be nice to reference a different one in the GCE examples

AmplabJenkins · 2018-01-23T06:53:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/556/
Test PASSed.

akmorrow13 · 2018-01-26T21:43:30Z

install.sh, run-notebook.sh, run-browser.sh

…o add_docs

AmplabJenkins · 2018-01-30T00:16:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/562/
Test PASSed.

akmorrow13 · 2018-02-02T21:51:11Z

bin/gce/run-submit.sh

@@ -0,0 +1,68 @@
+set -ex


change file to run-browser

AmplabJenkins · 2018-02-02T23:26:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/564/
Test PASSed.

akmorrow13 · 2018-02-07T23:20:04Z

looks good @Georgehe4 can you rebase?

Georgehe4 · 2018-02-08T04:43:20Z

Moved to #360.

Can you enable squash merges for this repo?

Georgehe4 added 3 commits December 11, 2017 22:19

Fix index underline warnings

4173ca3

Fix github repo

1817e1c

Add basic google cloud documentation

05fd42f

akmorrow13 reviewed Dec 12, 2017

View reviewed changes

Update google cloud instructions for running mango in docker

750dd09

Add docker install scripts

4f53b10

akmorrow13 reviewed Dec 14, 2017

View reviewed changes

Georgehe4 added 2 commits December 14, 2017 23:17

Update scripts directory for gce

6a65f77

merge with master

505705d

update gce scripts

0209b33

akmorrow13 reviewed Dec 15, 2017

View reviewed changes

Add command for running docker

770a02c

akmorrow13 reviewed Dec 19, 2017

View reviewed changes

akmorrow13 reviewed Jan 3, 2018

View reviewed changes

Add docker_run options to run from both mango-notebook and mango-submit

517c746

Georgehe4 mentioned this pull request Jan 18, 2018

Add maven dependency for gcloud NIO #346

Closed

Georgehe4 added 2 commits January 18, 2018 00:09

Update documentation to support gs

2916db0

Merge branch 'master' of https://github.com/bigdatagenomics/mango int…

11843f6

…o add_docs

Update documentation and add example file

90dd129

fnothaft mentioned this pull request Jan 21, 2018

Issues loading BAM files in Google FS bigdatagenomics/adam#1816

Closed

akmorrow13 reviewed Jan 22, 2018

View reviewed changes

Streamline installation, updated documentation and scripts

00b4d54

Georgehe4 added 3 commits January 29, 2018 19:01

Merge branch 'master' of https://github.com/bigdatagenomics/mango int…

49272a8

…o add_docs

Rename script names

872bb02

Remove example file

f214003

akmorrow13 reviewed Feb 2, 2018

View reviewed changes

bin/gce/run-submit.sh Outdated

@@ -0,0 +1,68 @@

set -ex

Copy link

Contributor

akmorrow13 Feb 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change file to run-browser

Update docs and scripts

0ab3d01

Georgehe4 mentioned this pull request Feb 8, 2018

Google Cloud Setup #360

Merged

Georgehe4 closed this Apr 13, 2018


		.. code:: bash

		wget https://gist.githubusercontent.com/Georgehe4/6bb1c142a9f68f30f38d80cd9407120a/raw/9b903e3b8746ee8f25911fe98925b53e9777002f/mango_install.sh

		gs://mango-initialization-bucket/mango_install.sh


		Once the above steps are completed, simply ssh into the master node to run Mango.


		hdfs dfs -put /<local machime path> /<hdfs path>

		An example docker startup script is available in the Mango `scripts directory <https://github.com/bigdatagenomics/mango/blob/master/bin/gce/google_cloud_docker_run.sh>`__ for running mango notebook [run with root permissions to work with docker].

		@@ -0,0 +1,37 @@
		export SPARK_HOME=/usr/lib/spark
		export SPARK_CONF_DIR=/usr/lib/spark/conf

		@@ -0,0 +1,84 @@
		Running Mango on Google Cloud
		=============================


		More information about available public datasets on Google cloud can be found `online <https://cloud.google.com/genomics/v1/public-data>`__

		More information on using the dataproc cluster's Spark interface is available through `Google Cloud documentation <https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces>`__

Add basic Documentation for setting up on Google Cloud #340

Add basic Documentation for setting up on Google Cloud #340

Conversation

Georgehe4 commented Dec 12, 2017

AmplabJenkins commented Dec 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Georgehe4 Dec 13, 2017 • edited Loading

Choose a reason for hiding this comment

AmplabJenkins commented Dec 13, 2017

AmplabJenkins commented Dec 14, 2017

Choose a reason for hiding this comment

Georgehe4 Dec 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Dec 15, 2017

AmplabJenkins commented Dec 15, 2017

Choose a reason for hiding this comment

AmplabJenkins commented Dec 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Jan 15, 2018

AmplabJenkins commented Jan 18, 2018

AmplabJenkins commented Jan 20, 2018

Georgehe4 commented Jan 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Jan 23, 2018

akmorrow13 commented Jan 26, 2018

AmplabJenkins commented Jan 30, 2018

Choose a reason for hiding this comment

AmplabJenkins commented Feb 2, 2018

akmorrow13 commented Feb 7, 2018

Georgehe4 commented Feb 8, 2018

Georgehe4 Dec 13, 2017 •

edited

Loading

Georgehe4 Dec 15, 2017 •

edited

Loading