QuickStart Installation for Local PhenoMeNal Workflow
This QuickStart guide demonstrates how to set up PhenoMeNal Workflow on your local machine.
This QuickStart guide can help you to set up PhenoMeNal Workflows, based on Galaxy running on top of Kubernetes, in your local machine.
Try it out! Here is a link to an example public instance of the PhenoMeNal Workflow, to get a grasp of what you would get locally.
- minikube and kubectl: For a single machine development environment on Linux, Windows or macOS, currently the official Kubernetes recommendation is to use Mini-Kube. This is the current preferred way, and the link provided has all instructions for both minikube and kubectl (the command line interface to interact with Kubernetes clusters, including minikube).
- Full Administrator access (root / sudo on Linux, full Admin on Windows) to the computer for installing Mini-Kube. You can add your preferred user to the group docker for using Mini-Kube without root provileges.
- Before installing the PhenoMeNal Workflow, make sure that minikube is running and the command
minikube startgives something likeKubernetes is available at https://192.168.99.100:8443as output.- On Linux, if having problems, use the following command to start minikube:
# For linux only
minikube start
Helm is a package manager for Kubernetes. We have created a Helm package to easily deploy Galaxy on Kubernetes, both for production and for development mode. Helm has been blessed as an official Kubernetes project (lives under the Kubernetes organization in GitHub).
- Install the Helm binary. You can find the latest releases for Mac, Linux and Windows here.
- Once helm is on the path and made executable, init your Kubernetes cluster with Helm (supposes that you have a running cluster accessible through
kubectlwhich is the case for minikube). This is run from your host machine, where you would runkubectl.
$ helm init
- Add our Helm repo:
$ helm repo add galaxy-helm-repo https://pcm32.github.io/galaxy-helm-charts
If you delete your minikube VM with minikube delete for any reason, you will need to do the helm init step again. Otherwise, the above steps need to be done only once.
The new galaxy-stable chart uses containers based on the community docker-galaxy-stable/compose images, which bring improved Galaxy web deployment (UWGSI based with multiple threads and multiprocessor), separate web process and job handling processes and sftp access among others. This also reduces the maintenance burden for PhenoMeNal, as we mostly rely on the community for the Galaxy images and only the flavouring/tools remains to be maintained.
The script simplified_galaxy_stable_container_creation.sh provided as part of container-galaxy-k8s-runtime allows you to create all the required docker images locally. This script also holds the current setup in terms of the desired commit to use from ansible-galaxy-extras config repo, the PhenoMeNal Galaxy checkout with the changes that haven't made it yet to a Galaxy release, and the correct commit to use from docker-galaxy-stable/compose. Make sure that you have made a recursive git pull of the repo to include submodules and that you update the submodule when changing branches:
git clone --recurse-submodules https://github.com/phnmnl/container-galaxy-k8s-runtime.git
git checkout feature/<where-you-are-working>
git submodule update
To build the images, set the following env variables:
export CONTAINER_USER=your-dockerhub-user
log in via docker login and then execute:
bash simplified_galaxy_stable_container_creation.sh
make sure that you are using a good internet connection and go an enjoy a coffee, as the first time it will take a while.
In the end, the script will output something like:
Relevant containers:
Web: pcm32/galaxy-web-k8s:v18.01-pheno-dev
Init Flavour: pcm32/galaxy-init-pheno-flavoured:v18.01-pheno-dev
Postgres: pcm32/galaxy-postgres:9.6.5_for_18.01
Proftpd: pcm32/galaxy-proftpd:for_galaxy_v18.01
where pcm32 will be changed for your dockerhub user (tags might be different as well, depending on how old this doc is). The script will push all images to your docker account.
Then, write a config yaml file like this one, available at https://github.com/galaxyproject/galaxy-kubernetes, where you change the repository and tag for galaxy.init.image (based on Init flavour above), galaxy.image (based on Web), postgres_for_galaxy.postgres_image and postgres_image_tag (based on Postgres) and proftpd.image.repository and tag (based on Proftpd above). The key of course is to set Web and Init flavour containers, as they will include the changes needed. After that, do:
helm install -f <your-config.yaml> --version 1.5.5 galaxy-helm-repo/galaxy-stable
Please note that we are working on version 2.0.0 of the Helm chart with the Galaxy community and it will bring a lot of changes in terms of variable naming.
Here we deploy Galaxy using helm injecting a path where your Galaxy development files (config and tools) are sitting, through the development_folder variable. The development_folder variable needs to point to the folder where you have the Galaxy config and tools folders, for development purposes. This could be, for instance, the directory resulting from git clone https://github.com/phnmnl/container-galaxy-k8s-runtime, which will contain a config and a tools folder populated with the adequate files. The deployment will fail if you use a directory that doesn't contain the expected files.
This setup will generate as well a Persistent Volume in minikube, inside /data/galaxy-data folder.
Please note below that jdoe is the account for "John Doe", a fictional character, you need to replace that accordingly.
$ helm install --set galaxy_admin_email="youremail@yourdomain",\
galaxy_admin_password="yourpassword",\
galaxy_api_key="qwertyuio",\
tool2container_mapping_file="phenomenal_tools2container.yaml",\
development_folder=/Users/jdoe/Development/where-galaxy-config-tools-are,pv_minikube="yes" \
galaxy-helm-repo/galaxy
Please note fields galaxy_admin_email and galaxy_admin_password require your attention. Also note the new field tool2container_mapping_file required for the new dynamic destination setting.
Please note that when running on linux, the mount point inside the minikube VM changes from /home to /hosthome (so files on /home/your-user/your-dev-files would be visible on minikube and Kubernetes on /hosthome/your-user/your-dev-files) so the development_folder variable needs to be set accordingly on linux:
$ helm install --set galaxy_admin_email="youremail@yourdomain",\
galaxy_admin_password="yourpassword",\
galaxy_api_key="qwertyuio",\
tool2container_mapping_file="phenomenal_tools2container.yaml",\
development_folder=/hosthome/jdoe/Development/where-galaxy-config-tools-are,pv_minikube="yes" \
galaxy-helm-repo/galaxy
Please note the difference on the development_folder setting, unique to Linux. Fields galaxy_admin_email and galaxy_admin_password require your attention. Also note the new field tool2container_mapping_file required for the new dynamic destination setting.
Please note that when running on Windows, the mount point inside the minikube VM changes from C:\Users to /c/Users (so files on C:\Users\your-user\your-dev-files would be visible on minikube and Kubernetes on /c/Users/your-user/your-dev-files) so the development_folder variable needs to be set accordingly on Windows:
$ helm install --set galaxy_admin_email="youremail@yourdomain",\
galaxy_admin_password="yourpassword",\
galaxy_api_key="qwertyuio",\
tool2container_mapping_file="phenomenal_tools2container.yaml",\
development_folder=/c/Users/jdoe/Development/where-galaxy-config-tools-are,pv_minikube="yes" \
galaxy-helm-repo/galaxy
Please note the difference on the development_folder setting, unique to Windows. Fields galaxy_admin_email and galaxy_admin_password require your attention. Also note the new field tool2container_mapping_file required for the new dynamic destination setting.
Sometimes you need to make modifications to your deployment, e.g. if your files are located in a custom folder. In this case you can mount the host directory to minikube via:
minikube mount /usr/src/:/usr/src
The you can deploy it manually with this custom directory:
$ helm install --set galaxy_admin_email="youremail@yourdomain",galaxy_admin_password="yourpassword",galaxy_api_key="qwertyuio",development_folder=/usr/src/where-galaxy-config-tools-are,pv_minikube="yes" galaxy-helm-repo/galaxy
This is an alternative to the previous, but instead we remove the development_folder variable:
$ helm install --set pv_minikube="yes" galaxy-helm-repo/galaxy
This is useful for demonstrations for instance.
If you want a local Helm deployment or using Helm on top of a Kubernetes cluster, you need to use the following line:
helm install --set galaxy_admin_email="youremail@yourdomain",galaxy_admin_password="yourpassword",galaxy_api_key="qwertyuio",galaxy_image_tag=":rc_v17.05-pheno_cv1.1.86" galaxy-helm-repo/galaxy
The most relevant part is …,galaxy_image_tag=":rc_v17.05-pheno_cv1.1.86" which in this case fixes the version to PhenoMeNal Galaxy release candidate rc_v17.05-pheno_cv1.1.86. We have tree types of images for Galaxy: development (image has tag dev prefix), release candidate (image tag hasrc prefix), and release (no prefix on tag). You can find existing tags at the CI server entry http://phenomenal-h2020.eu/jenkins/job/container-galaxy-k8s-runtime/.
If you haven't added the helm repo recently but some time has gone past, and you want to deploy using the newest available deployment in terms of helm functionality, you can run helm repo update to update your local repos.
To see other available options, see directly the Galaxy Helm chart documentation by invoking:
$ helm inspect galaxy-helm-repo/galaxy
Galaxy is now running on the computer-node you have installed PhenoMeNal and deployed Galaxy. You can test the running Galaxy instance with doing the following:
$ curl $(minikube ip):30700
Or just enter the IP address which the command minikube ip returns + Port 30700 in your webbrowser, usually it is: http://192.168.99.100:30700 (please note, it is http and not https). Please note that the first time that you run this, it will take a while to load as Galaxy is creating its database. You can check progress by asking for the logs of the pod running Galaxy:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
galaxy-k8s-z2sqp 1/1 Running 0 4m
$ kubectl logs -f galaxy-k8s-z2sqp
Once logs show something like
Starting server in PID 306.
serving on 0.0.0.0:8080 view at http://127.0.0.1:8080
it means it is ready for use. Alternatively, just reload your browser on the specified URL after a few minutes.
Note that the name of the pod used in the logs -f command will be different on each deployment, and needs to be queried as per the first command get pods.
First, check your active helm deployments:
$ helm list
NAME REVISION UPDATED STATUS CHART
tufted-heron 1 Mon Oct 24 15:10:21 2016 DEPLOYED galaxy-simple-0.1.2
Then using the name of the deployment, for this example it would be:
$ helm delete tufted-heron
You may have to delete the secrets too that have been created by helm before used in the development version.
$ kubectl delete secrets/galaxy-admin-secret
$ kubectl delete configmaps/galaxy-admin-user
$ kubectl delete configmaps/db-connection
$ kubectl delete secrets/galaxy-postgres-secret
Within PhenoMeNal, we also provide the ability to do some interactive data analysis through Jupyter in Python (2 or 3) and R. To lift Jupyter using Helm, you can use one of these alternatives:
- Deploy independently of Galaxy
- Deploy after Galaxy, sharing the PVC
Make sure that you have the needed repo for Jupyter:
$ helm repo add phenomenal-helm https://phnmnl.github.io/helm-charts
if you previously had this, please update the repo with:
$ helm repo update
You will need to set the password for Jupyter (through passwd_jupyter), the requirements in terms of CPU (jupyter_resource_req_cpu, keep it under 1 if deploying to minikube) and the requirements in terms of memory (jupyter_resource_req_memory, a few hundred MBs if deploying to minikube). If deploying on minikube, you can use pv_minikube variable to yes to activate a PersistenVolume based on host path.
helm install --set passwd_jupyter="your-pass",jupyter_resource_req_cpu=0.1,jupyter_resource_req_memory="100M",pv_minikube="yes" phenomenal-helm/jupyter
Resource requirement variables have defaults.
If you have previously deployed Galaxy, you might want to re-use its Persistent Volume Claim for Jupyter. You can do so by adding the following variables:
$ helm install -set ...,jupyter_pvc="galaxy-pvc",pvc_exists="yes",.... phenomenal-helm/jupyter
For more variations available, check the output of:
$ helm inspect phenomenal-helm/jupyter
Unless that you're using an ingress with a DNS strategy, you should be able to reach Jupyter on echo http://$(minikube ip):30712.
 |
Funded by the EC Horizon 2020 programme, grant agreement number 654241 |  |
|---|
