oswitch - virtual environments for flexible and reproducible bioinformatics analyses

More and more bioinformatics software are being "containerised" with docker, making them installable on personal computers and compute clusters alike with a single command in a reproducible manner. However, using containerised software can still be a challenge as the containers are effectively a different operating system altogether. oswitch enhances usability of docker containers by:

1. Making available the local file-system inside the container.
2. Ensuring you are the same user inside the container as you are on the host.
3. Running your login shell, including sourcing your config files when the container is booted.
4. Switching to the directory that you were in on the host when invoking oswitch.

The net effect is similar to entering "virtual environment" on the host system containing specific versions of software of interest.

mymacbook:~/2015-02-01-myproject> abyss-pe k=25 reads.fastq.gz
    zsh: command not found: abyss-pe

# List available images.
mymacbook:~/2015-02-01-myproject> oswitch -l
    yeban/biolinux:8
    ubuntu:14.04
    ontouchstart/texlive-full
    ipython/ipython
    hlapp/rpopgen
    bioconductor/release_sequencing

# Enter the continaer and run commands interactively.
mymacbook:~/2015-02-01-myproject> oswitch biolinux
    ###### You are now running: biolinux in container biolinux-7187. ######
biolinux-7187:~/2015-02-01-myproject> abyss-pe k=25 reads.fastq.gz
    [... just works on your files where they are...]
biolinux-7187:~/2015-02-01-myproject> exit
mymacbook:~/2015-02-01-myproject>
    [... output is where you expect it to be ...]


# Use a container non-interactively.
pixel:~/test/ $ oswitch yeban/biolinux blastp -remote -query mygene.fa -db nr > mygene_blastp_nr.txt

Installation

oswitch first requires a working docker installation.

Download and run oswitch

• Download for Linux (64-bit)
• Download for Linux (32-bit)
• Download for macOS

Install oswitch using RubyGems

If you have Ruby 2.0 or higher (available by default on Mac and through package managers on Linux), you can install oswitch systemwide by running the command below:

$ sudo gem install oswitch

Install oswitch using homebrew

If you are on Mac, you can also install oswitch using homebrew:

brew tap homebrew/science
brew install oswitch

This will install oswitch systemwide or only for your user depending on how homebrew is setup.

Currently this installs version 0.2.6 of oswitch.

On Ubuntu

A deb package of oswitch is available in BioLinux repository for Trusty, Vivid and Jessie.

$ sudo add-apt-repository ppa:nebc/bio-linux
$ sudo apt-get update
$ sudo apt-get install oswitch

This will install oswitch systemwide. Please note that the package may not be up to date as the first two options.

Currently this installs version 0.2.6 of oswitch.

Usage note

• Volume mounting on Mac OS hosts is imperfect.
• SELinux must be disabled on CentOS hosts for mounting volumes to work (check the SELinux documentation to see the implications of doing this).
• We have tested oswitch on Debian, Ubuntu, CentOS based docker images on the following hosts:
  • Mac OS X Yosemite, El Captain
  • Ubuntu 14.04.1
  • CentOS 7

FAQ

Q. Directories mounted within container on Mac host are empty.

The problem is, on Mac docker-machine is the real host, not OS X. oswitch can mount only what's available to it from docker-machine. Consider /Applications directory as an example. Run docker-machine ssh default ls /Applications and you will find it empty as well. The workaround is to correctly mount the directories you want inside docker-machine first:

docker-machine stop default
VBoxManage sharedfolder remove default --name Applications
VBoxManage sharedfolder add default --name Applications --hostpath /Applications --automount
docker-machine start default
docker-machine ssh default "sudo mkdir -p /Applications && sudo mount -t vboxsf -o uid=1000,gid=50 Applications /Applications"

Q. cwd is empty in the container

This means the said directory was not mounted by oswitch, or was incorrectly mounted. On Linux host, directories that can conflict with paths within container are not mounted. On Mac, docker-machine can get in the way.

Please report this on our issue tracker. To help us debug, please include:

1. the directory in question
2. the operating system you are running

Q. oswitch does not work with my docker image

Please report this on our issue tracker with oswitch's output. If the image you are using is not available via docker hub or another public repository, please include the Dockerfile as well.

Q. How does all this work?

We create a new image on the fly that inherits from the given image. While creating the new image we execute a shell script that installs packages required for oswitch to work and creates a user in the image (almost) identical to that on the host.

Q. How can I connect to an existing container?

In another shell, use docker ps to see which containers are already running. Copy the identifier from the CONTAINER ID (column this looks something like 37e4e6ada6a4), and use it to run docker attach 37e4e6ada6a4 (replace with your container's id). This will create a new ssh connection to your existing container.

Contribute

$ git clone https://github.com/yeban/oswitch
$ cd oswitch
$ gem install bundler && bundle
$ bundle exec bin/oswitch biolinux

Contributors & Funding

• Anurag Priyam - a.priyam@qmul.ac.uk | @yeban
• Bruno Vieira (@bmpvieira)
• Saurabh Kumar
• Richard Nichols - http://www.sbcs.qmul.ac.uk/staff/richardnichols.html | @qmwugbt112
• Yannick Wurm - http://wurmlab.github.io | @yannick__

---

Development funded as part of NERC Environmental Omics (EOS) Cloud at
Wurm Lab, Queen Mary University of London.