Parallel Data Analysis with Dask

Materials for the Dask tutorial at PyCon 2018.

First Time Setup

If you don't have git installed, you can download a ZIP copy of the repository using the green button above. Note that the file will be called dask-tutorial-pycon-2018-master, instead of dask-tutorial-pycon-2018. Adjust the commands below accordingly.

Install Miniconda or ensure you have Python 3.6 installed on your system.

# Update conda
conda update conda

# Clone the repository. Or download the ZIP and add `-master` to the name.
git clone https://github.com/TomAugspurger/dask-tutorial-pycon-2018

# Enter the repository
cd dask-tutorial-pycon-2018

# Create the environment
conda env create

# Activate the environment
conda activate dask-pycon

# Download data
python prep_data.py

# Start jupyterlab
jupyter lab

If you aren't using conda

# Clone the repository. Or download the ZIP and add `-master` to the name.
git clone https://github.com/TomAugspurger/dask-tutorial-pycon-2018

# Enter the repository
cd dsak-tutorial-pycon-2018

# Create a virtualenv
python3 -m venv .env

# Activate the env
# See https://docs.python.org/3/library/venv.html#creating-virtual-environments
# For bash it's
source .env/bin/activate

# Install the dependencies
python -m pip install -r requirements.txt

# Download data
python prep_data.py

# Start jupyterlab
jupyter lab

Connect to the Cluster

We have a pangeo deployment running that'll provide everyone with their own cluster to try out Dask on some larger problems. You can log into the cluster by going to: