Support conda packages & script.sh for system dependencies

docs/deployments/python-packages.md

@@ -1,10 +1,10 @@
-# Python packages
+# Python/Conda packages
 
 _WARNING: you are on the master branch, please refer to the docs on the branch that matches your `cortex version`_
 
 ## PyPI packages
 
-You can install your required PyPI packages and import them in your Python files. Cortex looks for a `requirements.txt` file in the top level Cortex project directory (i.e. the directory which contains `cortex.yaml`):
+You can install your required PyPI packages and import them in your Python files using pip. Cortex looks for a `requirements.txt` file in the top level Cortex project directory (i.e. the directory which contains `cortex.yaml`):
 
 ```text
 ./iris-classifier/
@@ -14,119 +14,68 @@ You can install your required PyPI packages and import them in your Python files
 └── requirements.txt
 ```
 
-Note that some packages are pre-installed by default (see "pre-installed packages" for your Predictor type in the [Predictor documentation](predictors.md)).
+If you want to use `conda` to install your python packages, see the [Conda section](#conda) below.
 
-## `setup.py`
+Note that some packages are pre-installed by default (see "pre-installed packages" for your Predictor type in the [Predictor documentation](predictors.md)).
 
-It is also possible to reference Python libraries that are packaged using `setup.py`.
+## Installing with Setup
 
-Here is an example directory structure:
+Python packages can also be installed by providing a `setup.py` that describes your project's modules. Here's an example directory structure:
 
 ```text
 ./iris-classifier/
 ├── cortex.yaml
 ├── predictor.py
 ├── ...
 ├── mypkg
-│   └── __init__.py
+│   └── __init__.py
 ├── requirements.txt
 └── setup.py
 ```
 
-In this case, `requirements.txt` can include a single `.`:
-
-```python
+In this case, `requirements.txt` will have this form:
+```text
 # requirements.txt
 
 .
 ```
 
-If this is the contents `setup.py`:
+## Installing from GitHub
 
-```python
-# setup.py
+You can also install public/private packages from git registries (such as GitHub) by adding them to `requirements.txt`. Here's an example for GitHub:
 
-from distutils.core import setup
-
-setup(
-    name="mypkg",
-    version="0.0.1",
-    packages=["mypkg"],
-)
-```
-
-And `__init__.py` looks like this:
+```text
+# requirements.txt
 
-```python
-# mypkg/__init__.py
+# public access
+git+https://github.com/<username>/<repo name>.git@<tag or branch name>#egg=<package name>
 
-def hello():
-    print("hello")
+# private access
+git+https://<personal access token>@github.com/<username>/<repo name>.git@<tag or branch name>#egg=<package name>
 ```
 
-You can reference your package in `predictor.py`:
+On GitHub, you can generate a personal access token by following [these steps](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line).
 
-```python
-# predictor.py
+## Conda
 
-import mypkg
+Cortex supports installing Conda packages. We recommend only using Conda when your required packages are not available in PyPI. Cortex looks for a `conda-packages.txt` file in the top level Cortex project directory (i.e. the directory which contains `cortex.yaml`):
 
-class PythonPredictor:
-    def predict(self, payload):
-        mypkg.hello()
+```text
+./iris-classifier/
+├── cortex.yaml
+├── predictor.py
+├── ...
+└── conda-packages.txt
 ```
 
-## Private packages on GitHub
-
-You can also install private packages hosed on GitHub by adding them to `requirements.txt` using this syntax:
+The `conda-packages.txt` file follows the format of `conda list --export`. Each line of `conda-packages.txt` should follow this pattern: `[channel::]package[=version[=buildid]]`.
 
+Here's an example of `conda-packages.txt`:
 ```text
-# requirements.txt
-
-git+https://<personal access token>@github.com/<username>/<repo name>.git@<tag or branch name>#egg=<package name>
+conda-forge::rdkit
+conda-forge::pygpu
 ```
 
-You can generate a personal access token by following [these steps](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line).
-
-## Conda packages
-
-You can install Conda packages by creating a custom Docker image that first installs Conda and then installs your Conda packages.
-
-Customize the template Dockerfile below with your desired Conda packages and follow these [instructions](./system-packages.md) to build and push your image to a container registry and configure Cortex to use your custom image.
-
-```dockerfile
-# Dockerfile
-
-FROM <BASE CORTEX IMAGE>
+In situations where both `requirements.txt` and `conda-packages.txt` are provided, Cortex installs Conda packages in `conda-packages.txt` followed by PyPI packages in `requirements.txt`. Conda and Pip package managers install packages and dependencies independently. You may run into situations where Conda and pip package managers install different versions of the same package because they install and resolve dependencies independently from one another. To resolve package version conflicts, it may be in your best interest to specify their exact versions in `conda-packages.txt`.
 
-# remove system-wide packages from the base image
-RUN pip freeze > req && for pkg in "$(cat req)"; do pip uninstall $pkg -y || true; done && rm req
-
-# add conda to path
-ENV PATH /opt/conda/bin:$PATH
-
-# install conda, it also includes py3.6.9
-RUN curl https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh --output ~/miniconda.sh && \
-    /bin/bash ~/miniconda.sh -b -p /opt/conda && \
-    rm ~/miniconda.sh && \
-    /opt/conda/bin/conda update conda && \
-    /opt/conda/bin/conda install --force python=3.6.9 && \
-    /opt/conda/bin/conda clean -tipsy && \
-    ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
-    echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
-    echo "conda activate base" >> ~/.bashrc
-
-# install pip dependencies
-RUN pip install --upgrade pip && \
-    pip install --no-cache-dir -r /src/cortex/lib/requirements.txt && \
-    pip install --no-cache-dir -r /src/cortex/serve/requirements.txt
-
-# ---------------------------------------------------------- #
-# Install your Conda packages here
-# RUN conda install --no-update-deps -c conda-forge rdkit
-
-# ---------------------------------------------------------- #
-
-# replace system python with conda's version
-RUN sed -i 's/\/usr\/bin\/python3.6/\/opt\/conda\/bin\/python/g' /src/cortex/serve/run.sh
-```
+Check the [best practices](https://www.anaconda.com/using-pip-in-a-conda-environment/) on using `pip` inside `conda`.

docs/deployments/system-packages.md

@@ -2,9 +2,39 @@
 
 _WARNING: you are on the master branch, please refer to the docs on the branch that matches your `cortex version`_
 
-Cortex uses Docker images to deploy your models. These images can be replaced with custom images that you can augment with your system packages and libraries. You will need to push your custom images to a container registry that your cluster has access to (e.g. [Docker Hub](https://hub.docker.com) or [AWS ECR](https://aws.amazon.com/ecr)).
+## Bash script
 
-## Create a custom image
+Cortex looks inside the root directory of the project for a file named `dependencies.sh`. (i.e. the directory which contains `cortex.yaml`).
+
+```text
+./iris-classifier/
+├── cortex.yaml
+├── predictor.py
+├── ...
+└── dependencies.sh
+```
+
+This `dependencies.sh` gets executed during the initialization of each replica. Typical use cases include installing required system packages to be used in Predictor, building python packages from source, etc.
+
+Sample `dependencies.sh` installing `tree` utility:
+```bash
+#!/bin/bash
+apt-get update && apt-get install -y tree
+```
+
+The `tree` utility can now be called inside your `predictor.py`:
+
+```python
+# predictor.py
+import subprocess
+
+class PythonPredictor:
+    def __init__(self, config):
+        subprocess.run(["tree"])
+    ...
+```
+
+## Custom Docker image
 
 Create a Dockerfile to build your custom image:
 
@@ -17,11 +47,11 @@ The Docker images used to deploy your models are listed below. Based on the Cort
 ### Base Cortex images for model serving
 
 <!-- CORTEX_VERSION_BRANCH_STABLE x5 -->
-* Python (CPU): cortexlabs/python-serve:master
-* Python (GPU): cortexlabs/python-serve-gpu:master
-* TensorFlow (CPU or GPU): cortexlabs/tf-api:master
-* ONNX (CPU): cortexlabs/onnx-serve:master
-* ONNX (GPU): cortexlabs/onnx-serve-gpu:master
+* Python (CPU): `cortexlabs/python-serve:master`
+* Python (GPU): `cortexlabs/python-serve-gpu:master`
+* TensorFlow (CPU or GPU): `cortexlabs/tf-api:master`
+* ONNX (CPU): `cortexlabs/onnx-serve:master`
+* ONNX (GPU): `cortexlabs/onnx-serve-gpu:master`
 
 Note that the Docker image version must match your cluster version displayed in `cortex version`.
 
@@ -38,7 +68,7 @@ RUN apt-get update \
     && apt-get clean && rm -rf /var/lib/apt/lists/*
 ```
 
-## Build and push to a container registry
+### Build and push to a container registry
 
 Create a repository to store your image:
 
@@ -62,7 +92,7 @@ docker build . -t org/my-api:latest -t <repository_url>:latest
 docker push <repository_url>:latest
 ```
 
-## Configure Cortex
+### Configure Cortex
 
 Update your cluster configuration file to point to your image:
 
@@ -79,18 +109,3 @@ Update your cluster for the change to take effect:
 ```bash
 cortex cluster update --config=cluster.yaml
 ```
-
-## Use system packages in workloads
-
-Cortex will use your custom image to launch workloads and you will have access to any packages you added:
-
-```python
-# predictor.py
-
-import subprocess
-
-class PythonPredictor:
-    def __init__(self, config):
-        subprocess.run(["tree"])
-        ...
-```

images/onnx-serve-gpu/Dockerfile

@@ -11,33 +11,42 @@ RUN apt-get update -qq && apt-get install -y -q \
         software-properties-common \
         unzip \
         zlib1g-dev \
-        python3.6-dev \
-        python3.6-distutils \
         git \
         libsm6 \
         libxext6 \
         libxrender-dev \
         libsndfile1 \
         locales \
-    && apt-get clean -qq && rm -rf /var/lib/apt/lists/* && \
-    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
-    python3.6 get-pip.py && \
-    pip install --upgrade pip && \
-    rm -rf /root/.cache/pip* && \
-    ln -s /usr/bin/python3.6 /usr/local/bin/python
+    && apt-get clean -qq && rm -rf /var/lib/apt/lists/*
 
 RUN locale-gen en_US.UTF-8
 ENV LANG=en_US.UTF-8 LANGUAGE=en_US.en LC_ALL=en_US.UTF-8
 
-ENV PYTHONPATH "${PYTHONPATH}:/src:/mnt/project"
+ENV PYTHONPATH="${PYTHONPATH}:/src:/mnt/project"
+ENV PATH=/opt/conda/bin:$PATH
+ENV PYTHONVERSION=3.6.9
+
+# conda needs an untainted base environment to function properly
+# that's why a new separate conda environment is created
+RUN curl "https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh" --output ~/miniconda.sh && \
+    /bin/bash ~/miniconda.sh -b -p /opt/conda && \
+    rm -rf ~/.cache ~/miniconda.sh
+
+# split the conda installations because the dev boxes have limited memory
+RUN /opt/conda/bin/conda create -n env -c conda-forge python=$PYTHONVERSION pip && \
+    /opt/conda/bin/conda clean -a && \
+    ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
+    echo ". /opt/conda/etc/profile.d/conda.sh" > ~/.env && \
+    echo "conda activate env" >> ~/.env && \
+    echo "source ~/.env" >> ~/.bashrc
+
+ENV BASH_ENV=~/.env
 
 COPY pkg/workloads/cortex/lib/requirements.txt /src/cortex/lib/requirements.txt
 COPY pkg/workloads/cortex/serve/onnx-gpu.requirements.txt /src/cortex/serve/requirements.txt
 
-RUN pip install --upgrade pip && \
-    pip install --no-cache-dir -r /src/cortex/lib/requirements.txt && \
-    pip install --no-cache-dir -r /src/cortex/serve/requirements.txt && \
-    rm -rf /root/.cache/pip*
+RUN /bin/bash -c \
+    "pip install --no-cache-dir -r /src/cortex/lib/requirements.txt -r /src/cortex/serve/requirements.txt"
 
 COPY pkg/workloads/cortex/consts.py /src/cortex
 COPY pkg/workloads/cortex/lib /src/cortex/lib

images/onnx-serve/Dockerfile

@@ -11,33 +11,42 @@ RUN apt-get update -qq && apt-get install -y -q \
         software-properties-common \
         unzip \
         zlib1g-dev \
-        python3.6-dev \
-        python3.6-distutils \
         git \
         libsm6 \
         libxext6 \
         libxrender-dev \
         libsndfile1 \
         locales \
-    && apt-get clean -qq && rm -rf /var/lib/apt/lists/* && \
-    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
-    python3.6 get-pip.py && \
-    pip install --upgrade pip && \
-    rm -rf /root/.cache/pip* && \
-    ln -s /usr/bin/python3.6 /usr/local/bin/python
+    && apt-get clean -qq && rm -rf /var/lib/apt/lists/*
 
 RUN locale-gen en_US.UTF-8
 ENV LANG=en_US.UTF-8 LANGUAGE=en_US.en LC_ALL=en_US.UTF-8
 
-ENV PYTHONPATH "${PYTHONPATH}:/src:/mnt/project"
+ENV PYTHONPATH="${PYTHONPATH}:/src:/mnt/project"
+ENV PATH=/opt/conda/bin:$PATH
+ENV PYTHONVERSION=3.6.9
+
+# conda needs an untainted base environment to function properly
+# that's why a new separate conda environment is created
+RUN curl "https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh" --output ~/miniconda.sh && \
+    /bin/bash ~/miniconda.sh -b -p /opt/conda && \
+    rm -rf ~/.cache ~/miniconda.sh
+
+# split the conda installations because the dev boxes have limited memory
+RUN /opt/conda/bin/conda create -n env -c conda-forge python=$PYTHONVERSION pip && \
+    /opt/conda/bin/conda clean -a && \
+    ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
+    echo ". /opt/conda/etc/profile.d/conda.sh" > ~/.env && \
+    echo "conda activate env" >> ~/.env && \
+    echo "source ~/.env" >> ~/.bashrc
+
+ENV BASH_ENV=~/.env
 
 COPY pkg/workloads/cortex/lib/requirements.txt /src/cortex/lib/requirements.txt
 COPY pkg/workloads/cortex/serve/onnx-cpu.requirements.txt /src/cortex/serve/requirements.txt
 
-RUN pip install --upgrade pip && \
-    pip install --no-cache-dir -r /src/cortex/lib/requirements.txt && \
-    pip install --no-cache-dir -r /src/cortex/serve/requirements.txt && \
-    rm -rf /root/.cache/pip*
+RUN /bin/bash -c \
+    "pip install --no-cache-dir -r /src/cortex/lib/requirements.txt -r /src/cortex/serve/requirements.txt"
 
 COPY pkg/workloads/cortex/consts.py /src/cortex
 COPY pkg/workloads/cortex/lib /src/cortex/lib