Creating custom python backend environment

[arshia-shakudo]

After creating a the triton custom environment using the steps listed below we get:

triton_python_backend_stub: ../nptl/pthread_mutex_lock.c:81: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.

Steps we took to create custom triton environment

1. First we installed conda and created and environment called python-3-8
2. We then installed numpy as required in the documentation along with our custom packages listed below:

apt-get update
apt-get -y install libgl1
apt-get install -yq libgtk2.0-dev
pip3 install torch==1.9.1+cpu torchvision==0.10.1+cpu torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cpu/torch1.9/index.html
pip install opencv-python==4.5.2.52

We also set the PYTHONNOUSERSITE environment variables as requested:
export PYTHONNOUSERSITE=True

3. We installed CMAKE and cuda-toolkit-11

install cmake and cuda

apt-get update
apt-get install -y apt-transport-https ca-certificates gnupg software-properties-common wget
wget -O -https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | apt-key add -

apt-add-repository 'debhttps://apt.kitware.com/ubuntu/ bionic main'
apt-get update
apt-get install -y cmake
##install rapidjson and libarchive
apt-get install -y rapidjson-dev libarchive-dev

apt-key adv --fetch-keys  http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

bash -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 /" > /etc/apt/sources.list.d/cuda.list'

apt update
apt install cuda-toolkit-11-0

4. We set the CUDA directory on our environment to the correct path:
   cmake -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.0 ..

5. we make the directory and create the custom backend stub

git clone https://github.com/triton-inference-server/python_backend -b r21.09
cd r21.09
mkdir build && cd build
cmake -DTRITON_ENABLE_GPU=ON -DCMAKE_INSTALL_PREFIX:PATH=pwd/install ..
make triton-python-backend-stub

6. We then uploaded the triton_backend_stub file to the model directory on our GCS bucket.
7. after this we used conda-pack to pack out environment and then uploaded it to our detectron2 model directory on our GCS bucket

Triton Information
Triton Version 21.09
Model: Detectron2

[arshia-shakudo]

@Tabrizian This is the complete steps we took to create out environment

[Tabrizian]

@arshiamalek Thanks for filing the ticket. I have started looking into this ticket and we'll update you as soon as more information is available.

[Tabrizian]

@arshiamalek Can you share your model.py and config.pbtxt file too?

[Tabrizian]

A couple of issues regarding creating execution environments for your model:

1. Looks like you are trying to compile the Python backend stub in Ubuntu 16.04 and running it in 20.04. Different versions of linux distributions use different versions of libraries which may not be compatible with each other (e.g. glibc and many others). I think this is the root cause for the error that you are seeing.

2. It is not required to compile the Python backend stub when the version of your Python environment matches what is available in Triton (which is 3.8). You only need to compile the stub, if the version of Python that you want to use is different from the version of Python that is shipped in Triton.

3. conda-pack can only capture your Python dependencies and may not be able to capture all the other dependencies installed using apt-get. Using conda to install those dependencies, can help you make them portable to other environments. I would also recommend that you create the conda-pack environment inside the official Triton NGC containers.

[Tabrizian]

Closing due to in-activity. Feel free to re-open if the issue persists.