Dakkak, A., Li, C., De Gonzalo, S. G., Xiong, J., & Hwu, W-M. W. (2019). TrIMS: Transparent and isolated model sharing for low latency deep learning inference in function-as-a-service. In E. Bertino, C. K. Chang, P. Chen, E. Damiani, M. Goul, & K. Oyama (Eds.), Proceedings - 2019 IEEE International Conference on Cloud Computing, CLOUD 2019 - Part of the 2019 IEEE World Congress on Services (pp. 372-382). [8814494] (IEEE International Conference on Cloud Computing, CLOUD; Vol. 2019-July). IEEE Computer Society. https://doi.org/10.1109/CLOUD.2019.00067
Setup your target location (make sure that LD_LIBRARY_PATH and PATH has been updated to include this)
export UPR_INSTALL_PREFIX=$HOME/.usr
export UPR_BASE_DIR=$HOME/carml/data/mxnet
make sure that both directories exist
mkdir -p $UPR_INSTALL_PREFIX
mkdir -p $UPR_BASE_DIR
Remember to update your PKG_CONFIG_PATH
export PKG_CONFIG_PATH=$UPR_INSTALL_PREFIX/lib/pkgconfig/:$PKG_CONFIG_PATH
wget https://c-ares.haxx.se/download/c-ares-1.13.0.tar.gz
tar -xf c-ares-1.13.0.tar.gz
cd c-ares-1.13.0
./buildconf
./configure --prefix=$UPR_INSTALL_PREFIX --disable-dependency-tracking --disable-debug
make install
wget https://github.com/google/protobuf/archive/v3.5.1.tar.gz
tar -xf v3.5.1.tar.gz
cd protobuf-3.5.1
./autogen.sh
./configure --prefix=$UPR_INSTALL_PREFIX --disable-dependency-tracking --disable-debug --with-zlib
make
make install
wget https://github.com/protobuf-c/protobuf-c/releases/download/v1.3.0/protobuf-c-1.3.0.tar.gz
tar -xf protobuf-c-1.3.0.tar.gz
cd protobuf-c-1.3.0
./configure --prefix=$UPR_INSTALL_PREFIX --disable-dependency-tracking --disable-debug
make install
wget https://github.com/grpc/grpc/archive/v1.9.1.tar.gz
tar -xf v1.9.1
cd v1.9.1
make install prefix=$UPR_INSTALL_PREFIX
make install-plugins prefix=$UPR_INSTALL_PREFIX
wget https://github.com/opencv/opencv/archive/3.4.1.tar.gz
tar -xf 3.4.1.tar.gz
cd opencv-3.4.1/
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$UPR_INSTALL_PREFIX -DWITH_CUDA=OFF -DWITH_OPENCL=OFF ..
make
make install
Other requirements can be installed using APT . The base requirements are listed in the (MXNet installation guide)[https://mxnet.apache.org/install/index.html].
It is not recommended to regenerate the protofile, since it's dependent on the protobuf version. The only time you need to regenerate it is after updating it.
cd src/c_api
make
The server is part of the MXNet build process.
| Name | Description | Default Value |
|---|---|---|
| UPR_ENABLED | true | |
| UPR_PROFILE_IO | only makes sense if UPR_ENABLED=false | true |
| UPR_RUN_ID | [undefined] | |
| git | build_git_sha | |
| UPR_GIT_BRANCH | build_git_branch | |
| UPR_GIT_TIME | build_git_time | |
| UPR_CLIENT | ||
| UPR_BASE_DIR | ||
| UPR_MODEL_NAME | ||
| UPR_PROFILE_TARGET | profile.json | |
| UPR_INITIALIZE_EAGER | false | |
| UPR_INITIALIZE_EAGER_ASYNC | false | |
| UPR_INPUT_CHANNELS | 3 | |
| UPR_INPUT_WIDTH | 224 | |
| UPR_INPUT_HEIGHT | 224 | |
| UPR_INPUT_MEAN_R | 0 | |
| UPR_INPUT_MEAN_G | 0 | |
| UPR_INPUT_MEAN_B | 0 | |
| UPR_ENABLE_MEMORY_PROFILE | false | |
| UPR_ENABLE_CUDA_FREE | false | |
| UPR_SHARING_GRANULARITY | model | |
| -------------------------- | ----------- | ------------- |
| UPRD_EVICTION_POLICY | LRU | |
| UPRD_ESTIMATION_RATE | 1.0 | |
| UPRD_MEMORY_PERCENTAGE | 0.8 | |
| UPRD_PERSIST_CPU | true | |
| UPRD_PERSIST_ONLY_CPU | only persist on cpu memory | false |
| UPRD_WRITE_PROFILE | write server profile file | false |
| UPRD_ESTIMATE_WITH_INTERNAL_MEMORY | use internal memory info for estimate | true |