Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Add CUDA version notes #5004

Merged
merged 5 commits into from
Oct 8, 2024
Merged

docs: Add CUDA version notes #5004

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0eb1611
Add CUDA notes
Sherlock113 Sep 27, 2024
f9060f0
Fix format
Sherlock113 Sep 27, 2024
2fad204
Update
Sherlock113 Sep 30, 2024
19cde79
Update
Sherlock113 Sep 30, 2024
23b60c1
Fix format
Sherlock113 Sep 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 22 additions & 24 deletions docs/source/guides/build-options.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -399,21 +399,15 @@ The following ``docker`` field contains some basic Docker configurations:
docker:
distro: debian
python_version: "3.11"
cuda_version: "11.6.2"
cuda_version: "11.6.2" # Deprecated
system_packages:
- libblas-dev
- liblapack-dev
- gfortran

.. note::

BentoML uses `BuildKit <https://github.com/moby/buildkit>`_, a cache-efficient builder toolkit, to containerize Bentos.

BuildKit comes with `Docker 18.09 <https://docs.docker.com/develop/develop-images/build_enhancements/>`_. This means
if you are using Docker via Docker Desktop, BuildKit will be available by default. If you are using a standalone version of Docker,
you can install BuildKit by following the instructions `here <https://github.com/docker/buildx#installing>`_.
BentoML uses `BuildKit <https://github.com/moby/buildkit>`_, a cache-efficient builder toolkit, to containerize Bentos. BuildKit comes with `Docker 18.09 <https://docs.docker.com/develop/develop-images/build_enhancements/>`_. This means if you are using Docker via Docker Desktop, BuildKit will be available by default. If you are using a standalone version of Docker, you can install BuildKit by following the instructions `here <https://github.com/docker/buildx#installing>`_.

The following sections provide detailed explanations of available Docker configurations.
The following sections provide detailed explanations of certain Docker configurations.

OS distros
""""""""""
Expand Down Expand Up @@ -519,21 +513,25 @@ Docker options table

The following table provides a full list of available configurations for the ``docker`` field.

+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Description |
+=====================+================================================================================================================================================+
| distro | The OS distribution on the Docker image. It defaults to ``debian``. |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| python_version | The Python version on the Docker image [``3.7``, ``3.8``, ``3.9``, ``3.10``]. It defaults to the Python version in the build environment. |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| system_packages | The system packages that will be installed in the container. |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| setup_script | A Python or Shell script that will be executed during the Docker build process. |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| base_image | A user-provided Docker base image. This will override all other custom attributes of the image. |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| dockerfile_template | Customize the generated Dockerfile by providing a Jinja2 template that extends the default Dockerfile. |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
.. list-table::
:header-rows: 1

* - Field
- Description
* - distro
- The OS distribution on the Docker image. It defaults to ``debian``.
* - python_version
- The Python version on the Docker image. It defaults to the Python version in the build environment.
* - cuda_version
- Deprecated. The CUDA version on the Docker image for running models that require GPUs. When using PyTorch or TensorFlow to run models on GPUs, we recommend you directly install them along with their respective CUDA dependencies, using ``pip``. This means you don't need to configure ``cuda_version`` separately. See :doc:`/guides/gpu-inference` for more information.
* - system_packages
- The system packages that will be installed in the container.
* - setup_script
- A Python or Shell script that will be executed during the Docker build process.
* - base_image
- A user-provided Docker base image. This will override all other custom attributes of the image.
* - dockerfile_template
- Customize the generated Dockerfile by providing a Jinja2 template that extends the default Dockerfile.

Build a Bento
-------------
Expand Down
21 changes: 21 additions & 0 deletions docs/source/guides/gpu-inference.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,27 @@ To make sure a Bento is using GPUs during deployment, configure its required CUD

If the desired CUDA version is not natively supported by BentoML, you can customize the installation of CUDA driver and libraries via ``system_packages``, ``setup_script``, or ``base_image`` options under the :ref:`docker-configuration` field.

When using PyTorch or TensorFlow to run models on GPUs, we recommend you directly install them along with their respective CUDA dependencies, via ``pip``. This ensures:

- **Minimal package size** since only the required components are installed.
- **Better compatibility** as the correct CUDA version is automatically installed alongside the frameworks.

To install PyTorch or TensorFlow with the appropriate CUDA version using ``pip``, use the following commands:

.. code-block:: bash

pip install torch
pip install tensorflow[and-cuda]

This means when building your Bento, you DO NOT need to specify ``cuda_version`` again in your ``bentofile.yaml`` to install the CUDA toolkit separately. Simply add PyTorch and TensorFlow under ``packages`` (or they are in the separate ``requirements.txt`` file).

.. code-block:: yaml

python:
packages:
- torch
- tensorflow[and-cuda]

BentoCloud
^^^^^^^^^^

Expand Down