Skip to content

Clarifying DockerSettings Usage #3558

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/book/getting-started/zenml-pro/self-hosted.md
Original file line number Diff line number Diff line change
@@ -782,6 +782,7 @@ If the TLS certificates used by the ZenML Pro services are signed by a custom Ce

docker_settings = DockerSettings(
parent_image=CUSTOM_BASE_IMAGE,
skip_build=True
)

@pipeline(settings={"docker": docker_settings})
Original file line number Diff line number Diff line change
@@ -74,6 +74,29 @@ steps:

Check out [this page](../pipeline-development/use-configuration-files/configuration-hierarchy.md) for more information on the hierarchy and precedence of the various ways in which you can supply the settings.

### DockerSettings Class Overview

The `DockerSettings` class in ZenML is designed to configure Docker-related settings for ZenML pipelines. It allows users to specify various options for building and using Docker images, including setting a parent image, Dockerfile, and build configurations.

#### Key Attributes
- **parent_image**: This attribute specifies the full name of the Docker image to be used as the parent for the image that will be built. It is specific to the `DockerSettings` class and not available in `BaseSettings`. By default, it uses a ZenML image built for the active Python and ZenML version.

#### Example Usage
To correctly use `DockerSettings`, you can specify a parent image and other configurations as shown below:

```python
from zenml.config import DockerSettings

docker_settings = DockerSettings(
parent_image="my_registry.io/image_name:tag",
skip_build=True
)

@pipeline(settings={"docker": docker_settings})
def my_pipeline(...):
...
```

### Specifying Docker build options

If you want to specify build options that get passed to the build method of the [image builder](../pipeline-development/configure-python-environments/README.md#image-builder-environment). For the default local image builder, these options get passed to the [`docker build` command](https://docker-py.readthedocs.io/en/stable/images.html#docker.models.images.ImageCollection.build).
Original file line number Diff line number Diff line change
@@ -6,6 +6,12 @@ description: You have the option to customize the Docker settings at a step leve

By default every step of a pipeline uses the same Docker image that is defined at the [pipeline level](./docker-settings-on-a-pipeline.md). Sometimes your steps will have special requirements that make it necessary to define a different Docker image for one or many steps. This can easily be accomplished by adding the [DockerSettings](https://sdkdocs.zenml.io/latest/core_code_docs/core-config.html#zenml.config.docker_settings) to the step decorator directly.

## DockerSettings Class

The `DockerSettings` class is used to configure Docker-related settings for ZenML pipelines. It allows you to specify a parent Docker image and other configurations necessary for building Docker images to run ZenML pipelines. The `parent_image` attribute is specific to `DockerSettings` and not available in `BaseSettings`.

### Example Usage

```python
from zenml import step
from zenml.config import DockerSettings
17 changes: 15 additions & 2 deletions docs/book/how-to/customize-docker-builds/use-a-prebuilt-image.md
Original file line number Diff line number Diff line change
@@ -14,9 +14,22 @@ Note that using this feature means that you won't be able to leverage any update

## How do you use this feature

The [`DockerSettings`](./docker-settings-on-a-pipeline.md#specify-docker-settings-for-a-pipeline) class in ZenML allows you to set a parent image to be used in your pipeline runs and gives the ability to skip building an image on top of it.
The `DockerSettings` class in ZenML is specifically designed for configuring Docker-related settings in ZenML pipelines. It allows you to set a parent image to be used in your pipeline runs and gives the ability to skip building an image on top of it. Unlike `BaseSettings`, `DockerSettings` includes attributes such as `parent_image` that are essential for Docker configurations.

To do this, just set the `parent_image` attribute of the `DockerSettings` class to the image you want to use and set `skip_build` to `True`.
To do this, instantiate the `DockerSettings` class and set the `parent_image` attribute to the image you want to use. Additionally, set `skip_build` to `True` if you wish to use the parent image directly without building on top of it. Here is an example:

```python
from zenml.config import DockerSettings

docker_settings = DockerSettings(
parent_image="my_registry.io/image_name:tag",
skip_build=True
)

@pipeline(settings={"docker": docker_settings})
def my_pipeline(...):
...
```

```python
docker_settings = DockerSettings(
Original file line number Diff line number Diff line change
@@ -387,6 +387,25 @@ def my_step() -> None:
...
```

### Understanding `DockerSettings`

The `DockerSettings` class is used to configure Docker-related settings for ZenML pipelines. It allows you to specify various options for building Docker images, including the `parent_image` attribute, which defines the base Docker image to use.

- **parent_image**: This attribute is specific to `DockerSettings` and not available in `BaseSettings`. It specifies the full name of the Docker image to be used as the parent for the image that will be built. If you set `skip_build=True`, you must provide a `parent_image`.

Example usage:

```python
from zenml.config import DockerSettings

docker_settings = DockerSettings(
parent_image="my_registry.io/image_name:tag",
skip_build=True
)
```

This example demonstrates how to instantiate `DockerSettings` with a specified `parent_image` and `skip_build` option.

With this change, all stack components (e.g. Orchestrators and Step Operators) that accepted a `docker_parent_image` as part of its Stack Configuration should now pass it through the `DockerSettings` object.

Read more [here](https://docs.zenml.io//how-to/customize-docker-builds/docker-settings-on-a-pipeline).
Original file line number Diff line number Diff line change
@@ -56,7 +56,7 @@ All steps running on GPU-backed hardware will be executed within a containerized

#### 1. **Specify a CUDA-enabled parent image in your `DockerSettings`**

For complete details, refer to the [containerization page](https://docs.zenml.io//how-to/customize-docker-builds) that explains how to do this. As an example, if you want to use the latest CUDA-enabled official PyTorch image for your entire pipeline run, you can include the following code:
The `DockerSettings` class is used to configure Docker-related settings for ZenML pipelines, including specifying a parent Docker image. The `parent_image` attribute is specific to `DockerSettings` and not available in `BaseSettings`. For complete details, refer to the [containerization page](https://docs.zenml.io//how-to/customize-docker-builds) that explains how to do this. As an example, if you want to use the latest CUDA-enabled official PyTorch image for your entire pipeline run, you can include the following code:

```python
from zenml import pipeline
Original file line number Diff line number Diff line change
@@ -21,7 +21,15 @@ You will learn about all of the above in more detail later, but for now, let's t
Settings are categorized into two types:

* **General settings** that can be used on all ZenML pipelines. Examples of these are:
* [`DockerSettings`](https://docs.zenml.io//how-to/customize-docker-builds) to specify Docker settings.
* [`DockerSettings`](https://docs.zenml.io//how-to/customize-docker-builds) to specify Docker settings. `DockerSettings` is used to configure Docker-related settings for ZenML pipelines, including specifying a parent Docker image through the `parent_image` attribute. This attribute is specific to `DockerSettings` and not available in `BaseSettings`. Here is an example of how to use `DockerSettings`:
```python
from zenml.config import DockerSettings

docker_settings = DockerSettings(
parent_image="my_registry.io/image_name:tag",
skip_build=True
)
```
* [`ResourceSettings`](https://docs.zenml.io//how-to/pipeline-development/training-with-gpus) to specify resource settings.
* **Stack-component-specific settings**: These can be used to supply runtime configurations to certain stack components (the key should be `<COMPONENT_CATEGORY>` or `<COMPONENT_CATEGORY>.<COMPONENT_FLAVOR>`). Settings for components not in the active stack will be ignored. Examples of these are:
* [`SkypilotAWSOrchestratorSettings`](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-skypilot_aws.html#zenml.integrations.skypilot_aws) to specify Skypilot settings (works for `SkypilotGCPOrchestratorSettings` and `SkypilotAzureOrchestratorSettings` as well).
Original file line number Diff line number Diff line change
@@ -206,6 +206,23 @@ settings:

```

#### Using `DockerSettings`

The `DockerSettings` class is used to configure Docker-related settings for ZenML pipelines. It allows you to specify various options for building and running Docker images.

- **parent_image**: This attribute is specific to `DockerSettings` and is used to specify the full name of the Docker image that should be used as the parent for the image that will be built. If you specify a custom image here, ensure it has ZenML installed.

Example usage:

```python
from zenml.config import DockerSettings

docker_settings = DockerSettings(
parent_image="my_registry.io/image_name:tag",
skip_build=True
)
```

{% hint style="info" %}
Find a complete list of all Docker Settings [here](https://sdkdocs.zenml.io/latest/core_code_docs/core-config.html#zenml.config.docker_settings). To learn more about pipeline containerization consult our documentation on this [here](https://docs.zenml.io//how-to/customize-docker-builds).
{% endhint %}