This is an example Dagster project with the dagster-nomad integration.
-
Install a Nomad cluster following the Nomad documentation.
-
Install Terraform following the Terraform documentation.
-
Install just for launching commands Just documentation.
-
Install Dagster and dagster-nomad in a python virtual environment:
python3.11 -m venv .venv just setup-dev
-
Install Docker compose
Build an image for the RunLauncher to use and adjust the tag in tf/dagster-executor.hcl
docker build . --tag dagster-nomad-example:local
In a terminal run the following
just docker-up
just nomad-up
Create a bucket dagster
in minio
In another terminal run the following.
just terraform-setup
just dagster-dev
Open in your browser dagster-webserver and nomad-ui
Following the convention for the DockerRunLauncher
and K8sRunLauncher
,
the DAGSTER_CURRENT_IMAGE
environment variable can used to specify the Docker image for the NomadRunLauncher
to use.
Build the different images for your code locations.
docker build . --tag dagster-nomad-example:local
Launch the multiple code locations:
DAGSTER_CURRENT_IMAGE=dagster-nomad-example:local just dagster-grpc
Uncomment the load_from.grpc_server
in the workspace.yaml
files
just dagster-dev