Skip to content

PayLead/dagster-nomad

Folders and files

NameName
Last commit message
Last commit date

Latest commit

d17c73e · Oct 17, 2023

History

5 Commits
Aug 23, 2023
Aug 29, 2023
Aug 23, 2023
Aug 23, 2023
Aug 29, 2023
Aug 23, 2023
Aug 23, 2023
Aug 23, 2023
Aug 23, 2023
Aug 29, 2023
Aug 23, 2023
Oct 17, 2023
Oct 17, 2023
Oct 17, 2023
Aug 29, 2023

Repository files navigation

Dagster Nomad

This is an example Dagster project with the dagster-nomad integration.

Setup

  1. Install a Nomad cluster following the Nomad documentation.

  2. Install Terraform following the Terraform documentation.

  3. Install just for launching commands Just documentation.

  4. Install Dagster and dagster-nomad in a python virtual environment:

    python3.11 -m venv .venv
    just setup-dev
  5. Install Docker compose

Run example

Build an image for the RunLauncher to use and adjust the tag in tf/dagster-executor.hcl

docker build . --tag dagster-nomad-example:local

In a terminal run the following

just docker-up
just nomad-up

Create a bucket dagster in minio In another terminal run the following.

just terraform-setup
just dagster-dev

Open in your browser dagster-webserver and nomad-ui

Multiple Code Locations example

Following the convention for the DockerRunLauncher and K8sRunLauncher, the DAGSTER_CURRENT_IMAGE environment variable can used to specify the Docker image for the NomadRunLauncher to use.

Build the different images for your code locations.

docker build . --tag dagster-nomad-example:local

Launch the multiple code locations:

DAGSTER_CURRENT_IMAGE=dagster-nomad-example:local  just dagster-grpc

Uncomment the load_from.grpc_server in the workspace.yaml files

just dagster-dev