Skip to content

pipekit/argo-workflows

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

slack CI CII Best Practices Artifact HUB Twitter Follow

What is Argo Workflows?

Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).

  • Define workflows where each step in the workflow is a container.
  • Model multi-step workflows as a sequence of tasks or capture the dependencies between tasks using a directed acyclic graph (DAG).
  • Easily run compute intensive jobs for machine learning or data processing in a fraction of the time using Argo Workflows on Kubernetes.

Argo is a Cloud Native Computing Foundation (CNCF) hosted project.

Argo Workflows in 5 minutes

Use Cases

Why Argo Workflows?

  • Argo Workflows is the most popular workflow execution engine for Kubernetes.
  • It can run 1000s of workflows a day, each with 1000s of concurrent tasks.
  • Our users say it is lighter-weight, faster, more powerful, and easier to use
  • Designed from the ground up for containers without the overhead and limitations of legacy VM and server-based environments.
  • Cloud agnostic and can run on any Kubernetes cluster.

Read what people said in our latest survey

Try Argo Workflows

Access the demo environment (login using Github)

Screenshot

Try Argo using Nix

Nix is a package manager / build tool which focusses on reproducible build environments. Argo Workflows has some basic support for Nix which is enough to get Argo Workflows up and running with minimal effort. Here are the steps to follow:

  1. Modify the hosts file according to this, don't worry about the other instructions.
  2. Setup a k8s cluster, k3d is the recommended solution here.
  3. Install Nix.
  4. Make a directory in the argo workflows folder called ".devenv".
  5. Run "nix develop --extra-experimental-features nix-command --extra-experimental-features flakes" (you can add this as a default in your nix.conf file).
  6. Run "devenv up".

As I said, this is still fairly barebones at the moment, any feature in the Makefile not mentioned here is excluded at the moment. In practice this means only a "make start UI=true" equivalent is supported at the moment, there are no LDFlags set in the build, so the UI will show something similar to "0.0.0-unknown"

Documentation

View the docs

Ecosystem

Just some of the projects that use or rely on Argo Workflows:

Client Libraries

Check out our Java, Golang and Python clients.

Quickstart

The following commands install Argo Workflows as well as some commmonly used components:

kubectl create ns argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/master/manifests/quick-start-postgres.yaml

These manifests are intended to help you get started quickly. They contain hard-coded passwords that are publicly available and are not suitable in production.

Who uses Argo Workflows?

Official Argo Workflows user list

Documentation

Features

  • UI to visualize and manage Workflows
  • Artifact support (S3, Artifactory, Alibaba Cloud OSS, Azure Blob Storage, HTTP, Git, GCS, raw)
  • Workflow templating to store commonly used Workflows in the cluster
  • Archiving Workflows after executing for later access
  • Scheduled workflows using cron
  • Server interface with REST API (HTTP and GRPC)
  • DAG or Steps based declaration of workflows
  • Step level input & outputs (artifacts/parameters)
  • Loops
  • Parameterization
  • Conditionals
  • Timeouts (step & workflow level)
  • Retry (step & workflow level)
  • Resubmit (memoized)
  • Suspend & Resume
  • Cancellation
  • K8s resource orchestration
  • Exit Hooks (notifications, cleanup)
  • Garbage collection of completed workflow
  • Scheduling (affinity/tolerations/node selectors)
  • Volumes (ephemeral/existing)
  • Parallelism limits
  • Daemoned steps
  • DinD (docker-in-docker)
  • Script steps
  • Event emission
  • Prometheus metrics
  • Multiple executors
  • Multiple pod and workflow garbage collection strategies
  • Automatically calculated resource usage per step
  • Java/Golang/Python SDKs
  • Pod Disruption Budget support
  • Single-sign on (OAuth2/OIDC)
  • Webhook triggering
  • CLI
  • Out-of-the box and custom Prometheus metrics
  • Windows container support
  • Embedded widgets
  • Multiplex log viewer

Community Meetings

We host monthly community meetings where we and the community showcase demos and discuss the current and future state of the project. Feel free to join us! For Community Meeting information, minutes and recordings please see here.

Participation in the Argo Workflows project is governed by the CNCF Code of Conduct

Community Blogs and Presentations

Project Resources

Security

See SECURITY.md.

Packages

No packages published

Languages

  • Go 80.3%
  • TypeScript 15.1%
  • Nix 2.3%
  • Makefile 0.8%
  • SCSS 0.7%
  • Shell 0.3%
  • Other 0.5%