[Proposal] Argo SOLO

[alexmt]

Is this a BUG REPORT or FEATURE REQUEST?:

FEATURE REQUEST

What happened:

While creating workflow I want to quickly run it locally using minikube or docker kubernetes .

What you expected to happen:

Instead of installing argo locally I would be nice to support argo execute command which connects to local kubernetes and executes workflow. Ideally, it should not require configuring s3 storage for artifacts.

This feature could simplify workflow development/troubleshooting: no need to install and manage local workflow controller. If argo is used for CI, then developers could build project locally without dependency on centralized argo instance.

[alexmt]

@jessesuen what do you think?

[mumoshu]

@alexmt Hi!

I'm interested in this feature, too.
Would you mind clarifying a bit more about your use-case and assumptions so that I could find any place to contribute.. 😉

Assuming it takes a reference to a workflow definition, would it be something like a stand-alone version of argo submit with - I guess - probably the facility to tail logs/watch statuses to see the progress of the workflow?

not require configuring s3 storage

I'm not an expert in argo yet but does it imply introducing an another artifact store backed by maybe a host volume mount?

[jessesuen]

I think this is a good idea. It may take quite a bit of refactoring to get it to work. My suggestion would be to name the command argo run similar to docker run and kubectl run

[alexmt]

Hello @mumoshu,

I have two use cases:

• workflow development/ troubleshooting. During workflow development I want to have more control over workflow execution and better access to logs/artifacts.
• use argo as makefile replacement. if I could run build workflow locally and get build artifacts in expected location it would be possible to use it as a main build tool.

As @jessesuen proposed we add command 'argo run':

argo run -f /path/to/workflow.yaml --output ./out - starts executing workflow, stream logs from all steps and saves artifacts into ./out directory. ctrl+c should stop process and workflow execution
argo run --continue <workflow-name> - starts controller and point it to existing workflow. We might focus controller on single workflow using 'instanceid' label.

[mumoshu]

@alexmt Thanks for the clarification.

I have never seen a single workflow engine or a CI system achieved that until now. CircleCI Local CLI is close, supports local builds within an env comparable to the cloud one, but lacks workflow support 😭

That being said, I'm not exactly sure what technical difficulty is implied by this.
But your expected UX seems awesome!

[mirdhyn]

+1 for that . This seems similar to concourse.ci execute feature. See https://concourse-ci.org/running-tasks.html and https://concoursetutorial.com/basics/task-hello-world/

[chriszs]

Or drone exec.

[jared-dtg]

+1 to this proposal -- would be great for debugging and probably lots of over clever situations

[cyclingwithelephants]

Does icebox mean this isn't happening? I'd love to see this feature implemented.

In addition to the use cases listed above, I have another; bootstrapping an environment that is supposed to contain argo. This would enable us to bootstrap argo using argo, or recover easily in some kind of DR scenario

[alexec]

You can kind of do this already:

Install the CRDs:

kubectl apply -f https://raw.githubusercontent.com/argoproj/argo/master/manifests/base/crds/minimal/argoproj.io_workflowtemplates.yaml
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo/master/manifests/base/crds/minimal/argoproj.io_clusterworkflowtemplates.yaml
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo/master/manifests/base/crds/minimal/argoproj.io_cronworkflows.yaml
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo/master/manifests/base/crds/minimal/argoproj.io_workflows.yaml

Create the config map with whatever executor you want and a suitable image to use, e.g.:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:
  containerRuntimeExecutor: pns
  executorImage: argoproj/argoexec:latest
EOF

We need to make the default service account admin so the executor can work (note - not safe for production!):

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default

In one tab, run the controller manually:

GO111MODULE=on go get github.com/argoproj/argo/cmd/workflow-controller@stable
workflow-controller

In another tab, create your workflow:

kubectl create -f https://raw.githubusercontent.com/argoproj/argo/master/examples/hello-world.yaml 

Finally, if you want to wait for completion:

kubectl wait wf --for=condition=True --all
kubectl get wf --watch

[cyclingwithelephants]

Oh sure, thanks! It seems obvious now you mention it! Am I right in assuming that you mean to run steps 1 and 2 against a local k8s provider e.g. k8s/minikube/etc?

My problem with that is that this breaks if artifacts are involved, I am assuming that the artifact configuration would exist inside the workflow, since it would be a property of the argo configuration and since we are using GCS as the artifact backend this would require that to already exist.

Of course it's no problem as long as we are constrained by not being able to use artifacts in this particular workflow but we will likely want to be able to use them here in the future, what would you recommend?