[feature] Allow jobs to be scheduled on AWS Fargate

[yuhuishi-convect]

Feature Area

/area backend

What feature would you like to see?

I am trying to run a large number of kubeflow pipeline jobs on AWS Fargate.

The kubeflow pipeline components are deployed on AWS EKS. While the EKS has a Fargate profile that allows scheduling pods onto virtual nodes, Kubeflow pipeline jobs contain privileged containers that prevent them from using Fargate machine resources (https://docs.aws.amazon.com/eks/latest/userguide/fargate.html).

What is the use case or pain point?

This feature enables more cost-efficient job scheduling since many jobs (e.g., hyperparameter tuning, scenario analysis ...) are ephermal, so scheduling them on a serverless machine pool such as provided by Fargate makes more sense. This avoids the need to reserve a pool of nodes upfront while supporting the burst type of workloads.

However, kubeflow pipeline jobs use privileged containers that are not supported by Fargate. For example, the wait container

  containers:
    - name: wait
      image: 'gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance'
      command:
        - argoexec
        - wait
      env:
        - name: ARGO_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
          value: pns
      resources: {}
      volumeMounts:
        - name: podmetadata
          mountPath: /argo/podmetadata
        - name: mlpipeline-minio-artifact
          readOnly: true
          mountPath: /argo/secret/mlpipeline-minio-artifact
        - name: input-artifacts
          mountPath: /mainctrfs/tmp/inputs/config/data
          subPath: config
        - name: input-artifacts
          mountPath: /mainctrfs/tmp/inputs/data/data
          subPath: convect-prepare-data-out_path
        - name: pipeline-runner-token-j2fm7
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        capabilities:
          add:
            - SYS_PTRACE

needs further configurations under securityContext.

I am wondering if there are any workarounds or better solutions to make the jobs schedulable on serverless resource pools such as Fargate.

Is there a workaround currently?

I do not see any solutions so far.

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[Bobgy]

@yuhuishi-convect we might want to switch to argo v3 emissary executor: https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary
which doesn't require privileged permission.

[Bobgy]

This can be resolved by #1654, when we switch to that executoor.

[yuhuishi-convect]

I got a walkaround under version 1.2 to allow scheduling jobs onto Fargate nodes. Here are the things I did:

1. Switch the Argo workflow executor to k8sapi.

kubectl edit cm workflow-controller-configmap  -n kubeflow

and change containerRuntimeExecutor from pns to k8sapi

2. Modify the components to use emptyDir as the output location.
   For example, I have a following helper function

def mount_empty_dir(task: kfp.dsl.ContainerOp) -> kfp.dsl.ContainerOp:
  from kubernetes import client as k8s_client
  task = task.add_volume(
    k8s_client.V1Volume(
      empty_dir={},
      name="output-empty-dir"
    )
  )

  task.container.add_volume_mount(
    k8s_client.V1VolumeMount(
      mount_path="/tmp/outputs",
      name="output-empty-dir"
    )
  )

  return task

Then apply the transformation to every op in the pipeline

pipeline_conf.add_op_transformer(
     mount_empty_dir
)

3. Hint an op can be scheduled on Fargate (this is specific to your Fargate settings). For my case, I am using the rule

Any pod that has a label fargate-schedulable=true under kubeflow namespace can be put on Fargate.

So in the pipeline

task.add_pod_label("fargate-schedulable", "true")

will hint the task can be scheduled on Fargate.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bobgy]

With KFP 1.7.0-rc.2, we support using argo emissary executor and it should be able to run on fargate.

(I verified it works on GKE autopilot)

[yuhuishi-convect]

With KFP 1.7.0-rc.2, we support using argo emissary executor and it should be able to run on fargate.

(I verified it works on GKE autopilot)

Thanks for the update @Bobgy. Will this executor mode be enabled by default under 1.17 or editing workflow-controller-configmap is still needed?

kubectl edit cm workflow-controller-configmap  -n kubeflow

[Bobgy]

1.7.0-rc.2 default to emissary, but I am reverting that.
I will add a separate manifest env for emissary.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rimolive]

Closing this issue. This seems resolved, but if it's not please open another issue.

/close

[google-oss-prow]

@rimolive: Closing this issue.

In response to this:

Closing this issue. This seems resolved, but if it's not please open another issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.