Configuring the default artifact repository in argo.libsonnet #513

Ark-kun · 2018-12-11T08:03:01Z

This is part of the work to clean-up the explicit artifact location specs from compiled workflows.

Right now the workflows are not portable. Even the namespace and service name:port is hard-coded:

Before:

  - container:
      args:
      - echo exit!
      command:
      - sh
      - -c
      image: python:3.5-jessie
    name: exiting
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact

After:

  - container:
      args:
      - echo exit!
      command:
      - sh
      - -c
      image: python:3.5-jessie
    name: exiting
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json

This change is

Ark-kun · 2018-12-11T08:10:22Z

/assign @IronPan

sdk/python/kfp/compiler/compiler.py

pipeline/pipeline/argo.libsonnet

IronPan · 2018-12-11T18:16:07Z

I don't think assign default artifact store to ml-pipeline specific would be the best option here, since argo is a shared component in the kubeflow cluster, not pipeline specific.

https://github.com/kubeflow/kubeflow/tree/master/kubeflow

IronPan · 2018-12-11T18:18:38Z

pipeline/pipeline/argo.libsonnet

+          executorImage: argoproj/argoexec:v2.2.0
+          artifactRepository:
+            s3:
+              bucket: mlpipeline


I don't think assigning mlpipeline as default for argo is an good option, given argo is not pipeline specific. Kubeflow users could use argo for various purpose.

also, if you make further change to ksonnet config, make sure do it in kubeflow ks registry here first,
https://github.com/kubeflow/kubeflow/tree/master/kubeflow
before making any change to our repo. the ks registry here is only used by bootstrapper image, which is not officially supported anymore.

What if we only add this if there is no existing default argo artifact repo?

LOL. We've been debating whether to add this config for Pipelines-only while someone just went and added this configMap for all Argo users. kubeflow/kubeflow#2238

Yes, all Kubeflow Argo installations since January have default artifact storage which is slightly pipeline-specific =)

Ark-kun · 2018-12-11T18:32:15Z

We can make the names more generic and only install the default repo if there is no existing one.
What do you think?

IronPan · 2018-12-11T21:38:13Z

The minio instance is pipeline specific. the name of the artifact store can be anything but the endpoint will point to pipeline specific minio instance. can't we inject the artifact manifest somewhere else, e.g. backend?

Ark-kun · 2018-12-13T08:22:02Z

The minio instance is pipeline specific. the name of the artifact store can be anything but the endpoint will point to pipeline specific minio instance.

We install both Argo instance and Minio instance. Shouldn't our Argo instance point to our Minio instance?

IronPan · 2018-12-21T05:48:44Z

the argo instance is not pipeline specific unfortunately.

Ark-kun · 2019-04-30T00:38:23Z

This is still relevant.

k8s-ci-robot · 2019-04-30T00:39:21Z

@Ark-kun: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
kubeflow-pipeline-e2e-test	`b2d4281`	link	`/test kubeflow-pipeline-e2e-test`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

IronPan · 2019-05-07T07:01:33Z

As discussed offline, the argo would support per-job configmap in v2.4 or later.
Before that's becoming available, it's ok to check in this change as temporary solution, as we don't expect people use argo extensively outside of pipeline in near future.

IronPan · 2019-05-07T07:01:49Z

/lgtm
/approve

k8s-ci-robot · 2019-05-07T07:01:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: IronPan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [IronPan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2019-05-07T07:02:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: IronPan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [IronPan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Ark-kun · 2019-05-14T00:11:05Z

Closing this issue since the same change was implemented in kubeflow/kubeflow#2238 which got picked up in our v0.1.9 release.

…beflow#513) * Auto deployed clusters are no longer recycling names; instead each auto deployed cluster will have a unique name * Use regexes to identify the appropriate auto deployed cluster * Only consider clusters with a minimum age; this is a hack to ensure clusters are properly setup. * Related to: kubeflow/testing#444

* KFP 1.5.0-rc.0 Rebase * Resolve backend and API conflicts * Resolve UI conflicts * Apply changes from KFP 1.5.0-rc.1 * Resolve backend and API conflicts from RC1 * Resolve UI conflicts from RC1 * Apply changes from KFP 1.5.0-rc.2 * Resolve backend conflicts from RC2 * Build SDK based on kfp RC2 from Github instead of PyPI * Regenerate unittest's Golden YAML files

Configuring the default artifact repository in argo.libsonnet

df86e32

k8s-ci-robot requested review from IronPan and vicaire December 11, 2018 08:03

k8s-ci-robot added the size/S label Dec 11, 2018

k8s-ci-robot assigned IronPan Dec 11, 2018

k8s-ci-robot added size/XL and removed size/S labels Dec 11, 2018

Removed the explicit artifact location from compiler and tests

a282764

gaoning777 reviewed Dec 11, 2018

View reviewed changes

sdk/python/kfp/compiler/compiler.py Show resolved Hide resolved

gaoning777 reviewed Dec 11, 2018

View reviewed changes

pipeline/pipeline/argo.libsonnet Outdated Show resolved Hide resolved

IronPan requested changes Dec 11, 2018

View reviewed changes

De-hardcoded the minio service namespace

b2d4281

Ark-kun closed this Dec 22, 2018

Ark-kun mentioned this pull request Apr 30, 2019

Added configArtifactRepository method into sdk/python/kfp/compiler/compiler.py #765

Closed

Ark-kun reopened this Apr 30, 2019

Ark-kun mentioned this pull request Apr 30, 2019

[Feature] Supports parameterized S3Artifactory for Pipeline and ContainerOp in kfp package #1064

Merged

k8s-ci-robot added the lgtm label May 7, 2019

k8s-ci-robot added the approved label May 7, 2019

Ark-kun closed this May 14, 2019

Ark-kun deleted the Deployment---Default-artifact-repository branch May 14, 2019 00:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configuring the default artifact repository in argo.libsonnet #513

Configuring the default artifact repository in argo.libsonnet #513

Ark-kun commented Dec 11, 2018 •

edited

Loading

Ark-kun commented Dec 11, 2018

IronPan commented Dec 11, 2018

IronPan Dec 11, 2018 •

edited

Loading

Ark-kun Dec 11, 2018

Ark-kun May 8, 2019

Ark-kun commented Dec 11, 2018 •

edited

Loading

IronPan commented Dec 11, 2018

Ark-kun commented Dec 13, 2018

IronPan commented Dec 21, 2018

Ark-kun commented Apr 30, 2019

k8s-ci-robot commented Apr 30, 2019

IronPan commented May 7, 2019

IronPan commented May 7, 2019

k8s-ci-robot commented May 7, 2019

k8s-ci-robot commented May 7, 2019

Ark-kun commented May 14, 2019

Configuring the default artifact repository in argo.libsonnet #513

Configuring the default artifact repository in argo.libsonnet #513

Conversation

Ark-kun commented Dec 11, 2018 • edited Loading

Ark-kun commented Dec 11, 2018

IronPan commented Dec 11, 2018

IronPan Dec 11, 2018 • edited Loading

Choose a reason for hiding this comment

Ark-kun Dec 11, 2018

Choose a reason for hiding this comment

Ark-kun May 8, 2019

Choose a reason for hiding this comment

Ark-kun commented Dec 11, 2018 • edited Loading

IronPan commented Dec 11, 2018

Ark-kun commented Dec 13, 2018

IronPan commented Dec 21, 2018

Ark-kun commented Apr 30, 2019

k8s-ci-robot commented Apr 30, 2019

IronPan commented May 7, 2019

IronPan commented May 7, 2019

k8s-ci-robot commented May 7, 2019

k8s-ci-robot commented May 7, 2019

Ark-kun commented May 14, 2019

Ark-kun commented Dec 11, 2018 •

edited

Loading

IronPan Dec 11, 2018 •

edited

Loading

Ark-kun commented Dec 11, 2018 •

edited

Loading