helm chart

[naseemkullah]

Are there any plans for a helm chart?

Preferably one that would leverage the already existing argo, ambassador, minio and any other component's community charts as dependencies of the kubeflow chart?

Would be great if we could deploy and manage kubeflow with the package manager we all love and know!

My 2 cents...

[issue-label-bot]

Issue-Label Bot is automatically applying the label improvement/enhancement to this issue, with a confidence of 0.74. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[naseemkullah]

Got started on something, is this worth moving forward on?

helm/charts#13454

[devonkinghorn]

I would like to see kubeflow in helm. We use helm for IAC and kubeflow is a massive project with no helm support.

[kkasravi]

@naseemkullah kubeflow is moving to kustomize for 0.6. Kustomize provides some support for helm through a helm plugin which would be a way to integrate new components. Your approach is interesting and goes the other way - moving everything to helm

[naseemkullah]

Thanks for the info @kkasravi

[dodizzle]

Is there an update to this? I am also woking with @naseemkullah to create a helm chart for kubeflow.
It won't take much work to get the manifests repo to be a helm chart. But I don't want to waste our efforts. Not sure why there would be resistance to deploying kubeflow via helm as it is by far the most common way to deploy services in kubernetes. Basically ever open source project can be deployed via helm. There are already 334 projects in this repo. https://github.com/helm/charts Kubeflow could easily be #335

[devonkinghorn]

@dodizzle even if the kubeflow org doesn't support helm. A community helm chart will be greatly appreciated.

[kkasravi]

kubeflow is a collection of components - eg the kustomization.yaml's under https://github.com/kubeflow/manifests are used to generate separate groups of resources. Each kustomize target is deployed separately.

if you do a kfctl init ~/kubeflow; cd ~/kubeflow; kfctl generate all
you'll see separate yamls under ~/kubeflow/kustomize.

Currently kfctl doesn't provide an option of deploying either using a helm chart or kustomize.
But one could deploy a helm chart separately rather than use the kustomize target.
As I noted earlier - kustomize has a helm plugin so you could use a helm chart that the plugin outputs as a kustomize target.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kyprifog]

@dodizzle curious where did the helm chart end up?

[maxisacoder]

helm is better!

[danielbraun89]

Why is this closed?

[jmandel1027]

I think that the resistance to helm is a huge bummer, we use helm + kustomize + flux pretty heavily and would love to deploy kubeflow but the lack of a stable chart and many interleaved components caused us to abandon this project for a different path.

Why not helm? Sure It might have some weird spots but it's an incredibly powerful and easy to use tool for automating complex deployments, I think this project would gain alot more traction if it embraced the tooling that the community was asking for.

[sonnysideup]

Just to restate what @jay-manday said:

Helm has its quirks, kustomize has its quirks, <fill-in-tool-name> has its quirks, ..., nothing is perfect. Realistically though, Helm has achieved widespread adoption in the K8s application deployment space, and the packaging format they use for distribution makes it easy to use. Running something like

export PIPELINE_VERSION=1.7.0
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/platform-agnostic-pns?ref=$PIPELINE_VERSION"

makes me feel like I'm forced to manually do things that Helm does for free with values, hooks, and CLI flags. I also have no interest in writing production-grade automation code using Bash and kubectl. 🤮

Secondly, in order to perform an upgrade, Helm has a well-defined procedure for rollbacks in the event of a failure; kustomize applies (overwrites) so hold onto your butts. 🤷‍♂️ I've used both Helm and kustomize deployment strategies in production and I will never repeat the latter given the pain I experienced there, but that's just me.

I'd use Kubeflow in anger if a Helm chart was available, otherwise I feel like I need to search elsewhere.

[sebastien-prudhomme]

@jay-manday @sonnysideup I hope to release an unofficial Helm chart for Kubeflow soon.

It will be more or less based on Kustomize files without the third-party components (Istio, cert-manager, an OIDC provider, ...)

[Legion2]

@sebastien-prudhomme did you already release your helm chart?

[sebastien-prudhomme]

@Legion2 just some components: see there https://github.com/cowboysysop/charts

I'm stuck with some problems, one of them is that some components are still using old APIs and are not compatible with K8S 1.22.

Fortunately it seems that things are changing since KServe which is remplacing KFServing now has an official Helm chart: https://github.com/kserve/kserve/tree/master/manifests/charts

[js-ts]

@sebastien-prudhomme the KServe Helm chart is not publised and will come with KServe 0.8 version

[mtparet]

There is here a helm chart for kubeflow-pipelines:
https://github.com/getindata/helm-charts/tree/main/charts/kubeflow-pipelines

Due to its complexity and high entry level, Data Scientists seem to be discouraged to even give it a go. At GetInData, we have developed a platform-agnostic Helm Chart for Kubeflow Pipelines, that will allow you to get started within minutes

https://getindata.com/blog/kubeflow-pipelines-running-5-minutes/

[js-ts]

@mtparet its not official and uptodate with latest KFP releases

[mtparet]

Indeed there is some lack for latest update but it is a good start for all people wishing an easy way to use kubeflow.
And with kindness contributions, I am sure this chart could be updated.

[Techn0logic]

Also, feel free to check this very simple helm chart for kubeflow pipelines. It is aimed at running smaller experiments locally. I mostly use it for my personal ml projects.

https://github.com/Techn0logic/kubeflow-pipelines-standalone

[typhoonzero]

Please feel free to try and join https://github.com/alauda/kubeflow-chart

[bhack]

This is was also identified as a priority from our survey outcome right?
https://blog.kubeflow.org/kubeflow-user-survey-2023/

[kromanow94]

Just to give you some update - I started work on KF Helm Chart and I think it looks promising but I want to finish this PR first: kubeflow/manifests#2544.

I was requested to do in-depth auth analysis for the kubeflow/manifests#2544 in respect to changes around istio and to understand how auth is currently working for kubeflow components. Also, I'll need to add some more tests in there...

[clayrisser]

@kromanow94 how are you building the kubeflow helm chart? Are you building it from scratch or are you using something like flux or argcd to create helm-release resources?

[kromanow94]

@clayrisser building from scratch. The idea is to create a strong base with meaningful and up to standard templates and template functions so they can be used across all the kf components. I'm focusing on single helm chart approach but I'm not using helm chart dependencies as I want to focus on decoupling various out-of-kubeflow components like istio and cert-manager that can be (and most often are) deployed and maintained by platform teams.

I'm having in mind that there must exist a standard, example installation like the example kf with kustomize so instructions or scripts or value files for helm charts that are installing istio and other components will definitely be a part of the proposed solution.

[clayrisser]

I think a helm chart that deploys helm charts might be an alternative approach. For example . . .

https://github.com/clayrisser/charts/blob/main/stable/openldap/templates/releases/openldap.yaml

@kromanow94 what do you think are the pros/cons to that approach?

[kromanow94]

@clayrisser I think such approach might be an alternative. But, as far as I understand, it would depend on fluxcd or other tool that allows configuring many helm charts as a single source and will require the approach of having a Helm Chart per KF Component. In my opinion this brings additional complication.

In my experience, the business should guide the technology in use. Having this in mind, I believe the approach "The fewer tools are needed to deploy Kubeflow, the better." gives more flexibility to the end user and helps in adoption.

But, this is just my opinion and you're very much welcome to challenge that. Best way would be to create a solution based on your proposal ot at least an MVP and then we can compare and discuss.

[Syntax3rror404]

@kromanow94

I would recommend using an umbrella chart to avoid redundancies and reduce maintenance efforts. For instance, Istio and Cert Manager etc are apps that already have existing Helm charts. These could be enabled through an 'if' statement using gotpl. In my scenario, a cert-manager and Istio are already installed.

https://artifacthub.io/packages/helm/cert-manager/cert-manager
https://artifacthub.io/packages/helm/istio-official/istiod

# Chart.yaml
dependencies:
- name: cert-manager
  version: "v1.13.3"
  repository: "https://artifacthub.io/packages/helm/cert-manager/cert-manager"
- name: istiod
  version: "1.20.1"
  repository: "https://artifacthub.io/packages/helm/istio-official/istiod"

You can modify the default values of these charts in the root values.yaml file of the Kubeflow Helm chart. I suggest adding 'enable' variables for each component to toggle them on or off as needed.

Thank you for your time! <3

[clayrisser]

@Syntax3rror404 I don't know if this is flexible enough because ideally values will need to be shared between the charts.

[Syntax3rror404]

@clayrisser

I know, but you can access this vars from the values files directly, or if the subchart generates something, you can also access this vars.

For example how I access in my chart a templated value from the bitnami mariadb chart in my main chart:
{{ template "mariadb.primary.fullname" .Subcharts.mariadb }}

And if its only the value you want, the subchart and your main chart shares the same values file which is located in your main chart.

[kromanow94]

Thank you for the insights, it's always nice to hear some ideas, alternatives and opposite approaches. I'm definitely going with the enable pattern for each component.

Right now I'm not planning to change my approach. I'm going with a single helm chart but I do put a big stress on having the right conventions and in-chart self-explanatory organization. I think I'm able to bring the benefits of a single helm chart and an umbrella chart together. Let's see...

Also, what I like in this approach is that the KF Helm Chart would focus only on the Kubeflow Components and then we can have an Example KF Platform Helm Chart defining only the dependencies. The dependencies could be kubeflow, istio, cert-manager, knative, whatever needed to deploy a ready-to-use platform. This would bring a great flexibility to the enterprise users who usually have all the infra components set and only need the installation tool that does one thing and does it good. Also, I've learned about scenarios where teams administrate multiple KF Installations so giving them the possibility to define a Helm Chart with their per-installation resources and use Kubeflow Helm Chart as a dependency would be great.

In a few weeks I should have something to share so we can have a discussion about it on real example.

[clayrisser]

@kromanow94 could you share the link? I might be interested in contributing.

[kromanow94]

@clayrisser hey, that would be great! There is still some stuff I want to finish before showing an mvp, then I'm planning to share to gather feedback and open for contributions. I think I need two weeks or so.

[Termibg22]

@clayrisser hey, that would be great! There is still some stuff I want to finish before showing an mvp, then I'm planning to share to gather feedback and open for contributions. I think I need two weeks or so.

Hi, I'm testing the deployment with Kustomize and I think the Helm chart is highly necessary. Is there a time schedule for the first test release?

[thesuperzapper]

Let's try to keep this thread about actual progress or proposals.

If anyone has any updates they want to share please do so.

PS: @Termibg22, if you're not already aware of deployKF, it will probably give you what you need. Even though it's not exactly the same as a helm chart, it still has values and supports every kind of config you can imagine.

[kromanow94]

Tomorrow I should have some more updates. There was a discussion about this Helm Chart in the last KF Manifests WG Meeting and so far it was received positively. Current plan is to bring this subject in tomorrows KF Community Meeting.

The idea to discuss it on the Community Meeting comes from the proposition of commitment to this feature from my company. I shared my work there and it was received positively with plan to invest further and share with the community. This is also a reason why I want to postpone sharing current code to align and have a clear perspective.

FYI, current idea is to bring this as an alpha feature and it will definitely not replace current kustomize setup, rather bring more options to the community.

[alex-treebeard]

Just chiming in as the three of us above were discussing in the community meeting.

• To the extent that it's possible @kromanow94 has done it justice with his chart
• I have also released a higher level helm chart which installs the whole thing, albeit with ArgoCD as a dependency
• @ronaldpetty made a lucid point that maybe in an ideal world we allow people to install kubeflow without istio/certmgr somehow which would be a much more standard helm install experience

[kromanow94]

Hello everyone,

I'm excited to share a preliminary Helm Chart for Kubeflow I've been working on with the community. While it's not a complete solution and some components are still under development, it provides a solid foundation for testing the most widely used features and the Helm Chart itself.

I'm looking for feedback and would like to spark a discussion to ensure this approach aligns with the community's needs. To facilitate this, I've created a Helm Chart Repository hosted on GitHub Pages. You can find it here, along with quickstart instructions for easy setup.

To see the repository, please visit:

• https://github.com/kromanow94/kubeflow-manifests
  To see the Helm Chart, please visit:
• https://github.com/kromanow94/kubeflow-manifests/tree/helmcharts/charts/kubeflow
  To see examples used for quickstart scripts and for reference, please visit:
• https://github.com/kromanow94/kubeflow-manifests/tree/helmcharts/example/helm

Assumptions

The primary focus of this Helm Chart is to streamline the installation of Kubeflow components. It's designed as a singular Helm Chart, marrying the benefits of individual charts for each component with the convenience of a unified installation process.

Quickstart Installation

Helm

This script will create 'kubeflow' namespace configured with istio injection and install helm releases for each kubeflow dependency and kubeflow itself in the correct order and will wait until the dependencies are ready.

1. Create a Kubernetes cluster:

   kind create cluster
   

2. Execute the quickstart script:

   curl -qs https://raw.githubusercontent.com/kromanow94/kubeflow-manifests/kubeflow-0.1.3/example/helm/quickstart.helm.sh | bash
   

ArgoCD Apps

This script installs ArgoCD if not installed, deploys dependencies and Kubeflow itself through ArgoCD Apps. It deploys the Apps in the correct order and will wait until the dependencies are ready within the script.

1. Create a Kubernetes cluster:

   kind create cluster
   

2. Execute the quickstart script:

   curl -qs https://raw.githubusercontent.com/kromanow94/kubeflow-manifests/kubeflow-0.1.3/example/helm/quickstart.argocd.sh | bash
   

ArgoCD App of Apps

This script installs ArgoCD, configures it for health assessment required for using sync-waves for ArgoCD Apps, and deploys an App of Apps manifest for Kubeflow and its dependencies. It uses sync-waves to cover for correct installation order of the dependencies and Kubeflow Components.

1. Create a Kubernetes cluster:

   kind create cluster
   

2. Execute the quickstart script:

   curl -qs https://raw.githubusercontent.com/kromanow94/kubeflow-manifests/kubeflow-0.1.3/example/helm/quickstart.apps-of-apps.sh | bash
   

If you already have ArgoCD installed and configured with health assessment for ArgoCD Apps, it's enough to execute this command:

kubectl apply -f https://raw.githubusercontent.com/kromanow94/kubeflow-manifests/kubeflow-0.1.3/example/helm/app.app-of-apps.yaml

To see more about the configuration for ArgoCD App health assessment, please visit:

• https://kubito.dev/posts/enable-argocd-sync-wave-between-apps/

Helm + EKS

This script is for reference only and will not work out of the box. Also, it assumes the script is run from a locally cloned git repository.

'kubeflow' and 'oauth2-proxy' Helm Charts needs to be parameterized for AWS IAM OIDC Issuer to enable access through M2M Service Account Tokens.

For required parameterization, please see the following files and replace 'https://oidc.eks..amazonaws.com/id/1234567890' with your AWS IAM OIDC Issuer URL.

• example/helm/values.kubeflow.eks.yaml
• example/helm/values.oauth2-proxy.eks.yaml

Assuming you've cloned this repository and are using it locally, you can parameterize the AWS IAM OIDC Issuer URL with:

OIDC_ISSUER_URL=https://oidc.eks..amazonaws.com/id/1234567890
grep -Rl 'https://oidc.eks..amazonaws.com/id/1234567890' --exclude 'quickstart*' | xargs sed "s;https://oidc.eks..amazonaws.com/id/1234567890;\$OIDC_ISSUER_URL;g" -i

Verifying the Installation

After completing the installation, use the following commands to verify your setup and perform initial configuration:

1. Forward the istio-ingressgateway port:

   nohup kubectl -n istio-ingress port-forward service/istio-ingressgateway 8080:80 &
   

2. See programmatic user id through Central Dashboard API:

   curl localhost:8080/api/workgroup/exists -H "Authorization: Bearer $(kubectl -n kubeflow-user-example-com create token default-editor)"

3. Create a new experiment in Kubeflow Pipelines:

   curl -XPOST localhost:8080/pipeline/apis/v1beta1/experiments -H "Authorization: Bearer $(kubectl -n kubeflow-user-example-com create token default-editor)" -d '{"name": "m2m-experiment", "resource_references": [{"key": {"id": "kubeflow-user-example-com", "type": "NAMESPACE"}, "relationship": "OWNER"}]}'

4. List experiments to verify the creation:

   curl "localhost:8080/pipeline/apis/v1beta1/experiments?resource_reference_key.type=NAMESPACE&resource_reference_key.id=kubeflow-user-example-com" -H "Authorization: Bearer $(kubectl -n kubeflow-user-example-com create token default-editor)"

What's included in the Kubeflow Helm Chart

• Central Dashboard
• Jupyter Web App and Notebooks Controller
• KF Pipelines
• Profiles Controller
• Integration manifests with Dex, oauth2-proxy, Istio,

What's not included

• Admission Webhook (PodDefaults)
• Katib
• PVC Viewer
• Tensorboard
• Training Operator
• Volumes Web App

Future of this Helm Chart

This Helm Chart will be developed and supported by myself and the company I work with. We rely on Kubeflow in multiple teams across multiple projects and our business solutions are already greatly leveraged by using Kubeflow. We started an internal initiative focused on sharing expertise and improving the ways of installing Kubeflow in enterprise environment. Kubeflow as a Platform is important to us. I started work on this Helm Chart myself and now seeing how positively it was received in my company and with some Kubeflow community members, I expect it to be continued.

My wish is to have this Helm Chart as part of the official Kubeflow project and some discussion across the community points to that it's quite possible.

[alex-treebeard]

@kromanow94 this is looking really good. My project is more focused on bootstrapping and would benefit from having your helm interface to depend on.

For my and others' benefit -- when do you recommend others start using this in production? I.e. Do you expect the repo URL/versioning/development to change radically in the near future?

+1 on this being part of Kubeflow and in an ideal world not having a hard Istio dependency.

[kromanow94]

Hey, sorry for not responding earlier. First I was traveling, then I was sick, then I was traveling again... 😄

I want to give an update on the Helm Chart :).

1. We're aligned in my company with the work and the current plan is to bring this Helm Chart into production state within a month.
   1. The work is aligned and distributed between the Kubeflow Inner Source team members in my company. We work in our private repository but the work will be synchronized and released in the GitHub repository I shared before: https://github.com/kromanow94/kubeflow-manifests
2. I was requested to create a recording with a demo for the Helm Chart. I did it and put to YouTube. It's a little longer than I anticipated but it also explains some of the assumptions and background, shows some parts of the Helm Chart with the conventions and abstractions, goes through the values that had to be configured for the dependencies (Istio, Dex, oauth2-proxy, and more) and shows the installation with the quickstart scripts:
   1. https://www.youtube.com/watch?v=VUrPJuGviaM

@alex-treebeard

when do you recommend others start using this in production?

If all goes well, I expect it to be production ready within a month. Then we have to align with the community if it's going to be merged into kubeflow/manifests in contrib directory or if there will be some time needed for investigation before that.

Do you expect the repo URL/versioning/development to change radically in the near future?

The URL might change if this work will be merged to kubeflow/manifests. I think creating some Helm Registry like kubeflow-contrib would make a lot of sense. For the versioning I'm not sure yet, it's an open discussion but I don't see radical changes ATM. Development - I don't see radical changes ATM. The assumption of having one Helm Chart to rule them all and focus on just the KF Components + Integration Manifests is proving to provide the installation method that's relatively easy to understand and maintain. Until this work is not merged into kubeflow/manifests, I'll be maintaining and releasing this Helm Chart in https://github.com/kromanow94/kubeflow-manifests.

To also cover for some common doubts I hear from various talks:

• What if the other WGs or KF Components start to provide Helm Charts of their own?
  • As far as I understand, this idea is currently not on the horizon. If there will be some discussion in that direction, let's align and find the best approach. Also, considering that the Helm Chart provides enable toggles in Values file, I'd encourage the other WGs to use this Helm Chart and use only the part that they're working on. For example, to deploy just the Central Dashboard, it should be possible to provide just this values.yaml file:

    centraldashboard:
      enabled: true
    
    notebooks:
      enabled: false
    
    profilesController:
      enabled: false
    
    pipelines:
      enabled: false
    
    certManagerIntegration:
      enabled: false
    
    istioIntegration:
      enabled: false
    
    dexIntegration:
      enabled: false
    

    • I run a quick test with that values file and there are a few manifests that shouldn't be there but it's just a few and it will be easy to fix.
• What about the maintenance for future KF releases?
  • The team of dedicated K8s Engineers in my company is willing to support and maintain at least a few future versions of Kubeflow. Putting that into time perspective, we want to continue this work for at least a few years and in that time get more people to contribute and maintain. If in that time this Helm Chart will finish it's incubation and will become an integral part of Kubeflow, that would be amazing and would definitely prove it's support in the KF Community.
• What if this work will not be accepted by the community?
  • It will not affect our dedication to continue this work, simply because we rely on Kubeflow too much and so far this was the easiest and most easy to parameterize installation method for KF between multiple projects in our enterprise environment, each having different needs and compliance policies.

[kromanow94]

Hey all, a short update before I start vacationing.

We have a new release of the Kubeflow Helm Chart!

This version is released as 0.2.0 and contains the following improvements over previous 0.1.3:

• possibility to provide database credentials as plaintext or through secret reference
  • related example:
    • charts/kubeflow/values.yaml#L905 db section
    • example/helm/secret.db-credentials.yaml
    • example/helm/values.kubeflow.yaml
    • example/helm/values.mysql.yaml
• possibility to provide objectstore credentials as plaintext or through secret reference
  • related example:
    • charts/kubeflow/values.yaml#L845 objectstore section
    • example/helm/secret.mlpipeline-minio-artifact.yaml
    • example/helm/values.kubeflow.yaml
    • example/helm/values.minio.yaml
• possibility to use minio with other endpoint than minio-service.kubeflow
  • kfp defaults to minio-service in many places and it required some configuration in Argo WF Controller as well to correctly propagate to KFP Steps for kfp-launcher.
  • related example:
    • charts/kubeflow/values.yaml#L880 pointing to the default svc from bitnami minio helm chart
      • this can also be referenced from
    • example/helm/values.argo-workflows.yaml#L86 with workflowDefaults patching every workflow pod to contain env variables pointing to minio endpoint
• validation for the hardcoded mlpipeline-minio-artifact secret name
  • if the credentials to the objectstore are provided as secret reference, the secret name has to be mlpipeline-minio-artifact. If this is not true, an error will be thrown.
    • charts/kubeflow/templates/_helpers/validation.tpl
• Volumes Web App
• PVC Viewer
• Admission Webhook (PodDefaults)
  • The example deployment from quickstart scripts is configured with:
    • PodDefaults in the kubeflow-user-example-com KF Profile that mounts the projected Service Account token with pipelines.kubeflow.org audience for KFP Auth
    • values.kubeflow.yaml with notebook spawner#L22 configured to add mentioned PodDefaults by default to created KF Notebooks
    • script to be run in KF Notebook that uses kfp client with Projected ServiceAccount Token that creates experiment and runs Pipeline
      • this example can also be called with curl calls. Here is an example gist where you have to change the ISTIO_INGRESSGATEWAY_NAMESPACE to istio-ingress but otherwise should work

The example deployment with this helm chart is defined in example/helm with all ArgoCD Apps, Quickstart and Destroy scripts and Values Files for every Helm Release and ArgoCD Apps (they use the same values.yaml files).

Quickstart scripts:

$ curl -qs https://raw.githubusercontent.com/kromanow94/kubeflow-manifests/kubeflow-0.2.0/example/helm/quickstart.app-of-apps.sh | bash
$ curl -qs https://raw.githubusercontent.com/kromanow94/kubeflow-manifests/kubeflow-0.2.0/example/helm/quickstart.argocd.sh | bash
$ curl -qs https://raw.githubusercontent.com/kromanow94/kubeflow-manifests/kubeflow-0.2.0/example/helm/quickstart.helm.sh | bash

[AndersBennedsgaard]

Hey @kromanow94 this looks really well, good job! I'm a big fan of the chart not including external dependencies likes Istio/cert-manager/Dex/MinIO/etc. and just assume that they are set up by the user!

Just a note: CustomResourceDefinitions placed in the crds/ folder will be installed by default, but won't be upgraded (docs). Systems like ArgoCD and Flux does have support for upgrading CRDs, but it might make sense to have a note about this in a README somewhere.

Also, won't the test container here always fail? As far as I understand, it will be rejected by Istio since it doesn't include a valid JWT

[juliusvonkohout]

/close

this belongs to kubeflow/manifests not kubeflow/kubeflow. Please recreate there.

[google-oss-prow]

@juliusvonkohout: Closing this issue.

In response to this:

/close

this belongs to kubeflow/manifests not kubeflow/kubeflow. Please recreate there.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[juliusvonkohout]

Please check kubeflow/manifests#2730