Multi-user support and allow user filter for pipelines, runs, and experiments owned by you

[FaustineLi]

Our use case internally involves sharing a cluster for a team of scientists. Since every scientist is working on their own different projects, the experiment / run / pipeline list gets cluttered quickly. We would love to have a filter to only (1) experiments you have created (2) runs you have started (3) pipelines you have uploaded if that is possible.

[eterna2]

I would love to have this feature too.

How I solve it now is deploying a kubeflow pipeline deployment for each team.

[FaustineLi]

@eterna2 Is there a way to have multiple pipeline deployments per kubeflow cluster or are you saying you are deploying a kubeflow cluster per team? If it's the later, that's too much underutilized overhead for us to stomach.

[eterna2]

Ya. U can deploy multiple kubeflow pipelines service per kubernetes cluster. However, u will need to deploy them in different namespace.

Note, I am referring to kubeflow pipelines only not the full featured kubeflow (haven't tried it, but shld be the same thou).

https://github.com/kubeflow/pipelines/blob/master/manifests/kustomize/README.md

Note: I did not use this manifest per se, cuz I am running on AWS EKS rather than GKE, my manifest is quite different, but conceptually they are the same.

A very important thing u need to note that is CRD and cluster role / cluster role bindings are cluster wide setting.

Remember not to delete them when removing a particular kfp deployment, cuz u will affect other kfp deployment in different namespaces.

i.e. dun just kubectl delete -f , u will remove some of these cluster wide resources.

[FaustineLi]

I see. I pretty much live on the data scientist user side, so honestly most of this information doesn't mean a whole lot to me. But I appreciate that it is possible!

[Bobgy]

@gaoning777 Is this sth that should be part of the multi tenant scenario?

[numerology]

@gaoning777 Is this sth that should be part of the multi tenant scenario?

I think it is. @gaoning777 WDYT?
That being said such use cases would be slightly different than the multi-tenant support. @FaustineLi for the multi-tenant support what is expected to happen is that, different users (or user groups) will be separated by different namespaces. It won't be as straightforward as adding an attr (say, author) to a pipeline and provide a way to group pipelines.