Scope TFJob operator to only claim jobs in a given namespace

[jlewi]

We'd like to be able to scope Kubeflow to a particular namespace to better support multi-tenant situations.
See kubeflow/kubeflow#1213

When deploying the operator we should be able to specify which namespaces to monitor and the operator should only claim/manage jobs in that namespace.

We'd also like to scope our RBAC permissions and service accounts to only the provided namespaces.

@gaocegege Any idea how this should be implemented? Do informers have an option to allow scoping to namespaces

@jessesuen I think Argo supports this; what do you do?

[jlewi]

Looks like informer can be scoped to a namespace. But by default we monitor all namespaces
https://github.com/kubeflow/tf-operator/blob/master/pkg/client/informers/externalversions/kubeflow/v1alpha2/tfjob.go#L65

So we need to add an option to monitor a namespace.

I wonder if we can easily monitor a subset of namespaces?

[jessesuen]

@jessesuen I think Argo supports this; what do you do?

Argo workflow controller supports the notion of an "instance ID". The way it works is:

• if the workflow-controller-configmap is not configured with an instanceID, then that controller instance will only watch workflow resources with the instanceID label missing.
• if the workflow-controller-configmap is configured with with an instanceID, then that controller instance will only watch resources label workflows.argoproj.io/controller-instanceid = XXXX, where XXXX is an arbitrary string that the user decides and sets in the configmap. This has the consequence of argo workflows being submitted using the --instanceid flag, for the proper controller to pick it up.

Here is the exact logic:

	var instanceIDReq *labels.Requirement
	var err error
	if wfc.Config.InstanceID != "" {
		instanceIDReq, err = labels.NewRequirement(common.LabelKeyControllerInstanceID, selection.Equals, []string{wfc.Config.InstanceID})
	} else {
		instanceIDReq, err = labels.NewRequirement(common.LabelKeyControllerInstanceID, selection.DoesNotExist, nil)
	}

I wonder if we can easily monitor a subset of namespaces?

No, not easily. The REST API supports either cluster scoped queries, or namespaced scoped queries. To monitor N namespaces, you'll have to have to establish N watches.

[jlewi]

Thanks @jessesuen