Fleet namespace creation and integration with Rancher Projects

[vCra]

Currently, fleet is able to create namespaces, but no functionality exists to add the created namespace to a rancher project.

Initial slack convo: https://rancher-users.slack.com/archives/C013SSBKB6U/p1615194570034900

Use Case:
A user wants to deploy a new application into a namespace which doesn't yet exist, but also wants to add this namespace to a rancher project so that it can inherit the PSPs/rbac configuration of the project. Currently fleet is able to create a namespace, but the user will still have to manually add that namespace into a project for each cluster we have, which is not scalable for large numbers of clusters

Workarounds:

Manually add the namespace to a project in each cluster using the UI

Random ideas/discussion points:

Should fleet have some functionality to customise namespaces - i.e. add annotations

• could these be customised on a per cluster basis? (That would mean that we could simply add the projectId annotation within fleet.yaml or simular)

Should fleet be managing namespace creation? or would it be better if users included a namespace maniest. (and use kustomize to add annotations). Could both work? - i.e. fleet creates a namespace, but then doesn't manage it - if a namespace manifest is included then it does

[vCra]

Also relates a bit to #285

[Jasstkn]

Is there any ETA when this feature will be implemented?

I want to seed some secrets like Registry auth using project features.

[nickgerace]

Hi @Jasstkn: I've added the "Release Candidates" label to help get this prioritized. Please stay subscribed if you would like further updates. Thank you for your patience, and I hope we can get it in for a future release!

[vkruoso]

Also thinking about this. As new namespaces are not assigned to any project, they won't be able to pull images from private registry if credentials are managed on a per-project basis, for instance. Right now I don't see a way to do it using Fleet as the project ids are not consistent from one cluster to another.

[skaven81]

As long as the Namespace gets created with the correct annotation to mark it as a member of the right Project, Rancher "adopts" it properly. From my experience, you just need this one annotation:

apiVersion: v1
kind: Namespace
metadata:
  name: wharrgarble
  annotations:
    field.cattle.io/projectId: c-9mj66:p-8v8fr   <-- clusterId:projectId

So as long as you can figure out the projectId at deploy time, you have several options.

Option 1: Static configuration
If the bundle always gets deployed the same (single) cluster and always goes into the same Project, then you can just hard-code the projectId and finalizer bits into the namespace YAML you include in your bundle.

Option 2: Per-cluster overlay
If you know the ProjectIDs for each cluster ahead of time, you can use per-cluster overlays in the bundle to set the projectId metadata in the Namespace as it's deployed

Option 3: Egregious hack
Create a Bundle that contains a ServiceAccount, ClusterRole, ClusterRoleBinding, and a Job in the kube-system namespace.
The ClusterRole grants the ServiceAccount permission to all verbs on the target namespace(s) and read/watch verbs on the kube-system namespace (or whichever namespace you want to use as your "template" for finding the ProjectID). The Job just runs a simple alpine-based container with a shell script inside that does kubectl get namespace kube-system and grabs the projectID, then kubectl create namespace <new-namespace> and then patches the new namespace with the right projectID annotation.

Then Create a second Bundle that contains the actual service to be deployed, that references the namespace created by the Job, but does not contain a Namespace resource itself.

On the first pass, the fleet-agent will report an error for the second Bundle because it can't find the referenced namespace. But then when the second pass comes around, it sees that the namespace is now present, and completes the deployment.

You can't include all of this in a single Bundle, because it seems the fleet-agent will abort the entire Bundle deployment if a resource in the Bundle references a non-existent namespace. So the namespace creation has to be in a separate Bundle as the namespace usage. But....this approach does work, if in a really hacky and ugly way.

[skaven81]

One warning I have about implementing this at the Fleet level, is that Project names are not guaranteed to be unique (see rancher/rancher#18222 and rancher/rancher#22761). So if Fleet gives you a method for selecting the Project for a namespace based on the Project's name (or maybe labels) how would Rancher/Fleet know which one is the "right" one to assign the namespace to?

[serverbaboon]

Pretty sure I tried precreating the namespace in the right project (Terraform ) for Rancher Monitoring and the Helm chart failed as the namespace pre-existed.

[kriswill]

Just hit this one myself. Project on rancher/local, namespace with annotation on target deployment cluster, and no way to reference the Rancher Project. My only thought on this is to hook into the object creation on the target cluster and signal a Webhooks somewhere to fix up the annotations, kind of like Option 3 above, but without granting rights to the Job runner.

Please fix this!

[nickgerace]

I'd love to get this issue scheduled. In the meantime, I wonder if a potential workaround is to create a management.cattle.io/v3 object of type FleetWorkspace? Those are automatically integrated with the continuous delivery UI and create the underlying namespace.

cc: @StrongMonkey

[skaven81]

@nickgerace I don't think that's a suitable workaround. If I have an application that I wish to be deployed to the "MyApp" Project in a number of different Rancher-managed clusters, how would a FleetWorkspace help with that? What I need is a way to tell Fleet, "go ask Rancher what projectId "MyApp" resolves to, and then create my namespace with the appropriate projectId labels/annotations".

Creating a FleetWorkspace would indeed make the namespace for me, and have it appear in the Rancher UI, but that's not really the point. I don't care whether the namespace appears in the UI -- I want it to be in a Project because there are certain administrative features (project-scoped secrets, project-scoped resource quotas, project-defined PodSecurityPolicies, project network isolation) that I want my Fleet-deployed application to be able to tap into.

[nickgerace]

Closely related: #633

[hazem-bouaziz]

Any updates about this issue ?

[manno]

Rancher is adding support for hierarchical namespaces. We will revisit the integration at that point

[manno]

There are no plans to integrate with projects. cc @olblak