Support for cluster-scoped CRDs and namespace-scoped resources in LambdaController #2
Comments
|
This isn't supported yet, but it's something I plan to add. Currently, the parent and children have to be in the same namespace. Part of this is because the standard way to express parent-child relationships for k8s API objects (OwnerReference) doesn't support cluster-scoped or cross-namespace references. This means we can't rely on the garbage collector to clean up children when the parent goes away, and we can't rely on ControllerRef to prevent fighting among controllers with overlapping label selectors. |
|
As requested by @enisoc my use case is automating the deployment of individual namespaces containing various deployments/services + an ingress on a multi-tenant platform. The namespaces come and go and each namespace belongs to a tenant for ease of management. Ideally we could return the namespace JSON (and an ingress-controller deployment) in a hook in response to the generation of a custom resource. Then have all child objects go into the same "parent" namespace. Make sense? |
|
This is also my use case. Thanks for describing it so concisely! |
|
Thanks for the details. The GC problem mentioned above is still the main blocker as far as I can tell. I haven't seen any indication that the core k8s GC is going to support cluster-scoped or cross-namespace cleanup any time soon, so I'm going to try to think of a way to make this possible without waiting for that. Just to start some brainstorming: One potential idea is for Metacontroller to place a finalizer on the parent object so it doesn't disappear immediately when deleted. Since Metacontroller presumably already had a way to decide which child objects it would have sent to your hook, it should know enough to be capable of cleaning those up for you before removing the finalizer. One limitation of this approach is that it doesn't prevent other controllers from stealing your children, because they would have no ControllerRef (since ControllerRef doesn't allow you to specify cross-namespace references) and would thus look like orphans. Another option might be for Metacontroller to create "proxy" parent objects in the same namespace as the children. This would prevent the children from looking like orphans. Then when the real parent is deleted, our finalizer will go and delete the proxy parents. Finally, the regular GC will delete the children since the proxy parent is now gone. @janetkuo @liyinan926 @crimsonfaith91 What do you think? |
|
Another idea is a "namespace controller" that lets you create singleton objects in each namespace. We would call your hook once for each namespace, as namespaces are added/updated, and let you return objects to create in that namespace. You could use labels or annotations on the namespaces to decide what to do, if you need extra parameters or if you only want to process certain namespaces. This would be a more limited pattern, but it bypasses the GC problem discussed above. For the use case that @marcwickenden described, this would mean you directly create the namespace (with whatever annotations you need) instead of creating a cluster-scoped CRD that in turn creates a namespace. |
|
I nearly wrote this exact idea in Slack! Wasn’t sure if that was deviating from the model too much. |
|
I lean towards creating "proxy" parent objects. As k8s grows, there is possibility for GC to support cross-namespace cleanup in future. We can add an annotation for all created "proxy" parent objects so that we can delete them easily when we have cross-namespace GC. This can be done with 2 steps:
Namespace controller is a good idea, but if we are adding labels or annotations to decide what to do with the namespaces, it may be easier to do so with the "proxy" parent objects. Furthermore, I don't think we should bypass GC as this involves parent-children relationship. |
|
It is possible that the proposed #59 changes work for this use case? It might require two controllers but the MapController seems like it would be good for bridging the Cluster -> Namespace scoped boundary. |
|
@rlguarino That's a good point. If MapController could work on Namespace objects, the "namespace controller" would be unnecessary. But since MapController supports an explicit parent object, we would still need to implement cross-namespace GC so that the output objects in various namespaces get cleaned up when the parent is deleted. Another alternative could be to add limited support for Namespace objects in DecoratorController. That is, Namespace would be the only cluster-scoped object that DecoratorController can add attachments to. This would bypass the fact that OwnerRef-based GC doesn't work across namespace boundaries, because when a namespace is deleted, the objects in that namespace get cleaned up by a separate GC that doesn't rely on OwnerRef. The downside is that this introduces a special case in DecoratorController -- one particular resource that behaves differently. The proposed NamespaceController API would essentially be this special case of DecoratorController, but pulled out to keep things clean. |
|
I did some experiments this morning with cluster scoped owner references on a GKE cluster (1.10.5-gke.0) and it looks like it's working to me. Reading through this issue and the PR that closed it it looks like Cluster scoped parents should be supported. I was able to create a create a CluterRoleBinding that references a ClusterRole owner, when I deleted the parent ClusterRole (foreground) the child was removed. When I deleted using
So to me it looks like cluster owners should always work, namespaced to namespaced should also work, but I'm not sure how namespaced parents of cluster resources would work.. WIthout too much magic, I think it would be possible to make ConpositeController/DecoratorController/MapController support look something like the above matrix by merging #62. (1) I haven't tested this, I don't expect it to make any sense. |
|
@rlguarino That's great news! Thanks for the research! Apparently my understanding has been out of date. I've merged #62. Let's see what else needs to change on the Metacontroller side to make this work. One problem is that the sync hook API assumes everything is in the same namespace. For example, the There are also likely other places where we need to relax the assumption that everything is in the same namespace, but that's definitely feasible if the built-in GC supports cluster-scoped OwnerRefs. |
|
Thanks for the extra details, sounds like it should work. I'm going to be
away on vacation for the upcoming week and I'm trying to avoid using my
computer. I should be able to take a stab at this the week after next if
someone doesn't get to it sooner!
…--RG
On Fri, Jul 6, 2018, 14:32 Anthony Yeh ***@***.***> wrote:
@rlguarino <https://github.com/rlguarino> That's great news! Thanks for
the research! Apparently my understanding has been out of date.
I've merged #62
<#62>. Let's
see what else needs to change on the Metacontroller side to make this work.
One problem is that the sync hook API assumes everything is in the same
namespace. For example, the children map is keyed only by name, not
namespace. We'll need to make an API change to support namespaced children
for a cluster-scoped parent, since the names might collide across
namespaces.
There are also likely other places where we need to relax the assumption
that everything is in the same namespace, but that's definitely feasible if
the built-in GC supports cluster-scoped OwnerRefs.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABQrakLts-wEea0dHAu-YtxkzaWPO4bjks5uD9dogaJpZM4REOxS>
.
|
|
@rlguarino It would be awesome if you're able to take a stab when you get back! Please update this thread if you do (or don't) so we can avoid overlap. Also feel free to bug me in the |
|
Based on our conversation in |
This CL adds initial support for clustered parents the decorator controller. The composite controller will work as long as you're not using rolling updates. This CL changes the behavior of the client map which is sent as `attachments` or `children` in the DecoratorController and CompositeController, respectively. The new keys can be thought of as a relative path to the child. When both the parent and child are at the same scope - either both namespaced or both clustered - the key is just the child's name. When the parent is clustered and the child is namespaced the relative are relative - the children's keys will always be prefaced with the namespace - this is to disambiguate between two children with the same name in different namespaces. To test this change this CL also adds two examples. The first example is of a decorator controller that creates a "reader" ClusterRole, similar to the default roles and rolebindings for each CRD with the `enable-default-roles` annotation. The second is a decorator controller which creates role bindings in the default namespace that bind the default service account to clusterrolebindings. Closes GoogleCloudPlatform#2
This CL adds initial support for clustered parents the decorator controller. The composite controller will work as long as you're not using rolling updates. This CL changes the behavior of the client map which is sent as `attachments` or `children` in the DecoratorController and CompositeController, respectively. The new keys can be thought of as a relative path to the child. When both the parent and child are at the same scope - either both namespaced or both clustered - the key is just the child's name. When the parent is clustered and the child is namespaced the relative are relative - the children's keys will always be prefaced with the namespace - this is to disambiguate between two children with the same name in different namespaces. To test this change this CL also adds two examples. The first example is of a decorator controller that creates a "reader" ClusterRole, similar to the default roles and rolebindings for each CRD with the `enable-default-roles` annotation. The second is a decorator controller which creates role bindings in the default namespace that bind the default service account to clusterrolebindings. Closes GoogleCloudPlatform#2
This CL adds initial support for clustered parents the decorator controller. The composite controller will work as long as you're not using rolling updates. This CL changes the behavior of the client map which is sent as `attachments` or `children` in the DecoratorController and CompositeController, respectively. The new keys can be thought of as a relative path to the child. When both the parent and child are at the same scope - either both namespaced or both clustered - the key is just the child's name. When the parent is clustered and the child is namespaced the relative are relative - the children's keys will always be prefaced with the namespace - this is to disambiguate between two children with the same name in different namespaces. To test this change this CL also adds two examples. The first example is of a decorator controller that creates a "reader" ClusterRole, similar to the default roles and rolebindings for each CRD with the `enable-default-roles` annotation. The second is a decorator controller which creates role bindings in the default namespace that bind the default service account to clusterrolebindings. Closes GoogleCloudPlatform#2
This CL adds initial support for clustered parents the decorator controller. The composite controller will work as long as you're not using rolling updates. This CL changes the behavior of the client map which is sent as `attachments` or `children` in the DecoratorController and CompositeController, respectively. The new keys can be thought of as a relative path to the child. When both the parent and child are at the same scope - either both namespaced or both clustered - the key is just the child's name. When the parent is clustered and the child is namespaced the relative are relative - the children's keys will always be prefaced with the namespace - this is to disambiguate between two children with the same name in different namespaces. To test this change this CL also adds two examples. The first example is of a decorator controller that creates a "reader" ClusterRole, similar to the default roles and rolebindings for each CRD with the `enable-default-roles` annotation. The second is a decorator controller which creates role bindings in the default namespace that bind the default service account to clusterrolebindings. Closes GoogleCloudPlatform#2
This CL adds initial support for clustered parents the decorator controller. The composite controller will work as long as you're not using rolling updates. This CL changes the behavior of the client map which is sent as `attachments` or `children` in the DecoratorController and CompositeController, respectively. The new keys can be thought of as a relative path to the child. When both the parent and child are at the same scope - either both namespaced or both clustered - the key is just the child's name. When the parent is clustered and the child is namespaced the relative are relative - the children's keys will always be prefaced with the namespace - this is to disambiguate between two children with the same name in different namespaces. To test this change this CL also adds two examples. The first example is of a decorator controller that creates a "reader" ClusterRole, similar to the default roles and rolebindings for each CRD with the `enable-default-roles` annotation. The second is a decorator controller which creates role bindings in the default namespace that bind the default service account to clusterrolebindings. Closes GoogleCloudPlatform#2
This CL adds initial support for clustered parents the decorator controller. The composite controller will work as long as you're not using rolling updates. This CL changes the behavior of the client map which is sent as `attachments` or `children` in the DecoratorController and CompositeController, respectively. The new keys can be thought of as a relative path to the child. When both the parent and child are at the same scope - either both namespaced or both clustered - the key is just the child's name. When the parent is clustered and the child is namespaced the relative are relative - the children's keys will always be prefaced with the namespace - this is to disambiguate between two children with the same name in different namespaces. To test this change this CL also adds two examples. The first example is of a decorator controller that creates a "reader" ClusterRole, similar to the default roles and rolebindings for each CRD with the `enable-default-roles` annotation. The second is a decorator controller which creates role bindings in the default namespace that bind the default service account to clusterrolebindings. Closes GoogleCloudPlatform#2
This CL adds initial support for clustered parents the decorator controller. The composite controller will work as long as you're not using rolling updates. This CL changes the behavior of the client map which is sent as `attachments` or `children` in the DecoratorController and CompositeController, respectively. The new keys can be thought of as a relative path to the child. When both the parent and child are at the same scope - either both namespaced or both clustered - the key is just the child's name. When the parent is clustered and the child is namespaced the relative are relative - the children's keys will always be prefaced with the namespace - this is to disambiguate between two children with the same name in different namespaces. To test this change this CL also adds two examples. The first example is of a decorator controller that creates a "reader" ClusterRole, similar to the default roles and rolebindings for each CRD with the `enable-default-roles` annotation. The second is a decorator controller which creates role bindings in the default namespace that bind the default service account to clusterrolebindings. Closes GoogleCloudPlatform#2
|
Hi, Guys Just to chime in, I was wondering how to understand the difference between the cluster-scoped and the namespaced CRD. Actually, we can get the Kind that defines by the namespaced CRD across the whole cluster. So, what's the apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ctHTTP path: /apis/stable.example.com/v1/namespaces/*/crontabs/status |
I saw this demo'ed at KubeCon and it gave me some inspiration to try writing a custom LambdaController sync hook.
I'm attempting to watch cluster-scoped custom resources to manage both cluster-scoped and namespace-scoped child resources (for now, namespaces and services). However, for all of the namespace-scoped resources in the returned children array, I get the following error:
I've confirmed that the response coming from my controller contains a namespace for these services, so I'm a bit confused.
Looking through the lambda_controller.go code, it appears that there's a limitation that child resources must be in the same namespace as the parent resource? Just want to confirm if what I'm trying to do is possible.
Thanks!
The text was updated successfully, but these errors were encountered: