Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

application resource can be applied in any namespace #3474

Closed
artbegolli opened this issue Apr 23, 2020 · 25 comments · Fixed by #9755
Closed

application resource can be applied in any namespace #3474

artbegolli opened this issue Apr 23, 2020 · 25 comments · Fixed by #9755
Labels
component:application-sets Bulk application management related enhancement New feature or request type:usability Enhancement of an existing feature
Milestone

Comments

@artbegolli
Copy link

artbegolli commented Apr 23, 2020

Summary

We would like the ability for argoCD to recognise any application resource that has been applied across the cluster and not just within the namespace where ArgoCD has been deployed.

Motivation

We are working on producing a multi-tenant implementation in our cluster and to do this, we have implemented a controller for handling all of these tenants. There is the use case where the same application will be applied in multiple different tenants (which will generally be a grouping of namespaces). Currently, as all application CRs are present in the argocd namespace, these cannot share the same name.

If applications were picked up from all namespaces, we could begin to store the application CR in the relevant tenants namespace instead of having to rename the application.

Proposal

How do you think this should be implemented?

In the ArgoCD spec allow specifying a scope field which allows argo to scan the entire cluster or just the namespace.

I'm sure you guys have a justification as to why you are limiting the application resource to only the namespace where argocd has been deployed and I'm curious as to why.

Cheers,

Art

@artbegolli artbegolli added the enhancement New feature or request label Apr 23, 2020
@rumstead
Copy link
Member

rumstead commented May 6, 2020

This was briefly discussed today at the Argo CD & Rollouts Community Meeting during the Application Set proposal given by @jessesuen.

@jannfis jannfis added component:application-sets Bulk application management related type:usability Enhancement of an existing feature labels May 14, 2020
@george-angel
Copy link

#2557 (comment)

Thank you

@peterbosalliandercom
Copy link

I would like to add the following suggestion. If you set a sort of scrape label like prometheus has on the namespace i.e. argocd-scrape: true then it is fully customizable.

@yhaenggi
Copy link

@rumstead any update on this? We also need it...

@rumstead
Copy link
Member

@rumstead any update on this? We also need it...

Last I heard the app set hasn't been moved into AroCD yet. Though I am not the best person to give an update.

@mvbakker
Copy link

Is there any progress? Similar to @yhaenggi we would also really like to have this implemented. Centrally we're controlling the AppProject resources, but we'd like to give developer teams control over their own Application resources, within their AppProject. This can now only be done if we also give them access to the argocd namespace, which is something we wouldn't like to do.

@peterbosalliandercom
Copy link

@yhaenggi @mvbakker What risk is involved if we allow developers to deploy application objects in argocd if we allow then via the project destination? Do they have permission to delete other applications?

@sbose78
Copy link
Contributor

sbose78 commented Apr 7, 2021

Hi @alexmt ! I've started taking a look at this.

In the upcoming milestone , I'm planning to make an enhancement proposal ( we can all feel free to +1/-1 it :) ) on this area. We may not get to implementing it, but I'd like to get a design going. Could we mark it for the upcoming milestone as 'design only' effort ?

@alexmt
Copy link
Collaborator

alexmt commented Apr 7, 2021

Hello @sbose78 , I also think this is important.

The next milestone already got few important tickets related to multi-tenancy (#5283 and #2251 ). It is good to tackle the issue described in this ticket at well.

@juv
Copy link

juv commented Apr 7, 2021

Can someone explain why the current behaviour was implemented as it is?
What are the benefits of keeping all of those resources within the argocd namespace?

@alexmt
Copy link
Collaborator

alexmt commented Apr 7, 2021

@juv, Argo CD provides a single control plane that manages resources in multiple clusters. Having Applications CRDs in control plane cluster in one namespace allows us to implement UI/API that presents the state of all managed resources at once.

If we store applications CRDs in multiple managed clusters it will be really difficult to fetch application status information. However I totally agree with the use case described in this ticket: it would be very nice to let end users create "apps" in target namespaces and "self service" argocd management. I've been working on PoC that resolves that issue for couple weeks. Hope to finish it this weekend and will present to the community.

@sbose78
Copy link
Contributor

sbose78 commented Apr 7, 2021

There are two distinct areas :

  • Watch Application CRs in the remote cluster instead of the cluster being specified as a destination. I like what we do today.
  • Application CR in non-ArgoCD namespace : This is the current issue at hand.

What risk is involved if we allow developers to deploy application objects in argocd if we allow then via the project destination?

Here's how I see it :)
If ArgoCD reconciles an Application CR and AppProject CR in namespace foo that had been created using kubectl, ArgoCD has no good way of determining whether the user is allowed to created the derived kubernetes objects :)

@jannfis
Copy link
Member

jannfis commented Apr 8, 2021

I had some thoughts on this as well, and I think some of the major problems to solve are pointed out below:

The relationship between AppProject and Application.

The AppProject provides governance to what an Application is allowed to do in the cluster, which repositories it may access etc. In the current state, Application CRs can be created in the cluster without enforcing what values the .spec.project field may hold. So basically, any Application can define to use any project when created via the K8s control plane, effectively circumventing any RBAC and governance provided by Argo CD. We assume that some who can create Application CRs in the argocd namespace is an Argo CD admin. So we would need a way to limit what values are allowed in an Application's .spec.project field.

One possible solution would be to either enhance the AppProject CRD with a new field, e.g. .spec.applicationSources (or maybe even .spec.sources) that defines possible sources where Application CRs may exist and specify this AppProject as .spec.project value. For example,

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: default
spec:
  applicationSources:
    - cluster: '*'
      namespace: '*'

would allow Applications from each namespace in each cluster to specify the default project as value in .spec.project. We could argue to re-use the .spec.destinations field of AppProject CR for this purpose (so, Application CRs may exist where Applications are allowed to be deployed), but I feel that would open up too much of a hole for people not wanting to have Application CRs exist outside argocd namespace.

Also, all that being written, I think we should really have the AppProject CRs to exist only in the argocd namespace and not allow them to exist in arbitrary locations. These are central to governance of Argo CD after all.

Missing connection to a remote cluster.

What happens when the connection to a remote cluster that holds Application CRs goes away? I think this is what @alexmt is tackling right now, and I believe this could be mitigated using some caching mechanisms on the Argo CD control plane side. However, since we consider Argo CD stateless, and the cache to be rebuildable at any time, this could pose some very nasty surprises imho and Applications to temporarily disappear.

@sbose78
Copy link
Contributor

sbose78 commented Apr 8, 2021

From my perspective, the experience I'm looking for is :

  1. User logs in to namespace foo.
  2. User creates an Application CR using kubectl with a reference to the Git repo where the user has manifests
  3. Application Controller reconciles it and ...
  4. that triggers creation/sync of the resources in namespace foo/ outside namespace foo / cluster-scoped.

What determines whether (4) is allowed ?
User's existing permissions. Example, if the user wasn't allowed to create pods in the first place, she still doesn't get create to one!

@jannfis
Copy link
Member

jannfis commented Apr 8, 2021

What determines whether (4) is allowed ?
User's existing permissions. Example, if the user wasn't allowed to create pods in the first place, she still doesn't get create to one!

I'm wondering how that could possibly work. I think there are a multitude of constraints that prevent that from working:

  • Once a resource is created, how do you find out by whom it was created? What if another user modifies that resource? What if the Kubernetes RBAC of that user change?
  • What about applying least privilege concept? With GitOps, possibly you don't want any "interactive" SA to have permissions to actually create application related workloads, such as Pods. Maybe you want them to only have required permissions to gain read-only visibility into the cluster, and deploy everything else using Argo CD
  • What if the Application CR is coming from Git, deployed via Argo CD? Or through some kind of CI solution?

@sbose78
Copy link
Contributor

sbose78 commented Apr 8, 2021

Once a resource is created, how do you find out by whom it was created?

The information could be extracted from an admission controller payload and persisted in an annotation on the Application CR. Only the webhook will be allowed to modify that annotation.

What if the Kubernetes RBAC of that user change?

Before applying changes to a cluster, always do an SubjectAccessReview to see if user Jane is allowed to do X

{
  "apiVersion": "authorization.k8s.io/v1",
  "kind": "SubjectAccessReview",
  "spec": {
    "resourceAttributes": {
      "namespace": "team-toucans",
      "verb": "create",
      "resource": "pods",
      "version": "v1"
    },
    "user": "jane",
    "groups": [
      "system:authenticated",
      "devops",
      "team-toucans",
    ]
  }
}

What about applying least privilege concept? With GitOps, possibly you don't want any "interactive" SA to have permissions to actually create application related workloads, such as Pods. Maybe you want them to only have required permissions to gain read-only visibility into the cluster, and deploy everything else using Argo CD

We are potentially letting a cluster user do things which she wasn't allowed to do in the first place by the cluster admin. Based on what the ArgoCD admin decided, we are letting the user create workloads. Effectively, we are trusting the Argo CD RBAC system to make a good decision. I'm suggesting that if the user exists on the cluster, why not use the Kubernetes RBAC system to make the decision. :-)

What if the Application CR is coming from Git, deployed via Argo CD? Or through some kind of CI solution?

The CI would be authenticating to the cluster in some way. The user/service account with which the CI authenticates should have the necessary privileges.

@jannfis
Copy link
Member

jannfis commented Apr 8, 2021

We are potentially letting a cluster user do things which she wasn't allowed to do in the first place by the cluster admin. Based on what the ArgoCD admin decided, we are letting the user create workloads. Effectively, we are trusting the Argo CD RBAC system to make a good decision. I'm suggesting that if the user exists on the cluster, why not use the Kubernetes RBAC system to make the decision. :-)

But what if the organization doesn't even want to have users with those kind of privileges? IMHO, one of the neat things of having a system like Argo CD is, that you can get rid of all of those users jumping around with kubectl on the control plane to do all kind of things, right? Why should those users have more privileges on the cluster than they need to? The sufficient permissions would be to operate on Application CRs in their given namespaces.

The CI would be authenticating to the cluster in some way. The user/service account with which the CI authenticates should have the necessary privileges.

From my point of view, the CI would have the right to create an Application CR in a namespace. Not more. Again, why should the CI's user be granted privileges to create anything else?

The application controller makes sure that only resources can be created that are allowed by the AppProject associated with any application. I do agree that the current approach of having the application controller having full cluster level access for every operation is not good, however, I think inheriting permissions from the user who creates an Application CR is not a good idea either. Not only would it require additional privileges for users that they wouldn't need in the first place, but also would add a whole lot of complexity as far as I understood (like, an additional admission controller, etc).

There is this (unfortunately) stale PR (#3377) which proposed user impersonation for the application controller, bound to an AppProject. I haven't tested whether this is possible, or what side-effects it would bring, but it could allow the application controller to operate with a given ServiceAccount (and therefore, K8s RBAC restrictions) when reconciling applications. With such a method, we could get rid of resource restrictions within Argo CD (e.g. those imposed by the AppProject), and manage & impose those restrictions using K8s native methods.

@sbose78
Copy link
Contributor

sbose78 commented Apr 8, 2021

But what if the organization doesn't even want to have users with those kind of privileges?

Agreed, the existing model solves the problem for organizations that don't necessarily want their users to be exposed to Kubernetes.
However, I'm talking about the scenario where a user already has access to a cluster and one or more namespaces there.

From my point of view, the CI would have the right to create an Application CR in a namespace. Not more. Again, why should the CI's user be granted privileges to create anything else?

The CI already has access to create other things, not by itself, rather via ArgoCD Application CR. So, effectively the permission management is not the native Kubernetes one, rather the ArgoCD one.

Not only would it require additional privileges for users that they wouldn't need in the first place, but also would add a whole lot of complexity as far as I understood (like, an additional admission controller, etc).

As an example, I am allowed to do everything on three namespaces in a cluster - I need something that would sync manifests within the realm of the namespaces I already have access to. :)

@jannfis
Copy link
Member

jannfis commented Apr 8, 2021

Agreed, the existing model solves the problem for organizations that don't necessarily want their users to be exposed to Kubernetes.
However, I'm talking about the scenario where a user already has access to a cluster and one or more namespaces there.

Isn't this a scenario we want to get away from? :)

The CI already has access to create other things, not by itself, rather via ArgoCD Application CR.

That's correct, and the CI (or any other user) would have these privileges through an intermediary, Argo CD. And only for resources stored in Git, and every change would be traceable and auditable. By granting the user corresponding privileges on the control plane, you would effectively allow changes outside of this pattern.

So, effectively the permission management is not the native Kubernetes one, rather the ArgoCD one.

That's also correct, and that's why I wanted to bring #3377 into play :)

@mvbakker
Copy link

mvbakker commented Apr 9, 2021

Great to see that this thread has gotten some traction! I really like the discussion. To come back on the question from @peterbosalliandercom:

@yhaenggi @mvbakker What risk is involved if we allow developers to deploy application objects in argocd if we allow then via the project destination? Do they have permission to delete other applications?

Yes, right now developers of different teams/products, and therefore different namespaces, all have essentially the power to modify the Application CRs of another team/product. Of course, us being all friends, this is unlikely to happen, but from a least privilege point of view, ideally you'd restrict teams to only control their own Application CR.

From @sbose78:

Application CR in non-ArgoCD namespace : This is the current issue at hand.

Exactly, and to provide some more context why we would like to see this, let me explain our use-case. Centrally, from the Platform/Admin team, we control the Argo CD installation/configuration. However, we try to give as much 'do-it-yourself' control to teams to control their own deployments using Argo CD. We control centrally what a team is allowed to synchronize (i.e. the repo, the namespace, specific non-namespaced resources, or even blacklist namespaced resources) with an AppProject CR. In this way, teams can deploy their own Application resource, triggering the deployment of their product whenever and however they would like, within the boundaries controlled centrally. Allowing them to deploy their Application CR in their own namespace will avoid the need to give them permissions to the argocd namespace.

From my point of view, the CI would have the right to create an Application CR in a namespace. Not more. Again, why should the CI's user be granted privileges to create anything else?

100% agreed. In our set up, generally users only have view and list rights to the cluster. The CI has the privilege to deploy Application CRs. Argo CD is allowed to deploy resources as described by the AppProject CRs.

What I have been wondering though, is there a reason why AppProject resources are namespaced? If Applications CRs are allowed to be deployed in other namespaces, then the AppProject would also need to be there? Or should it remain in the argocd namespace? Or would it be possible to make an AppProject a cluster-wide resource? Especially looking at the new 'Global AppProject' feature, which sounds like a cluster-wide resource (though these could also just be deployed in the argocd namespace of course, as long as other AppProject CRs can still inhered from it).

Another point to consider, imagine that Application CRs are also picked up from other namespaces other then argocd. If a team is only allowed to deploy Application CRs in their own namespace (controlled via K8s RBAC or CI), and they do not have permissions to the argocd namespace. Then they are not able to use the default AppProject CR to deploy anything and there must be an AppProject deployed for their namespace, am I right? It is something to consider, but I would like this behaviour.

@jannfis
Copy link
Member

jannfis commented Apr 10, 2021

Another point to consider, imagine that Application CRs are also picked up from other namespaces other then argocd. If a team is only allowed to deploy Application CRs in their own namespace (controlled via K8s RBAC or CI), and they do not have permissions to the argocd namespace. Then they are not able to use the default AppProject CR to deploy anything and there must be an AppProject deployed for their namespace, am I right? It is something to consider, but I would like this behaviour.

I think the AppProject would still live in a central namespace (i.e. argocd), and that users should not be able to create, modify or delete them. As you already pointed out, AppProjects will define what kind of resources an Application is allowed to deploy, and where it is allowed to deploy them. I see no reason why an Application created in namespace foo should not be able to use an AppProject created in namespace argocd, because the relationship between the two is made by Argo CD, not K8s. But as already discussed in this thread, it would be really awesome if we could map an AppProject to K8s RBAC instead of configuring its restrictions using a possibly redundant method in Argo CD.

@mvbakker
Copy link

mvbakker commented Apr 12, 2021

What do you guys think about an AppProject not being namespaced at all? Any major reasons why it should be namespaced? Couldn't it be, and wouldn't it make sense, to make it a cluster-wide resource? From a permissions point of view an Application CR wouldn't be able to sync non-namespaced resources already any way by default, unless whitelisted. So in that case, dev teams managing their own Application wouldn't be able to sync and/or adjust their own AppProject resource, unless explicitly whitelisted.

But as already discussed in this thread, it would be really awesome if we could map an AppProject to K8s RBAC instead of configuring its restrictions using a possibly redundant method in Argo CD.

Additionally, it would be awesome if Argo CD builds upon native K8s RBAC. Though I think that is a topic to discuss outside of the issue at hand regarding the namespacing of Application CRs and AppProject CRs, wouldn't you agree?

@dimitarKiryakov
Copy link

Hi @mvbakker
I think that making Application and AppProject cluster wide is not a good option, at least in our case. Currently we are using namespace installation of argocds in the same cluster (https://github.com/argoproj/argo-cd/tree/master/manifests). This way we achieve tenant isolation for our stakeholders and at the same time managing only one cluster.
I think that we are not the only ones using it this way, there is also argocd operator which you could use to create argocd instances in the same cluster.

@yydzhou
Copy link

yydzhou commented Jun 21, 2021

@juv, Argo CD provides a single control plane that manages resources in multiple clusters. Having Applications CRDs in control plane cluster in one namespace allows us to implement UI/API that presents the state of all managed resources at once.

If we store applications CRDs in multiple managed clusters it will be really difficult to fetch application status information. However I totally agree with the use case described in this ticket: it would be very nice to let end users create "apps" in target namespaces and "self service" argocd management. I've been working on PoC that resolves that issue for couple weeks. Hope to finish it this weekend and will present to the community.

@alexmt Any progressing on this? This becomes a main pain point for our use case. We have tested that users can easily destroy or crush others' service by creating an application with same name and setup auto-sync policy. In a multi-tenant environment, It's a big risk and also it's hard to prevent without sacrificing user experience/flexibility.

We need either global naming mechanism or allowing the application CR to be in different namespaces. So that users' impact/interaction are isolated.

@jannfis
Copy link
Member

jannfis commented Jun 21, 2021

@yydzhou There are ongoing proposals being discussed at #6409 and #6425

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:application-sets Bulk application management related enhancement New feature or request type:usability Enhancement of an existing feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.