trouble using --aws-role-arn option when adding EKS cluster with argocd CLI #2347
Comments
|
I'm not sure of the minimal permissions necessary, but I was able to get argocd to add an external cluster (cross account) by doing the following: In account B (external cluster), created a role argocd-test with AdministratorAccess policy attached, and created a trust relationship between it and account A (running argocd in an eks cluster). In the external cluster, 'kubectl edit -n kube-system configmap/aws-auth' to edit the iam role to RBAC mappings, under mapped roles added: Note that the username has to be the same as the rolearn; if not, when trying to add the external cluster to argocd, will get a 'the server has asked for the client to provide credentials' error In account A, that's running argocd, attached a policy allowing assumption of the argocd-test role in account B to the iam role on the EC2 nodes running argocd. (May be possible to map the role to a service account used by the argocd pods; I didn't test whether argocd is using a new enough aws-sdk for that support yet.) In my kubeconfig, the external cluster in account B is registered as 'arn:aws:eks:us-east-1:accountB_number:cluster/kube_remote'. Note that both --aws-role-arn and --aws-cluster-name are needed to make argocd use the aws-iam-authenticator to assume the role in account B. Also, --aws-cluster-name has to be just the name of the external cluster - not the full arn. I am using Kube 1.14 in EKS, and argocd 1.2.2. |
jannfis
added
the
component:multi-cluster
|
I have the same issue but I'm using terraform to create the secret just after the cluster has been created but I can't get it working I followed this structure but I can't find any logs related to detecting this new secret, cluster name, cluster endpoint on the I was also trying to guess how does ArgoCD uses
But this it not documented and I'm struggling to automate this process I'd happily document (crating a PR) the IAM configuration required if I could understand how all this works |
|
when I try to add the cluster on the CLI, I am seeing the following in the argocd-server logs: need to know how to setup a role that can be assumed by a role! |
|
Ok, so if you are doing this in the same account, here is what I did. I created a role to put in my target clusters aws-auth configmap.... Like so: This role trusts the root, eg: The worker node role of the argocd cluster need to have STS:assumeRole permissions to this new role. This has now gotten me passed the AWS error above, but now I have a new error: |
|
ok, it looks like this error is a 'user issue', eg: my fault. ;) I am trying to add the cluster like so: the --aws-cluster-name was incorrect, should be 'my-cluster-name', not 'kube_remote'. Once I discovered this: and in the argocd logs: but! in the UI it is still failed: but I can see no other sign to why it is failed. |
|
ok, I have it working now... I updated my 1.6 install to 'latest'.. it works in master/latest. |
|
antoher thing of note, I tried not using the IAM role of the nodes but using the OIDC features in EKS: I still received the same error: Maybe this can be supported? |
|
@jurgenweber any luck in getting this to work with a ServiceAccount? |
|
I am struggling with the same thing. EKS with ArgoCD in one cluster, wanting to add external EKS cluster. I have setup a role in IAM that has an attached trust policy that grants trust via OICD to the two service accounts used by ArgoCD (server and app-controller). These service accounts have been annotated with the IAM role. In the external cluster the role has been added to auth-cm. The pods are already using the role that has rights in the external cluster, so no need to assume a role. I am using declarative secrets for adding the external clusters, and not specifying the ARN for the role (as it does not need to assume one). The clusters show up in the UI, but as failed. No messages in log on argo-cd-server pod. |
|
From the argo-server pod I am able to use use |
|
Here's a rough recipe for what I've managed to get working with IAM roles scoped to a Kubernetes ServiceAccount and a single AWS account:
|
|
@helenabutowatcisco many thanks for the instructions! Worked for me even with external cluster in another account, just had to create argocd-manager role in external account, add it to Worth noting that after creating Secret with cluster's info, it's status becames |
|
Looks like IRSA roles work even with 1.6.2 images. Found out while experimenting and switching between latest and 1.6.2. I suspect that after adding SA annotation with role name pods has to be recreated to pick up changes. latest image appears to have some issues with resource tracking and constantly shows apps out of sync. One example: cert-manager deployed from proxy helm chart with cluster issuer created from manifest in templates folder (can share details if anybody interested). Important note on multi-cluster management: application names have to be unique in argocd, so to deploy same app to multiple clusters different app names has to be used. Solution that worked for me:
Final configs (essentials): argocd/kustomization.yaml: overlays/argocd-application-controller-deployment.yaml: overlays/argocd-application-controller-sa.yaml: overlays/argocd-cm.yaml: overlays/argocd-server-deployment.yaml: overlays/argocd-server-sa.yaml.yaml: Where ARGOCD_ROLE is IRSA IAM role for argocd's cluster with permissions to assume target clusters admin user IAM roles App config example: |
|
I think there should be better documentation on how to properly configure |
So you were able to make it work with just one role for ArgoCD SA? You didn't have to create any roles for the clusters that ArgoCD was managing? |
Correct. Just one IAM role. |
We are struggling to make it work, the cluster just shows up as
Not sure what we are missing 😢 |
|
Tackling the issue with @musabmasood, I have copied this kubeconfig in the # /home/argocd/.kube/config
apiVersion: v1
kind: Config
preferences: {}
clusters:
- cluster:
certificate-authority-data: <base64-encoded ca-data>
server: https://<eks-cluster-specific-host>.eks.amazonaws.com
name: arn:aws:eks:ap-south-1:<account-id>:cluster/<cluster-name>
users:
- name: arn:aws:eks:ap-south-1:<account-id>:cluster/<cluster-name>
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- eks
- get-token
- --cluster-name
- <cluster-name>
command: aws
contexts:
- context:
cluster: arn:aws:eks:ap-south-1:<account-id>:cluster/<cluster-name>
user: arn:aws:eks:ap-south-1:<account-id>:cluster/<cluster-name>
name: arn:aws:eks:ap-south-1:<account-id>:cluster/<cluster-name>And I am able to list pods in the target cluster: argocd@argocd-server-5cc5c44949-8rqhc:~$ kubectl cluster-info --context arn:aws:eks:ap-south-1:<account-id>:cluster/<cluster-name>
Kubernetes master is running at https://<eks-cluster-specific-host>.eks.amazonaws.com
CoreDNS is running at https://<eks-cluster-specific-host>.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://<eks-cluster-specific-host>.eks.amazonaws.com/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.But with the cluster secret which apparently does the same, it does not work: argo-cd/pkg/apis/application/v1alpha1/types.go Lines 2182 to 2186 in 3d1f37b apiVersion: v1
kind: Secret
metadata:
name: cluster-<cluster-name>
namespace: argocd
labels:
argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
name: <cluster-name>
server: https://<eks-cluster-specific-host>.eks.amazonaws.com
config: |
{
"awsAuthConfig": {
"clusterName": "<cluster-name>",
},
"tlsClientConfig": {
"caData": "<base64-encoded ca-data>",
"insecure": false,
}
}We only see it as "Failed" when listing the cluster in the UI. We have increased the log level to debug, but it does not add any new message. We have been able to get |
|
Tested both v1.6.2 and v1.7.0-rc1 Target cluster nonprod-useast1-cluster2 aws-auth configmap: ArgoCD cluster config (notice NO role specified here, it's not needed as we allowed access to argocd-managed cluster directly for ArgoCD SA IRSA role): ArgoCD v1.6.2 ArgoCD v1.7.0-rc1 So I'm pretty sure it was UI bug or bug related to cluster discovery mechanism and it's resolved in v1.7. |
|
Confirmed. It works in 1.7 rc1. We only specify the caData and cluster name
(not cluster ARN).
We also added the role arn to application-controller and server SAs.
…On Sun., Aug. 23, 2020, 16:49 savealive ***@***.***> wrote:
Tested both v1.6.2 and v1.7.0-rc1
Role config in account where we run ArgoCD
module "iam_assumable_role" {
source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
version = "~> v2.18.0"
create_role = true
role_name = "k8s-argocd-admin"
provider_url = replace(data.aws_eks_cluster.this.identity.0.oidc.0.issuer, "https://", "")
role_policy_arns = []
oidc_fully_qualified_subjects = ["system:serviceaccount:argocd:argocd-server", "system:serviceaccount:argocd:argocd-application-controller"]
}
data "aws_eks_cluster" "this" {
name = var.argocd_cluster_name
}
Target cluster *nonprod-useast1-cluster2* aws-auth configmap:
- "groups":
- "system:masters"
"rolearn": "arn:aws:iam::369115111111:role/k8s-argocd-admin"
"username": "argocd-admin"
ArgoCD cluster config (notice NO role specified here, it's not needed as
we allowed access to argocd-managed cluster directly for ArgoCD SA IRSA
role):
apiVersion: v1
kind: Secret
metadata:
name: nonprod-useast1-cluster2
labels:
argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
name: nonprod-useast1-cluster2
server: https://8BB8308BEFD883211111111111111.gr7.us-east-1.eks.amazonaws.com
config: |
{
"awsAuthConfig": {
"clusterName": "nonprod-useast1-cluster2"
},
"tlsClientConfig": {
"insecure": false,
"caData": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EVXlOekUzTkRFeU1Gb1hEVE13TURVeU5URTNOREV5TUZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTmJZCkR4cGdGMHFPdGpWSWhUUVJ0cXdnWC8weEtwdjhRMWJkejdHM0tyU1Z0bWtjYXBIbE94Yit2ODUyRXM5T2liYWMKT0I3eEk4NFpoWnRIZlRJZlNNUG5mSDFsdFUvSXcwZzZ6T0prNGl5bHVrdWxISHJOcXdDY2hvMmRTQno5Sm9NcApuNU1mSGpmV2I5RVppeERZeW5sZXdCK1dWTGIwdkNvTzNNeEloT3RTVG00djB4ZEJMQzBzM29Dd1lmNy9kaFd6CkZsZVZFNmYvY0xkNW1aclRjdlF6TzFzYVIrcEQ4T1FCblVjSXUrT2lXSTV5c2d4SUphWEFJRkZYSU5mWWF5Y2gKZGVJVUdYazAxMjNtMWNRaGJ3eGtEYzhnZkNVMlMxenFHZGVQNElOSGhTcCthSFN4cmJIa3dRYWxO"
}
}
*ArgoCD v1.6.2*
Cluster created but reporting error:
https://8BB8308BEFD883211111111111111.gr7.us-east-1.eks.amazonaws.com nonprod-useast1-cluster2 Failed Unable to connect to cluster: the server has asked for the client to provide credentials
However, it's possible to deploy applications to it:
gb2 https://8BB8308BEFD883211111111111111.gr7.us-east-1.eks.amazonaws.com guestbook1 default Synced Healthy Auto-Prune <none> https://github.com/argoproj/argocd-example-apps.git kustomize-guestbook HEAD
*ArgoCD v1.7.0-rc1*
New version has different cluster health discovery logic: it shows a
cluster with "unknown status" unless you deploy an app into it.
Once app is deployed cluster becomes "green" in UI. App is deployed
successfully.
https://8BB8308BEFD883211111111111111.gr7.us-east-1.eks.amazonaws.com nonprod-useast1-cluster2 1.17+ Successful
So I'm pretty sure it was UI bug or bug related to cluster discovery
mechanism and it's resolved in v1.7.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2347 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB6KUUSTQPEVQ3T6JNDRJCLSCF6DJANCNFSM4IZP7LNA>
.
|
|
@maxbrunet try to shell into the pod and run |
|
@okdas yes, the |
@helenabutowatcisco yes, I got it all working. I hope you did also. |
@helenabutowatcisco just to see If I understand: on step 6 you added to the aws auth cm running on account B the same role you created on account A with and annotated to the argo service roles? |
|
I recently implemented this, the above comments and references to the documentation helped me a lot. It still took me quite some time to get a working setup. I wrote a tutorial describing a basic setup that works and is fairly flexible: https://www.modulo2.nl/blog/argocd-on-aws-with-multiple-clusters Hope this helps someone. |
This was extremely helpful. There needs to be clearer examples of this in the documentation. |
|
Will ahh the above, I'm still getting |
|
I merely added the service role of admin (where argocd is running) to the cluster that Argo wants to "manage". This worked, but need to get opinion on "best security practice" here.. |
|
I follow this tutorial : https://www.modulo2.nl/blog/argocd-on-aws-with-multiple-clusters (With my own EKS cluster) But i have this error : |
|
Just some quick notes on setting this up while sifting through a lot of confused comments above.
#2347 (comment) is a great guide, and hopefully I've answered some of the ambiguities in there in a couple spots that had a bit of doubt. |
|
we had the below config working great but this pr appears to have introduced a new CLI for auth without any docs for how to update the config for EKS. Still experimenting and will post back if we can figure out new config. |
|
Hi, I have EKS clusters in the same account(the cluster where argocd is running and the external clusters are in the same account). Approach 1:I am able to add an external cluster by:
Versions: However, the approach above means using a long lived token(the Approach 2:So I am trying to use the behaviour summary:When I deploy the external cluster secret, the cluster status is Versions: The full error message when using argocd v2.4.5: I did increase log level to debug, I didn't notice anything new around that error log line(the same info log lines around it, same as before increasing the log level). Steps to reproduce:Assume the following:
Then:
My questions:
Thanks. |
|
Hi @jannfis, sorry if I have disturbed you. Do you reckon I should open a separate issue for my previous comment above? Thanks. |
|
Hey @AmmarovTou |
|
clusterName is the cluster name, not an ARN. {
"awsAuthConfig": {
"clusterName": "<CLUSTER_NAME>",
"roleARN": "arn:aws:iam::<AWS_ACCOUNT_NUMBER>:role/argocd-deployer"
},
"tlsClientConfig": {
"insecure": false,
"caData": "<BASE64_DATA>"
}
}And if the external cluster is in a different region, you must specify the region. {
"execProviderConfig": {
"apiVersion": "client.authentication.k8s.io/v1beta1",
"command": "argocd-k8s-auth",
"args": [
"aws",
"--cluster-name",
"<CLUSTER_NAME>",
"--role-arn",
"arn:aws:iam::<AWS_ACCOUNT_NUMBER>:role/argocd-deployer"
],
"env": {
"AWS_REGION": "eu-west-1"
}
},
"tlsClientConfig": {
"insecure": false,
"caData": "<BASE64_DATA>"
}
} |
|
Can someone outline what troubleshooting steps they may have taken? I'm not clear yet how to debug it. I'm getting the following error: |
|
I added some docs on howto configure Argo with EKS (see #14187), hope this helps |
|
Is there any way to dynamically get the Background: If the target account's IAM Role has permission to perform describe-cluster and within Equivalent AWS CLI Command : |
|
@sachin-net if you will be creating secret with terraform you can leverage terraform for this as well. |
|
Hi, I'm having similar issues adding a cluster to ArgoCD. The target cluster is in a different region (but in the same AWS account) that the one running ArgoCD. I'm getting I have created IAM roles and annotated the service accounts and edited the when I run the command in the argocd-server pod it appears to be getting a token: could you please help? I have spent a number of hours troubleshooting this issue and I'm running out of ideas |
|
I am not sure what example you’re following to use execProviderConfig but if you go back to just using awsAuthConfig there is no need to care about cross region details. The endpoint is in another region, but IAM is global and argocd can grab the creds from its local region and use them to talk to any cluster endpoint that understands them regardless of region or account. |
|
@mmerickel I was following the instructions in the post above #2347 (comment) |
|
I've been using my comments here plus the linked comment/instructions to manage cross-account cross-region clusters without issues for a while now. All of the configs look like: apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: test-cluster
namespace: argocd
labels:
argocd.argoproj.io/secret-type: cluster
data:
name: "cluster name shown in argocd",
server: "https://eks-control-plane-endpoint",
config: |
{
"awsAuthConfig": {
"clusterName": "<actual-name-of-eks-cluster>"
},
"tlsClientConfig": {
"caData": "the base64 ca data string"
}
}This is all you need assuming:
|
|
@mmerickel thanks for your help, I have changed the setup to what you suggested and checked the IAM role: (trust relationship, no policies attached) annotated the service accounts, and updated the could it be a bug with the version of argocd I'm using? or perhaps something to do with the EKS version? |
|
First, confirm the cluster is showing up in argocd's UI. Probably in a disconnected state. Then at least you know the secret is being read correctly. Then look at the argocd-server logs and see if you can find some useful error messages. I think you might be able to click invalidate cache on the cluster to force it to reconnect and see what logs it outputs while it tries to do that. |
|
As an addition to what @mmerickel said, I'd also take a look in Cloudtrail and check for anything weird going on with IAM there (e.g if the role cannot be assumed for whatever reason) |
|
Closing this issue for now since this is something which has been working for some time (both cross-account and cross-region), and with #14187 there's documentation on how to configure it. |
Just checking, @mmerickel your saying you are not using cross-account Roles in each EKS cluster account and instead just allowing your main ArgoCD IAM Role in each clusters aws-auth configmap so that ArgoCD can just directly request creds to the cluster? Reading through the cross-account setup instructions that have landed, I had started questioning why we need the cross-account role when the aws-auth configmap just allows X IAM role access with seemingly no regard to what account it is from. |
|
That's correct, it's how I'm doing it right now. So in my example the IRSA role given to ArgoCD's server is the same one that the cross-account clusters are granting access to, such that there is no extra assume role hops required. So the proposal is: There are some theoretical advantages to making multiple IAM roles but I didn't think it was worthwhile for me. If you do want an extra IAM role in the middle then you'll need to specify that role in the I'll leave you to decide which makes more sense for your org. |
|
I managed to glue everything mainly based on what is shared here, except for adding argocd-manager role in the main cluster (or argocd-deployer in the child cluster, if you wanna use a second role). I couldn't make it work with manually adding the role to |
|
@akefirad would you mind sharing how you set up the IAM Access Entry? Does the username matter? |
|
@markandrus which one, IAM Access entry or IRSA entry? The former doesn't need a username. This is our CDKTF code: const argocdEntry = new EksAccessEntry(this, "cluster-argocd-access-entry", {
clusterName: clusterName,
principalArn: argocdRoleArn, // including the path.
// kubernetesGroups: ["admin"], // What is this? Why doesn't it working?
});
new EksAccessPolicyAssociation(this, "cluster-argocd-access-policy-association", {
clusterName: clusterName,
policyArn: "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy",
principalArn: argocdRoleArn,
accessScope: { type: "cluster" },
dependsOn: [argocdEntry],
});(I finally managed to make it working with IRSA too, but in my case the issue was that our role has a path, and when setting the role in was-auth configmap, you should drop the path. See the bug reported above.) Does that help? |
|
Can we accomplish this without using the However with using Access Entries and skipping the use of |
|
@fideloper I'm not sure, the issue could be some restriction with access entries. I remember there were some restriction around the feature. |
|
I got it working without Using Access Entries worked great, no Within Terraform, setting up an access entries per cluster looked like this: (where this is done on each "spoke" cluster, which is an EKS cluster that ArgoCD will manage): # Create an access entry on a "spoke" EKS cluster so that ArgoCD ("hub" cluster)'s assumed role
# has RBAC permissions to administrate the spoke cluster
resource "aws_eks_access_entry" "argocd_rbac" {
cluster_name = "spoke-cluster-name"
principal_arn = "arn-of-role-being-assumed-by-argocd"
kubernetes_groups = []
type = "STANDARD"
}
resource "aws_eks_access_policy_association" "argocd_rbac" {
access_scope {
type = "cluster"
}
cluster_name = "spoke-cluster-name"
policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
principal_arn = "arn-of-role-being-assumed-by-argocd"
depends_on = [
aws_eks_access_entry.argocd_rbac
]
} |
and others
I am trying to use the
--aws-role-arnoption when adding an EKS cluster to ArgoCD, as described in #1304. I have not been able to get it to work and the error messages are difficult to interpret and I am not sure how to debug.I have ArgoCD running in one AWS account and my EKS cluster is in another AWS account
I have set up the
acme-production-deploy-roleso that it can be assumed both by the AWS role that I am using to runargocd cluster add ...and by the EC2 instances in my ArgoCD cluster (I am confused about which IAM identity is used to assume the role so I tried to allow both to work).Here is what I see when I try to add the cluster. (I have redacted the AWS account numbers and the EKS id, but confirmed that I used the correct values for these):
Note that I am able to successfully add the cluster using
argocd cluster add https://<eks-cluster-id>.yl4.us-west-2.eks.amazonaws.comThanks
The text was updated successfully, but these errors were encountered: