kube2iam issues ...cannot assume role correctly in AWS SQS scaler in EKS

[bmacauley]

Specifications

• KEDA Version: 1.3.0
• Platform & Version: AWS/EKS
• Kubernetes Version: 1.14
• Scaler(s): AWS SQS

Having problems setting the SQS scaler up on a kubernetes cluster using kube2iam. We are using v 1.3.0.

Followed @ncrothe approach in Add AWS pod identity support

• kube2iam role on both operator (for setup, I guess) and metrics-apiserver that is allowed to assume the role configured in the TriggerAuthentication (the kube2iam annotation is being written to the wrong section in the charts)
• TriggerAuthentication with awsRoleArn reference in secret as explained by Zach
• no podIdentity involved in my use case anywhere
• created a keda role(k8s-keda), which contains a trust relationship for the kube2iam role attached to the kubernetes worker node(same as the many other roles we have)

Get the following error in the keda-operator...
{"level":"debug","ts":1585482843.978178,"logger":"scalehandler","msg":"Error getting scale decision","ScaledObject.Namespace":"hydra-sbx1-mobilelogins", "ScaledObject.Name":"hydra-ingest-outbound-etl","ScaledObject.ScaleType":"deployment", "Error":"AccessDenied: User: arn:aws:sts::123456789123:assumed-role/k8s-keda/e7450c49-k8s-keda is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123456789123:policy/k8s-keda\n\n\tstatus code: 403,

kube2iam provides the role for the pod(uses the annotation)...so session should be available inside pod?

Any thoughts?

Looking at the code...it seems to use a different technique to consume the kiam/kube2iam role compared to other tools eg externaldns
aws_sqs_queue_scaler.go.
aws_iam_authorization.go

externaldns. ...uses session.NewSessionWithOptions
external-dns/provider/aws.go

[bmacauley]

@ncrothe suggestions in Add AWS pod identity support
#499

• you add the role as trust to itself or
• you split this into two roles: one role A that the pod assumes via kube2iam and a role B that role A can assume (via Keda) that has the actual permissions.

[ncrothe]

To clarify: This would be how to make it work in the current solution, due to needing two roles (or a self-referential).
I don't think kube2iam support is great within the current solution and would prefer a much more direct way of assuming a pod role. Most other tools can just assume the pod role and rely on standard AWS SDK functions (which has the default provider chain which typically picks up a pod role provided by kube2iam).

[tomkerkhove]

What would you suggest to have then @ncrothe?

[ncrothe]

We've touched a bit on that in the discussion in #499 while I was trying to figure out what was going on. While there is some merit to splitting the permissions into multiple roles, which could be tuned according to individual "things to be scaled", I reckon the majority of users would prefer to simply set up a single pod role and be done with it.

As far as I understand the current auth flow, there is some logic that always programatically assumes an explicitly configured IAM role (either via hard-coded credentials or the awsRoleArn), despite this not strictly being necessary in a kube2iam environment. Here, the AWS SDK could simply pick up and assume the role configured via kube2iam without any explicit logic needed. This is made possible with the default provider chain of the AWS SDK which looks for env vars, credentials file or instance profile (see https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials) - with kube2iam hooking into the instance profile case.

Thus my suggestion for kiam/kube2iam mode would be:

• configure kube2iam to provide a role which has access to SQS/Cloudwatch metrics as needed
• have some setting in keda which bypasses the whole "assume some role" logic and simply let the SDK take care of that

I was under the initial assumption that this was what aws-kiam was meant to do, which Zack corrected me on.

In a perfect world we would also keep the current behaviour as optional for those edge cases (my assessment) where a more fine-grained, ScaledObject-specific permissions control is needed.

[tomkerkhove]

Thanks for the input - We should make this easier indeed, thanks!

[bmacauley-reward]

Other observations...

The annotations for using kube2iam are in the wrong location in helm chart package...

• 12-keda-deployment.yaml
• 22-metrics-deployment.yaml

should be at the spec level in the deployment

[ncrothe]

Sorry about that, missed that when reviewing kedacore/charts#27

Seems you're right. @tomkerkhove - can you fix that?

[tomkerkhove]

Created kedacore/charts#35 for that and we are open for PRs.
If it doesn't get fixed from the community I'll try to make some time for it this week.

[ncrothe]

@tomkerkhove beat me by a few minutes :-)

Check out kedacore/charts#36

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.