KEDA scaler not working on AKS with trigger authentication using pod identity #2178
Assignees
Comments
|
Hi @iarunpaul , |
KEDA 2.4.0 |
|
Hi @iarunpaul, |
Hi @JorTurFer Its an immediate failure as we try to change the trigger auth from connection string secret to pod identity of KEDA |
|
Hi @JorTurFer, Any updates on your run? |
|
hi @iarunpaul , |
|
hi again @iarunpaul , Reviewing the logs that you sent, the issue seems that is related with any misconfiguration in AAD Pod Identity side. You should take a look there because probably anything is missing there. Is the MSI correctly assigned to the node virtual machines? Is there any error in the MIC or NMI pod logs (aad-pod-identity)? For me it is weird that the error says |
|
Hi @JorTurFer, |
|
How did you deploy AAD Pod Identity? |
|
I am installing the AAD Pod Identity using helm, assigning required roles for the cluster, "Managed Identity Operator" and "Virtual Machine Contributor"to the and installing keda using helm |
|
The pod aad-pod-identity-mic-6497465fcb-6v7x8 logs are: And Not looking promising for me. |
|
Hi @iarunpaul , |
|
aad-pod-identity-mic-6497465fcb-6v7x8 logs: |
|
hi @iarunpaul |
|
the logs for aad-pod-identity-mic-6497465fcb-pjn5t: |
|
Iran this command to assign roles to the cluster: |
|
But still the AAD Pod Identity pods shows roles errors :( |
|
This line is strange: The message basically says that your identity can't be assigned to the VMSS. I have several problems in the past configuring AAD Pod Identity. I think that you should also grant the roles and permissions to kubelet MSI (the MSI inside the node resource group). I know that it's not documented, but it's the only way that I found to solve the issue, granting to both MSI (k8s and kubelet) the roles that are in the examples. Once the role assignment are done (and after giving a few minutes to refresh the tokens and propagate the changes), if you restart the mic pod you should see that now it works |
|
Okay...Is this a rare case while configuring pod identity? |
|
This ms documentation, says there is a limitation to bring your own kubelet id on system-assigned managed cluster.
and assign the roles to both cluster id and kubelet id. |
|
No no, don't worry about setting the ids. This MSI is in your case, which has the id If you can't see any MSI in your nodes' resource group, maybe it's because you have created your cluster using Service Principal instead MSI. |
|
|
|
When I check it the MSI fe0d7679-8477-48e3-ae7d-43e2a6fdb957 is assigned the roles correctly as the screen shot. |
|
hi @iarunpaul , E1110 02:31:09.031633 1 mic.go:1129] failed to apply binding keda-system/autoscaler-id-binding node aks-agentpool-14229154-vmss000002 for pod keda-system/keda-operator-7c9b598554-tchmd, error: failed to update identities for aks-agentpool-14229154-vmss in MC_Arun_democluster_westeurope, error: compute.VirtualMachineScaleSetsClient#Update: Failure sending request: StatusCode=0 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'fe0d7679-8477-48e3-ae7d-43e2a6fdb957' with object id 'fe0d7679-8477-48e3-ae7d-43e2a6fdb957' has permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/write' on scope '/subscriptions/f3786c6b-8dca-417d-af3f-23929e8b4129/resourceGroups/MC_Arun_democluster_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-14229154-vmss'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/f3786c6b-8dca-417d-af3f-23929e8b4129/resourcegroups/arun/providers/microsoft.managedidentity/userassignedidentities/autoscaler-id' or the linked scope(s) are invalid."
E1110 02:31:09.031664 1 mic.go:1129] failed to apply binding keda-system/autoscaler-id-binding node aks-agentpool-14229154-vmss000002 for pod keda-system/keda-operator-metrics-apiserver-97cd665d4-kw4hm, error: failed to update identities for aks-agentpool-14229154-vmss in MC_Arun_democluster_westeurope, error: compute.VirtualMachineScaleSetsClient#Update: Failure sending request: StatusCode=0 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'fe0d7679-8477-48e3-ae7d-43e2a6fdb957' with object id 'fe0d7679-8477-48e3-ae7d-43e2a6fdb957' has permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/write' on scope '/subscriptions/f3786c6b-8dca-417d-af3f-23929e8b4129/resourceGroups/MC_Arun_democluster_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-14229154-vmss'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/f3786c6b-8dca-417d-af3f-23929e8b4129/resourcegroups/arun/providers/microsoft.managedidentity/userassignedidentities/autoscaler-id' or the linked scope(s) are invalid."In the end of the lines, you could see the identity id that is trying to assign a permission, and what permission is needed. Before solving these problems, KEDA will not be able to use MSI because it can't get the token from AAD Pod Identity |
|
Let me create the cluster and install the AAD Pod Id again and update then today! |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
|
@JorTurFer when you said: could you drop installation process, please? I kind of hard to find details around it. I understand is the above correct? I'm sure step by step with examples will help Azure geeks to undertsand. |
stale
bot
removed
the
stale
|
Hi @dariuszbz , |
|
@JorTurFer thank You! I'm not sure about that: "After that, we create an orders queue in our namespace: ❯ az servicebus queue create --namespace-name --name orders --resource-group ❯ az servicebus queue create --namespace-name --name orders --resource-group two lines (az command) are the same. Should the second line (az command) be different? both commands does the same. |
|
I think that you could ignore one of them (it seems a typo) but maybe @tomkerkhove could confirm it |
|
Yes, you can leave out the second one |
|
ok. So to make my life simple I created below .... and it won't work :( what is wrong below? |
|
Hi @dariuszbz , |
|
@tomkerkhove @dariuszbz @JorTurFer This is a workaround to read. |
|
Thanks for letting us know, I was not aware of it! Once you have the outcome of CNI I will update our FAQ/troubleshooting. |
|
I tried the workaround for kubenet , but not in success yet but pretty close, I think! Keda pod id with Managed IdentityRecreate IssueWe create the aks with the commands: Checked the providers are registered Create the cluster Install AAD Pod Identity on cluster
AAD Pod Identity is disabled by default on Clusters with Kubenet starting from release v1.7. We need to look at it if the cluster is configured with kubenet.
This is a workaround to read. I edited the daemonset Create the required identitiesAssigning roles to the MIsWe create the service bus thenAssign role to MIInstall KEDA using HelmApply autoscaling infrastructure in
|
|
@JorTurFer I have followed documentation and I have dropped you the e2e script. Let me run that again and I'll drop you logs. Just which logs would you need, please? There is a plenty from K8s :) I don't want to keep you busy for a next few winters ;) as I need to test it and recommend for production or not recommend for productions environments. |
|
@JorTurFer is it somehow usefull? 0117 09:28:06.034191 1 mic.go:608] No AzureIdentityBinding found for pod keda-system/keda-operator-584cc6777-2fw9j that matches selector: autoscaler-aad-identity. it will be ignored something you can help with? It from: aad-pod-identity namespace, deployment: aad-pod-identity-mic. keda-system namespace, autoscaler yaml definition: apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
name: autoscaler-aad-identity
spec:
type: 0 # 0 means User-assigned MSI
resourceID: $autoscaler_aad_identity_resourceId
clientID: $autoscaler_aad_identity_clientId
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
name: autoscaler-aad-identity-binding
spec:
azureIdentity: autoscaler-aad-identity
selector: app-autoscalerI think the selector here is somehow problematic, but it is educated guess |
Tring everything I have assigned the
|
|
Hi @iarunpaul contrats! If you have e2e script, could you share with me, please? |
|
Your |
|
@tomkerkhove @JorTurFer educated guess but 100% hit. selector is a problem there. I fixed and it works now. attaching e2e script it does everything except creating aks cluster. prvide aks name, aks rosource group name, node group name, and happy keda-ing :). |
|
|
Finally I found it! Then I had to edit Now the Still the order application doesn't work. You need to delete and redeploy the application and wait for a few minutes... |
|
@iarunpaul would be nice to see your e2e example for kubenet - it can save some time for other geek :) |
Posting e2e Script for AKS with kubenet plugin, if it is of any assistance to some:Note: Read this instruction before you run AAD Identity On a Kubenet powered AKS. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
|
This issue has been automatically closed due to inactivity. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Report
KEDA scaler not scales with scaled object defined with trigger using pod identity for authentication for service bus queue.
I'm following this KEDA service bus triggered scaling project.
The scaling works fine with the connection string, but when I try to scale using the pod identity for KEDA scaler the keda operator fails to get the azure identity bound to it with the following keda operator error message log:
My scaler objects' definition is as below:
Im deploying the azure identity to the
namespace kedawhere my keda deployment resides.And installs KEDA with the following command to set the
pod identity bindingusing helm:Expected Behavior
The KEDA scaler should have worked fine with the assigned pod identity and access token to perform scaling
Actual Behavior
The KEDA operator could not be able to find the azure identity assigned and scaling fails
Steps to Reproduce the Problem
Logs from KEDA operator
KEDA Version
No response
Kubernetes Version
1.20
Platform
Microsoft Azure
Scaler Details
Azure Service Bus
Anything else?
No response
The text was updated successfully, but these errors were encountered: