Roadmap: KEDA Scale with Managed Identity

[SophCarp]

No ETA yet

[Ralle1986]

any ETA on this ?

[preardon]

Anything we can do to increase the priority on this?

[andreaskasc]

We would really need this feature. Is there any update for it?

[preardon]

We would really need this feature. Is there any update for it?

Agreed, this is a bit of a blocker for us moving to ACA

[onionhammer]

Yes please, this would be tremendous

[KamilLach]

This will be a great improvement for our ACA architecture

[superzer0]

+1

[mikocot]

are there any particular difficulties implementing that?

[ezYakaEagle442]

Hi @SophCarp could you please share an ETA ?

[coin-op]

Please can this be sorted out @SophCarp. 1 year on a backlog!

[mjrousos]

Needing a SAS-based connection string for KEDA scaling is the only reason I need local authentication enabled on a Service Bus namespace for the solution I'm working on. Everything else authenticates with managed identity. It would be great to add support for that here.

[preardon]

Needing SAS for KEDA is the reason why we've not moved to ACA or KEDA as yet, I refused to lower security

[jemrpo]

I ended up implementing a microservice to get around this restriction, I'd be great if the product team gives this a higher priority. I'd help a lot of people to simplify their applications.

[preardon]

I ended up implementing a microservice to get around this restriction, I'd be great if the product team gives this a higher priority. I'd help a lot of people to simplify their applications.

@jemrpo

I'm assuming you implemented a hosted service with a timer that used the managed Identity with the Service Bus Administration client, what did you use for the ACA side, just HTTP or is there a programetic way to tell it to - scale

[jemrpo]

@preardon my app is not using service bus.

KEDA is forcing me to use some info and a token for authentication in order to scale, I did not want to use a regular token because this would be insecure.
I ended up creating a microservice which uses a User Assigned Identity to request an ephemeral bearer token from Azure, once it gets the token, it stores it on a KV that is linked to the ACA secrets, once all this is done it monitors the token life, and it will request a new one once it expires.

I've been using this approach with ACA apps and Jobs, it has been working fine and I've been able to scale from 0 to 30 containers in a few seconds.

The life of the bearer token varies but it's never valid for more than 24 hours. It is not perfect, but it's way better than manually rotating tokens every 3 or 6 months.

Please let me know if this answers your question or if you want me to add more details.

[DenisBalan]

Missing feature, eta?

[fgheysels]

We're using ACA as well, and are using KEDA scalers on a service bus topic. Unfortunately, our client / customer who's ISO certified doesn't like it that we need to use a connection-string with shared access key here.

[verschaevesiebe]

Any update on this ? We've setup KEDA scaler to service bus but customer doesn't allow SAS keys to be used.
Microsoft pushes towards identity usage & rbac management but this blocks us from using Container Jobs tbh.

Please update ?

[vturecek]

Hi everyone, thanks for your patience on this. KEDA support for managed identity is rolling out, docs are getting prepped for publishing, and we will be announcing availability before the end of the month.

[Hi-Fi]

Seems that KEDA got updated already at least in Sweden Central region. Should managed identities already work or is there some wrapper in between that requires also changes @vturecek?

[vturecek]

@Hi-Fi yes this now available in all regions with API version 2024-02-02-preview.

I am still waiting on editing and publication of additional documentation, but this should get you started: https://learn.microsoft.com/en-us/azure/container-apps/managed-identity?tabs=portal%2Cdotnet#use-managed-identity-for-scale-rules

Thanks for your patience. I'll post links to additional documentation once it is published. Please let us know if you have any feedback.

[vturecek]

Additional documentation is now available:
https://learn.microsoft.com/en-us/azure/container-apps/scale-app?pivots=azure-cli#authentication-1

API reference: https://learn.microsoft.com/en-us/rest/api/containerapps/container-apps/create-or-update?view=rest-containerapps-2024-02-02-preview&tabs=HTTP#scalerule

[alexpkent]

I can confirm this is working however the docs on the above links are targeted at Storage Accounts and incorrect for Service Bus queue scaling for example.

Instead of the Scale Rule requiring a metadata property called accountName as the docs state, it needs a metadata property called namespace instead.

[vturecek]

@alexpkent the example here shows a Service Bus queue scale rule with the namespace set: https://learn.microsoft.com/en-us/azure/container-apps/scale-app?pivots=azure-cli#example-3

The other examples we have show Azure Storage queues with accountName, for example: https://learn.microsoft.com/en-us/azure/container-apps/managed-identity?tabs=portal%2Cdotnet#scale-rules

Where do you see an example Service Bus queue scale rule with accountName instead of namespace set in the metadata?

[haflidif]

@vturecek Are there any examples on how to utilize this with the event driven configuration for azure-pipelines KEDA Scaler in Container App Jobs, or is that still not supporting Managed Identity to monitor the queue in Azure DevOps ? and you still need to use PAT there ?

[drdr-vincentvm]

@haflidif I got this working today after a bit of trial and error as I couldn't see how to do it from the docs.

Don't use organizationURLFromEnv in the metadata as you might expect. Keep using the auth element to specify the org, just remove the PAT from it.

See this bicep:

rules: [
  {
    name: 'azure-pipelines'
    type: 'azure-pipelines'
    metadata: {
      poolName: azureDevopsAgentPool
    }
    auth: [
      {
        secretRef: 'azure-devops-organization-url'
        triggerParameter: 'organizationURL'
      }
    ]
    identity: userManagedIdentity.id
  }
]

Set the org URL in your job secrets as well (just as a direct value, no key vault needed) and you're good to go.

I've now completely eliminated PATs in my self hosted agents on Container App Jobs, it's great.

[haflidif]

@haflidif I got this working today after a bit of trial and error as I couldn't see how to do it from the docs.

Don't use organizationURLFromEnv in the metadata as you might expect. Keep using the auth element to specify the org, just remove the PAT from it.

See this bicep:

rules: [

  {

    name: 'azure-pipelines'

    type: 'azure-pipelines'

    metadata: {

      poolName: azureDevopsAgentPool

    }

    auth: [

      {

        secretRef: 'azure-devops-organization-url'

        triggerParameter: 'organizationURL'

      }

    ]

    identity: userManagedIdentity.id

  }

]

Set the org URL in your job secrets as well (just as a direct value, no key vault needed) and you're good to go.

I've now completely eliminated PATs in my self hosted agents on Container App Jobs, it's great.

Awesome 👏 Thank you very much!

I'll try it out at keep you updated

[jemrpo]

@haflidif I got this working today after a bit of trial and error as I couldn't see how to do it from the docs.

Don't use organizationURLFromEnv in the metadata as you might expect. Keep using the auth element to specify the org, just remove the PAT from it.

See this bicep:

rules: [
  {
    name: 'azure-pipelines'
    type: 'azure-pipelines'
    metadata: {
      poolName: azureDevopsAgentPool
    }
    auth: [
      {
        secretRef: 'azure-devops-organization-url'
        triggerParameter: 'organizationURL'
      }
    ]
    identity: userManagedIdentity.id
  }
]

Set the org URL in your job secrets as well (just as a direct value, no key vault needed) and you're good to go.

I've now completely eliminated PATs in my self hosted agents on Container App Jobs, it's great.

I confirm this works perfectly fine.

[andreaskasc]

Hi @vturecek,
I am trying to make it work with Azure Event Hubs scaler. The ACA deployment is done with bicep. The initial rule using connection strings is as following:

I tried replacing the auth section with just the identity but I haven't succeeded.

First off, I get a bicep syntax warning: The property "identity" is not allowed on objects of type "CustomScaleRule". Permissible properties include "auth". And when deploying it, I don't see the identity anywhere in the scaling rule definition and I get errors from KEDA logger, like "unable to get eventhub metadata: no storage connection string given".
Do you have any example using managed identity with Azure Event Hubs scaler?

Edit: I also tried putting the identity inside the metadata section, but still no luck.

[drdr-vincentvm]

@andreaskasc did you update the API version of your ACA resource to 2024-02-02-preview?

[rajravat]

is this only available for cli/api at the minute and not bicep? i cant see a 2024-02-02-preview for bicep and havent been able to get it to work (unless i use cli)

[drdr-vincentvm]

Yes, I've used it in bicep, I just ignore the warning that it doesn't know about 2024-02-02-preview, it doesn't stop the deployment from going through.

[rajravat]

Thanks, i managed to get it working with bicep + ignoring warning

[andreaskasc]

Thanks @drdr-vincentvm I made it work with the changed ACA API version.

For anyone using the Event Hubs scaler, this is how it is supposed to be:

However, since I am using bicep for deployments and this version is not there yet on the bicep module (https://learn.microsoft.com/en-us/azure/templates/microsoft.app/containerapps?pivots=deployment-language-bicep) I will not use it yet.
And the reason is that on one hand I am getting warnings on the deployment and I also do not have any validation on the bicep module itself. And on the other hand, I am getting some others errors not related to KEDA scaling on the probes of the ACA itself. Maybe I need to make some other changes in the bicep module definition for it, but I will wait until 2024-02-02-preview (or better an official version) is available on the bicep module.
@vturecek what is the plan for adding the new ACA version to the bicep module?

[mjrousos]

Has anyone got this working with Service Bus yet? I'm using version 2024-02-02-preview of the ACA API and the container app provisions without error but it doesn't scale.

Scale rule looks like this:

scaleRules: [
      {
        name: 'service-bus-queue-length-rule'
        custom: {
          type: 'azure-servicebus'
          metadata: {
            messageCount: '10'
            namespace: renderRequestServiceBusNamespace
            queueName: renderRequestServiceBusQueueName
          }
          identity: managedIdentity.id
        }
      }
    ]

Using that rule, I got warnings in the ACA system log: "error parsing azure service bus metadata: no connection setting given" so I tried adding the secretRef back and just removing the access key from the Service Bus connection information. It still didn't scale, though, so I may not be approaching it right.

[alexpkent]

This worked for service bus for me, the only difference I can see if the messageCount property

"scale": {
                    "minExecutions": 0,
                    "maxExecutions": 10,
                    "pollingInterval": 30,
                    "rules": [
                        {
                            "name": "myqueuerule",
                            "type": "azure-servicebus",
                            "metadata": {
                                "messageCount": "1",
                                "namespace": "myservicebusnamespace",
                                "queueName": "myqueue"
                            }
                        }
                    ]
                }

[onionhammer]

When will this be released beyond the '-preview' tag?

[mjrousos]

This worked for service bus for me, the only difference I can see if the messageCount property

@alexpkent, did it work for you when using managed identity to authenticate with the service bus? I don't see an 'identity' property in your rule definiton.

[alexpkent]

This worked for service bus for me, the only difference I can see if the messageCount property

@alexpkent, did it work for you when using managed identity to authenticate with the service bus? I don't see an 'identity' property in your rule definiton.

Yes I am using a user assigned managed identity with 'Azure Service Bus Data Owner' on the service bus, I just didn't include that in the sample above. This identity is then assigned to the ACA Job itself not the ACA Environment.

[rckvwijk]

@haflidif I got this working today after a bit of trial and error as I couldn't see how to do it from the docs.

Don't use organizationURLFromEnv in the metadata as you might expect. Keep using the auth element to specify the org, just remove the PAT from it.

See this bicep:

rules: [
  {
    name: 'azure-pipelines'
    type: 'azure-pipelines'
    metadata: {
      poolName: azureDevopsAgentPool
    }
    auth: [
      {
        secretRef: 'azure-devops-organization-url'
        triggerParameter: 'organizationURL'
      }
    ]
    identity: userManagedIdentity.id
  }
]

Set the org URL in your job secrets as well (just as a direct value, no key vault needed) and you're good to go.

I've now completely eliminated PATs in my self hosted agents on Container App Jobs, it's great.

Is it possible that you share your bicep code? I'm trying to get it up and running but it's not really working, at least not reliably, for me. It can scale up but then it keeps the agent in Idle status instead of scaling down again. I'm not sure what the actual problem is tho.

[Choi-jk]

    scalingRules: [
      {
        name: 'topic-based-scaling'
        custom: {
          type: 'azure-servicebus'
          metadata: {
            topicName: serviceBusTopicName
            subscriptionName: serviceBusSubscriptionName
            namespace: serviceBusName
            messageCount: '2'
            activationMessageCount: '1'
          }
          // auth: [
          //   {
          //     secretRef: 'sb-connectionstring'
          //     triggerParameter: 'connection'
          //   }
          // ]
          identity: managedIdentityResourceId
        }
      }
    ]

I'm having issue using managed identity for scaling ACA.

what I'm experiencing

1. Scaling container app's replica based on azure service bus doesn't works when using managed identity
2. Warning states "Msg":"error parsing azure service bus metadata: no connection setting given","Reason":"KEDAScalerFailed"

what I've confirmed

1. ACA api version set to 2024-02-02-preview
2. managed identity has proper rbac for Azure service bus [Azure Service Bus Data Owner]
3. ACA also has same managed identity assigned (not sure this is related but)
4. authentication using "connection string" works with no problem (Just commenting out "identity" and give "auth" enables scaling)

If anyone have clues to this problem, please feel free to give advice.