[FEATURE REQ] DefaultAzureCredential for local docker testing

[et1975]

Azure.Identity
Testing code that uses DefaultAzureCredential in a container locally seems to require a lot of effort, unless one is willing to supply username/password into the environment.
Creating a service principal and supplying the clientID + Secret is not much better, but also requires a whole lot of additional effort - like setting up the SP, granting the permissions that the developer account already has, etc.

There should be a way to use VS/VSCode/CLI tokens simply by mounting ~/.azure into /root/.azure of the container, unfortunately this does not work today. #12749 mentions installation of the CLI as a working solution, but I just tried this on Alpine and
a) it's a hassle - installing all that stuff on Alpine is error-prone experience and takes a long time (on each build!)
b) it doesn't work, as I still get the exception

SharedTokenCacheCredential authentication failed: Persistence check failed. Inspect inner exception for details
---> Azure.Identity.AuthenticationFailedException: SharedTokenCacheCredential authentication failed: Persistence check failed. Inspect inner exception for details
---> Microsoft.Identity.Client.Extensions.Msal.MsalCachePersistenceException: Persistence check failed. Inspect inner exception for details
---> System.DllNotFoundException: Unable to load shared library 'libsecret-1.so.0' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library liblibsecret-1.so.0: No such file or directory
at Microsoft.Identity.Client.Extensions.Msal.Libsecret.secret_schema_new(String name, Int32 flags, String attribute1, Int32 attribute1Type, String attribute2, Int32 attribute2Type, IntPtr end)
at Microsoft.Identity.Client.Extensions.Msal.LinuxKeyringAccessor.GetLibsecretSchema()
at Microsoft.Identity.Client.Extensions.Msal.LinuxKeyringAccessor.Write(Byte[] data)
at Microsoft.Identity.Client.Extensions.Msal.MsalCacheStorage.VerifyPersistence()
--- End of inner exception stack trace ---
at Microsoft.Identity.Client.Extensions.Msal.MsalCacheStorage.VerifyPersistence()
at Microsoft.Identity.Client.Extensions.Msal.MsalCacheHelper.VerifyPersistence()
at Azure.Identity.MsalClientBase1.GetClientAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.MsalClientBase1.GetClientAsync(Boolean async, CancellationToken cancellationToken)
at Azure.Identity.MsalPublicClient.GetAccountsAsync(Boolean async, CancellationToken cancellationToken)
at Azure.Identity.SharedTokenCacheCredential.GetAccountAsync(Boolean async, CancellationToken cancellationToken)
at Azure.Identity.SharedTokenCacheCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)

[et1975]

I figured a workaround just now:

• have a Dockerfile just for running stuff locally (not a great start, but easier than the alternatives)
• that uses mcr.microsoft.com/azure-cli as the base image and
• in docker-compose has:

volumes:
 - ~/.azure:/root/.azure 

And in the code sets up the auth chain:

ChainedTokenCredential(ManagedIdentityCredential() or EnvironmentCredential(), AzureCliCredential())

This works, but would be great if we didn't need az cli in the first place.

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jongio]

See here for how I do it, which is the same as you, but checkout the CLI install script in my dev container, it's a one liner.

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

VIDEO: https://youtu.be/oDNGs7B2g1A
CODE: https://github.com/jongio/azureclicredentialcontainer

The only thing better than this would be local ManagedIdentity, but that isn't available right now. We have discussed it, but it opens issues that need to be fleshed out.

[nhart12]

Agreed, to be able use/mount IDE azure credentials when local testing would be awesome. Azure CLI bloats images by almost a gig

[jdthorpe]

See here for how I do it, which is the same as you, but checkout the CLI install script in my dev container, it's a one liner.

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

VIDEO: https://youtu.be/oDNGs7B2g1A CODE: https://github.com/jongio/azureclicredentialcontainer

The only thing better than this would be local ManagedIdentity, but that isn't available right now. We have discussed it, but it opens issues that need to be fleshed out.

@jongio, This worked for me up until I upgraded my Azure CLI to 2.33. Now it seems the windows host machine encrypts the tokens in a .bin file, but the linux azure CLI inside the container expects the unencrypted .json file, so I get a message inside the container stating Please run 'az login' from a command prompt to authenticate before using this credential. inside the container, but the same code running on the windows host fetches an access token without issue.

[philipwolfe]

@et1975 @jdthorpe @jongio @christothes I am running into this too. Here is what I came up with. We too need ways for a container running on a QA engineer machine to authenticate to Azure without checking credentials into SCC in a YAML file. Would love some feedback. My goal is to take the access token from the engineer and use it for this session...doesn't need to be long term like the EnvironmentCredential.
philipwolfe@5dff08d
based on ideas from: https://stackoverflow.com/a/61498506/13122820

[jdthorpe]

@philipwolfe this solution may work for you for now. It essentially requires installing a previous version of the Azure CLI onto both the host machine and in the container, logging into Azure (az login) on the host machine, mapping the ~/.azrue directory into the container.

[goenning]

We're also using the CLI solution, but the az cli on developer machines is auto updating to the 2.33 version, so that means every day developers have to downgrade to 2.29.

Based on az cli docs, it's not meant to auto-upgrade by default, but apparently it is...

By default, auto-upgrade for Azure CLI is disabled. If you would like to keep up with the latest version, you can enable auto-upgrade through [configuration](https://docs.microsoft.com/en-us/cli/azure/config).

[tpanum]

Surreal to read that no progress has been made on such a fundamental problem for over a year. Much like the Python counter part (azure-identities), this package simply seems to be poorly designed, as it relies on some unversioned binary to function. Thus this binary dependency has to be baked in to the container images, despite serving no use in production.

The least destructive hack I have come up with is simply to retrieve secrets (e.g. access token) from my host machine (using Azure CLI) and pass it into my docker container using environment variables, and overrule the azure-identity clients, like so:
docker run -e TOKEN=$(az account get-access-token --resource <resource-id> | jq -r .accessToken) my/fantastic-image.

[paupal1]

Also running into this issue... Is there a recommended workaround other than downgrading AzCli version?

[dmitriyse]

Was forced to write a tool that proxies the local tokens for local user (obtained from the DefaultAzureCredential) to the container through the same protocol as MSI are delivered to the ARC enabled servers.
https://github.com/ClrCoder/ClrPro.AzureFX/releases/tag/v0.1.0

This tool should be executed from a developer account on port 40342

$env:ASPNETCORE_URLS="http://+:40342"
.\ClrPro.Azure.LocalCredentialBridge.exe

Then container should have the next env, volumes:

docker ....
      -e IDENTITY_ENDPOINT=http://host.docker.internal:40342/metadata/identity/oauth2/token
      -e IMDS_ENDPOINT=http://host.docker.internal:40342 
      -v %USERPROFILE%/.LocalCredentialBridgeTokens:/var/opt/azcmagent/tokens:ro
      ...

And the DefaultAzureCredential will work inside the container.

[dmitriyse]

Ideally such functionality should be inside Visual Studio out of the box.

1. Docker containers development is a first-class feature of the Visual Studio
2. Azure secret-less resource access is a first-class feature of the Azure SDK
3. Azure connectivity from Visual-Studio again is a first class feature

Why developers should do the IDE enhancement job for the first class features to make them works together ?

Lack of support of zero secrets connectivity is appearing here and there. For example here there was also a problem dotnet/efcore#26491

Please increase the priority of this feature request.
It's spanning a year already.

[esimkowitz]

Hi @jongio, any updates here? While we would like to get all our developers working in Docker containers to improve compatibility with our production environments, requiring a complicated login process versus just running in VS is too much of a burden.

[asanjabi]

@esimkowitz one workaround is to mount a volume that's shared between all containers, you'd have to connect to one and login once, but the rest will be fine after that. You would need to install the CLI on all the images, so there is that. Not ideal, but workable sample

[esimkowitz]

Frankly that seems like more work to explain to my devs and write troubleshooting docs for than to just tell them to test their changes separately against our Linux environments.

[stefanolsenn]

We fixed it by injecting the environment variables into the containers: in our docker-compose file and using InTune to set the environment variables on all developer pc's.

      [...]
        environment:
         - AZURE_TENANT_ID
         - AZURE_CLIENT_ID
         - AZURE_CLIENT_SECRET
      [...]

[esimkowitz]

That kind of fix won't work for us. We do not store client credentials on local dev boxes, we need to have RBAC set up to someone's own account for any dev resources. We are able to use DefaultAzureCredential in Visual Studio with no issue, ideally this should pipe automatically into Docker when running locally.

[dmitriyse]

#19167 (comment)

Please try this approach. Works good enough in our team.
Visual Studio Credential get passed into containers.

~ 1/2 Year, all good, we forgot about this problem.

[mykola-yakovliev]

Just to add another argument to this problem: for someone (like me), who is new to development of cloud solutions using Azure and wants to try things out, it is a little bit frustrating experience to get an exception after you generate the project from a template and just want it to run with zero-configuration needed. Of course, it is not really much critical in my case, but from my point of view, people would expect it to work locally out-of-box equally with or without Docker.

[NCarlsonMSFT]

@jaredpar the volume mount is not for tokens, it is for a Proxy that VS uses to handle Token Requests: https://learn.microsoft.com/en-us/visualstudio/containers/container-tools-configure?view=vs-2022#configure-azure-authentication

[NCarlsonMSFT]

@krukowskid Can you try with the latest preview: https://visualstudio.microsoft.com/vs/preview/ I believe the Alpine issue has been fixed there.

[krukowskid]

@NCarlsonMSFT installed preview version 17.7.0 Preview 5.0 and problem with alpine image still persist.
its working! :)

[NCarlsonMSFT]

@krukowskid could you open an issue over at https://github.com/microsoft/DockerTools with a link to a repro project? I just did a quick test retargeting my sample (https://github.com/NCarlsonMSFT/VisualStudioCredentialExample) to mcr.microsoft.com/dotnet/runtime:6.0-alpine and was able to get a token.

[krukowskid]

@NCarlsonMSFT I've missed that after installing preview version, VS asked me to "Re-enter my credentials". I can confirm that alpine images are working with preview version. Can't wait for the stable release!

[henriksen]

Any immediate plans of making the VS proxy work with AzureCliCredential outside of VS? So I don't have to run the container(s) though Visual Studio?

[kjreuter]

@krukowskid @NCarlsonMSFT Does this only work with WSL? Will it work with the mcr.microsoft.com/dotnet/aspnet:6.0 image?

I followed @krukowskid's blog post's instructions using Docker Desktop (minus WSL) with mcr.microsoft.com/dotnet/aspnet:6.0 and the latest versions of the required packages and Visual Studio (17.7.1), but am getting a 403 on a ManagedIdentityCredential requests to 169.254.169.254. I understand that IMDS uses that IP in Azure, but it is not routable and should not be being used locally, and based on the blog post, I would expect that VisualStudioIdentity would be used locally.

One other item potentially of note, if I run Fiddler, the request succeeds.

Here is the full error:
Azure.Identity.AuthenticationFailedException: 'ManagedIdentityCredential authentication failed: Service request failed.
Status: 403 (connecting to 169.254.169.254:80: connecting to 169.254.169.254:80: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.)

[christothes]

@kjreuter This 403 response from Docker is a known issue - we should have a fix soon for it.

[kjreuter]

@christothes Your workaround and downgrading to version 1.9.0 of Azure.Identity both worked. Thank you! I will keep an eye out for a new version with a fix...

[jonnekleijer]

Some teammates are on mac's and since VS will be retired next year for mac, it would be great to see the VS proxy feature land in VSCode as well.

[svrooij]

Drumroll 🥁 ..... I've managed to use DefaultAzureCredentials in docker with the use of an open-source proxy (build by myself). https://svrooij.io/2023/08/03/emulate-managed-identities/

I know this is not a real solution but at least you can start testing your apps running in docker.

[flaxh]

I tried solution proposed by @krukowskid via example from https://github.com/NCarlsonMSFT/VisualStudioCredentialExample.
I successfully authenticate but only in isolated Azure function app. Do you have any idea why I'm getting "Visual Studio Token provider can't be accessed at /home/.IdentityService/AzureServiceAuth/tokenprovider.json" for in-process function app ?

[NCarlsonMSFT]

@flaxh the error message is that the VisualStudioCredential isn't configured properly. The underlying issue is that the Integrated Azure Functions images don’t have a dotnet runtime so the proxy isn’t able to run. I’ve updated the sample with a new project FunctionAppIntegrated that demonstrates a work-around to unblock debugging.

[rhythmnewt]

@NCarlsonMSFT I tried your visual studio credential and it worked properly for me with the isolated function project, I was able to run the function both locally and in docker using your solution. However, I hit a snag when trying this approach in my solution.

In my solution I have a service bus queue trigger that authenticates using managedidentity (configured in the local.settings.json)
"MyServiceBusConnection__fullyQualifiedNamespace": "[myservicebus].servicebus.windows.net"

In this case when debugging or running the project within docker, the authentication to servicebus fails with this exception in console (it works with VisualStudioCredential when executing natively in VS2022). It just can't authenticate the listener, even though it should be using same VisualStudioCredential under the hood.

Azure.Identity: ManagedIdentityCredential authentication failed: Managed Identity response was not in the expected format. See the inner exception for details.

I was able to create a basic replication using your solution.
In the FunctionAppIsolated project add nuget package:
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.14.0" />
Add new function definition in Function1.cs

        [Function("ProcessMessage")]
        public async Task ProcessMessageAsync([ServiceBusTrigger("%QueueName%", Connection = "ServiceBusConnection")] ServiceBusReceivedMessage receivedMessage, CancellationToken cancellationToken, ServiceBusMessageActions messageActions, FunctionContext context)
        {

            IDictionary<string, string> messageProperties = new Dictionary<string, string>();

            var message = receivedMessage.Body.ToString();
            Console.WriteLine(message);
        }

update local.settings.json

   "QueueName": "[your sample queue]",
   "ServiceBusConnection__fullyQualifiedNamespace": "[sample service bus instance].servicebus.windows.net"

[NCarlsonMSFT]

@rhythmnewt If you're only getting an error from ManagedIdentityCredential then the other identity providers aren't being used. I've found some docs on how this is supposed to work locally: Guidance for developing Azure Functions. If those docs don't help, you're best bet is to contact the Azure Functions team : Azure Functions questions

[github-actions]

Hi @et1975, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

[verydarkmagic]

Hmm, closed by bot after 3 years :D
I think this issue is a really accurate picture of development for Azure as a whole.
Rich on marketing promises, confusing on pricing, and downright abysmal on actual features.

[LL-SRN]

If the az client could retrieve the relevant env variables - or read them out from my local ~/.azure - I could easily write a makefile or similar to pass them to docker run and automate the whole thing without having to commit any secrets to a repo.

As is, it is very strange that I have, on my PC, everything I need to do dotnet run and have everything work, but I don't have any way to pass those necessary things into my container.

[jsheetzmt]

@NCarlsonMSFT Are you aware of any issues with TokenService.Proxy not working when using 17.10.0 Preview 5.0? The TokenService.Proxy bind mount isn't appearing when using latest preview version of VS. Works fine with 17.9

[NCarlsonMSFT]

@jsheetzmt in 17.10 the various helper CLIs for supporting the container tools got moved into one mount /VSTools. Does that appear? Are you having an issue with getting Tokens in 17.10? If so, please open an issue in microsoft/DockerTools.

[svrooij]

I never saw this working, so I checked my settings and it's turned on, I never turned it on, so I guess its on by default. Does not work, never did.

Windows 10
Visual Studio Enterprise 2022 17.9

Back to using my open-source implementation.... https://svrooij.io/2023/08/03/emulate-managed-identities/
Which just works everywhere, without any complicated mounts, just setting a single environment variable and poof it works.

[github-actions]

Hi @et1975, we deeply appreciate your input into this project. Regrettably, this issue has remained unresolved for over 2 years and inactive for 30 days, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

[jongio]

Reopening as we still need to figure this out