Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Azure Identity 1.9.0 but fails in v 1.10 fails #38218

Closed
BC89 opened this issue Aug 15, 2023 · 19 comments · Fixed by #38321
Closed

[BUG] Azure Identity 1.9.0 but fails in v 1.10 fails #38218

BC89 opened this issue Aug 15, 2023 · 19 comments · Fixed by #38321
Assignees
@christothes
Labels
Azure.Identity Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-author-feedback Workflow: More information is needed from author to address the issue. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Milestone
2023-09

Comments

@BC89
Copy link

BC89 commented Aug 15, 2023

Library name and version

Azure.Identity v1.10.0

Describe the bug

This code works in Azure Identity 1.9.0 but fails in v 1.10 fails:

var credential = new DefaultAzureCredential(new DefaultAzureCredentialOptions
{
    TenantId = tenantid,
    ManagedIdentityClientId = managedidentityclientid,
});


var token = credential.GetToken(new TokenRequestContext(new string[] { $"https://{keyvaultname}.vault.azure.net/" }));
var client = new SecretClient(new Uri($"https://{keyvaultname}.vault.azure.net/"), credential);

If I upgrade the nuget and rollback w/out any other changes in the codebase this breaks every time. The error is as follows:

Azure.Identity.AuthenticationFailedException: 'ManagedIdentityCredential authentication failed: Service request failed.
Status: 403 (connecting to 169.254.169.254:80: connecting to 169.254.169.254:80: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.)

Expected behavior

Functionality would not break.

Actual behavior

Does not work.

Reproduction Steps

Upgrade the nuget and watch it fail.

Environment

<PackageReference Include="Azure.Identity" Version="1.10.0" />
<PackageReference Include="Azure.Monitor.OpenTelemetry.AspNetCore" Version="1.0.0-beta.6" />
<PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="7.0.10" />
<PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.19.5" />
<PackageReference Include="Swashbuckle.AspNetCore" Version="6.5.0" />
<PackageReference Include="Azure.Security.KeyVault.Secrets" Version="4.5.0" />

net7.0

Running inside a docker-compose on windows but linux containers. Have logged in between sessions to ensure local token cache is up to date from within VS 2022 IDE Account Settings -

Microsoft Visual Studio Professional 2022 (64-bit) - Preview
Version 17.8.0 Preview 1.0
@github-actions github-actions bot added Azure.Identity Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-triage Workflow: This issue needs the team to triage. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Aug 15, 2023
@BC89 BC89 changed the title [BUG] [BUG] Azure Identity 1.9.0 but fails in v 1.10 fails Aug 15, 2023
@christothes
Copy link
Member

Hi @BC89 -
Would you mind providing the logging output after reproducing this with logging enabled?

@christothes christothes added the needs-author-feedback Workflow: More information is needed from author to address the issue. label Aug 15, 2023
@github-actions
Copy link

Hi @BC89. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

@github-actions github-actions bot removed the needs-team-triage Workflow: This issue needs the team to triage. label Aug 15, 2023
@BC89
Copy link
Author

BC89 commented Aug 15, 2023 

An unhandled exception of type 'Azure.Identity.AuthenticationFailedException' occurred in System.Private.CoreLib.dll: 'ManagedIdentityCredential authentication failed: Service request failed.
Status: 403 (connecting to 169.254.169.254:80: connecting to 169.254.169.254:80: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.)

Headers:
Connection: close

See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot'
Stack trace:
 >   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
 >   at Azure.Identity.ManagedIdentityCredential.<GetTokenImplAsync>d__16.MoveNext()
 >   at System.Threading.Tasks.ValueTask`1.get_Result()
 >   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
 >   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask`1 task)
 >   at Azure.Identity.ManagedIdentityCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
 >   at Azure.Identity.DefaultAzureCredential.<GetTokenFromSourcesAsync>d__14.MoveNext()
 >   at System.Threading.Tasks.ValueTask`1.get_Result()
 >   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
 >   at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext()
 >   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
 >   at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext()
 >   at System.Threading.Tasks.ValueTask`1.get_Result()
 >   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
 >   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask`1 task)
 >   at Azure.Identity.DefaultAzureCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
 >   at Program.<Main>$(String[] args) in C:\Users\brendan.carroll\source\repos\Bryx\Marketplace-Services\Services\Email\Program.cs:line 37
Loaded '/app/bin/Debug/net7.0/Microsoft.Identity.Client.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

@github-actions github-actions bot added needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team and removed needs-author-feedback Workflow: More information is needed from author to address the issue. labels Aug 15, 2023
@BC89
Copy link
Author

BC89 commented Aug 15, 2023 

[Informational] Azure-Identity: DefaultAzureCredential.GetToken invoked. Scopes: [ https://k......vault.azure.net/ ] ParentRequestId: 
[Informational] Azure-Identity: EnvironmentCredential.GetToken invoked. Scopes: [ https://k.......vault.azure.net/ ] ParentRequestId: 
[Informational] Azure-Identity: EnvironmentCredential.GetToken was unable to retrieve an access token. Scopes: [ https://k.......vault.azure.net/ ] ParentRequestId:  Exception: Azure.Identity.CredentialUnavailableException (0x80131500): EnvironmentCredential authentication unavailable. Environment variables are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/environmentcredential/troubleshoot
[Informational] Azure-Identity: WorkloadIdentityCredential.GetToken invoked. Scopes: [ https://k.......vault.azure.net/ ] ParentRequestId: 
[Informational] Azure-Identity: WorkloadIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://k.......vault.azure.net/ ] ParentRequestId:  Exception: Azure.Identity.CredentialUnavailableException (0x80131500): WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/workloadidentitycredential/troubleshoot
[Informational] Azure-Identity: ManagedIdentityCredential.GetToken invoked. Scopes: [ https://k.......vault.azure.net/ ] ParentRequestId: 
[Informational] Azure-Identity: False MSAL 4.54.1.0 MSAL.NetCore .NET 7.0.10 Linux 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 [2023-08-15 20:37:14Z - e4997ec2-79f6-4a66-9b91-69761dfeeadd] MSAL MSAL.NetCore with assembly version '4.54.1.0'. CorrelationId(e4997ec2-79f6-4a66-9b91-69761dfeeadd)
[Informational] Azure-Identity: False MSAL 4.54.1.0 MSAL.NetCore .NET 7.0.10 Linux 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 [2023-08-15 20:37:14Z - e4997ec2-79f6-4a66-9b91-69761dfeeadd] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False

@thremendus
Copy link

I am also experiencing the same issue. I just upgraded from 1.9.0 to 1.10.0 and started getting a 403 exception when using DefaultCredentials() against AzureKeyVault. I triple checked my permissions and my user is fine. I downgraded back to 1.9.0 and it works again. Worth mentioning I run my .NET 7 Web API from VS 2022 Enterprise in a docker container (linux).

Code in question:

builder.Configuration.AddAzureKeyVault(
       new Uri($"{builder.Configuration["AzureKeyVaultConfiguration:BaseUrl"]}"),
       new DefaultAzureCredential(),
       new AzureKeyVaultConfigurationOptions
       {       
           ReloadInterval = TimeSpan.FromHours(24)
       });

Error:

Service request failed.
Status: 403 (connecting to 169.254.169.254:80: connecting to 169.254.169.254:80: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.)

Headers:
Connection: close

   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueFromCredentialAsync(TokenRequestContext context, Boolean async, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequestAsync(HttpMessage message, TokenRequestContext context)
   at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.AuthorizeRequestOnChallengeAsyncInternal(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.KeyVaultPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.KeyVaultPipeline.GetPageAsync[T](Uri firstPageUri, String nextLink, Func`1 itemFactory, String operationName, CancellationToken cancellationToken)
   at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+MoveNext()
   at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync()
   at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync()
   at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.AzureKeyVaultConfigurationExtensions.AddAzureKeyVault(IConfigurationBuilder configurationBuilder, SecretClient client, AzureKeyVaultConfigurationOptions options)
   at Microsoft.Extensions.Configuration.AzureKeyVaultConfigurationExtensions.AddAzureKeyVault(IConfigurationBuilder configurationBuilder, Uri vaultUri, TokenCredential credential, AzureKeyVaultConfigurationOptions options)
   at Program.<Main>$(String[] args) in C:\Develop\Elephant-Api\Elephant.WebApi\Program.cs:line 36

Screenshot of the code breaking:

image

@christothes
Copy link
Member

Thanks for the additional information.

@BC89 - I have a few things that would help isolate the issue, if you'd be willing to try them. I'll also work on setting up a repro locally, but these things may help short circuit the investigation.

  • Starting from a working state, Try taking a reference to the latest Azure.Core only and see if you can reproduce with just that change.
  • The logging output you provided is missing the [Informational] Azure-Core: Request and [Informational] Azure-Core: Response trace events. Could you provide the logs with those events also? I'd like to confirm that the 403 is coming from the IMDS endpoint request.
  • Please provide the same logging with the fully working configuration. I'd like to confirm whether the response status code from the IMDS endpoint is the same as when this fails.

@thremendus - If you are willing to try the same, please do.

@christothes christothes added the needs-author-feedback Workflow: More information is needed from author to address the issue. label Aug 15, 2023
@github-actions
Copy link

Hi @BC89. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

@github-actions github-actions bot removed the needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team label Aug 15, 2023
@joshfree joshfree moved this from Untriaged to In Progress in Azure Identity SDK Improvements Aug 16, 2023
@joshfree joshfree added this to the 2023-09 milestone Aug 16, 2023
@christothes
Copy link
Member

Just a quick update - I have a local repro and understand the reason we are seeing this new behavior. I'll update this issue when we decide on how to address this behavior.

@christothes
Copy link
Member

We're still deciding how we want to address this issue, since the issue seems to be how Docker Desktop response to requests to an unreachable network. For example, here is a curl command to the IMDS endpoint run on my repro machine across 3 different environments

curl -I http://169.254.169.254

On Windows:
curl: (7) Failed to connect to 169.254.169.254 port 80 after 0 ms: Couldn't connect to server

On WSL2 Ubuntu:
curl: (28) Failed to connect to 169.254.169.254 port 80: Connection timed out

On Docker Desktop from a Linux container:
HTTP/1.0 403 connecting to 169.254.169.254:80: connecting to 169.254.169.254:80: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.
Connection: close

But, as a workaround, when running in the local environment, excluding the ManagedIdentityCredential should avoid this problem.

var options = new DefaultAzureCredentialOptions { ExcludeManagedIdentityCredential = true };
var credential = new DefaultAzureCredential(options);

This was referenced Aug 22, 2023
Open
Merged
This was referenced Aug 22, 2023
Open
Closed
This was referenced Aug 22, 2023
Closed
Closed
Closed
@github-project-automation github-project-automation bot moved this from In Progress to Done in Azure Identity SDK Improvements Aug 22, 2023
@KijoKim76
Copy link

KijoKim76 commented Sep 5, 2023 •
edited
Loading

Thanks. I had the same issue for Open AI + Cognitive Search but christothes's fix seems to working.
https://github.com/Azure-Samples/azure-search-openai-demo-csharp/tree/feature/embeddingSearch
Thanks

@RobertPaulson90
Copy link

Why was this closed? It is still a major issue for me and my team.

@RobertPaulson90
Copy link

But, as a workaround, when running in the local environment, excluding the ManagedIdentityCredential should avoid this problem.

This workaround is not viable in Python when my Function input trigger is service bus using managed identity. I guess I need to start using connection strings in the meantime, until a fix is provided?

@christothes
Copy link
Member

Why was this closed? It is still a major issue for me and my team.

This should be fixed in version 1.10.1 , which was recently released.

@aconsolati
Copy link

aconsolati commented Nov 23, 2023 •
edited
Loading

This is not working when attempting to connect to SQL Azure from within a docker-compose project using Authentication=Active Directory Default in the connection string. The same code works fine when targetting the local Windows OS.

Dockerfile base image: mcr.microsoft.com/dotnet/sdk:8.0
Microsoft Visual Studio Enterprise 2022 (64-bit) - Current
Version 17.8.1

<TargetFramework>net8.0</TargetFramework>
<PackageReference Include="Azure.Identity" Version="1.10.4" />

Connection String: Server=tcp:{redacted}.database.windows.net,1433;Authentication=Active Directory Default;Database={redacted};

docker-compose output stack-trace (abbreviated):

1>#18 8.395 See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
1>#18 8.395  ---> Azure.RequestFailedException: Managed Identity response was not in the expected format. See the inner exception for details.
1>#18 8.395 Status: 403 (connecting to 169.254.169.254:80: connecting to 169.254.169.254:80: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.)
1>#18 8.395
1>#18 8.395 Headers:
1>#18 8.395 Connection: close
1>#18 8.395
1>#18 8.395  ---> System.Text.Json.JsonReaderException: 'c' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
1>#18 8.395    at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
1>#18 8.395    at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
1>#18 8.395    at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first)
1>#18 8.395    at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
1>#18 8.395    at System.Text.Json.Utf8JsonReader.Read()
1>#18 8.395    at System.Text.Json.JsonDocument.Parse(ReadOnlySpan`1 utf8JsonSpan, JsonReaderOptions readerOptions, MetadataDb& database, StackRowStack& stack)
1>#18 8.395    at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedArrayPoolBytes, PooledByteBufferWriter extraPooledByteBufferWriter)
1>#18 8.395    at System.Text.Json.JsonDocument.ParseAsyncCore(Stream utf8Json, JsonDocumentOptions options, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.ManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
1>#18 8.395    --- End of inner exception stack trace ---
1>#18 8.395    at Azure.Identity.ManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.ImdsManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
1>#18 8.395    --- End of inner exception stack trace ---
1>#18 8.395    at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage)
1>#18 8.395    at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage)
1>#18 8.395    at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
1>#18 8.395    at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
1>#18 8.395    at Microsoft.Data.SqlClient.ActiveDirectoryAuthenticationProvider.AcquireTokenAsync(SqlAuthenticationParameters parameters)
1>#18 8.395    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.<>c__DisplayClass147_1.<<GetFedAuthToken>b__1>d.MoveNext()
...

@BC89
Copy link
Author

BC89 commented Dec 27, 2023 •
edited
Loading

Seems like there was a regression here. Running
Microsoft Visual Studio Professional 2022 (64-bit) - Preview
Version 17.9.0 Preview 2.0
with Azure.Identity 1.10.4 and Azure.Security.KeyVault.Secrets 4.5.0

As of a few days ago started getting the 403 error while attempting to get a token from KeyVault. Fixed by simply adding ExcludeManagedIdentityCredential = true but prior it was working fine without. Removing/adding ExcludeManagedIdentityCredential is like an feature error switch now :)

Running inside a docker-compose on windows but linux containers. .Net 8 -FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base

@christothes
Copy link
Member

Seems like there was a regression here. Running Microsoft Visual Studio Professional 2022 (64-bit) - Preview Version 17.9.0 Preview 2.0 with Azure.Identity 1.10.4 and Azure.Security.KeyVault.Secrets 4.5.0

As of a few days ago started getting the 403 error while attempting to get a token from KeyVault. Fixed by simply adding ExcludeManagedIdentityCredential = true but prior it was working fine without. Removing/adding ExcludeManagedIdentityCredential is like an feature error switch now :)

Running inside a docker-compose on windows but linux containers. .Net 8 -FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base

@BC89 - Could you share the details of the exception including the full error message?

@ockert-pretorius
Copy link

ockert-pretorius commented Jan 11, 2024 •
edited
Loading

Im experiencing a similar issue when connecting to EventHubs using the Function Binding EventHubTrigger with an Identity based connection string. The error appears to be related to ManagedIdentity. When running the function locally, the error is logged as a Credential failure, but the credential is successfully obtained via VisualStudioCredential and the function continues to work as expected. However, once the function is deployed, it stops working because the server uses ManagedIdentityCredential. Reverting Azure.Identity back to version 1.9.0 resolves the issue

[2024-01-11T11:05:23.175Z] EnvironmentCredential.GetToken invoked. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:
[2024-01-11T11:05:23.180Z] EnvironmentCredential.GetToken was unable to retrieve an access token. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:  Exception: Azure.Identity.CredentialUnavailableException (0x80131500): EnvironmentCredential authentication unavailable. Environment variables are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/environmentcredential/troubleshoot
[2024-01-11T11:05:23.188Z] WorkloadIdentityCredential.GetToken invoked. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:
[2024-01-11T11:05:23.190Z] WorkloadIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:  Exception: Azure.Identity.CredentialUnavailableException (0x80131500): WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/workloadidentitycredential/troubleshoot
[2024-01-11T11:05:23.196Z] ManagedIdentityCredential.GetToken invoked. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:
[2024-01-11T11:05:23.259Z] False MSAL 4.56.0.0 MSAL.NetCore .NET 6.0.24 Microsoft Windows 10.0.22621 [2024-01-11 11:05:23Z - c4cb8c80-5109-4966-8596-5859a79390a8] MSAL MSAL.NetCore with assembly version '4.56.0.0'. CorrelationId(c4cb8c80-5109-4966-8596-5859a79390a8)
[2024-01-11T11:05:23.275Z] False MSAL 4.56.0.0 MSAL.NetCore .NET 6.0.24 Microsoft Windows 10.0.22621 [2024-01-11 11:05:23Z - c4cb8c80-5109-4966-8596-5859a79390a8] === AcquireTokenForClientParameters ===
[2024-01-11T11:05:23.278Z] SendX5C: False
[2024-01-11T11:05:23.279Z] ForceRefresh: False
[2024-01-11T11:05:23.280Z]
[2024-01-11T11:05:23.287Z] False MSAL 4.56.0.0 MSAL.NetCore .NET 6.0.24 Microsoft Windows 10.0.22621 [2024-01-11 11:05:23Z - c4cb8c80-5109-4966-8596-5859a79390a8]
[2024-01-11T11:05:23.288Z] === Request Data ===
[2024-01-11T11:05:23.289Z] Authority Provided? - True
[2024-01-11T11:05:23.290Z] Scopes - https://eventhubs.azure.net/.default
[2024-01-11T11:05:23.291Z] Extra Query Params Keys (space separated) -
[2024-01-11T11:05:23.294Z] ApiId - AcquireTokenForClient
[2024-01-11T11:05:23.296Z] IsConfidentialClient - True
[2024-01-11T11:05:23.296Z] SendX5C - False
[2024-01-11T11:05:23.297Z] LoginHint ? False
[2024-01-11T11:05:23.298Z] IsBrokerConfigured - False
[2024-01-11T11:05:23.299Z] HomeAccountId - False
[2024-01-11T11:05:23.300Z] CorrelationId - c4cb8c80-5109-4966-8596-5859a79390a8
[2024-01-11T11:05:23.301Z] UserAssertion set: False
[2024-01-11T11:05:23.301Z] LongRunningOboCacheKey set: False
[2024-01-11T11:05:23.302Z] Region configured:
[2024-01-11T11:05:23.303Z]
[2024-01-11T11:05:23.305Z] False MSAL 4.56.0.0 MSAL.NetCore .NET 6.0.24 Microsoft Windows 10.0.22621 [2024-01-11 11:05:23Z - c4cb8c80-5109-4966-8596-5859a79390a8] === Token Acquisition (ClientCredentialRequest) started:
         Scopes: https://eventhubs.azure.net/.default
        Authority Host: login.microsoftonline.com
[2024-01-11T11:05:23.331Z] False MSAL 4.56.0.0 MSAL.NetCore .NET 6.0.24 Microsoft Windows 10.0.22621 [2024-01-11 11:05:23Z - c4cb8c80-5109-4966-8596-5859a79390a8] [Region discovery] Not using a regional authority.
[2024-01-11T11:05:23.333Z] False MSAL 4.56.0.0 MSAL.NetCore .NET 6.0.24 Microsoft Windows 10.0.22621 [2024-01-11 11:05:23Z - c4cb8c80-5109-4966-8596-5859a79390a8] [Instance Discovery] Skipping Instance discovery because it is disabled.
[2024-01-11T11:05:23.337Z] False MSAL 4.56.0.0 MSAL.NetCore .NET 6.0.24 Microsoft Windows 10.0.22621 [2024-01-11 11:05:23Z - c4cb8c80-5109-4966-8596-5859a79390a8] [ClientCredentialRequest] Sending Token response to client credential request endpoint ...
[2024-01-11T11:05:23.352Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=REDACTED
[2024-01-11T11:05:23.358Z] Metadata:REDACTED
[2024-01-11T11:05:23.359Z] x-ms-client-request-id:40f919aa-9f8c-4bb7-bae1-7be24d269181
[2024-01-11T11:05:23.361Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:23.362Z] User-Agent:azsdk-net-Identity/1.10.4 (.NET 6.0.24; Microsoft Windows 10.0.22621)[2024-01-11T11:05:23.363Z] client assembly: Azure.Identity
[2024-01-11T11:05:23.458Z] Retrieving properties for Event Hub: xxxxxxxxxxxx.
[2024-01-11T11:05:23.462Z] DefaultAzureCredential.GetToken invoked. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:
[2024-01-11T11:05:23.493Z] Request [55e1ed99-2a07-420f-abcd-9354df9c6c4d] HEAD http://127.0.0.1:10000/devstoreaccount1/azure-webjobs-eventhub/xxxxxxxxxxxx.servicebus.windows.net/xxxxxxxxxxxx/eme/checkpoint/-1
[2024-01-11T11:05:23.496Z] x-ms-version:2022-11-02
[2024-01-11T11:05:23.496Z] Accept:application/xml
[2024-01-11T11:05:23.497Z] x-ms-client-request-id:55e1ed99-2a07-420f-abcd-9354df9c6c4d
[2024-01-11T11:05:23.498Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:23.499Z] User-Agent:azsdk-net-Storage.Blobs/12.16.0 (.NET 6.0.24; Microsoft Windows 10.0.22621)
[2024-01-11T11:05:23.500Z] x-ms-date:Thu, 11 Jan 2024 11:05:23 GMT
[2024-01-11T11:05:23.501Z] Authorization:REDACTED
[2024-01-11T11:05:23.502Z] client assembly: Azure.Storage.Blobs
[2024-01-11T11:05:23.554Z] Error response [55e1ed99-2a07-420f-abcd-9354df9c6c4d] 404 The specified blob does not exist. (00.1s)
[2024-01-11T11:05:23.556Z] Server:Azurite-Blob/3.26.0
[2024-01-11T11:05:23.557Z] x-ms-error-code:BlobNotFound
[2024-01-11T11:05:23.558Z] x-ms-request-id:52fd0618-632b-4aca-b302-3b73a8489fc7
[2024-01-11T11:05:23.559Z] Date:Thu, 11 Jan 2024 11:05:23 GMT
[2024-01-11T11:05:23.560Z] Connection:keep-alive
[2024-01-11T11:05:23.561Z] Keep-Alive:REDACTED
[2024-01-11T11:05:23.562Z]
[2024-01-11T11:05:23.596Z] Request [4ca7c86f-1a84-47b8-956b-c9f4e14bdcaf] GET http://127.0.0.1:10000/devstoreaccount1/azure-webjobs-eventhub/xxxxxxxxxxxx.servicebus.windows.net/xxxxxxxxxxxx/eme/-1
[2024-01-11T11:05:23.602Z] x-ms-range:bytes=0-268435455
[2024-01-11T11:05:23.602Z] x-ms-version:2022-11-02
[2024-01-11T11:05:23.603Z] Accept:application/xml
[2024-01-11T11:05:23.604Z] x-ms-client-request-id:4ca7c86f-1a84-47b8-956b-c9f4e14bdcaf
[2024-01-11T11:05:23.605Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:23.606Z] User-Agent:azsdk-net-Storage.Blobs/12.16.0 (.NET 6.0.24; Microsoft Windows 10.0.22621)
[2024-01-11T11:05:23.607Z] x-ms-date:Thu, 11 Jan 2024 11:05:23 GMT
[2024-01-11T11:05:23.608Z] Authorization:REDACTED
[2024-01-11T11:05:23.609Z] client assembly: Azure.Storage.Blobs
[2024-01-11T11:05:23.664Z] Error response [4ca7c86f-1a84-47b8-956b-c9f4e14bdcaf] 404 The specified blob does not exist. (00.1s)
[2024-01-11T11:05:23.666Z] Server:Azurite-Blob/3.26.0
[2024-01-11T11:05:23.667Z] x-ms-error-code:BlobNotFound
[2024-01-11T11:05:23.668Z] x-ms-request-id:b7279030-f3fc-4fa8-be21-84addba08fc1
[2024-01-11T11:05:23.669Z] Date:Thu, 11 Jan 2024 11:05:23 GMT
[2024-01-11T11:05:23.670Z] Connection:keep-alive
[2024-01-11T11:05:23.670Z] Keep-Alive:REDACTED
[2024-01-11T11:05:23.672Z] Transfer-Encoding:chunked
[2024-01-11T11:05:23.672Z] Content-Type:application/xml
[2024-01-11T11:05:23.674Z]
[2024-01-11T11:05:24.427Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:00:01. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
[2024-01-11T11:05:24.430Z]  ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
[2024-01-11T11:05:24.431Z]    at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
[2024-01-11T11:05:24.433Z]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[2024-01-11T11:05:24.434Z]    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
[2024-01-11T11:05:24.435Z]    at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
[2024-01-11T11:05:24.436Z]    at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:24.437Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:24.439Z]    --- End of inner exception stack trace ---
[2024-01-11T11:05:24.440Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
[2024-01-11T11:05:24.441Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:24.443Z]    at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:25.268Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] attempt number 1 took 01.1s
[2024-01-11T11:05:25.275Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=REDACTED
[2024-01-11T11:05:25.277Z] Metadata:REDACTED
[2024-01-11T11:05:25.278Z] x-ms-client-request-id:40f919aa-9f8c-4bb7-bae1-7be24d269181
[2024-01-11T11:05:25.279Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:25.282Z] User-Agent:azsdk-net-Identity/1.10.4 (.NET 6.0.24; Microsoft Windows 10.0.22621)[2024-01-11T11:05:25.283Z] client assembly: Azure.Identity
[2024-01-11T11:05:26.315Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:00:01. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
[2024-01-11T11:05:26.316Z]  ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
[2024-01-11T11:05:26.318Z]    at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
[2024-01-11T11:05:26.319Z]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[2024-01-11T11:05:26.320Z]    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
[2024-01-11T11:05:26.321Z]    at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
[2024-01-11T11:05:26.322Z]    at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:26.323Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:26.325Z]    --- End of inner exception stack trace ---
[2024-01-11T11:05:26.325Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
[2024-01-11T11:05:26.326Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:26.327Z]    at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:27.368Z] Request [6ad14344-40dc-41bb-9ee5-5ec95a787f4f] HEAD http://127.0.0.1:10000/devstoreaccount1/azure-webjobs-hosts/locks/csjhbn89vzjg3-2090646957/host
[2024-01-11T11:05:27.376Z] x-ms-version:2021-08-06
[2024-01-11T11:05:27.382Z] Accept:application/xml
[2024-01-11T11:05:27.386Z] x-ms-client-request-id:6ad14344-40dc-41bb-9ee5-5ec95a787f4f
[2024-01-11T11:05:27.386Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:27.387Z] User-Agent:azsdk-net-Storage.Blobs/12.13.0 (.NET 6.0.24; Microsoft Windows 10.0.22621)
[2024-01-11T11:05:27.388Z] x-ms-date:Thu, 11 Jan 2024 11:05:27 GMT
[2024-01-11T11:05:27.389Z] Authorization:REDACTED
[2024-01-11T11:05:27.390Z] client assembly: Azure.Storage.Blobs
[2024-01-11T11:05:27.432Z] Response [6ad14344-40dc-41bb-9ee5-5ec95a787f4f] 200 OK (00.0s)
[2024-01-11T11:05:27.433Z] Server:Azurite-Blob/3.26.0
[2024-01-11T11:05:27.434Z] x-ms-creation-time:Fri, 22 Dec 2023 10:17:04 GMT
[2024-01-11T11:05:27.436Z] x-ms-meta-FunctionInstance:REDACTED
[2024-01-11T11:05:27.437Z] x-ms-blob-type:BlockBlob
[2024-01-11T11:05:27.437Z] x-ms-lease-state:expired
[2024-01-11T11:05:27.438Z] x-ms-lease-status:unlocked
[2024-01-11T11:05:27.439Z] ETag:"0x253ADA9DEF85160"
[2024-01-11T11:05:27.440Z] x-ms-client-request-id:6ad14344-40dc-41bb-9ee5-5ec95a787f4f
[2024-01-11T11:05:27.441Z] x-ms-request-id:da60a674-bcf8-4b62-9a2e-d2fcf93e0eae
[2024-01-11T11:05:27.442Z] x-ms-version:2023-08-03
[2024-01-11T11:05:27.443Z] Date:Thu, 11 Jan 2024 11:05:27 GMT
[2024-01-11T11:05:27.444Z] Accept-Ranges:bytes
[2024-01-11T11:05:27.445Z] x-ms-server-encrypted:true
[2024-01-11T11:05:27.445Z] x-ms-access-tier:Hot
[2024-01-11T11:05:27.446Z] x-ms-access-tier-inferred:true
[2024-01-11T11:05:27.447Z] x-ms-access-tier-change-time:Fri, 22 Dec 2023 10:17:04 GMT
[2024-01-11T11:05:27.448Z] Connection:keep-alive
[2024-01-11T11:05:27.449Z] Keep-Alive:REDACTED
[2024-01-11T11:05:27.449Z] Last-Modified:Thu, 11 Jan 2024 11:03:51 GMT
[2024-01-11T11:05:27.450Z] Content-Length:0
[2024-01-11T11:05:27.451Z] Content-Type:application/octet-stream
[2024-01-11T11:05:27.452Z] Content-MD5:1B2M2Y8AsgTpgAmY7PhCfg==
[2024-01-11T11:05:27.453Z]
[2024-01-11T11:05:27.465Z] Request [1102e74e-0c26-474c-8749-50e136be1a31] PUT http://127.0.0.1:10000/devstoreaccount1/azure-webjobs-hosts/locks/csjhbn89vzjg3-2090646957/host?comp=lease
[2024-01-11T11:05:27.466Z] x-ms-lease-action:acquire
[2024-01-11T11:05:27.467Z] x-ms-lease-duration:15
[2024-01-11T11:05:27.469Z] x-ms-proposed-lease-id:000000000000000000000000CD5B5480
[2024-01-11T11:05:27.470Z] x-ms-version:2021-08-06
[2024-01-11T11:05:27.471Z] Accept:application/xml
[2024-01-11T11:05:27.472Z] x-ms-client-request-id:1102e74e-0c26-474c-8749-50e136be1a31
[2024-01-11T11:05:27.473Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:27.475Z] User-Agent:azsdk-net-Storage.Blobs/12.13.0 (.NET 6.0.24; Microsoft Windows 10.0.22621)
[2024-01-11T11:05:27.476Z] x-ms-date:Thu, 11 Jan 2024 11:05:27 GMT
[2024-01-11T11:05:27.477Z] Authorization:REDACTED
[2024-01-11T11:05:27.477Z] client assembly: Azure.Storage.Blobs
[2024-01-11T11:05:27.522Z] Response [1102e74e-0c26-474c-8749-50e136be1a31] 201 Created (00.0s)
[2024-01-11T11:05:27.524Z] Server:Azurite-Blob/3.26.0
[2024-01-11T11:05:27.525Z] ETag:"0x253ADA9DEF85160"
[2024-01-11T11:05:27.526Z] x-ms-lease-id:000000000000000000000000CD5B5480
[2024-01-11T11:05:27.527Z] x-ms-client-request-id:1102e74e-0c26-474c-8749-50e136be1a31
[2024-01-11T11:05:27.528Z] x-ms-request-id:3c62ce2a-36fd-4b03-abd2-8b90524d9858
[2024-01-11T11:05:27.528Z] x-ms-version:2023-08-03
[2024-01-11T11:05:27.529Z] Date:Thu, 11 Jan 2024 11:05:27 GMT
[2024-01-11T11:05:27.530Z] Connection:keep-alive
[2024-01-11T11:05:27.531Z] Keep-Alive:REDACTED
[2024-01-11T11:05:27.532Z] Last-Modified:Thu, 11 Jan 2024 11:03:51 GMT
[2024-01-11T11:05:27.533Z] Content-Length:0
[2024-01-11T11:05:27.534Z]
[2024-01-11T11:05:27.539Z] Request [05c13415-4d81-429d-a46a-693110c5c7c8] HEAD http://127.0.0.1:10000/devstoreaccount1/azure-webjobs-hosts/locks/csjhbn89vzjg3-2090646957/host
[2024-01-11T11:05:27.540Z] x-ms-version:2021-08-06
[2024-01-11T11:05:27.541Z] Accept:application/xml
[2024-01-11T11:05:27.542Z] x-ms-client-request-id:05c13415-4d81-429d-a46a-693110c5c7c8
[2024-01-11T11:05:27.543Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:27.544Z] User-Agent:azsdk-net-Storage.Blobs/12.13.0 (.NET 6.0.24; Microsoft Windows 10.0.22621)
[2024-01-11T11:05:27.545Z] x-ms-date:Thu, 11 Jan 2024 11:05:27 GMT
[2024-01-11T11:05:27.546Z] Authorization:REDACTED
[2024-01-11T11:05:27.546Z] client assembly: Azure.Storage.Blobs
[2024-01-11T11:05:27.587Z] Response [05c13415-4d81-429d-a46a-693110c5c7c8] 200 OK (00.0s)
[2024-01-11T11:05:27.588Z] Server:Azurite-Blob/3.26.0
[2024-01-11T11:05:27.590Z] x-ms-creation-time:Fri, 22 Dec 2023 10:17:04 GMT
[2024-01-11T11:05:27.590Z] x-ms-meta-FunctionInstance:REDACTED
[2024-01-11T11:05:27.591Z] x-ms-blob-type:BlockBlob
[2024-01-11T11:05:27.592Z] x-ms-lease-duration:fixed
[2024-01-11T11:05:27.593Z] x-ms-lease-state:leased
[2024-01-11T11:05:27.593Z] x-ms-lease-status:locked
[2024-01-11T11:05:27.594Z] ETag:"0x253ADA9DEF85160"
[2024-01-11T11:05:27.595Z] x-ms-client-request-id:05c13415-4d81-429d-a46a-693110c5c7c8
[2024-01-11T11:05:27.596Z] x-ms-request-id:6511ed54-fae6-4c49-85b9-66d6adf3b0f7
[2024-01-11T11:05:27.597Z] x-ms-version:2023-08-03
[2024-01-11T11:05:27.602Z] Date:Thu, 11 Jan 2024 11:05:27 GMT
[2024-01-11T11:05:27.603Z] Accept-Ranges:bytes
[2024-01-11T11:05:27.604Z] x-ms-server-encrypted:true
[2024-01-11T11:05:27.605Z] x-ms-access-tier:Hot
[2024-01-11T11:05:27.606Z] x-ms-access-tier-inferred:true
[2024-01-11T11:05:27.607Z] x-ms-access-tier-change-time:Fri, 22 Dec 2023 10:17:04 GMT
[2024-01-11T11:05:27.608Z] Connection:keep-alive
[2024-01-11T11:05:27.609Z] Keep-Alive:REDACTED
[2024-01-11T11:05:27.609Z] Last-Modified:Thu, 11 Jan 2024 11:03:51 GMT
[2024-01-11T11:05:27.610Z] Content-Length:0
[2024-01-11T11:05:27.611Z] Content-Type:application/octet-stream
[2024-01-11T11:05:27.612Z] Content-MD5:1B2M2Y8AsgTpgAmY7PhCfg==
[2024-01-11T11:05:27.612Z]
[2024-01-11T11:05:27.627Z] Request [1f47b46c-b097-43ad-84e7-1187227ff5be] PUT http://127.0.0.1:10000/devstoreaccount1/azure-webjobs-hosts/locks/csjhbn89vzjg3-2090646957/host?comp=metadata
[2024-01-11T11:05:27.630Z] x-ms-meta-FunctionInstance:REDACTED
[2024-01-11T11:05:27.631Z] x-ms-lease-id:000000000000000000000000CD5B5480
[2024-01-11T11:05:27.632Z] x-ms-version:2021-08-06
[2024-01-11T11:05:27.633Z] Accept:application/xml
[2024-01-11T11:05:27.634Z] x-ms-client-request-id:1f47b46c-b097-43ad-84e7-1187227ff5be
[2024-01-11T11:05:27.635Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:27.636Z] User-Agent:azsdk-net-Storage.Blobs/12.13.0 (.NET 6.0.24; Microsoft Windows 10.0.22621)
[2024-01-11T11:05:27.637Z] x-ms-date:Thu, 11 Jan 2024 11:05:27 GMT
[2024-01-11T11:05:27.638Z] Authorization:REDACTED
[2024-01-11T11:05:27.639Z] client assembly: Azure.Storage.Blobs
[2024-01-11T11:05:27.681Z] Response [1f47b46c-b097-43ad-84e7-1187227ff5be] 200 OK (00.0s)
[2024-01-11T11:05:27.683Z] Server:Azurite-Blob/3.26.0
[2024-01-11T11:05:27.684Z] ETag:"0x224985FF23CAA40"
[2024-01-11T11:05:27.686Z] x-ms-client-request-id:1f47b46c-b097-43ad-84e7-1187227ff5be
[2024-01-11T11:05:27.686Z] x-ms-request-id:dc8a7da4-1e7c-4b9c-9fd5-adae91bea70e
[2024-01-11T11:05:27.687Z] x-ms-version:2023-08-03
[2024-01-11T11:05:27.688Z] Date:Thu, 11 Jan 2024 11:05:27 GMT
[2024-01-11T11:05:27.689Z] x-ms-request-server-encrypted:true
[2024-01-11T11:05:27.690Z] Connection:keep-alive
[2024-01-11T11:05:27.692Z] Keep-Alive:REDACTED
[2024-01-11T11:05:27.693Z] Last-Modified:Thu, 11 Jan 2024 11:05:27 GMT
[2024-01-11T11:05:27.693Z] Content-Length:0
[2024-01-11T11:05:27.694Z]
[2024-01-11T11:05:27.701Z] Host lock lease acquired by instance ID '000000000000000000000000CD5B5480'.
[2024-01-11T11:05:27.893Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] attempt number 2 took 01.1s
[2024-01-11T11:05:27.897Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=REDACTED
[2024-01-11T11:05:27.900Z] Metadata:REDACTED
[2024-01-11T11:05:27.902Z] x-ms-client-request-id:40f919aa-9f8c-4bb7-bae1-7be24d269181
[2024-01-11T11:05:27.903Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:27.904Z] User-Agent:azsdk-net-Identity/1.10.4 (.NET 6.0.24; Microsoft Windows 10.0.22621)[2024-01-11T11:05:27.905Z] client assembly: Azure.Identity
[2024-01-11T11:05:28.949Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:00:01. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
[2024-01-11T11:05:28.952Z]  ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
[2024-01-11T11:05:28.953Z]    at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
[2024-01-11T11:05:28.954Z]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[2024-01-11T11:05:28.958Z]    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
[2024-01-11T11:05:28.959Z]    at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
[2024-01-11T11:05:28.960Z]    at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:28.961Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:28.962Z]    --- End of inner exception stack trace ---
[2024-01-11T11:05:28.963Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
[2024-01-11T11:05:28.964Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:28.965Z]    at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:32.543Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] attempt number 3 took 01.1s
[2024-01-11T11:05:32.551Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=REDACTED
[2024-01-11T11:05:32.553Z] Metadata:REDACTED
[2024-01-11T11:05:32.554Z] x-ms-client-request-id:40f919aa-9f8c-4bb7-bae1-7be24d269181
[2024-01-11T11:05:32.555Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:32.556Z] User-Agent:azsdk-net-Identity/1.10.4 (.NET 6.0.24; Microsoft Windows 10.0.22621)[2024-01-11T11:05:32.557Z] client assembly: Azure.Identity
[2024-01-11T11:05:33.576Z] Request [40f919aa-9f8c-4bb7-bae1-7be24d269181] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:00:01. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
[2024-01-11T11:05:33.577Z]  ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
[2024-01-11T11:05:33.578Z]    at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
[2024-01-11T11:05:33.580Z]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[2024-01-11T11:05:33.581Z]    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
[2024-01-11T11:05:33.582Z]    at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
[2024-01-11T11:05:33.583Z]    at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.584Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.585Z]    --- End of inner exception stack trace ---
[2024-01-11T11:05:33.586Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
[2024-01-11T11:05:33.587Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.589Z]    at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.607Z] False MSAL 4.56.0.0 MSAL.NetCore .NET 6.0.24 Microsoft Windows 10.0.22621 [2024-01-11 11:05:33Z - c4cb8c80-5109-4966-8596-5859a79390a8] Exception type: Azure.Identity.CredentialUnavailableException
[2024-01-11T11:05:33.608Z] ---> Inner Exception Details
[2024-01-11T11:05:33.609Z] Exception type: System.AggregateException
[2024-01-11T11:05:33.610Z] ---> Inner Exception Details
[2024-01-11T11:05:33.611Z] Exception type: System.Threading.Tasks.TaskCanceledException
[2024-01-11T11:05:33.612Z] ---> Inner Exception Details
[2024-01-11T11:05:33.613Z] Exception type: System.Threading.Tasks.TaskCanceledException
[2024-01-11T11:05:33.614Z]
[2024-01-11T11:05:33.615Z]    at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
[2024-01-11T11:05:33.616Z]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[2024-01-11T11:05:33.617Z]    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
[2024-01-11T11:05:33.618Z]    at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
[2024-01-11T11:05:33.621Z]    at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.623Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.624Z] === End of inner exception stack trace ===
[2024-01-11T11:05:33.625Z]
[2024-01-11T11:05:33.625Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
[2024-01-11T11:05:33.627Z]    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.628Z]    at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.629Z]    at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.631Z]    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.632Z] === End of inner exception stack trace ===
[2024-01-11T11:05:33.633Z]
[2024-01-11T11:05:33.635Z]    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
[2024-01-11T11:05:33.636Z]    at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
[2024-01-11T11:05:33.637Z]    at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
[2024-01-11T11:05:33.638Z] === End of inner exception stack trace ===
[2024-01-11T11:05:33.639Z]
[2024-01-11T11:05:33.640Z]    at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
[2024-01-11T11:05:33.641Z]    at Azure.Identity.ManagedIdentityClient.AuthenticateCoreAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
[2024-01-11T11:05:33.642Z]    at Azure.Identity.ManagedIdentityClient.AppTokenProviderImpl(AppTokenProviderParameters parameters)
[2024-01-11T11:05:33.643Z]    at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.SendTokenRequestToAppTokenProviderAsync(ILoggerAdapter logger, CancellationToken cancellationToken)
[2024-01-11T11:05:33.644Z]    at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.GetAccessTokenAsync(CancellationToken cancellationToken, ILoggerAdapter logger)
[2024-01-11T11:05:33.647Z]    at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
[2024-01-11T11:05:33.648Z]    at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
[2024-01-11T11:05:33.657Z] ManagedIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:  Exception: Azure.Identity.CredentialUnavailableException (0x80131500): ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint.
[2024-01-11T11:05:33.659Z]  ---> System.AggregateException (0x80131500): Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:01. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:01. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:01. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:01. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
[2024-01-11T11:05:33.660Z]  ---> System.Threading.Tasks.TaskCanceledException (0x8013153b): The operation was cancelled because it exceeded the configured timeout of 0:00:01. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
[2024-01-11T11:05:33.661Z]  ---> System.Threading.Tasks.TaskCanceledException (0x8013153b): A task was canceled.
[2024-01-11T11:05:33.669Z] VisualStudioCredential.GetToken invoked. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:
[2024-01-11T11:05:39.740Z] Request [0ada2945-0b45-4fa5-a22b-f65bc079f867] PUT http://127.0.0.1:10000/devstoreaccount1/azure-webjobs-hosts/locks/csjhbn89vzjg3-2090646957/host?comp=lease
[2024-01-11T11:05:39.747Z] x-ms-lease-action:renew
[2024-01-11T11:05:39.749Z] x-ms-lease-id:000000000000000000000000CD5B5480
[2024-01-11T11:05:39.750Z] x-ms-version:2021-08-06
[2024-01-11T11:05:39.751Z] Accept:application/xml
[2024-01-11T11:05:39.752Z] x-ms-client-request-id:0ada2945-0b45-4fa5-a22b-f65bc079f867
[2024-01-11T11:05:39.753Z] x-ms-return-client-request-id:true
[2024-01-11T11:05:39.754Z] User-Agent:azsdk-net-Storage.Blobs/12.13.0 (.NET 6.0.24; Microsoft Windows 10.0.22621)
[2024-01-11T11:05:39.755Z] x-ms-date:Thu, 11 Jan 2024 11:05:39 GMT
[2024-01-11T11:05:39.756Z] Authorization:REDACTED
[2024-01-11T11:05:39.757Z] client assembly: Azure.Storage.Blobs
[2024-01-11T11:05:39.796Z] Response [0ada2945-0b45-4fa5-a22b-f65bc079f867] 200 OK (00.0s)
[2024-01-11T11:05:39.799Z] Server:Azurite-Blob/3.26.0
[2024-01-11T11:05:39.801Z] ETag:"0x224985FF23CAA40"
[2024-01-11T11:05:39.803Z] x-ms-lease-id:000000000000000000000000CD5B5480
[2024-01-11T11:05:39.806Z] x-ms-client-request-id:0ada2945-0b45-4fa5-a22b-f65bc079f867
[2024-01-11T11:05:39.807Z] x-ms-request-id:a3372dc5-9f54-40f9-8200-10baeeb4e79e
[2024-01-11T11:05:39.809Z] x-ms-version:2023-08-03
[2024-01-11T11:05:39.810Z] Date:Thu, 11 Jan 2024 11:05:39 GMT
[2024-01-11T11:05:39.811Z] Connection:keep-alive
[2024-01-11T11:05:39.812Z] Keep-Alive:REDACTED
[2024-01-11T11:05:39.813Z] Last-Modified:Thu, 11 Jan 2024 11:05:27 GMT
[2024-01-11T11:05:39.815Z] Content-Length:0
[2024-01-11T11:05:39.816Z]
[2024-01-11T11:05:40.170Z] VisualStudioCredential.GetToken succeeded. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:  ExpiresOn: 2024-01-11T12:05:05.0000000+00:00
[2024-01-11T11:05:40.178Z] DefaultAzureCredential credential selected: Azure.Identity.VisualStudioCredential
[2024-01-11T11:05:40.181Z] DefaultAzureCredential.GetToken succeeded. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:  ExpiresOn: 2024-01-11T12:05:05.0000000+00:00
[2024-01-11T11:05:40.183Z] VisualStudioCredential.GetToken invoked. Scopes: [ https://eventhubs.azure.net/.default ] ParentRequestId:
[2024-01-11T11:05:43.688Z] Completed retrieving properties for Event Hub: xxxxxxxxxxxx.
[2024-01-11T11:05:43.739Z] Request [dae24d15-58a5-4efb-8a12-3c340c0fc369] GET http://127.0.0.1:10000/devstoreaccount1/azure-webjobs-eventhub?restype=container&comp=list&prefix=xxxxxxxxxxxx.servicebus.windows.net%2Fxxxxxxxxxxxx%2Feme%2Fownership%2F&include=Metadata

@whynot1597
Copy link

whynot1597 commented Jan 12, 2024 •
edited
Loading

@christothes

I'm getting this error with the same setup as @BC89. Here is my full error.

Azure.Identity.AuthenticationFailedException
  HResult=0x80131500
  Message=ManagedIdentityCredential authentication failed: Service request failed.
Status: 403 (connecting to 169.254.169.254:80: connecting to 169.254.169.254:80: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable host.)

Headers:
Connection: close

See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
  Source=Azure.Identity
  StackTrace:
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.ManagedIdentityCredential.<GetTokenImplAsync>d__16.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Identity.ManagedIdentityCredential.<GetTokenAsync>d__14.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Identity.DefaultAzureCredential.<GetTokenFromSourcesAsync>d__14.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext()
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Identity.DefaultAzureCredential.<GetTokenAsync>d__11.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueFromCredentialAsync>d__9.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueAsync>d__6.MoveNext()
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueAsync>d__6.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<AuthenticateAndAuthorizeRequestAsync>d__12.MoveNext()
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<ProcessAsync>d__11.MoveNext()
   at Azure.Core.Pipeline.RedirectPolicy.<ProcessAsync>d__7.MoveNext()
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__5.MoveNext()
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__5.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.UserAgentHeaderPolicy.<ProcessAsync>d__3.MoveNext()
   at Azure.Core.PageableHelpers.PageableImplementation`1.<GetNextResponseAsync>d__20.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Azure.Core.PageableHelpers.PageableImplementation`1.<GetAsyncEnumerator>d__15.MoveNext()
   at Azure.Core.PageableHelpers.PageableImplementation`1.<GetAsyncEnumerator>d__15.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass31_2.<<LoadSelectedKeyValues>b__2>d.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass31_2.<<LoadSelectedKeyValues>b__2>d.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.TracingUtils.<CallWithRequestTracing>d__3.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<LoadSelectedKeyValues>d__31.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass30_0.<<InitializeAsync>b__0>d.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<>c__DisplayClass43_0.<<ExecuteWithFailOverPolicyAsync>b__0>d.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<ExecuteWithFailOverPolicyAsync>d__42`1.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<ExecuteWithFailOverPolicyAsync>d__43.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<InitializeAsync>d__30.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<TryInitializeAsync>d__29.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.<LoadAsync>d__28.MoveNext()
   at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at <internal code>

  This exception was originally thrown at this call stack:
    [External Code]

Inner Exception 1:
RequestFailedException: Service request failed.
Status: 403 (connecting to 169.254.169.254:80: connecting to 169.254.169.254:80: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable host.)

Headers:
Connection: close

@christothes
Copy link
Member

Looks like this is due to the error message text changing slightly. Will look at a fix that matches the error message more loosely.

This was referenced Jan 16, 2024
Merged
Closed
Open
Closed
Closed
Closed
Open
Open
@github-actions github-actions bot locked and limited conversation to collaborators Apr 15, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@christothes christothes

Labels
Azure.Identity Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-author-feedback Workflow: More information is needed from author to address the issue. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
Azure Identity SDK Improvements
Archived in project
Milestone
2023-09
Development

Successfully merging a pull request may close this issue.

Handle Docker Desktop 403 response for IMDS endpoint christothes/azure-sdk-for-net
9 participants
@christothes @joshfree @RobertPaulson90 @BC89 @aconsolati @whynot1597 @KijoKim76 @thremendus @ockert-pretorius