Unreachable network (169.254.169.254:80) to request Access Token outside Azure

[johan-v-r]

Describe the bug

Trying to Authenticate Using Active Directory Managed Identity authentication from local machine (or any other environment outside Azure) gives the below exception.

Initially I created this SO issue that lead me to this breaking change/bug.

Exception message: Tried to get token using Managed Identity. Access token could not be acquired. A socket operation was attempted to an unreachable network. (169.254.169.254:80)

Stack trace:
   at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)
   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open()
   at NetLatest.Program.<TestConnectionAsync>d__2.MoveNext() in C:\SourceCode\NetLatest\NetLatest\Program.cs:line 43
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()

To reproduce

Executing this whilst logged in with AAD

static async Task TestConnectionAsync(string server, string db)
{
	// Use your own server and database.
	string ConnectionString1 = @$"Server={server}; Database={db}; Authentication=Active Directory Managed Identity";

	using (SqlConnection conn = new SqlConnection(ConnectionString1))
	{
		conn.Open();
	}
}

Expected behavior

Get the token... This should be possible from outside Azure right?
The following works 100% like the original tutorial suggested.

static async Task TestConnectionAsync(string server, string db)
{
	// Use your own server and database.
	var connString = @$"Server={server}; Database={db}";
	var token = await new AzureServiceTokenProvider().GetAccessTokenAsync("https://database.windows.net");

	using (SqlConnection conn = new SqlConnection(connString))
	{
		conn.AccessToken = token;
		conn.Open();
	}
}

Further technical details

Microsoft.Data.SqlClient version: 2.1.0
.NET target: 5.0
SQL Server version: Azure SQL
Operating system: Windows 10

Additional context
The environment is fully connected to Azure SQL using AAD, but not running in Azure.

[cheenamalhotra]

Hi @johan-v-r

Managed Identity based authentication only applies to Azure resources that support Managed Identities. This isn't a breaking change but a newly added support in Microsoft.Data.SqlClient.

AzureServiceTokenProvider on the other hand uses various other token providers under the hood:
https://github.com/Azure/azure-sdk-for-net/blob/2b594335737a65cb8f36543ef817deaba5c43f4b/sdk/mgmtcommon/AppAuthentication/Azure.Services.AppAuthentication/AzureServiceTokenProvider.cs#L104-L111

Managed Identity is one of them: new MsiAccessTokenProvider().
It's not comparable with our driver as we only try Managed Identity when that is requested and don't fallback to others.

Related discussion: #730 (comment)
Suggested alternative solution: #730 (comment)

[johan-v-r]

Thanks @cheenamalhotra - I'll implement as suggested, although still unclear as to why it's only accessible in Azure..?

I get that the managed identity only applies to Azure (like my Azure SQL) resources, but now I can't host my API (running under a Managed Identity) on premise or anywhere other than inside Azure, because it can't hit that IP.

Wouldn't making that endpoint "public" also solve our issues here?

[cheenamalhotra]

That endpoint is only available in Azure environments, read more: Azure IMDS Service

IMDS is a REST API that's available at a well-known, non-routable IP address (169.254.169.254). You can only access it from within the VM. Communication between the VM and IMDS never leaves the host. Have your HTTP clients bypass web proxies within the VM when querying IMDS, and treat 169.254.169.254 the same as 168.63.129.16.

[johan-v-r]

Well that's unfortunate. So effectively this authentication has a hard dependency on apps being hosted in the Azure network, with the only alternative to implement your own custom auth provider to bypass all your work...

Maybe worth a note in the docs..? Are there any discussions on changing this going forward?
You can close this issue, thanks for the info!

[cheenamalhotra]

That's actually what this mode was meant to do, support only Managed Identities in Azure. Docs are everywhere (2.1 release notes), we never mentioned it's going to support anything else but Managed Identities in Azure.

What gets confusing is when you somehow associate it to 'AzureServiceTokenProvider' which is actually doing multiple authentication modes at a time. Yes it works with Managed Identity, but that's only one part of the story. It does not change meaning of Managed Identity either.

Why should 'ActiveDirectoryManagedIdentity' ever do everything else, when it's purposely designed to authenticate with Azure's Managed Identities?

Both libraries have different behaviours and so do their APIs. Neither of them tend to mislead, but they're not comparable that way.

Also, this is consistent design in all SQL Client Drivers that support Identity based authentication (JDBC, ODBC, PHP, etc.) with ActiveDirectoryManagedIdentity or ActiveDirectoryMSI keywords. It's another federated authentication mode, and is not meant to do anything else.

[johan-v-r]

Yup you're right. My fundamental understanding of the Azure accounts were wrong.
I was trying to use the same connection string format for all my environments (local dev, build & test, host).

Apologies for the confusion, but I really appreciate this feedback.

[cheenamalhotra]

What you could try is DefaultAzureCredential with Azure.Identity library that offers exact experience you're looking for, as I mentioned in the alternative above.

We will be adding the same support very soon internally with a new mode so your transition to our new authentication mode will become much easier and you will be able to work seamlessly without changing your environments or connection string in future.