feat(csharp/test/Drivers/Databricks): Add mandatory token exchange

[alexguo-db]

Motivation

Databricks will eventually require that all non-inhouse OAuth tokens be exchanged for Databricks OAuth tokens before accessing resources. This change implements mandatory token exchange before sending Thrift requests. This check and exchange is performed in the background for now to reduce latency, but it will eventually need to be blocking if non-inhouse OAuth tokens will fail to access Databricks resources in the future.

Key Components

1. JWT Token Decoder - Decodes JWT tokens to inspect the issuer claim and determine if token exchange is necessary
2. MandatoryTokenExchangeDelegatingHandler - HTTP handler that intercepts requests and performs token exchange when required
3. TokenExchangeClient - Handles the token exchange logic with the same /oidc/v1/token endpoint as token refresh, with slightly different parameters

Changes

• Added new connection string parameter: IdentityFederationClientId for service principal workload identity federation scenarios
• Implemented token exchange logic that checks JWT issuer against workspace host
• Introduced fallback behavior to maintain backward compatibility if token exchange fails

Testing

dotnet test --filter "FullyQualifiedName~MandatoryTokenExchangeDelegatingHandlerTests"

[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.1+bf6400fd51 (64-bit .NET 8.0.7)
[xUnit.net 00:00:00.06]   Discovering: Apache.Arrow.Adbc.Tests.Drivers.Databricks
[xUnit.net 00:00:00.15]   Discovered:  Apache.Arrow.Adbc.Tests.Drivers.Databricks
[xUnit.net 00:00:00.16]   Starting:    Apache.Arrow.Adbc.Tests.Drivers.Databricks
[xUnit.net 00:00:01.77]   Finished:    Apache.Arrow.Adbc.Tests.Drivers.Databricks
  Apache.Arrow.Adbc.Tests.Drivers.Databricks test net8.0 succeeded (2.6s)

Test summary: total: 11, failed: 0, succeeded: 11, skipped: 0, duration: 2.6s

dotnet test --filter "FullyQualifiedName~TokenExchangeClientTests"

[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.1+bf6400fd51 (64-bit .NET 8.0.7)
[xUnit.net 00:00:00.06]   Discovering: Apache.Arrow.Adbc.Tests.Drivers.Databricks
[xUnit.net 00:00:00.14]   Discovered:  Apache.Arrow.Adbc.Tests.Drivers.Databricks
[xUnit.net 00:00:00.15]   Starting:    Apache.Arrow.Adbc.Tests.Drivers.Databricks
[xUnit.net 00:00:00.23]   Finished:    Apache.Arrow.Adbc.Tests.Drivers.Databricks
  Apache.Arrow.Adbc.Tests.Drivers.Databricks test net8.0 succeeded (0.8s)

Test summary: total: 19, failed: 0, succeeded: 19, skipped: 0, duration: 0.8s

dotnet test --filter "FullyQualifiedName~JwtTokenDecoderTests"

[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.1+bf6400fd51 (64-bit .NET 8.0.7)
[xUnit.net 00:00:00.06]   Discovering: Apache.Arrow.Adbc.Tests.Drivers.Databricks
[xUnit.net 00:00:00.14]   Discovered:  Apache.Arrow.Adbc.Tests.Drivers.Databricks
[xUnit.net 00:00:00.15]   Starting:    Apache.Arrow.Adbc.Tests.Drivers.Databricks
[xUnit.net 00:00:00.19]   Finished:    Apache.Arrow.Adbc.Tests.Drivers.Databricks
  Apache.Arrow.Adbc.Tests.Drivers.Databricks test net8.0 succeeded (0.8s)

Test summary: total: 10, failed: 0, succeeded: 10, skipped: 0, duration: 0.8s

Also tested E2E manually with AAD tokens for Azure Databricks workspaces, AAD tokens for AWS Databricks workspaces, and service principal workload identity federation tokens

[CurtHagenlocher]

Can this also be added to the documentation at csharp\src\Drivers\Databricks\readme.md?

I haven't always been paying attention to that, so there may be a bit of a gap now which could be addressed all at once with a separate PR.