[Identity] Enable CAE toggle per token request

[pvaneck]

Motivation

Access tokens are typically valid until the time they expire. For more security, Azure AD offers a feature, Continuous Access Evaluation (CAE), that provides near real-time access control where it continuously monitors and re-evaluates user access.

For CAE to work, both the client and resource API must be CAE-enabled. If a resource API implements CAE and a Python SDK declares itself as CAE ready, the client receives CAE tokens for that resource. The client is expected to handle CAE claim challenges that are given. If CAE responses aren't handled, it's possible that an app could end up in a loop of retrying an API call with a token that is still in the returned lifespan of the token but has been revoked due to CAE.

The current design for CAE-enablement in Python SDKs is to always enable CAE by default (mainly just for user-credentials) even if a service isn't CAE-enabled. This is done by adding "CP1" to as an underlying capability to the underlying MSAL application used for authentication. A user can, optionally, globally disable CAE tokens with the AZURE_IDENTITY_DISABLE_CP1 env variable. However, we want to change this design to be more flexible, allowing the same credential instance to get both CAE tokens and non-CAE tokens.

Sample scenario: a single application could require talking to five different services/resources. Each of these services may or may not be CAE-enabled. We want to be able to create one credential and use it with all of the service clients, and internally, the service SDK's get_token call should be able to choose if a CAE-token should be requested or not.

Modifications

• All relevant credentials (User Credentials + Service Principal Credentials + SharedTokenCacheCredential) now accept and honor an enable_cae keyword argument. This denotes that the token request should include "CP1" client capabilities indicating that the SDK is ready to handle CAE claims challenges.
• Two token caches are now maintained — one for non-CAE tokens and one for CAE-tokens.
  • Two caches are needed because each cache is agnostic to the claims of a token, so CAE-tokens would keep overwriting non-CAE tokens if they were to share the same cache (and vice-versa).
  • If a persistent cache is used. the non-CAE cache filename will have suffix .nocae, and the CAE-cache filename will have suffix .cae.
• The AZURE_IDENTITY_DISABLE_CP1 environment variable is removed since the behavior of the CP1 capability being "always-on" has been changed.

Result

Any get_token calls inside a BearerTokenCredentialPolicy/AsyncBearerTokenCredentialPolicy can now toggle the enable_cae flag depending on if the corresponding service supports CAE or not. This allows more flexibility in when/where CAE tokens are requested/issued.

---

Changelog entry

### Breaking Changes

- CP1 client capabilities (CAE) is no longer always-on by default for user credentials. This capability will now be configured as-needed in each `get_token` request by each SDK.   ([#30777](https://github.com/Azure/azure-sdk-for-python/pull/30777))
  - Since CAE is no longer always-on, the `AZURE_IDENTITY_DISABLE_CP1` environment variable is no longer supported.

Corresponding Azure Core PR: #31012
Closes: #26484

[azure-sdk]

API change check

APIView has identified API level changes in this PR and created following API reviews.

azure-identity
azure-core

[pvaneck]

@xiangyan99 , Would be great to get a review pass on this 😄

[xiangyan99]

Does this reply on azure-core change?

[pvaneck]

Does this reply on azure-core change?

Not necessarily, but would be good to get that one in soon, as well 😄.