Board Review: Introducing an @azure/identity extension API, @azure/identity-persistence and @azure/identity-vscode

[witemple-msft]

This issue is for a board review of the new extension API for @azure/identity as well as two companion packages:

• @azure/identity-persistence, providing access to persistent token caching
• @azure/identity-vscode, extracting the existing VisualStudioCodeCredential into a separate package

This is a scenario and client packages that are unique to the Azure SDK for JavaScript. The scenarios are explained below. We are primarily seeking advice about the names of the packages and the overall architectural approach of creating an extensible Azure Identity API.

Our immediate goal is to reach a consensus on the names of the packages and the extension approach, so that we can proceed with a beta release and user studies.

Contacts and Timeline

• Team responsible for the client library: Azure SDK for JavaScript (Identity)
• Main contacts: Will Temple (witemple), Daniel Rodriguez (daniel.rodriguez)
• Expected GA date for this library: August 10th, 2021

About the client library

This review consists of modifications to the existing @azure/identity library, as well as two new packages that have not previously been released (and therefore have no available reference documentation):

• @azure/identity-persistence
• @azure/identity-vscode

Understanding the Scenario

The Azure Identity package for JavaScript utilizes native machine-code dependencies to support certain functionality. Specifically:

• Storing credentials in an operating system-specific secure cache (DPAPI, macOS Keychain, or libsecret).
• Interacting with the credential storage that the Azure Account VS Code extension uses to store its authentication session information.

However, requiring our downstream customers to install these native module depdendencies has proven problematic. Some of the native dependencies do not offer prebuilt binaries, either, so customers wishing to use that functionality are required to install C/C++ build tools to install those dependencies. We are proposing this alternative extension-based architecture to provide this functionality optionally for customers who want to use Azure Identity, but don't wish to use any machine code dependencies. For example, we have customers who:

• cannot or are unwilling to trust native module dependencies altogether.
• do not wish to or are unable to install a C/C++ toolchain in their build environment (for example, some secure containerized build environments make this difficult).
• would prefer a smaller installation footprint, or are building an application for the browser where they have no use for machine code extensions in the first place.

In the past, we have listed our machine code dependencies as "optional" dependencies. However, changes to NPM v7 have rendered that strategy ineffective, as NPM v7 now attepmpts to install optional dependencies by default. We have also considered leaving the dependencies unlisted altogether, but we are concerned about discoverability issues, and doing so would violate core principles of the JavaScript packaging ecosystem.

In order to alleviate these concerns, we are proposing the following changes:

• Add an extension API to @azure/identity that will enable other package to export "plugin" objects that can be used to enhance @azure/identity.
• [Breaking] Remove VisualStudioCodeCredential from @azure/identity and move it to a new package, @azure/identity-vscode.
  • Also from @azure/identity-vscode, export a plugin object that, when added to @azure/identity at runtime will add VisualStudioCodeCredential to the list of credentials tried by DefaultAzureCredential.
• Create a package @azure/identity-persistence that exports a plugin that will enable persistent token caching through OS-native secure storage when it is added to @azure/identity.

Sample Programs

The proposed design and implementation is currently in a Draft PR state, and links to our most up-to-date draft samples can be found at the following links:

• @azure/identity-persistence sample
• @azure/identity-vscode samples

In general, you can observe that each of the samples follows this pattern:

import { useIdentityExtension, SomeCredential }  from "@azure/identity";
import extension from "@azure/identity-<extension name>";

useIdentityExtension(extension);

// Now Identity is extended with new functionality

JavaScript Prior Art

Some other packages use similar extension systems, and our primary example of such a package in the JavaScript ecosystem is the Chai assertion framework, which we use as part of our testing strategy and which boasts 3.3 million weekly downloads. It has a very similar extension system, for example, the chai-as-promised extension package extends chai with support for asynchronous operations:

import chai from "chai";
import chaiAsPromised from "chai-as-promised";

chai.use(chaiAsPromised);

After "using" the extension, chai is extended with additional functionality as specified by the chaiAsPromised plugin. Our proposal is to extend Azure Identity for JavaScript with a similar system and leverage that system to solve the problem of machine code dependencies in the mainline @azure/identity package.

[lilyjma]

scheduled for 6/3 2-4pm pst

[tg-msft]

I've got a couple of questions I'd love to discuss during the review today:

1. Are there any scenarios where removing VisualStudioCodeCredential could break at runtime? Or is it only breaking the developer workflow?
   • Is there any way we can warn customers who might be affected either at development time or maybe even runtime?
2. Is this an approach we're only using to factor Identity apart?
   • Or are we trying to create a pattern to use across the JS SDK?
3. What is this doing under the hood?
   • Modifying the client's prototype?
   • How are conflicts resolved?
     • Last extension applied wins?
     • Throw an error?
   • Is it possible to use both the modified client and the original client in the same app?
4. Who will we support creating these extensions?
   • Is it just for us?
   • Can other second parties like Graph create supported extensions?
   • Will we allow/enable/encourage external third parties?
5. How does this version?
   • Do we support using new extensions with older clients (back compat)?
     • Is there any support required from the client or is this extension purely additive?
     • And if there is support, how does that version over time as we expand it for new features? Is it part of the public API contract?
   • Do we support using old extensions with newer clients (forward compat)?
     • Do these extensions effectively block the same API from being added on the client in the future?
     • Do we need to treat extensions as if they were part of the client's API contract?
   • Does the extension's package.json define the supported range of clients?
   • Can I use multiple versions of the same extension side by side?
6. How will customers discover these extensions?
   • Pointers in our README/docs?
   • Or can we release identity as a "meta package" that automatically applies our supported extensions and ship a separate identity-core for folks who don't want them?
7. What does the docs experience look like?

[witemple-msft]

@tg-msft some pre-emptive answers below:

Are there any scenarios where removing VisualStudioCodeCredential could break at runtime? Or is it only breaking the developer workflow?

Yes, this is an API breaking change across a major version boundary. A user can no longer import { VisualStudioCodeCredential } from "@azure/identity"; in 2.x (Identity 2.0 is still beta for JavaScript). In addition, VisualStudioCodeCredential is no longer part of DefaultAzureCredential until the extension is registered, which is a behavioral breaking change. While I'm not excited about removing a credential type from the API surface of @azure/identity (and believe me, I looked for many ways to make it possible to inject VisualStudioCodeCredential back into its API surface), it's the only sane way to relieve the tension of this machine-code dependency problem.

Is this an approach we're only using to factor Identity apart? (or a pattern)

While it's primarily about factoring the Identity SDK apart, it sets a precedent in our repo as well by establishing a pattern for doing this kind of dependency injection. The implementation is tightly coupled with the Identity package implementation, though, so it's not a generally reusable abstraction.

What is this doing under the hood?

In general, there is a representation of an "extension" that is exported from the extension package (an "extension provider"), and the identity package contains logic for consuming those extension representations. The extension is basically a function that is called with an "extension context" that allows mutating certain, sanctioned internal properties of the @azure/identity package. For example, they can set a cache plugin for the Node auth flows, and they can add credentials to the DefaultAzureCredential list.

How are conflicts resolved?

The last extension applied has final authority, but they do not necessarily conflict. If it sets a cache plugin, it will overwrite any previously configured cache plugins. However, two different extensions could add separate credentials to the default credential list, for example.

Possible to use the original client and the modified client?

Yes and no. Yes because you could explicitly alias and require two different versions of it. I don't really know what to make of that scenario (@xirzec). An extension mutates static properties of the module. When an extension is registered it is registered for all consumers of that module within the process (however, there could be multiple copies of the package naturally if the version constraints require it, for example I could load Identity 2.0.0-beta.4 but one of my dependencies could load Identity 1.3.0, and the extension would only be applied to the 2.0.0-beta.4 copy).

Who will we support creating these extensions?

The current implementation makes it hard for 3rd party extensions, and the API contract for extensions isn't flexible enough to do much more than what we've already provided (though someone could theoretically use it to provide a custom implementation of a cache plugin, for example). Someone would have to look at our code and see how we interact with the extension system. We use various tricks to hide the implementation details of the extension subsystem from the public API surface, so they'd effectively need a copy of our extensionProvider.ts sources in their own package. To summarize: it's not great for third or even second parties with the draft implementation, but @xirzec and I have some ideas about how to make it a more apparent contract.

How does this version?

The extension API becomes part of our API contract. If we break it, we'd have to treat it as a breaking change, so it's important that we get it right through the beta cycles.

New extensions with old clients?

This would be supported as long as there haven't been breaking changes in the extension API. The identity-vscode package, for example, doesn't rely on any internals from @azure/identity other than the extension API. As long as it doesn't break, there's no reason that a newer VSCC wouldn't be registerable with an old Identity package (as long as we're constraining that to >=2.0.0, since the API doesn't exist in the 1.x series), though it could create some "interesting" dependency graphs.

Old extension, new clients?

Same story as above. As long as the extension API is stable, a newer Identity package should never break an old extension. For example, if the MSAL definition of a cache plugin were to change, we would have to treat that as a breaking change, because the extension API provides access to that system.

Does the extension define version ranges in its package.json?

Yes. The extension packages depend on @azure/identity, and so they specify versions they work with using SemVer like any other packages would.

Can I use multiple versions of the same extension side-by-side?

No. Well, maybe. You can alias packages in NPM and get two versions of the same extension package. With our existing packages, there isn't much point to this, and two versions of the persistence extension would overwrite one another. Two versions of VisualStudioCodeCredential would just result in two copies of that credential being added to the DefaultAzureCredential list.

Discoverability?

We will have pointers in the README files that explain the extensions, and we will also add a "stub" credential to @azure/identity that, if DefaultAzureCredential fails, will display a message explaining the identity-vscode package. When the extension is registered, it'll replace the stub with the real credential, so users will only see the message if DAC fails and VisualStudioCodeCredential wasn't available.

identity-core, identity meta package?

We did consider this option. We also considered the inverted alternative of having @azure/identity and something like @azure/identity-plus. There are certainly advantages to that pattern (it's simpler because you could just have one package be a drop-in replacement for the other), but this pattern of "use " has a lot of purchase in the JS ecosystem already with libraries like Chai and also Express, where you register plugins from other packages using app.use(<plugin>).

What does the docs experience look like?

Personally, I can't wait to find out! At the end of the day, these are just packages, and so they'll get their own pages on docs.ms. The extension API is currently hidden (effectively) from the API listing, so it won't show up, but the useIdentityExtension function will show up as a top-level export from the package.

[tg-msft]

Recording[MS INTERNAL ONLY]