[Bug] Bearer error="invalid_token", error_description="The issuer '(null)' is invalid" in v1.14.0

[ThomasBarnekow]

Which version of Microsoft Identity Web are you using?
v1.14.0

Where is the issue?

• Web app
  • Sign-in users
  • Sign-in users and call web APIs
• Web API
  • Protected web APIs (validating tokens)
  • Protected web APIs (validating scopes)
  • Protected web APIs call downstream web APIs
• Token cache serialization
  • In-memory caches
  • Session caches
  • Distributed caches
• Other (please describe)

Is this a new or an existing app?
This is an app under active development for which I have successfully used v1.12.0.

Repro
As there is no change in my code, the repro would be to update a working Web API from v1.12.0 to v1.14.0 and observe that it suddenly fails to validate tokens generated by both Postman v8.7.0 and MSAL for Angular:

"@azure/msal-angular": "^2.0.0",
"@azure/msal-browser": "^2.14.2",

On the server side, I am using ASP.NET Core 5.0 with the following configuration:

appsettings.json

"AzureAd": {
  "Instance": "https://login.microsoftonline.com/",
  "ClientId": "[guid]",
  "ClientSecret": "[removed]",
  "TenantId": "organizations"
},

Startup.ConfigureServices(IServiceCollection services)

services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApi(Configuration.GetSection("AzureAd"))
    .EnableTokenAcquisitionToCallDownstreamApi()
    .AddMicrosoftGraph(Configuration.GetSection("Graph"))
    .AddInMemoryTokenCaches();

services.AddAuthorization(options =>
{
    options.AddPolicy("MinimumScope", policy => policy.RequireClaim("scp", "access_as_user"));
});

Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env)

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }

    app.UseSerilogRequestLogging();

    app.UseHttpsRedirection();
    app.UseRouting();

    // The UseCors call must be placed between UseRouting and UseAuthentication.
    // See https://docs.microsoft.com/en-us/aspnet/core/fundamentals/middleware/?view=aspnetcore-5.0#middleware-order.
    app.UseCors(CorsPolicyName);

    app.UseAuthentication();
    app.UseAuthorization();

    // Custom middleware must be placed between UseAuthorization and UseEndpoints.
    app.UseRoleBasedAccessControl();
    app.UseJsonApi();

    app.UseEndpoints(endpoints =>
    {
        // Require callers to be authenticated and have the minimum scope claim.
        endpoints.MapControllers().RequireAuthorization("MinimumScope");
    });
}

With app.UseRoleBasedAccessControl(), I am inserting my own custom middleware into the pipeline. With v1.14.0, the requests no longer reach my custom middleware.

I've successfully decoded and validated the tokens using both jwt.ms and jwt.io:

{
  "aud": "api://[guid]",
  "iss": "https://sts.windows.net/[guid]/",
  "iat": 1626041859,
  "nbf": 1626041859,
  "exp": 1626045759,
  "acr": "1",
  "aio": "[removed]",
  "amr": [
    "pwd"
  ],
  "appid": "[guid]",
  "appidacr": "0",
  "email": "[email]",
  "family_name": "Barnekow",
  "given_name": "Thomas",
  "ipaddr": "[IP]",
  "name": "Thomas Barnekow",
  "oid": "[guid]",
  "preferred_username": "[email]",
  "rh": "[removed]",
  "scp": "access_as_user",
  "sub": "[removed]",
  "tid": "[guid]",
  "unique_name": "[email]",
  "upn": "[email]",
  "uti": "[guid]",
  "ver": "1.0"
}

As you see, the iss claim is not null. I am wondering why I am getting "ver": "1.0" tokens and do not know how to change that or whether that is an issue.

Expected behavior
Token validation works as in v1.12.0 and no error is returned.

Actual behavior
With v1.14.0, the Web API only returns error responses with status code 401 Unauthorized and a WWW-Authenticate header with a value of Bearer error="invalid_token", error_description="The issuer '(null)' is invalid".

Other information
This issue is likely related to #1167. I added a comment and only then saw it was already closed. However, it seems the root cause is not removed.

[jmprieur]

Thanks @ThomasBarnekow for the heads-up and detailed repros.
We might have overlooked a case, when we have done #1260. We'll have a look

Meanwhile, if you want to set the access token version of your web API to v2.0, you need to do it in the app registration. See https://docs.microsoft.com/azure/active-directory/develop/scenario-protected-web-api-app-registration#accepted-token-version

[ThomasBarnekow]

Thanks, @jmprieur for the super-quick response. I have quickly checked whether using the v2.0 endpoint makes a difference. It does! Using "accessTokenAcceptedVersion": 2, in the manifest to specify the v2.0 endpoint (rather than the v1.0 endpoint) fixes the issue immediately. Now, only my custom middleware complains that the upn claim is no longer present. But that's on my end and can be fixed easily.

[jmprieur]

Thanks for the update, @ThomasBarnekow
We have tested with v1.0 tokens. But I guess something special happens. We'll investigate.
instead of the upn claim, you could use prefered_username. or add an optional claim for your web API: See https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-optional-claims

[ThomasBarnekow]

Thanks for the hint, @jmprieur.

I needed an additional claim (and decided to use the upn) to provide the option for application administrators to add users to my app, using a name or (better) email address that is known to the user (e.g., because that is how they log in or receive email).

I used the upn based on the understanding that this is the name used when logging in (and therefore known to the user), which is also equal to the email address in most cases. Looking at the other claims, it seems like the email claim might be the best claim for my purpose. The preferred_username claim does not have to be an email address.

Would you agree?

[jmprieur]

I guess so, @ThomasBarnekow
@hpsin will confirm.

[ThomasBarnekow]

@jmprieur and @hpsin, am I right in assuming that the preferred_username and the upn always have identical values?

I created test users for development and test purposes, which do not have email addresses. For those users, there is obviously no email claim. Therefore, the preferred_username claim that @jmprieur originally suggested looks like the better claim. The documentation is just not terribly helpful and I don't understand how exactly it relates to the upn claim. Further, I don't know how the upn and preferred_username claims could be (sensibly) different from the user's email claim if there is one.

When creating a user in Azure AD, you specify a UPN. When managing the username in Microsoft 365, you also specify the username portion of an email address and select an email domain to basically define the UPN, right? I've not seen a way for the user to specify a preferred username, so I can only assume (and actually hope) the upn and preferred_username are identical.

I'm happy to ask those questions elsewhere if this is not the place for such questions.

[hpsin]

preferred_username and upn are not guaranteed to be equal. The same is true of email and upn. Some tenants use different UPNs and email addresses in a belief that it is more secure, which is why the claims can be mismatched.

Preferred_username is only suitable for displaying a user-recognizable name to the user in the local UX, as it can change with each login (e.g. if they sign in with a UPN one day, and an email address the next). UPN is the "correct" value for a user-readable identifier for a user, but may not be recognizable to the human.

[ThomasBarnekow]

Thanks, @hpsin. So what you are saying is that the preferred_username will always reflect the "name" last used by the user to sign in, whatever he or she used (e.g., UPN, email, phone number). There is no separate "field" stored anywhere in AAD or elsewhere from which the preferred_username value would be sourced. Right?

Say the UPN and email are different and the user can sign in using only the UPN or email (but no phone number or other "name"). In that case, the preferred_username claim would always be equal to either the upn or email claim. Right?

[jlpstolwijk]

To add to the above: I added logging for the OnAuthenticationFailed event, and I saw that the error message is this:

Microsoft.IdentityModel.Tokens.SecurityTokenInvalidIssuerException: IDW10303: Issuer: 'https://sts.windows.net/<guid>/', does not match any of the valid issuers provided for this application.

So for testing purposes, I changed the value of TenantId in appSettings.json and put a real tenantId there (guid) instead of the current value "organizations". And boom, authentication is working again with version 1.14.0 , without changing access token version of my web API to v2.0.

Unfortunately, this workaround is not an option for me (and for most users, probably), because I need to authenticate users from different AAD tenants.

Anyway, it looks like the problem is related to "TenantId": "organizations" in appSettings.json for v1.0 tokens.

[jmprieur]

Thanks @jlpstolwijk for this investigation.

[jmprieur]

@jlpstolwijk, I've been trying to repro this problem. I changed the manifest of my web API to have it request for a v1.0 token, and changed the TenantId to be common, but I still don't have any error.

Would you have set the issuer yourself?
Do you have repro steps

[hpsin]

Say the UPN and email are different and the user can sign in using only the UPN or email (but no phone number or other "name"). In that case, the preferred_username claim would always be equal to either the upn or email claim. Right?

Almost - unfortunately in guest scenarios, they could sign in using their foreign-tenant UPN, and then you'd have preferred_username = name@domain.com and upn = user_domain.com#EXT#yourTenant.com

If you're looking to ACL based off of human-readable names, it'll generally be a rough time as values change over time. If possible, I'd suggest using app role assignment within the Azure Portal instead, which gives you reliable connection between the user object and the permission you want to grant them. This works best for low-resolution granularity - e.g. "reader" and "writer" roles. If you need to use high resolution rules, you may need to build a widget for the admin that reliably turns human-names into OIDs.

[jlpstolwijk]

@jmprieur tenantId should be “organizations”, not common. Maybe that helps?

[jmprieur]

@jlpstolwijk : I tried organizations too

[jlpstolwijk]

Hmm strange. My manifest for both angular spa (which retrieves token) and .net core api (which cheks token) have "accessTokenAcceptedVersion": null. Dont know if that makes a difference. I’ll try to see if I can post more code later, not on my laptop now. A little more code is in this other comment:
#1167 (comment)

[ThomasBarnekow]

@hpsin, thanks. I only need this for first-time user registration to match a piece of information (e.g., the username or email) that is known to the user and given to an application admin for adding the user to the application. Once the user signs in, the OID is retrieved and used for any further authentication and authorization decision.

The desire is to ask a straightforward question such as "what is your email address" and be able to retrieve that piece of information from the token. However, I also need a solution for test users (in my own tenant) who don't have an email address, which is why the UPN comes into play. For all other cases, I would guess the email address is the best way.

Just to confirm, in your guest scenario, is "user_domain.com" the representation of the guest user's email address (e.g., user@domain.com) or UPN? I would expect this to be the email address. Anyhow, in guest scenarios, the email claim will be present and I can use that.

[ThomasBarnekow]

@jlpstolwijk, did you try to change "accessTokenAcceptedVersion": null to "accessTokenAcceptedVersion": 2? That fixed it for me. In the v2.0 token, the issuer will then no longer be https://sts.windows.net/<guid>/.

[jmprieur]

@jlpstolwijk : I tried organizations too

Actually sorry, I had changed it on the client web app, not the web API. We can repro it. Thanks so much!

[jlpstolwijk]

@jlpstolwijk, did you try to change "accessTokenAcceptedVersion": null to "accessTokenAcceptedVersion": 2? That fixed it for me. In the v2.0 token, the issuer will then no longer be https://sts.windows.net/<guid>/.

Yes I tried that and it seemed to work. However, I prefer not to switch to v2, because of existing clients being logged out etc

[hpsin]

This is a bit tricky. If you need a valid email address, you need to ask for it explicitly. We can't guarantee that the email addresses are addressable, nor the email claim that we create for guest users. I believe the email claim on guest users is the upn of the original account. If you're looking up details against Graph, we should pull in folks from that team to give a better set of options there (which would be best)

Your best bet is pulling in the oid before auth (e.g. syncing down all of the details to your app first) based on what the admin looked up and approved, and then comparing that to the oid in the token.

[ThomasBarnekow]

@hpsin, the documentation says that the email claim is "the addressable email for this user if the user has one." It further says:

This value is included by default if the user is a guest in the tenant. For managed users (the users inside the tenant), it must be requested through this optional claim or, on v2.0 only, with the OpenID scope. For managed users, the email address must be set in the Office admin portal.

Therefore, my very clear expectation would be that, if I configure the optional claim, the email claim contains the email address that a user would provide when asked to provide his or her email address. I confirmed that with different accounts from different tenants. However, in all cases, the UPNs and email addresses are also the same, so I could not confirm that for non-UPN email addresses.

[hpsin]

I'll fix the documentation there, as it's not always the case. My apologies.

[ThomasBarnekow]

What exactly do you mean by "it's not always the case?" In Section 2.5 (Standard Claims), the OpenID Connect Basic Client Implementer's Guide 1.0 states that the email claim is the:

End-User's preferred e-mail address. Its value MUST conform to the RFC 5322 [RFC5322] addr-spec syntax. The RP MUST NOT rely upon this value being unique, as discussed in Section 2.5.3.

[hpsin]

The email claim will always conform to the textual definition of the addr-spec syntax when present, that's not a concern at least. The problem is that the email claims isn't guaranteed to be the user's actual "preferred" email address - simply the email address string that's been recorded for them in the directory, potentially sourced from an incorrect origin (i.e. it could be sourced from a guest account, a federated IDP, manual editing by an admin, etc). We are investigating work to make this a validated field, but that is very future looking.

[jennyf19]

Included in 1.14.1 release