TOTP NullReferenceException

[gtbuchanan]

I'm encountering a NullReferenceException any time I try to log in with TOTP on the latest version. Unfortunately, the error is not getting logged on any server in my farm so I don't have much useful information to provide. It occurs on this page after I enter the code, PIN, and click "Sign In". It still occurs when I disable the PIN requirement.

Here's my exported configuration with sensitive information redacted:

adfsmfa.txt

[redhook62]

Hi @gtbuchanan

I looked at your setup, it looks correct. You use a WID farm of 2 servers.
The attributes are stored in ADDS with the "MFA" Schema.

The encryption keys are in RNG Format.
I think that this is where the problem is, and which will explain that the key is invalid.
Are you sure that the key format has not been changed? by you or because of the update.

You can check the format of the stored key for a user by checking the value of the ADDS attribute "MFA-TOTPKey".
If the key has a prefix:
rng: //: this is the RNG format
rsa: //: This is an RSA key common to all users (Certificate)
rsa2: //: this is an RSA key specific to each user (certificate)
aes: //: This is an AES256 symmetric key
ecdh: //: this is an AES256 key generated with ECDH_P256 (Elliptic-Curve Diffie-Hellman)

If it does not fit you can change the parameter in the configuration.

regards

[gtbuchanan]

@redhook62 It doesn't look like the format changed. Here is an example of a test user I created after I upgraded that exhibits the same problem:

I'll try changing to the one of the other formats and see if that resolves the issue.

[redhook62]

Hi, @gtbuchanan

In this case, check the passphrase that you entered during your configuration. the initial default is "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890".
If this still does not work, it may be better to replay the wizard to register a new key ...
On the other hand, does this apply to all users? or else a key had to be copied, an upn checksum is added in the key which normally prevents this (in this case an event is entered in the logs).

Obviously, we were unable to reproduce your problem. No NullReferenceException

regards

[gtbuchanan]

@redhook62

In this case, check the passphrase that you entered during your configuration

I used my own passphrase during setup. I just reset it to make sure it didn't get set back to the default at some point.

If this still does not work, it may be better to replay the wizard to register a new key

I just went through this too but it didn't seem to help.

On the other hand, does this apply to all users?

Correct, no user is able to use TOTP.

---

For the sake of completeness, I have been through the following steps trying to reset my encryption configuration:

1. Set passphrase to a custom value (Security Configuration -> Passphrase)

2. Set RSA to "Use one certificate per user" (Security Configuration -> Asymmetric Keys RSA)

3. Set TOTP Security Key Mode to "Asymmetric Keys RSA" (MFA Providers -> TOTP Multi-Factor Provider)

4. Reset RSA master keys with Register-MFASystemMasterKey -Deploy (tried with and without -Deploy)

5. Manually clear MFA-RSACertificate and MFA-TOTPKey attribute in Active Directory

6. Generate new TOTP key for user (Users management)

7. Restart ADFS and "MFA Notifications" service

At this point, the user has a new certificate and a new TOTP key but still receives the same error when trying to log in (Object reference not set to an instance of the object). Nothing shows up in the event log when the error occurs. I have also tried completing the steps in PowerShell only but it doesn't seem to make a difference.

Is there anything from the above steps that might be missing or out of order? I'm trying to avoid complete settings reset so biometric devices don't need to be re-enrolled (they all still work).

[redhook62]

HI,

The biometric part is not affected by the modification of the keys.

Are your servers on time?

You can find logs in the ADFS eventlog, this can give us the possible error stack.

On our side this is seen as a user error (even if NullReferenceException ...) and it is not logged, only security alerts are logged (eg: key corruption)

Last thing, do you have activated the password policy feature ? if yes, try to deactivate it.

Can you send me a new export of configuration, i want to see if new properties are well provisionned.

regards

[gtbuchanan]

@redhook62 I doubled checked my servers are on the correct time and they are (NTP is working fine). I did have the password policy feature enabled but disabling it did not make a difference. I have tried just about everything I can think of at this point, including running UnRegister-MFASystem and reconfiguring everything from scratch (i.e. not importing config). Same error and nothing in the event log.

Here are the AD attributes for a test user (non-sensitive) that receives the error:

And here is the updated configuration (AFTER manually reconfiguring): adfsmfa.txt

[redhook62]

Hi @gtbuchanan

First, Sorry, i was in holidays.... with not internet.

Of sure, i was not able to reproduce your problem.

With the current logs, I know in which procedure the error occurs. But that's not enough.
I therefore suggest that you send you a "custom build" next week, with a multitude of logs in this procedure.
this will allow us to understand the reasons for your concern..

regards

[redhook62]

Hi @gtbuchanan, @Martin-Gardner

Install and test this build, I put some logs/traces in it, to understand your problem.
Indicate me in return the different stages passed correctly (EventLog / Application)

regards

adfsmfa.zip

[gtbuchanan]

Hi @redhook62 . Here is the event log: adfs-mfa-error.zip

The last message before displaying the error seems to be: "STEP 06 : Identification HTML Gen - Checking NOCODE"

[redhook62]

Hi, @gtbuchanan

Thank you !

The problem seems to occur when you have to test if one or more "Providers" are active, in order to display "I Have no code".
However the method behind is really basic ...
The provider list may not be correctly initialized ...
Investigating !

regards

[redhook62]

Hi,

Just one thing...
Can you disable one of your ADFS servers from your NLB and test ?

Can you send me the logs after clicking on "Sign In" (STEPS starting at 11). We couldn't find any trace of the post-back.

I think it's more of a javascript error (browser side).
Can you verify that there are no errors using your browser debugging

regards

[redhook62]

Hi @gtbuchanan

Sorry to come back to you so late. but we have failed to reproduce your error.
However, and thanks to @apr-un, a javascript error could occur depending on the nanvigator used.
We therefore corrected the problem and tested with Edge, Chrome, Brave and Firefox (yes Firefox generated an error preventing the page from displaying correctly)
I am therefore attaching you a new build (without logs), so that you can assess if this corrects your problem.

regards

adfsmfa.zip

[gtbuchanan]

@redhook62 I have tested out the new build. Unfortunately, my issue persists... There is no JavaScript error on my end. There also aren't any cookies:

I also tried Unregister-MFAComputer on my secondary server as you asked and removed the server from DNS but that didn't seem to make a difference. I even removed my AAAA records just in case it is related to IPv6 but still nothing. I'm at a loss. I'll go back to the build with extra logging and see if I can get something useful.

[apr-un]

@redhook62 is IdentificationForm defined as javascript object somewhere? I can't see any reference to it, so it probably should be found first with document.getElementById('IdentificationForm');" ?

string result = "<form method=\"post\" id=\"IdentificationForm\" >";
//.... some code, 
// I belive that somewhere near is STEP 06 : Identification HTML Gen - Checking NOCODE log
if (RuntimeAuthProvider.GetActiveProvidersCount() > 1) 
                result += "<a class=\"actionLink\" href=\"#\" id=\"nocode\" name=\"nocode\" onclick=\"return SetLinkData(IdentificationForm, '3')\"; style=\"cursor: pointer;\">" + Resources.GetString(ResourcesLocaleKind.Html, "HtmlUIMNoCode") + "</a>";

SetLinkData js method is using this IdentificationForm value as object (frm) to do frm.submit() in code, frm could be not defined if IdentificationForm is not really an javascript/dom object

[redhook62]

Hi @apr-un , @gtbuchanan

I think the form variable is described as well, and it's passed to the javascript function. moreover in debug, this value does exist.

Can you confirm that the problem occurs after validating the TOTP code, and not while viewing the form. if this is the case, the logs show that post is never received (no call on the server side, no logs starting with # 11)

We tested with your config file and found no issues (since we fixed a duplicate ID with Microsoft HTML, as well as a poorly formatted cookie option)

The problem can come from the browser (which browser are you using?) Or from the environment (Antivirus program, Threat Prevention and others).

Also check the ADFS logs, because if we have no traces on our side, it is in the ADFS code that the problem must occur.

Interestingly in your screenshot, this is a SAML 2.0 (SAMLP) application.
There is something in our code concerning SAML / WSFED for the management of Themes, can you test with an application using another protocol (WSFED or openid-connect (which does not manage the themes))

regards

[gtbuchanan]

@redhook62

Can you confirm that the problem occurs after validating the TOTP code, and not while viewing the form. if this is the case, the logs show that post is never received (no call on the server side, no logs starting with # 11)

The error occurs after I click "Sign In" with the correct Code and PIN filled in. There are no logs past STEP 06.

The problem can come from the browser (which browser are you using?)

The problem occurs on both Firefox and Chromium Edge. The browser doesn't seem to matter.

Or from the environment (Antivirus program, Threat Prevention and others).

The environment didn't change. The only thing that changed was upgrading ADFS MFA.

Also check the ADFS logs, because if we have no traces on our side, it is in the ADFS code that the problem must occur.

I have. There are no warnings/errors logged there either when the error occurs.

There is something in our code concerning SAML / WSFED for the management of Themes, can you test with an application using another protocol (WSFED or openid-connect (which does not manage the themes))

I just tested with Open ID Connect and the same error occurs. However, I just tried the latest build you provided in this thread again in Firefox and now I see some warnings (these don't show up in Edge):

[redhook62]

Hi @gtbuchanan

Thanks for these informations.

So, no worries related to the themes.

For Firefox, yes, these are warnings. However ADFS does not use the "SameSite" attribute. so I think we'll take it out ... because no real gain in using it.

At the beginning of next week, I will send you a build, with logs and having removed this attribute on a cookie, which in fact is only used for asynchronous calls to the providers (return sms or approval).
this would correspond since its introduction to the concerns that you're running into, I hope !.

regards

[redhook62]

Hi @gtbuchanan

In order to evacuate the problems one by one, I am attaching you a new build in which we have removed the "SameSite" attribute on the "showoptions" cookie although this does not occur in the totp process.
We are in the process of putting logs to trace the entire TOTP validation (for tomorrow).
Can you tell us, by activating the email provider if the problem also exists, because the validation process is the same up to the provider (TOTP side there are keys and encryption, which is not the case for the email side) )
We still suspect a problem reading the totp key.
Let's move on like this, a new buil tomorrow

regards
adfsmfa.zip

[gtbuchanan]

@redhook62 I enabled the Email provider, set an email on my user, and set the user's default to "Email" but I don't receive the option to use email (not even biometrics even though I have devices configured):

I'm not sure if this is a bug or what. I don't remember having this problem before. I'd think all 3 options I have available should show up (TOTP, Email, Biometrics).

Yes, very strange !
I have the impression here that access to your ADDS is refused, only the mandatory provider appears.
I saw that you had extended the ADDS schema with the MFA specific attributes, did you use the attributes marked as "CONFIDENTIAL"? does the ADDS data access account really have the correct rights? are these rights that worked before still valid ?

regards

[apr-un]

@gtbuchanan I can be wrong but I think that there You'll see only options which were configured (present at moment of user registration). Please try to delete and register this user again (if possible).
Please also look at browser console (f12) if there are any js errors

Edit: Clearing browser cache can help too!

[redhook62]

Hi @gtbuchanan, @apr-un

Attached is a new build which includes Traces.
In order not to pollute the eventlog, you just have to launch "DbgView" by Mark Russinovich (Sysinternals) to capture the OUTPUTSTRINGS of the System.

However, I still do not understand your problem. Which version remains functional for you?
Because we think we haven't made any major changes for a long time.

adfsmfa.zip

Please save the logs and send them to us.

regards

[redhook62]

Hi @gtbuchanan , @apr-un

If you can't see the positioned traces, then the page postback is never received, having no exceptions in our code, this probably happens in ADFS code.

But, I think that the cause of the problem could be upstream, for example a too intrusive firewall, analyzing the flows and modifying them.
If you think that this is the case, you have to put this element in "Learning" mode, or good for the test phases to deactivate its filtering.

on the other hand, make sure that your secret key and your passwords are correct (encryption). you can export your configuration and enter this information in plain text, then import the modified configuration file.

regards

[redhook62]

Hi,

No news for me... really i don't know where your problem occurs.
There's a new release. perhaps it can help you.

regards

[Phfas]

oops, commented on the wrong issue

I've also started receiving these errors. They most commonly pop-up in Firefox. To get it to work, I need to clear my browsing history, restart my browser, and sometimes open a new tab. It can also happen in Chrome, and Chromium Deriviates(Opera,Vivaldi,Edge). But usually the Chrome based browsers are easier to recover without wiping browsing history.

This started happening really frequently as of the latest update for me.
Is there anything I can do to help out, by providing logs or anything?

[redhook62]

oops, commented on the wrong issue

I've also started receiving these errors. They most commonly pop-up in Firefox. To get it to work, I need to clear my browsing history, restart my browser, and sometimes open a new tab. It can also happen in Chrome, and Chromium Deriviates(Opera,Vivaldi,Edge). But usually the Chrome based browsers are easier to recover without wiping browsing history.

This started happening really frequently as of the latest update for me.
Is there anything I can do to help out, by providing logs or anything?

Yes, I know that there are some JS errors reported by firefox in the MFA but also in ADFS ..., but with Edge there are none. All without causing me any problems of use (Edge, Chrome, Brave, FireFox) so logs why not, but I have already looked at them for a long time.
As for FireFox, much too rigid and limited, I would not try too hard for firefox. On the other hand, if you have some solutions, I invite you to share them with pull requests.

regards

[redhook62]

Hi @gtbuchanan

New version 3.1.2109.0.

One of my clients seemed to have the same problem as you. diagnosed as a problem of configuration of the rights on their ADDS without being able to clearly identify their problem.
The creation of the MFA service startup cache files by the system account resolved the issue.

regards

[Phfas]

I eventually found only one of my ADFS servers was throwing this error, so we had evicted it from the load balance group and just ran on the other one.
I confirmed it was still throwing the error this morning and then performed the update.

1. Uninstalled 3.1.2108.0
2. Installed 3.1.2109.0

initial test still showed the error, so I tried a reinstall with a reboot in between steps

1. Uninstalled 3.1.2109.0
2. Reboot
3. Installed 3.1.2109.0
4. Reboot

After this process, it seems to have resolved the issue for us.
We've been running it for about 6 hours now, so I could just be a little premature. heh.

[redhook62]

Hi, @Phfas, @gtbuchanan and all

It still seems that there is a difference in the rights between your 2 servers.
The root of the problem was that the "system.db" file which stores and hides security information was created by the first invocation of the MMC or PowerShell modules. in this context, if the "administrator" was unable to retrieve this information (SIDs from ADDS and Information about the administration group in ADFS). As a general rule, this does not pose a problem if the prerequisites are met.

There is also an important difference depending on the configuration of adfs (SQLServer or WID). To simplify, a secondary server (WID) cannot access local information in its SQL Express database with powershell commands. the secondary servers therefore make requests to the primary servers. this remains true today.

The modification made in last version, ensures that this "system.db" file will be correctly recreated each time by the "MFA Notification Hub" when the service (mfanotifhub) is started, and dispatch normally to the other servers.
When the MFA service is started on a secondary server, it tries to retrieve a copy of this file.
After the system.db file is properly stored on each server, MMC, ADFS service, PowerShell commands only use the cache file (subject to ACLs on directories)

I think that the order of installation of the component (since the service starts at the end of the installation) must be, first to install on the Primary servers, and then only on the secondary servers.

In principle, there is no need to restart the servers. restarting the "mfanotifhub" service on a primary server will suffice to dispatch this file to all the other servers (provided that the components are installed, that the firewalls are open (port 5987) and that the services are started (the mfanotifhub service stops and also restart ADFS services)

On the other hand, as indicated, having never succeeded in observing this problem, the proposed evolution is more coherent and more reliable. without providing the reason for this malfunction.

In October there will be an update regarding WebAuth

regards

[parisarcane]

Hello,
I had the same error. I updated the MFA certificate security settings and restarted the Federation Service, and the problem was resolved.

[redhook62]

Hi @parisarcane

Your ADFS configuration does not seem correct!
There is no Identifier for your federation platform.
This is not normal, please fix and test again

regards

[parisarcane]

what is identifire?