The site crashes because of Keyset does not exist

[hung-doan]

About the application

• Version: OrchardCore.Application.Cms.Core.Targets; 1.0.0-rc2-13450
• I use OrchardCore as a Headless CMS to feed data to a Single Page Web Application.
• So that I use OrchardCore as a administration page only.
• I use OpenIdConnect with JWT token.
• I uses the default setting for the Certificate Storage as "None", that means It will generate a certificate in the App_Data
• I deployed the app to Azure, I use the Free Windows Service Plan.

Problem

• I randomly hit the error [ERR] An exception was thrown attempting to execute the error handler. Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Keyset does not exist
• The Administration page (/Admin) is completely down. I can't access the site due to the Http 500 error.

NOTE:

• I hit the error on Sep 22, and the Certificate in App_Data was created on Sep 19. The expiration is on December, so that It should not the expiration issue.
• Restart the app can resolve the problem.
• I'm not able to preproduce the problem. It sometime happen.
• I found this topic but It's not the case as I don't use a custom module for user management. And my site is completely down. Ref: OIDC implicit flow client login fails when Orchard Core hosted on Azure App Service #3451

Exceptions

Exception 1

2020-09-22 09:45:19.942 +00:00 [ERR] An exception was thrown attempting to execute the error handler.
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Keyset does not exist
   at Internal.NativeCrypto.CapiHelper.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeProvHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeKeyHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 keySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(CspParameters parameters)
   at Internal.Cryptography.Pal.CertificatePal.<>c.<GetRSAPrivateKey>b__66_0(CspParameters csp)
   at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
   at Internal.Cryptography.Pal.CertificatePal.GetRSAPrivateKey()
   at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
   at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKey()
   at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKeyStatus()
   at Microsoft.Extensions.DependencyInjection.OpenIdConnectServerExtensions.AddKey(IList`1 credentials, SecurityKey key)
   at OrchardCore.OpenId.Configuration.OpenIdServerConfiguration.Configure(String name, OpenIddictServerOptions options) in C:\projects\orchardcore\src\OrchardCore.Modules\OrchardCore.OpenId\Configuration\OpenIdServerConfiguration.cs:line 113
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at Microsoft.Extensions.Options.OptionsMonitor`1.<>c__DisplayClass11_0.<Get>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location where exception was thrown ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd(String name, Func`1 createOptions)
   at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at OrchardCore.Diagnostics.DiagnosticsStartupFilter.<>c__DisplayClass3_0.<<Configure>b__1>d.MoveNext() in C:\projects\orchardcore\src\OrchardCore.Modules\OrchardCore.Diagnostics\DiagnosticsStartupFilter.cs:line 47
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)

Exception 2

2020-09-22 09:45:19.943 +00:00 [ERR] Connection ID "17293822575008287453", Request ID "80000ae6-0001-f000-b63f-84710c7967bb": An unhandled exception was thrown by the application.
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Keyset does not exist
   at Internal.NativeCrypto.CapiHelper.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeProvHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeKeyHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 keySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(CspParameters parameters)
   at Internal.Cryptography.Pal.CertificatePal.<>c.<GetRSAPrivateKey>b__66_0(CspParameters csp)
   at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
   at Internal.Cryptography.Pal.CertificatePal.GetRSAPrivateKey()
   at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
   at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKey()
   at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKeyStatus()
   at Microsoft.Extensions.DependencyInjection.OpenIdConnectServerExtensions.AddKey(IList`1 credentials, SecurityKey key)
   at OrchardCore.OpenId.Configuration.OpenIdServerConfiguration.Configure(String name, OpenIddictServerOptions options) in C:\projects\orchardcore\src\OrchardCore.Modules\OrchardCore.OpenId\Configuration\OpenIdServerConfiguration.cs:line 113
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at Microsoft.Extensions.Options.OptionsMonitor`1.<>c__DisplayClass11_0.<Get>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location where exception was thrown ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd(String name, Func`1 createOptions)
   at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at OrchardCore.Diagnostics.DiagnosticsStartupFilter.<>c__DisplayClass3_0.<<Configure>b__1>d.MoveNext() in C:\projects\orchardcore\src\OrchardCore.Modules\OrchardCore.Diagnostics\DiagnosticsStartupFilter.cs:line 47
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
   at OrchardCore.Modules.ModularTenantRouterMiddleware.Invoke(HttpContext httpContext) in C:\projects\orchardcore\src\OrchardCore\OrchardCore\Modules\ModularTenantRouterMiddleware.cs:line 63
   at OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func`2 execute) in C:\projects\orchardcore\src\OrchardCore\OrchardCore.Abstractions\Shell\Scope\ShellScope.cs:line 204
   at OrchardCore.Modules.ModularTenantContainerMiddleware.Invoke(HttpContext httpContext) in C:\projects\orchardcore\src\OrchardCore\OrchardCore\Modules\ModularTenantContainerMiddleware.cs:line 59
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()

Any thoughts and suggestion will be appreciated, thanks everyone!

[hishamco]

/cc @kevinchalet

[hung-doan]

I'm not sure if this is the cause but I found some references:

References

Kudu team: According to this NOTE from the kudu team: https://github.com/projectkudu/kudu/wiki/Configurable-settings#add-user-profile-support-for-a-site

If you new up X509Certificate2 instance from a pfx file or cert encoded containing private key, you may run into CryptographicException: The system cannot find the file specified.. The reason is Windows stores private key as a file under a user profile directory. By default, Azure Web App (AppService) does not load user profile (avoid overhead for majority scenarios where it is not needed). Hence, the The system cannot find the file specified. issue. To work around, set the below appSetting to enable User Profile.

Microsoft: It is also mentioned in the Microsoft's documentation https://docs.microsoft.com/en-us/azure/app-service/configure-ssl-certificate-in-code#load-certificate-from-file

ASP.NET and ASP.NET Core on Windows must access the certificate store even if you load a certificate from a file. To load a certificate file in a Windows .NET app, load the current user profile with the following command in the Cloud Shell.
...
This approach to using certificates in your code makes use of the TLS functionality in App Service, which requires your app to be in Basic tier or above.

Community blogs/discussions:

• https://hajekj.net/2018/07/30/working-with-certificates-in-azure-app-service/
• Reading PFX fails in Azure projectkudu/kudu#2820
• https://stackoverflow.com/questions/61674483/identity-server-4-cant-find-certificate-when-deployed-to-azure-appservice

In summary (my assumptions)

• To read pfx file or cert encoded containing the private key via the code. The "current user profile" must be enabled & loaded.
• However, in Azure, the "Load user profile" feature is disabled by default.

Workaround 1:

• Follow this guide to use the Certificate Management in the Azure Portal, which requires us to generate the certificate and upload it to Azure. Then we get the certificate from the code via the cert Thumbprint. Ref https://docs.orchardcore.net/en/dev/docs/reference/modules/OpenId/#use-the-certificate-in-azure
• NOTE: The Certificates feature only support isolated instance only - which is the Basic instance or above. We can't use this feature in a free/shared instance.

Workaround 2:

• We still read the certificate from the pfx file.
• Then we need to enable the "Load user profile" feature.
  In your Azure Configuration, add this key

WEBSITE_LOAD_USER_PROFILE=1

• NOTE: The Certificates feature only support isolated instance only - which is the Basic instance or above. We can't use this feature in a free/shared instance.

Actually, I don't really understand why the "Load user profile" is disabled by default and I'm still able to read the pfx file successfully and randomly fail. It should be a failure all the time.

Anyway, I will try the Workaround 2 and will update the status later if It works.

[hung-doan]

Beside the X509 certificate. From my point of view, The x509 certificate is a bit complex for a simple site.
I think we should also consider a simpler approach for the private key.
e.g. use a simple 256 or 2048 bits as the private key, which can be easily configured via a string without any complex configuration relates to the certificate.

[kevinchalet]

@hung-doan I initially opted for X.509 certificates as they are perfect containers for private keys. Generation and storage works generally well, but certain "hostile" environments like Azure's free instances indeed make using X.509 certificates a real PITA.

Moving to raw RSA keys instead of X.509 RSA certificates is certainly an option. Would you like to send a PR?

[kevinchalet]

Note: it may also be a good opportunity to store the keys in the database rather than on the file system.

Note 2: keep using this method to ensure we stay "user-profile-loading-disabled"-friendly:

OrchardCore/src/OrchardCore.Modules/OrchardCore.OpenId/Services/OpenIdServerService.cs

Lines 583 to 605 in 954a90d

	private static RSA GenerateRsaSecurityKey(int size)
	{
	// By default, the default RSA implementation used by .NET Core relies on the newest Windows CNG APIs.
	// Unfortunately, when a new key is generated using the default RSA.Create() method, it is not bound
	// to the machine account, which may cause security exceptions when running Orchard on IIS using a
	// virtual application pool identity or without the profile loading feature enabled (off by default).
	// To ensure a RSA key can be generated flawlessly, it is manually created using the managed CNG APIs.
	// For more information, visit https://github.com/openiddict/openiddict-core/issues/204.
	if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
	{
	// Warning: ensure a null key name is specified to ensure the RSA key is not persisted by CNG.
	var key = CngKey.Create(CngAlgorithm.Rsa, keyName: null, new CngKeyCreationParameters
	{
	ExportPolicy = CngExportPolicies.AllowPlaintextExport,
	KeyCreationOptions = CngKeyCreationOptions.MachineKey,
	Parameters = { new CngProperty("Length", BitConverter.GetBytes(size), CngPropertyOptions.None) }
	});
	return new RSACng(key);
	}
	return RSA.Create(size);
	}

[sebastienros]

@deanmarcussen is already working on a feature/service to store secret in various stores. Including certificates. Database will be supported. So this might be the solution to use this service instead of adding a custom way here. This way you get the UI and deployment steps of these secrets for free.

[Piedone]

If anybody stumbles on this: indeed, setting the WEBSITE_LOAD_USER_PROFILE app setting on the Azure Portal to 1 for the site fixed this for me. Thank you @hung-doan to so diligently explain your findings above.

[Piedone]

Wouldn't we want to simply store the certificates (at least for now) in the Azure Data Protection storage, if it's enabled? @kevinchalet what do you think?

[kevinchalet]

@Piedone I'm not sure how that would work: that module is for storing ASP.NET Core Data Protection master keys on Azure (it's based on Azure.Extensions.AspNetCore.DataProtection.Blobs, which basically implements ASP.NET Core Data Protection's IXmlRepository and nothing else) ; you can't store arbitrary data like X.509 certificates with that 😃

[Piedone]

I mean, we could just store the certificates in the same Blob Storage account/container, or similarly in Blob Storage. Basically, we could have something like IOpenIdServerCertificateStorage, with a default implementation that accesses local files like OpenIdServerService does today, and another feature that provides a Blob Storage-based implementation.

[kevinchalet]

Certainly, but how would that solve the issue discussed in this thread? The issue isn't about where the certificates are stored, it's about how they are loaded by the server OS.

[Piedone]

Yeah, sorry, I mixed things in here, because in our case, the original issue here is tightly connected to the certificates being wiped out on server reset. But having its own issue is better: #13205

[kevinchalet]

It's worth mentioning that the recommended approach is to create the signing/encryption certificates yourself and store them in the machine store, instead of relying on the automatic generation stuff (once the certificates are added, you can then select them using the interactive dropdowns via the UI).

Also, using raw RSA keys instead of RSA keys embedded in X.509 certificates (for which we tend to see fewer issues in hostile environments) in conjunction with #7891 might be a better approach than making the storage of the current feature replaceable.

[Piedone]

I see, thanks!