Glacier2/router client crashes in IceSSL with stress testing

[bentoi]

This error occurred with distribution testing on the Windows 2017 x64 VM. I was able to recover the dump of the client crash (dump available in C:\Users\vagrant\LocalDumps).

The client crashes in the SSL plugin initialization, here's the stack:

>	crypt32.dll!_guard_dispatch_icall_nop�()	Unknown
 	crypt32.dll!TlgAggregateInternalRegisteredProviderEtwCallback(struct _GUID const *,unsigned long,unsigned char,unsigned __int64,unsigned __int64,struct _EVENT_FILTER_DESCRIPTOR *,void *)	Unknown
 	crypt32.dll!_TlgEnableCallback()	Unknown
 	ntdll.dll!EtwpEventApiCallback()	Unknown
 	ntdll.dll!EtwpUpdateEnableInfoAndCallback()	Unknown
 	ntdll.dll!EtwpSetProviderTraits()	Unknown
 	crypt32.dll!TraceLoggingRegisterEx()	Unknown
 	crypt32.dll!TlgRegisterAggregateProviderEx�()	Unknown
 	crypt32.dll!I_PFXDecrypt�()	Unknown
 	crypt32.dll!Decrypt_Private_Key(struct _CRYPT_ALGORITHM_IDENTIFIER,struct _CRYPTOAPI_BLOB,unsigned char *,unsigned long *,void *)	Unknown
 	crypt32.dll!CryptImportPKCS8�()	Unknown
 	crypt32.dll!ImportCAPIKey()	Unknown
 	crypt32.dll!CertImportSafeContents()	Unknown
 	crypt32.dll!I_PFXImportCertStoreEx()	Unknown
 	crypt32.dll!PFXImportCertStore�()	Unknown
 	icessl37.dll!IceSSL::SChannel::SSLEngine::initialize() Line 783	C++
 	ice37.dll!Ice::PluginManagerI::initializePlugins() Line 89	C++
 	ice37.dll!IceInternal::Instance::finishSetup(int & argc, const char * * argv, const IceInternal::Handle<Ice::Communicator> & communicator) Line 1550	C++
 	ice37.dll!Ice::CommunicatorI::finishSetup(int & argc, const char * * argv) Line 565	C++
 	ice37.dll!Ice::initialize(const Ice::InitializationData & initData, int version) Line 354	C++
 	client.exe!StressClient::run() Line 192	C++
 	ice37.dll!startHook(void * arg) Line 411	C++
 	[External Code]	

The test initialize multiple communicators concurrently, it sounds like it could be some sort of Windows thread safety issue.

Similar reports on MSDN forums appears to indicate this is new:
https://social.msdn.microsoft.com/Forums/en-US/3c76728b-5159-4273-8456-2b449540ef9e/random-crashes-in-crypt32dll-with-latest-windows-update-1803?forum=netfxbcl

Not that great but I guess we could have a global mutex for SSL initialization... the probability of this issue hitting an application is quite low however given that you have to create communicators concurrently.

[bentoi]

This failure also showed up with CI testing.