CoInitializeEx return value RPC_E_CHANGED_MODE not handled correctly

[hrydgard]

On Windows, in NFD_OpenDialog, COM is initialized like this:

HRESULT coResult = ::CoInitializeEx(NULL,
                                        ::COINIT_APARTMENTTHREADED |
                                        ::COINIT_DISABLE_OLE1DDE );

Unfortunately, if COM has already been initialized in the process through COINIT_MULTITHREADED, the above call will fail (but COM will still work). See https://docs.microsoft.com/en-us/windows/win32/api/combaseapi/nf-combaseapi-coinitializeex.

And if the above call fails, NFD_OpenFileDialog just bails. This breaks our application.

I can think of three solutions:

• There could be an option to specify COINIT_MULTITHREADED (or that could be the default)
• CoInitialize should happen separately in some kind of NFD_Init, and the failure RPC_E_CHANGED_MODE should be tolerated
• NFD_OpenFileDialog should recognize the RPC_E_CHANGED_MODE error and just roll with it - COM still works if that is returned.

We use this library through https://github.com/saurvs/nfd-rs , but this is easily reproducible by adding a a CoInitializeEx(NULL, COINIT_MULTITHREADED); call to main() in test_opendialog.c.

[gunhaxxor]

Thanks for finding this!
I had the same issue.
I changed the relevant lines in nfd_win.cpp to:

if (autoCoInit.Result() != RPC_E_CHANGED_MODE
		&& !SUCCEEDED(autoCoInit.Result()))
    {
....

so that it won't bail out if that specific error is returned from the CoInitializeEx

[mlabbe]

Fix in master branch. Version bumped to 1.1.5.