64 bit VST on Linux/Wine

[zonkmachine]

Wine supports 64-bit since wine-1.2 but it doesn't seem to be working under linux/wine/lmms.
#4102, https://lmms.io/forum/viewtopic.php?f=7&t=26647#p51235

• Assess the status/quality of 64-bit VST on Linux/Wine
• If possible, activate 64-bit VST on Linux/Wine
• Warn if needed/possible when running 64-bit VST on Linux/Wine
• Update documentation

---

Wine 3.0 is in the works and rc5 was recently released.
https://www.winehq.org/

[lukas-w]

I did some work to make building 64-bit RemoteVstPlugin work on Linux in the msvc/vst branch, but it fails loading VSTs with the message

RemotePlugin::DebugMessage: mainEntry procedure returned NULL

[lukas-w]

Fixed the error via 631a3db. 64bit VSTs load perfectly now when hard-coding LMMS to always use the 64-bit version of RemoteVstPlugin.

Unfortunately, there's still something going wrong with the trial-and-error approach we're using for starting the right version:

lmms/plugins/vst_base/VstPlugin.cpp

Lines 100 to 109 in 631a3db

	#ifdef LMMS_BUILD_WIN64
	tryLoad( "RemoteVstPlugin64" );
	if( m_badDllFormat )
	{
	m_badDllFormat = false;
	#endif
	tryLoad( "RemoteVstPlugin32" );
	#ifdef LMMS_BUILD_WIN64
	}
	#endif

The IdVstBadDllFormat message isn't always received for some reason and even when it is, starting RemoteVstPlugin32 after RemoteVstPlugin64 has already been started makes LMMS hang indefinitely.

[tresf]

@lukas-w if we do a quick file read on the DLL, we should be able to tell if it's 64-bit or not without the trial-and-error approach. https://stackoverflow.com/a/35418180/3196753.

[lukas-w]

@tresf Thanks for sharing. That would work around the issue and I think we should implement it, but I suspect some deeper IPC bug causing this that may resurface, so I think this is worth investigating first.

Update: IdVstBadDllFormat isn't received because LMMS stops processing messages from the plugin as soon as the plugin quits. This was partially fixed in 4fd8ecd and is fully fixed in 9db8cbf. Even after that fix, LMMS hangs, which is due to missing cleanup in VstPlugin.cpp when re-starting. This is too much work fixing so I implemented the approach suggested by @tresf via 9db8cbf.

[Spekular]

Closing and consolidating into #5433