Symbol not found: _aligned_alloc on MacOS 10.9.5

[jonoomph]

A user reported this to me on MacOS 10.9.5 running our latest Mac builds. When launching OpenShot, an error during loading PyQt5.QtCore.so, looking for a missing symbol: Symbol not found: _aligned_alloc. It is expecting that symbol in /usr/lib/System.B.dylib.

We are currently building with GCC 8.4 on MacOS 10.15 (Catalina), and using the MacOS 10.9 SDK while building. Hopefully this can be solved with some compiler flags, or something simple. Our current CMake command used on our Mac builder:

cmake -DCMAKE_CXX_FLAGS=-I\ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -D"CMAKE_INSTALL_PREFIX:PATH=$CI_PROJECT_DIR/build/install-x64" -DCMAKE_CXX_COMPILER=/usr/local/opt/gcc@8/bin/g++-8 -DCMAKE_C_COMPILER=/usr/local/opt/gcc@8/bin/gcc-8 -DCMAKE_PREFIX_PATH=/usr/local/qt5/5.5/clang_64 -DPYTHON_INCLUDE_DIR=/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m -DPYTHON_LIBRARY=/Library/Frameworks/Python.framework/Versions/3.6/lib/libpython3.6.dylib -DPYTHON_MODULE_PATH=python -DPython_FRAMEWORKS=/Library/Frameworks/Python.framework/ -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_OSX_SYSROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk" -D"CMAKE_OSX_DEPLOYMENT_TARGET=10.9" -D"CMAKE_INSTALL_RPATH_USE_LINK_PATH=1" -D"ENABLE_RUBY=0" ../

[jonoomph]

@ferdnyc I would love your thoughts on this, if you have any. This just feels like a "CMake" or "GCC flag" type fix... your specialty! haha.

[jonoomph]

I can also verify this error happens on MacOS 10.13 (High Sierra). But interestingly, the error message includes a hint: libstdc++.6.dylib (which was built for Mac OS X 10.15):

ImportError: dlopen(/Applications/OpenShot Video Editor.app/Contents/MacOS/lib/python3.6/PyQt5.QtCore.so, 2): Symbol not found: _aligned_alloc
  Referenced from: /Applications/OpenShot Video Editor.app/Contents/MacOS/libstdc++.6.dylib (which was built for Mac OS X 10.15)
  Expected in: /usr/lib/libSystem.B.dylib
 in /Applications/OpenShot Video Editor.app/Contents/MacOS/libstdc++.6.dylib

[ferdnyc]

Hmm. First thought: Support for Python 3.6 was officially added in cx_Freeze 5.0.1. cx_Freeze 5.0 might just be too old.

[ferdnyc]

What happens if you take cx_Freeze out of the mix and just try to run OpenShot from the source tree? That is, from a Mac Terminal, do a

cd /path/to/the/cloned/openshot-qt/src
PYTHONPATH=/path/to/libopenshot/build_dir/src/bindings/python/ python3 launch.py

?

[ferdnyc]

Hmm. It looks like neither PyQt5.QtCore.so nor anything else should be linked against libstdc++, which was deprecated in XCode 10 and has been on Apple's "avoid" pile for longer than that. They want people to migrate to their libc++, and have apparently been forcing the issue more of late.

How/when was PyQt5 built for this system? If it wasn't compiled on the MacOS release currently installed, it may need to be rebuilt. Finding a way to work -stdlib=libc++ (or -DCMAKE_CXX_FLAGS="-stdlib=libc++") into its build configs might also help.

[ferdnyc]

Also, regarding aligned_alloc specifically, that's apparently a not-uncommon source of pain for people building code on MacOS. But just doing a sed -e 's/aligned_alloc/malloc/g' is apparently effective at working around its availability issues, at least sometimes.

[jonoomph]

I think the PyQt and Qt releases are older and almost certainly were compiled on an ancient toolchain. Perhaps that is a great place to start, updating Python, cx_freeze, Qt, Sip, and PyQt. Of course, this will break our Mac builds again for a few days while I sort through the chaos, but it makes since that I need to update the dependencies.

[jonoomph]

@ferdnyc Okay, this is actually a nightmare situation, similar to our older Linux builder. Qt 5.5 is the last official release to contain qtwebkit. Of course, I was using the prebuild MacOS version from qt.io (brew versions had issues with packaging and deployment), and manually compiling Sip and PyQt. The oldest official MacOS build I can get is Qt 5.9.9, which of course, has long since removed QtWebKit. I'm having trouble finding source code for any of these old versions, from qt.io or riverbank. So...

Long story short, I think it's finally time for QtWebEngine!!!! This is too painful...

[ferdnyc]

@jonoomph Nahh, in some ways QtWebKit is better supported now than when it was part of Qt. It's just an external project, which lives here: https://github.com/qtwebkit/qtwebkit

[ferdnyc]

You may even be able to use their binary package: qtwebkit-MacOS-MacOS_10_13-Clang-MacOS-MacOS_10_13-X86_64.7z

Binary packages should be unpacked inside installation of Qt 5.14.1 which you can obtain at https://www.qt.io/download-qt-installer.

...If that doesn't work, AIUI building it is fairly easy if you have a working Qt install, it's just a matter of unpack source, qmake, make, make install and it becomes part of Qt. Then you need to rebuild PyQt5 on top of that Qt, to pick up the QtWebKit and QtWebKitWidgets modules..

[jonoomph]

FYI: I have a QtWebEngine version of OpenShot running locally now! However, I'm converting all the synchronous calls to async, with callback methods. Might take a few hours to fully flush through it all. But so far, seems to be working great.

[jeffski]

FYI: I have a QtWebEngine version of OpenShot running locally now! However, I'm converting all the synchronous calls to async, with callback methods. Might take a few hours to fully flush through it all. But so far, seems to be working great.

@jonoomph, does this mean you are rebuilding the entire interface using HTML?

[ferdnyc]

@jeffski No, just the timeline, which is already in HTML. (He's switching the backend from QtWebKit to QtWebEngine.)

[ferdnyc]

@jonoomph

You know this means we're going to HAVE to upgrade the Linux builder too, now, right? QtWebEngine was pretty much broken in Qt 5.2. (It was broken until Qt 5.6, from what I hear.)

...Not that we shouldn't upgrade the Linux builder regardless, to move it off an EOL'd Ubuntu release, of course.

[jonoomph]

@ferdnyc Yup, this is probably also the time to drop support for some older Linux distros with old versions of glib. I'm open to recommendations on what version of Ubuntu (but I would prefer we keep it on Ubuntu, since I most experienced with it). Not sure what distro AppImage recommends these days... I guess I need to do some research.

I'm already so deep in this from completely rebuilding the Mac builder, I figure, bring it on. Let's rebuild all the builders!

[ferdnyc]

@jonoomph The AppImage recommendation is still "use the oldest supported", AFAIK, but... I've somewhat lost faith in that recommendation.

(Actually, it's fairer to say that I've become somewhat disillusioned with AppImages, period. Because it's become apparent that their recommendation — which is essential to the entire AppImage concept — is based on a faulty.premise. The notion that software which uses older dependencies will run on newer systems any better than newer software will run on older systems may be true, for simple userspace applications, but it breaks down the moment you need to access any of the underlying system resources, and especially if you need to talk to hardware. Then, incompatibilities in either direction are equally likely to screw you over.)

The reason that hardware acceleration completely blew up in the AppImage builds (OpenShot/openshot-qt#3210), and why ultimately it had to be disabled, is because our AppImage bundles libvdpau.so.1 and libva.so.1 from Ubuntu 14.04, and both of those libraries contain compiled-in paths to the GPU hardware drivers. (/usr/lib/x86_64-linux-gnu/vdpau/ and /usr/lib/x86_64-linux-gnu/dri, respectively.)

• Why do we bundle libvdpau.so.1 and libva.so.1?

  • Because our bundled ffmpeg libavutil is linked with those libs, so they have to be present for OpenShot to run.

• Why don't all systems have libva.so.1 and libvdpau.so.1 available?

  • Well, in the case of libvdpau.so.1, it's because it only supports Nvidia hardware, so systems with other GPUs may not have it installed. (Actually, that's wrong, I keep forgetting that despite originating with Nvidia, VDPAU is a generic interface that supports other GPUs as well. It actually is likely to be present on most systems.)
  • But in the case of libva.so.1, it's because newer systems (meaning, any distro released in the past 3 years) have all moved on to libva.so.2 as their VA-API implementation.

• Why doesn't hardware acceleration at least work on Debian systems, where the hardcoded path in our bundled libs is correct?

  • Oh, that's the fun one: Because the hardware drivers themselves are linked with the system's libva. (Which is, again, libva.so.2.) So even if the bundled libva.so.1 is able to open a driver, that driver will then try to load the incompatible libva.so.2 from the system, causing a version conflict — which led to the crashes reported in Crash when launching Preferences openshot-qt#3210.

• What happens if we start building AppImages on a distro that uses libva.so.2?
  Well, that's the billion-dollar question. Truth is, I'm not sure.

  • If we start bundling libva.so.2 and libvdpau.so.1, then there's at least a chance that hardware-acceleration might start working for users running on recent Debian-based systems (where the hardcoded paths in the library are correct, and point to possibly-compatible hardware drivers).
  • If we're able to exclude both libraries from the AppImage, then there's at least a chance that hardware acceleration will begin to work for everyone. (At least if they have those libraries installed.) Since libavutil will use the system libraries, they'll be able to find the drivers even if, as on my Fedora system, they're located at /usr/lib64/vdpau/ and /usr/lib64/dri/.
  • The AppImage will definitely not work on any system running a distro released more than 3 years ago. But that strikes me as a better tradeoff than having things break for users with relatively current systems, simply because we're targeting the oldest possible distro.

...So, personally, I think if we're going to upgrade the builder to a newer Ubuntu release, it should be 18.04 (Bionic). I do see some value in sticking with the oldest release that's workable, so there's no point in going with the just-released Focal — that's too new. But the actual "oldest supported", Xenial (16.04) (which still technically has almost a year left before EOL), is a bad idea because it was still on libva.so.1, so it doesn't even have a chance of solving that problem.

Bionic does bring us libva.so.2. So there's at least the possibility of getting hardware accel sorted out for the AppImages, if we build on that release.

[eisneinechse]

Just a little information. I had this libva problem already while programming the hardware acceleration and wrote to Jonathan about it way back. But it doesn't stop there, the SVT_* encoders tent you only compile for CPU with AVX2 commands. Unfortunately on new multi core CPU they are quite fast, like really fast.
We could compile our ffmpeg ourselves and include it in the AppImage, but will it include the SVT_* encoders (and decoders) or would we also have two versions, one for CPUs with AVX2 and one for CPUs without it (those CPUs are not that old)?
BTW on CPUs with only a few cores rav1e is as fast or faster than SVT_AV1 (There is also SVT_HEVC and SVT_VP9).
I personally like ffmpeg 4.3 with rav1e included on my older PC and with rav1e, SVT_AV1, SVT_HEVC, and SVT_VP9 included on my newer PC. And aomenc, x.265, x264, .... all the standard stuff.
My suggestion would be ffmpeg 4.3 without the SVT stuff self compiles included in the AppImage with libva.so.2
Just my 2ct.

[SuslikV]

Want to cover more PCs or to make soft more reliable? AppImage is step back. Old Mac is step back too.

Really, programmers are living in other world than users.

Interesting, is the switch to the QtWebEngine (and dependencies update) actually solves the Symbol not found: _aligned_alloc error for the Mac 10.9.5? Was this solution tested by the end user that reported the original bug?

[ferdnyc]

@eisneinechse

Just a little information. I had this libva problem already while programming the hardware acceleration and wrote to Jonathan about it way back.

Yeah, I saw the notes about it in the doc/HW-ACCEL.md file, eventually. (Only after I'd re-chased all of the same info myself, naturally.) Unfortunately, there's no good solution except to leave users on older systems in the dust, in hopes that, as a result, we can rely more on system resources for users on newer systems.

But it doesn't stop there, the SVT_* encoders tent you only compile for CPU with AVX2 commands. Unfortunately on new multi core CPU they are quite fast, like really fast.

Some Fedora developers tried to push moving the distro baseline to AVX2 around a year ago. After a rather deafening outcry from the community as a whole, the proposal was soundly rejected. They're still trying to "explore" AVX2-only builds in roundabout ways, but frankly they're just nuts.

The proposal was made following "discussions with CPU vendors" (who are of course gung-ho about it, can't imagine why ), but as was pointed out in the feedback email thread flamewar, those same CPU vendors also have products currently still available for purchase that don't even support AVX2! NONE of my three systems at home would be compatible with a Fedora that included a kernel built with AVX2 instructions. It became very clear, based on the feedback from that proposal, that the vendors had vastly overestimated (or overstated) the penetration of AVX2-capable hardware.

I don't doubt that the benefits are significant, but moving to an AVX2 build just feels like a non-starter, especially for the AppImage which is meant to be sort of lowest-common-denominator. And it's not like hardware-accel, a "be nice if it worked, but it doesn't so you're stuck with software encoding" kind of thing. If your hardware doesn't support AVX2 (which most doesn't, today) the app just doesn't run, period.

We could compile our ffmpeg ourselves and include it in the AppImage, but will it include the SVT_* encoders (and decoders) or would we also have two versions, one for CPUs with AVX2 and one for CPUs without it (those CPUs are not that old)?

There is one way we could possibly make that work, though it's a tricky tightrope act even still and it's debatable whether the hassle would be worth it. This is something I'd come up with in thinking about the libva.so issue. It could work for AVX2 as well, though it would make things even more complicated.

Imagine an AppImage with multiple library directories, instead of everything dumped in usr/bin:

1. usr/lib contains the lion's share of the bundled libraries, the ones we use unconditionally
2. usr/lib-ffmpeg contains a set of FFmpeg libraries built without AVX2 instructions
3. usr/lib-ffmpeg_avx2 contains the same libraries, but built so that an AVX2-capable CPU is required
4. usr/lib-va-2 contains a build of libva.so.2 compatible with our bundled libs/binaries
5. usr/bin/probe-libva is a small program linked against a shared libva.so.2, that just confirms successful loading of libva.so.2. We might even be able to just include the vainfo tool. All it needs to do is not crash, to confirm that libva.so.2 is available for use on the system.
6. usr/bin/probe-avx2 is — I don't know exactly, a small binary, a shell script, etc... that "does something" (we'd have to figure out what, exactly) to determine conclusively whether the CPU the code is currently running on supports AVX2 instructions.

Currently when the AppImage launches, what happens is:

1. The AppRun binary gets kicked off
2. It runs /tmp/.mount_random/usr/bin/openshot-qt-launch (a shell script)
3. The script sets LD_LIBRARY_PATH="."
4. Finally, it execs openshot-qt (which is the cx_Frozen Python wrapper)

But in our new setup, the openshot-qt-launch script would first use the probe-... tools to construct its LD_LIBRARY_PATH based on the detected local capabilities:

1. The starting point is /tmp/.mount_random/usr/lib, that's always in the path
2. If probe-libva crashes, guess we can't use the system libva.so.2. So :/tmp/.mount_random/usr/lib-va-2 gets tacked on to LD_LIBRARY_PATH.
3. probe-avx2 confirms CPU support? Great! :/tmp/.mount_random/usr/lib-ffmpeg-avx2 gets tacked on to the loader path.
4. probe-avx2 failed? :/tmp/.mount_random/usr/lib-ffmpeg gets added to LD_LIBRARY_PATH instead.

Once LD_LIBRARY_PATH is assembled, containing up to three directories inside the AppImage that contain libraries determined necessary to run on the current hardware, then openshot-qt is launched — and, hopefully, the dynamic loader picks up all of the correct shared library pieces for them all to be compatible.

There's at least the possibility that a setup like that could get us a little closer to actually achieving that (frankly, overstated/sorely-lacking) holy grail of universal binary compatibility. Which, even though it's the entire premise upon which AppImages were based, is something they've never really lived up to so far.

[ferdnyc]

@SuslikV

Want to cover more PCs or to make soft more reliable? AppImage is step back.

Well, you're the Linux expert.

Interesting, is the switch to the QtWebEngine (and dependencies update) actually solves the Symbol not found: _aligned_alloc error for the Mac 10.9.5?

It's not intended to. Or, rather, you've got that the wrong way around. The intention is to move to a newer version of Qt / PyQt5, because the one we're currently shipping for Macs is Qt 5.5 which is ancient, And it does bad things like link with libstdc++, which Apple deprecated like 3 years ago.

The belief is that a newer Qt build — something actually-modern like 5.12 at a minimum — can be built with support for modern systems, whereas simply rebuilding Qt 5.5 won't fix all of the deprecated, outdated, and just plain wrong things that cause problems when running it on newer Macs.

The move to a Qt version > 5.5, though, means having to deal with the fact that QtWebKit was removed from the Qt distribution starting with the 5.6 release. QtWebEngine is Qt's still-supported replacement for QtWebKit.

Was this solution tested by the end user that reported the original bug?

How do you propose they do that, when it isn't even finished yet? For a user to test an OpenShot build that uses Qt > 5.5 without QtWebKit available, there needs to be some alternative solution for how it can run the timeline.

[SuslikV]

@ferdnyc it is simple management. You deviate from the issue itself. And you are spending a lot of time on side tasks. Qt itself is major issue in libopenshot by the way.

[jonoomph]

Really, programmers are living in other world than users.

@SuslikV Please refer once again to https://github.com/OpenShot/openshot-qt/wiki/Code-of-Conduct. I've been very patient with your rudeness and lack of respect, but after 3 warnings to be respectful and follow our Code of Conduct, your behavior remains unchanged. You are now blocked from contributing on OpenShot's GitHub repos. Thanks for the positive things you contributed, and I hope you find a better outlet for yourself.

[jonoomph]

The belief is that a newer Qt build — something actually-modern like 5.12 at a minimum — can be built with support for modern systems, whereas simply rebuilding Qt 5.5 won't fix all of the deprecated, outdated, and just plain wrong things that cause problems when running it on newer Macs.

The move to a Qt version > 5.5, though, means having to deal with the fact that QtWebKit was removed from the Qt distribution starting with the 5.6 release. QtWebEngine is Qt's still-supported replacement for QtWebKit.

@ferdnyc Well said!

[eisneinechse]

@ferdnyc
I agree 100%. I have systems that are pre Ryzen AMD, no AVX2; that's why I know that SVT_ is, lets say, intel developed. When I want to check SVT_AV1 improvements I have to use the Ryzen PC.
OT: The 64 bit intel (actually AMD) patents are gonna run out soon, I think November. From then on cheap, old command, but still Windows compatible 64 bit CPU can be made by whoever is interested (and has the money). No AVX naturally! But SSE2. Maybe not a good idea to stop supporting them now.
As for SVT: Well, the three SVT encoders are all only includable into ffmpeg with patches. So, they are nice to have at high core number CPUs but not for average users right now. (Still funny to see encoding speeds that are higher than the ones I had 5 years ago with x.264).
You might have noticed that I try to make some time free for Openshot development but the times are still unpredictable.

[eisneinechse]

@ferdnyc
Just look how long this patch is sitting there:
OpenVisualCloud/SVT-VP9#67
And it would really be nice to have a fast VP9 encoder, which SVT_VP9 is BTW.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.