Linux wheels: manylinux2014 tracking issue

[skvark]

The manylinux1 images are starting show their age and therefore make the maintenance of the Linux wheels quite troublesome. OpenCV 3.4.10 and 4.3.0 cannot be built on manylinux1 images due to too old glibc:

/io/opencv/modules/core/src/parallel.cpp:943:56: error: ‘CPU_COUNT’ was not declared in this scope

         unsigned cpu_count_cpu_set = CPU_COUNT(&cpu_set);

In practice this means that the manylinux1 (CentOS 5) needs to be updated to manylinux2010 (CentOS 6) or manylinux2014 (CentOS 7). Considering that the CentOS 6 will be deprecated later this year it's better move straight to manylinux2014. The manylinux2014 tag support is not yet fully rolled out (pypa/manylinux#338). However, there are only some minor issues remaining and they will get solved likely during the next few months.

This upgrade will most likely have positive effects to the OpenCV performance (see: #287). Additionally, external dependencies, such as Qt, can be updated to more recent versions.

What needs to be done:

• Prepare new custom Dockerfiles which extend from manylinux2014 similarly to the current manylinux1 Dockerfiles
  • Install Qt 5 (preferably via package manager, currently Qt 4.8.7 is built from sources)
  • Check if Cmake is present. If not, it could be probably set as a Python dependency similarly to numpy
  • Check if custom OpenSSL build is needed
  • Build latest FFmpeg from sources
  • Check if other often requested 3rd party dependencies can be enabled easily by installing them via package manager (must check license terms) Edit: this needs more work, needs to be done later when first working versions of the new wheels have been released
  • Clean up other stuff from the Dockerfiles
• Check setup.py custom rules and patches
  • Enable IPP if possible
  • Remove other custom rules which are not needed
  • Check if OpenEXR patch is still needed

[johnthagen]

The manylinux2014 tag support is not yet fully rolled out

It looks like as of this comment, the rollout has been completed.

[skvark]

Thanks for the info. I'll update the Docker images and will run some tests.

[skvark]

The current status is that Linux builds have an issue with Qt5. I get this on Ubuntu 20.04 when trying to run very simple script with imshow:

Found metadata in lib /home/username/test/lib/python3.8/site-packages/cv2/qt/plugins/platforms/libqxcb.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "xcb"
        ]
    },
    "className": "QXcbIntegrationPlugin",
    "debug": false,
    "version": 329991
}


Got keys from plugin meta data ("xcb")
QFactoryLoader::QFactoryLoader() checking directory path "/usr/bin/platforms" ...
loaded library "/home/username/test/lib/python3.8/site-packages/cv2/qt/plugins/platforms/libqxcb.so"
[New Thread 0x7fffbb3b0700 (LWP 5905)]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffbe3abe05 in _xcb_in_expect_reply () from /home/username/test/lib/python3.8/site-packages/cv2/qt/plugins/platforms/../../../../opencv_python.libs/libxcb-65da195c.so.1.1.0

The Qt xcb plugin crashes on _xcb_in_expect_reply() method.

[skvark]

If someone would like to test the manylinux2014 wheels they can be downloaded by following this guide: https://github.com/skvark/opencv-python/releases/tag/34

[ychechik]

Hi, any updates? looking forward to 4.3 :)

[skvark]

I'll have to compile Qt5 probably manually to get rid of that error but haven't had the time to do it yet.

[Red-Eyed]

Regarding manual build, have u look at C++ package managers?
this one is actively contributed https://github.com/microsoft/vcpkg

[Red-Eyed]

BTW, I built opencv4.3 on ubuntu 20.04 for python 3.7.6 (from conda) and it works fine.
Is the issue only on CentOS ?

[skvark]

Yeah, locally built OpenCV will work of course just fine but that's not what this repository is trying to do. Manylinux wheels must be built on very specific systems to make them compatible with different Linux distributions (https://www.python.org/dev/peps/pep-0513/). The problem is not in package managers but in binary level compatibility across different distributions.

The OpenCV builds related to this issue work just fine, only a subset of the methods related to displaying GUI windows is broken. As we can see above, Qt5 segfaults when it tries to use the bundled xcb plugin which has been included from the CentOS Qt5 build.

Additionally, building for example Qt5 manually is not a problem, it just takes time when it's done for the first time. After that it can be used from the Docker images without any build overhead.

[johnthagen]

@skvark In case any of these resources are helpful:

• Building Qt5 from Git
• A Dockerfile for Qt5 on CentOS 7

Another interesting thing I found: PyQt5 builds manylinux2014 wheels already.

• The source is listed here: https://riverbankcomputing.com/software/pyqt/download5 but at first glance I don't see any kind of Docker build environment for making the wheels. Maybe they don't release that.
• Perhaps for a project as important as OpenCV, they would consider answering a question: https://riverbankcomputing.com/support/pyqt

[Red-Eyed]

I guess we don't need pyqt5 but instead we need native Qt5 with cmake and header files.
I will try to build Qt5 and opencv with this docker

[johnthagen]

I guess we don't need pyqt5 but instead we need native Qt5 with cmake and header files.

Right, I just assumed that if PyQt5 managed to build a manylinux2014 wheel, as part of that they must have figured out how to built Qt5 on a CentOS 7 image.

[Red-Eyed]

Oh, I see

[skvark]

I'm pretty sure that building Qt5 on the manylinux image is quite straightforward process (same as Qt4 in current images, probably mostly copy pasting from there). After that the next thing is to get OpenCV built against it and testing the resulting wheels in other systems. Again, the main issue is that all those things take multiple evenings (or weekends) of my time. Anyone can of course do that as this is a open source project, it does not have to be me. The Dockerfiles are in the docker directory in case someone is interested in contributing. Copying the Qt4 example from master branch Dockerfiles to manylinux2014 branch Dockerfiles and modifying it for Qt5 is a good starting point. https://doc.qt.io/qt-5/build-sources.html#linux-x11

[johnthagen]

@skvark Would you consider setting up something like GitHub Sponsors for people who would like to say thank you for your work?

[skvark]

@johnthagen I'll have a look at it. There are some complicated things related to Finnish taxation system which I need to check before even applying for example to Github Sponsors.

[skvark]

I couldn't get the latest Qt5 work with OpenCV (cv2 import hanged forever in 64 bit builds and segfaulted in 32-bit builds) so downgrading back to Qt4 which hopefully solves the issue for now. I will test the builds this week. macOS has also some build failures which need to be solved.

[skvark]

Linux builds are now ok. Fixing macOS issues now.

[skvark]

Homebrew apparently fixed itself so the release builds can be now made. 3.4.10 is now in progress, 4.3.0 will come after they are done.