Linux wheels: manylinux2014 tracking issue

[skvark]

The manylinux1 images are starting show their age and therefore make the maintenance of the Linux wheels quite troublesome. OpenCV 3.4.10 and 4.3.0 cannot be built on manylinux1 images due to too old glibc:

/io/opencv/modules/core/src/parallel.cpp:943:56: error: ‘CPU_COUNT’ was not declared in this scope

         unsigned cpu_count_cpu_set = CPU_COUNT(&cpu_set);

In practice this means that the manylinux1 (CentOS 5) needs to be updated to manylinux2010 (CentOS 6) or manylinux2014 (CentOS 7). Considering that the CentOS 6 will be deprecated later this year it's better move straight to manylinux2014. The manylinux2014 tag support is not yet fully rolled out (pypa/manylinux#338). However, there are only some minor issues remaining and they will get solved likely during the next few months.

This upgrade will most likely have positive effects to the OpenCV performance (see: #287). Additionally, external dependencies, such as Qt, can be updated to more recent versions.

What needs to be done:

• Prepare new custom Dockerfiles which extend from manylinux2014 similarly to the current manylinux1 Dockerfiles
  • Install Qt 5 (preferably via package manager, currently Qt 4.8.7 is built from sources)
  • Check if Cmake is present. If not, it could be probably set as a Python dependency similarly to numpy
  • Check if custom OpenSSL build is needed
  • Build latest FFmpeg from sources
  • Check if other often requested 3rd party dependencies can be enabled easily by installing them via package manager (must check license terms) Edit: this needs more work, needs to be done later when first working versions of the new wheels have been released
  • Clean up other stuff from the Dockerfiles
• Check setup.py custom rules and patches
  • Enable IPP if possible
  • Remove other custom rules which are not needed
  • Check if OpenEXR patch is still needed

[johnthagen]

The manylinux2014 tag support is not yet fully rolled out

It looks like as of this comment, the rollout has been completed.

[skvark]

Thanks for the info. I'll update the Docker images and will run some tests.

[skvark]

The current status is that Linux builds have an issue with Qt5. I get this on Ubuntu 20.04 when trying to run very simple script with imshow:

Found metadata in lib /home/username/test/lib/python3.8/site-packages/cv2/qt/plugins/platforms/libqxcb.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "xcb"
        ]
    },
    "className": "QXcbIntegrationPlugin",
    "debug": false,
    "version": 329991
}


Got keys from plugin meta data ("xcb")
QFactoryLoader::QFactoryLoader() checking directory path "/usr/bin/platforms" ...
loaded library "/home/username/test/lib/python3.8/site-packages/cv2/qt/plugins/platforms/libqxcb.so"
[New Thread 0x7fffbb3b0700 (LWP 5905)]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffbe3abe05 in _xcb_in_expect_reply () from /home/username/test/lib/python3.8/site-packages/cv2/qt/plugins/platforms/../../../../opencv_python.libs/libxcb-65da195c.so.1.1.0

The Qt xcb plugin crashes on _xcb_in_expect_reply() method.

[skvark]

If someone would like to test the manylinux2014 wheels they can be downloaded by following this guide: https://github.com/skvark/opencv-python/releases/tag/34

[ychechik]

Hi, any updates? looking forward to 4.3 :)

[skvark]

I'll have to compile Qt5 probably manually to get rid of that error but haven't had the time to do it yet.

[Red-Eyed]

Regarding manual build, have u look at C++ package managers?
this one is actively contributed https://github.com/microsoft/vcpkg

[Red-Eyed]

BTW, I built opencv4.3 on ubuntu 20.04 for python 3.7.6 (from conda) and it works fine.
Is the issue only on CentOS ?

[skvark]

Yeah, locally built OpenCV will work of course just fine but that's not what this repository is trying to do. Manylinux wheels must be built on very specific systems to make them compatible with different Linux distributions (https://www.python.org/dev/peps/pep-0513/). The problem is not in package managers but in binary level compatibility across different distributions.

The OpenCV builds related to this issue work just fine, only a subset of the methods related to displaying GUI windows is broken. As we can see above, Qt5 segfaults when it tries to use the bundled xcb plugin which has been included from the CentOS Qt5 build.

Additionally, building for example Qt5 manually is not a problem, it just takes time when it's done for the first time. After that it can be used from the Docker images without any build overhead.

[johnthagen]

@skvark In case any of these resources are helpful:

• Building Qt5 from Git
• A Dockerfile for Qt5 on CentOS 7

Another interesting thing I found: PyQt5 builds manylinux2014 wheels already.

• The source is listed here: https://riverbankcomputing.com/software/pyqt/download5 but at first glance I don't see any kind of Docker build environment for making the wheels. Maybe they don't release that.
• Perhaps for a project as important as OpenCV, they would consider answering a question: https://riverbankcomputing.com/support/pyqt

[Red-Eyed]

I guess we don't need pyqt5 but instead we need native Qt5 with cmake and header files.
I will try to build Qt5 and opencv with this docker

[johnthagen]

I guess we don't need pyqt5 but instead we need native Qt5 with cmake and header files.

Right, I just assumed that if PyQt5 managed to build a manylinux2014 wheel, as part of that they must have figured out how to built Qt5 on a CentOS 7 image.

[Red-Eyed]

Oh, I see