Build ITK Python Packages with gcc-toolset-12 toolset fails

[tbirdso]

Summary

We are in the process of updating ITK build scripts in ITKPythonPackage to accommodate targeting ARM architectures. Module builds currently fail on the first CastXML generation step. With that said, I am not 100% certain whether the issue lies in CastXML.

Expected behavior

XML files are generated without error, wrapping succeeds

Observed behavior

[8/17] Completed 'castxml'
[9/17] Generating /work/ITK-cp38-cp38-manylinux_2_28_aarch64/Wrapping/castxml_inputs/itkSplitComponentsImageFilter.xml
FAILED: /work/ITK-cp38-cp38-manylinux_2_28_aarch64/Wrapping/castxml_inputs/itkSplitComponentsImageFilter.xml 
cd /work/_skbuild/linux-aarch64-3.8/cmake-build/Wrapping/Modules/SplitComponents && /work/_skbuild/linux-aarch64-3.8/cmake-build/Wrapping/Generators/CastXML/castxml/bin/castxml -o /work/ITK-cp38-cp38-manylinux_2_28_aarch64/Wrapping/castxml_inputs/itkSplitComponentsImageFilter.xml --castxml-gccxml --castxml-start _wrapping_ --castxml-cc-gnu "(" /opt/rh/gcc-toolset-12/root/usr/bin/c++ -std=c++14 ")" -w -c @/work/ITK-cp38-cp38-manylinux_2_28_aarch64/Wrapping/castxml_inputs/.castxml.inc /work/ITK-cp38-cp38-manylinux_2_28_aarch64/Wrapping/castxml_inputs/itkSplitComponentsImageFilter.cxx
In file included from /work/ITK-cp38-cp38-manylinux_2_28_aarch64/Wrapping/castxml_inputs/itkSplitComponentsImageFilter.cxx:1:
In file included from /work/ITK-source/ITK/Modules/Core/Common/include/itkCommand.h:21:
In file included from /work/ITK-source/ITK/Modules/Core/Common/include/itkObject.h:31:
In file included from /work/ITK-source/ITK/Modules/Core/Common/include/itkLightObject.h:21:
In file included from /work/ITK-source/ITK/Modules/Core/Common/include/itkMacro.h:47:
In file included from /opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/string:41:
In file included from /opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/bits/allocator.h:46:
In file included from /opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/aarch64-redhat-linux/bits/c++allocator.h:33:
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/bits/new_allocator.h:158:2: error: call to '__builtin_operator_delete' selects non-usual deallocation function
        _GLIBCXX_OPERATOR_DELETE(_GLIBCXX_SIZED_DEALLOC(__p, __n));
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/bits/new_allocator.h:103:35: note: expanded from macro '_GLIBCXX_OPERATOR_DELETE'
# define _GLIBCXX_OPERATOR_DELETE __builtin_operator_delete
                                  ^
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:496:13: note: in instantiation of member function 'std::__new_allocator<char>::deallocate' requested here
      { __a.deallocate(__p, __n); }
            ^
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/bits/basic_string.h:292:24: note: in instantiation of member function 'std::allocator_traits<std::allocator<char>>::deallocate' requested here
      { _Alloc_traits::deallocate(_M_get_allocator(), _M_data(), __size + 1); }
                       ^
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/bits/basic_string.h:286:4: note: in instantiation of member function 'std::basic_string<char>::_M_destroy' requested here
          _M_destroy(_M_allocated_capacity);
          ^
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/bits/basic_string.h:794:9: note: in instantiation of member function 'std::basic_string<char>::_M_dispose' requested here
      { _M_dispose(); }
        ^
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/bits/basic_string.h:4019:12: note: in instantiation of member function 'std::basic_string<char>::~basic_string' requested here
    string __str(__neg + __len, '-');
           ^
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/aarch64-redhat-linux/12/../../../../include/c++/12/new:135:6: note: non-usual 'operator delete' declared here
void operator delete(void*, std::size_t) _GLIBCXX_USE_NOEXCEPT
~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Build output available at https://github.com/tbirdso/ITKSplitComponents/actions/runs/3604194121/jobs/6073281299

Platform Information

manylinux_2_28_aarch64 image running on an Ubuntu 20.04 Github Actions runner with aarch64 emulation.

Manylinux image: https://quay.io/repository/pypa/manylinux_2_24_aarch64?tab=info , tag=2022-11-28-5d13db4
Emulation tools: https://hub.docker.com/r/tonistiigi/binfmt

Able to reproduce on local Ubuntu 20.04 instance with aarch64 emulation + manylinux_2_28_aarch64 image.

Not yet attempted on native ARM machine.

Versions

CastXML 0.4.8

-- Using src='https://github.com/CastXML/CastXMLSuperbuild/releases/download/v0.4.8/castxml-linux-aarch64.tar.gz'

SWIG 4.0.2
ITK v5.3.0

Other Notes

• ITKPythonPackage ARM build scripts under test are here: COMP: Consolidate Linux ARM build script ITKPythonPackage#236
• ITKSplitComponents is a small header-only ITK module often used for testing build script development
• I was able to successfully build and run ITKSplitComponents tests in a local emulated ARM container with ITK_WRAP_PYTHON:BOOL=OFF and BUILD_TESTING:BOOL=ON.

xpost from CastXML/CastXML#228

[tbirdso]

@bradking @jcfr @thewtex Any thoughts on how to proceed are appreciated.

[bradking]

I tried running castxml on a simple #include <string> source file with gcc-toolset-12 in an almalinux:9 container on a native aarch64 host, and it works fine. Please try to demonstrate the problem without all the scripting and build abstraction layers (ITKPythonPackage, dockcross, manylinux, etc.).

[tbirdso]

Hi @bradking , thanks for taking a look. @thewtex pointed out in InsightSoftwareConsortium/ITKPythonPackage#236 (comment) that this is likely due to our recent update from gcc-toolset-11 to gcc-toolset-12 for building wheels. I will close the CastXML issue and consolidate discussion in this issue.

[tbirdso]

Consolidating discussion from ITKPythonPackage here:

@tbirdso I am wondering if this is an incompatibility with the ITKPythonBuilds tarball (built with the gcc-toolset-11) and the gcc-toolset-12 in the manylinux aarch64 latest image?

Hi @thewtex, good call, I had not connected those dots. I am able to recreate the error in an ITKSplitComponents development branch bumped for InsightSoftwareConsortium/ITKPythonPackage#237. The error occurs for an manylinux_2_28-x64 build which shows that this is not limited to Arm, thus I am moving discussion out of that Arm-specific PR.

https://github.com/tbirdso/ITKSplitComponents/actions/runs/3625465984/jobs/6113545703

I recommend rebasing on master to integrate InsightSoftwareConsortium/ITKPythonPackage#237. Then rebuild the tarball on blaster. This will take a while. Then test the module wheel script locally with the built tarball.

Can do.

Since we will need updated itk-* aarch64 wheels, I suggest we defer this update to 5.3.1.

It now seems that this issue is not limited to aarch64 wheels, but also impacts at least _2_28 and probably also 2014 wheels. With that knowledge, would it be prudent to release 5.3.1 sooner rather than later, or at least a postfix?

My understanding of the impact is as follows:

1. Most ITK module wheels may still be built. However, build scripts will be limited to pre- ENH: Update manylinux images to latest tagged version 20221201-fd49c08 ITKPythonPackage#237, i.e. dockcross images with gcc-toolset-11 to match the available v5.3.0 tarballs.
2. ITKIOOMEZarrNGFF will be stuck in limbo. My understanding is that ITKIOOMEZarrNGFF requires NASM for its build process which is first made available in ITKPythonPackage in that same image which updated to gcc-toolset-12. Therefore we will not be able to build ITKIOOMEZarrNGFF Python wheels until updated tarballs are made available. @dzenanz may be able to comment on whether there is urgency related to that package.

[thewtex]

We can continue to build with and against gcc-toolset-11 with the remote module packages / CI. We can build remote module wheels against 5.3.0 for aarch64 by building explicitly against quay.io/pypa/manylinux_2_28_x86_64:2022-11-19-1b19e81. For x86_64, I pushed a dockcross image to DockerHub that includes the NASM patch but is fixed to a version of the image that uses gcc-toolset-11. This is dockcross/manylinux_2_28-x64:20221205-459c9f0. manylinux2014 uses an older toolchain and will not be an issue.

[tbirdso]

Excellent, thanks for the dockcross patch @thewtex. I will update module workflows accordingly.

EDIT: I have confirmed that the updated container still includes the fix for dockcross/dockcross#746 . 👍

[tbirdso]

@thewtex For future reference, is the gcc version on the system sufficient to tell what gcc toolset will be used for building in a given manylinux container? Or is there a better way to determine this?

I see the following in respective containers:

IMAGE_TAG=20221205-459c9f0

$ gcc --version
gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

IMAGE_TAG=20221201-fd49c08

$ gcc --version
gcc (GCC) 12.1.1 20220628 (Red Hat 12.1.1-3)
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I am wondering if it would be reasonable to add a check in ITK module build workflows in the future to verify that the gcc toolset matches that used in generating tarballs, i.e. that v5.3.0 packages are built with gcc 11, v5.3.1 and onward with gcc 12, etc

[dzenanz]

Did I spend hours barking up the wrong tree? See google/tensorstore#69.

[tbirdso]

@dzenanz If I understand correctly the errors you've discussed in google/tensorstore#69 occur in non-ITK dependencies prior to building the ITK module wheels. Under the current plan we will revert to gcc 11 for v5.3.0 wheels and upgrade to gcc 12 for v5.3.1 wheels, meaning that there is a bit more time for those dependencies to reach a resolution for gcc 12.

Even if not immediate, there will still be a need for those fixes in the long term and it certainly doesn't sound like time wasted.

[dzenanz]

I just tried zarrCI with GCC-11 PythonPackage, and the compile error is the same, which means it is unrelated to this issue.

[thewtex]

Addressed via the Docker image used for remote module builds in the ITKPythonPackage repository.