Fix Python 3 wheels cross-compilation

[ymartin59]

At the moment application packages publication are stuck because Python 3.8 cross-compiled wheels are badly generated with "x64" architecture in their shared objects (so) file names, which prevent their loading at runtime.

Example: file share/wheelhouse/immutables-0.11-cp36-none-any.whl containes /immutables/_map.cpython-36m-x86_64-linux-gnu.so for all compiled archs.

# file immutables/_map.cpython-36m-x86_64-linux-gnu.so 
immutables/_map.cpython-36m-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

Some application installation just fail because of this invalid file naming. It is expected wheels file to match target runtime architecture like _map.cpython-36m-aarch64-linux-gnueabi.so according to toolchain TC_TARGET value.

Pending packages because of this:

• borgbackup [Update Request] borg #3934 Borgbackup and python3 no longer install as intended #3644 [Update Request] borgmatic 1.1.15 (part of borg) #3334
• homeassistant Home Assistant update? #3740 HomeAssistant 0.87.1-3 won't start #3643 Home Assistant - Error while setting up platform broadlink #3573 Home Assistant - Error while setting up Xiaomi vacuum 2 #3584
• rdiff-backup Update rdiff-backup and librsync #3916
• deluge (move to python3)
• flexget (move to python3)
• duplicity (move to python3) Update duplicity with oauth and Python 3 #3823

And many other Python applications to come...

@roleoroleo and @smaarn did progress with #3726 #3731 #3739 #3833
But there is still trouble according

Related issues: #3737

Objectives: fix current 3.6.10 to be published, and prepare for next version, either 3.7 or directly 3.8

[smaarn]

@ymartin59 open question: is there any specific reason to skipping the intermediary 3.7 version ?
I mean, in the "master" branch only 3.6 version seems to be present. On my side I'm working on having a 3.7 version running (turns out that bazarr is upgrading its requirements to 3.7 ...) and I was wondering if there was a specific reason which would imply the dismissal of that one.

[ymartin59]

@smaarn Simply because it is to much work. Publishing 3.6.10 is still in pending/failed state because of that issue. As far as I know, all applications' package list is expected to run on Python 3.8

[smaarn]

@ymartin59 ok I was misled by the title of this issue to be honest !

I'm having a look to see how this works.

[smaarn]

Ok after a "little" of digging:

• The file name format is actually determined by the EXT_FILE parameter computed when building the python executing the build. So here it's the native guy which, obviously, depends on the ABIFLAGS.
• From what you can read here: https://stackoverflow.com/questions/38523941/change-cythons-naming-rules-for-so-files and basically here: https://www.python.org/dev/peps/pep-3149/ it's not recommend to play with this

From what I see

1. It's "pretty easy" to find all the files to look for (basically their name end up with that <EXT_SUFFIX>.so)
2. We need to plug ourselves after the compilation has occurred and before the wheel is being generated to perform the renaming
   or we need to patch the native python so it can be overriden (using an environment variable maybe ?)
   or we need to patch every single wheel we're generating so that it enforces a custom value (yuck but maybe feasible...)

What do you think ?

The "guilty code" is under Lib/distutils/sysconfig.py and the variable is defined in configure (line 15280).

[smaarn]

(I checked it on python 3.7.7 to be honest)

[ymartin59]

Investigate if crossenv https://pypi.org/project/crossenv/ would be an easier and more efficient way to cross-compile wheels.

[Safihre]

This seems fixed with #3953? Yes, the whl file is still wrongly named, but the so files inside have the right name.

[ymartin59]

PR #3953 has been merged.

• beet needs to be rebuilt so a version update would be relevant
• borgmatic (borgbackup package) requires attention too, so with version update too

borgmatic --version
Traceback (most recent call last):
  File "/usr/local/bin/borgmatic", line 8, in <module>
    sys.exit(main())
  File "/volume1/@appstore/borgbackup/env/lib/python3.7/site-packages/borgmatic/commands/borgmatic.py", line 380, in main
    print(pkg_resources.require('borgmatic')[0].version)
  File "/volume1/@appstore/borgbackup/env/lib/python3.7/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/volume1/@appstore/borgbackup/env/lib/python3.7/site-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (ruamel.yaml 0.16.10 (/volume1/@appstore/python3/lib/python3.7/site-packages), Requirement.parse('ruamel.yaml<0.16.0,>0.15.0'), {'borgmatic'})

[smaarn]

@ymartin59 issue reported for borgmatic seems to be fixed by #4094 (just tested it manually). Whether or not borgbackup is now totally operational is, unfortunately, outside of my area of expertise unfortunately.

[ymartin59]

@smaarn Thanks for your contribution on Python 3. borgbackup fails to cross-compile for powerpc (qoriq-6.1) with message:

building 'borg.algorithms.msgpack._packer' extension
/spksrc/toolchains/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-linux-gnuspe-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/spksrc/toolchains/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include -mcpu=8548 -mhard-float -mfloat-gprs=double -I/spksrc/spk/borgbackup/work-qoriq-6.1/install//var/packages/borgbackup/target/include -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -L /spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/lib -I /spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include -I/spksrc/toolchains/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include -mcpu=8548 -mhard-float -mfloat-gprs=double -I/spksrc/spk/borgbackup/work-qoriq-6.1/install//var/packages/borgbackup/target/include -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -L /spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/lib -I /spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include -I/spksrc/toolchains/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include -mcpu=8548 -mhard-float -mfloat-gprs=double -I/spksrc/spk/borgbackup/work-qoriq-6.1/install//var/packages/borgbackup/target/include -I/spksrc/toolchains/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include -mcpu=8548 -mhard-float -mfloat-gprs=double -I/spksrc/spk/borgbackup/work-qoriq-6.1/install//var/packages/borgbackup/target/include -fPIC -D__LITTLE_ENDIAN__=1 -I/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include -Isrc/borg/algorithms/lz4/lib -Isrc/borg/algorithms/zstd/lib -Isrc/borg/algorithms/zstd/lib/common -Isrc/borg/algorithms/zstd/lib/compress -Isrc/borg/algorithms/zstd/lib/decompress -Isrc/borg/algorithms/zstd/lib/dictBuilder -Isrc/borg/algorithms/blake2/ref -I/spksrc/spk/borgbackup/work-qoriq-6.1/crossenv/cross/include -I/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include/python3.7m -c src/borg/algorithms/msgpack/_packer.cpp -o build/temp.linux-powerpc-3.7/src/borg/algorithms/msgpack/_packer.o
In file included from /spksrc/toolchains/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include/endian.h:36:0,
                 from /spksrc/toolchains/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include/bits/waitstatus.h:64,
                 from /spksrc/toolchains/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include/stdlib.h:42,
                 from /spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include/python3.7m/Python.h:34,
                 from src/borg/algorithms/msgpack/_packer.cpp:4:
/spksrc/toolchains/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include/bits/endian.h:26:4: error: #error Both BIG_ENDIAN and LITTLE_ENDIAN defined!
 #  error Both BIG_ENDIAN and LITTLE_ENDIAN defined!
    ^
error: command '/spksrc/toolchains/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-linux-gnuspe-gcc' failed with exit status 1

May you help about it?

[ymartin59]

@hgy59 @th0ma7 If I remember well, you have already faced powerpc compilation troubles which required patching original build chain or source. May you please help about this borgbackup ppc compilation failure?

[th0ma7]

@ymartin59 I can certainly have a look at it but I'll be offline for a few days, perhaps next week. Cheers!

[smaarn]

@ymartin59 @th0ma7 @hgy59 I actually may have found a way to fix it (see #4135 ). Unfortunately I cannot really test it to be sure on the corresponding powerpc architectures.

[ymartin59]

@smaarn Many thanks. I take it as-is, I have no way to test either.

[ymartin59]

Thanks everybody. I consider it as fixed - published applications run well as far as I know

[rgodinezp]

@ymartin59 Sorry to bring bad news, but the homeassistant package release 0.114.2-7 still has the same problems in my DS214play (Intel Atom processor, DSM 6.2.3-25426 Update 2):

• It doesn't finish cleanly the initial config after installation,
  2020-08-24 00:40:33 ERROR (SyncWorker_36) [homeassistant.util.package] Unable to install package hass-nabucasa==0.35.0: ERROR: Command errored out with exit status 1: command: /volume1/@appstore/homeassistant/env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wa7y4lp8/yarl/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wa7y4lp8/yarl/setup.py'"'"';f=getattr(tok enize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-f7x9emxg cwd: /tmp/pip-install-wa7y4lp8/yarl/
• web interface says "Home Assistant is starting, not everything will be available until it finishes."
• second screen (location) of "/onboarding.html" gives error "Failed to save: undefined" and cannot continue with initial setup.
• log excerpt with some of the errors:
  /spksrc/toolchains/syno-evansport-6.1/work/i686-pc-linux-gnu/bin/i686-pc-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/spksrc/toolchains/syno-evansport-6.1/work/i686-pc-linux-gnu/i686-pc-linux-gnu/sy s-root/usr/include -I/spksrc/spk/python3/work-evansport-6.1/install//var/packages/python3/target/include -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -L /spksrc/spk/python3/work-evansport-6.1/install/var/packages/python3/target/lib -I /sp ksrc/spk/python3/work-evansport-6.1/install/var/packages/python3/target/include -I/spksrc/toolchains/syno-evansport-6.1/work/i686-pc-linux-gnu/i686-pc-linux-gnu/sys-root/usr/include -I/spksrc/spk/python3/work-evansport-6.1/install//var/pa ckages/python3/target/include -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -L /spksrc/spk/python3/work-evansport-6.1/install/var/packages/python3/target/lib -I /spksrc/spk/python3/work-evansport-6.1/install/var/packages/python3/target/inc lude -fPIC -I/volume1/@appstore/homeassistant/env/include -I/var/packages/python3/target/include/python3.7m -c yarl/_quoting.c -o build/temp.linux-i686-3.7/yarl/_quoting.o unable to execute '/spksrc/toolchains/syno-evansport-6.1/work/i686-pc-linux-gnu/bin/i686-pc-linux-gnu-gcc': No such file or directory error: command '/spksrc/toolchains/syno-evansport-6.1/work/i686-pc-linux-gnu/bin/i686-pc-linux-gnu-gcc' failed with exit status 1

Attached the logs
homeassistant.log
homeassistant_install.log

Hope this helps finding root cause and fixing the issue.