Compile and distribute immutables along with Python3

[nbarrientos]

Hi,

Not sure I'll be able to convince you that this is a good idea but it's worth a shot. Here I go.

I install Flexget on my Synology using a Python virtual environment. Unfortunately for me, recent versions of Flexget require immutables (via contextvars) which has C extensions. As I don't have any toolchain installed on the Synology and there are no precompiled wheels for aaarch64 when it's time to install this dependency via pip install everything fails for obvious reasons.

Having this library distributed by Python3 would make my life much easier. Creating a symlink from the Python virtual environment to @appstore/python3/lib/python3.6/site-packages/immutables is enough to make pip happy and get the latest Flexget installed. Also, this might be useful is somebody gives a push to the Synocommunity Flexget package in the future.

I'd totally understand if you didn't want this here but please don't delete the PR, just close it. It might be useful for my future me as a recipe for whenever I have to build Python3 again. I guess it's the price I have to pay for not having bought an Intel Synology where I could just run my stuff in Docker containers and forget about all this 😁.

The patch has been tested with a build for arch-rtd1296.

https://cern.ch/nacho/srcpkg/python3/

Thanks for your time 🤞

[ymartin59]

Your proposal sounds like a good idea. Synocommunity Flexget package requires an update itself, and so this dependency is probably a requirement for update too.

[hgy59]

I incorporated my own code review.
But found general issues with compiled wheels.

[hgy59]

@ymartin59 I fixed this to integrate immutables as compliled wheel.
But I suppose there is a general issue with compiled wheels on python3.

I analyzed the compiled wheels of all-supported archs on the PR and all arch specific compiled shared objects (*.so) have x86_64 in the name. Even these files have different size (and I assume the correct arch) they have a wrong name (IMO).

example:
the file share/wheelhouse/immutables-0.11-cp36-none-any.whl containes /immutables/_map.cpython-36m-x86_64-linux-gnu.so for all compiled archs.

I can analyze self compiled packages only (synocommunity.com is not reachable) but as far as I see this is the same with the working version of Python3 (3.6.8-9).

So please @nbarrientos and @ymartin59 (or any one else) can you test whether Python3 and immutable of this PR work with non intel archs (other than x64 and x86)?

[hgy59]

@ymartin59 as Python 3.6.10 is not yet published I didn't increment SPK_REV.

[nbarrientos]

Thanks for the review.

So please @nbarrientos and @ymartin59 (or any one else) can you test whether Python3 and immutable of this PR work with non intel archs (other than x64 and x86)?

Will generate a build with your patch for rtd1296 an report back.

[nbarrientos]

Will generate a build with your patch for rtd1296 an report back.

WFM.

foo@bar:/somewhere/@appstore/python3$ uname -m
aarch64
foo@bar:/somewhere/@appstore/python3$ ./bin/python3
Python 3.6.10 (default, Apr  5 2020, 20:07:11) 
[GCC 4.9.4 20150629 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import immutables
>>> 

Indeed the wrong arch is in the name of the shared object but it seems to be loaded just fine.

# file immutables/_map.cpython-36m-x86_64-linux-gnu.so 
immutables/_map.cpython-36m-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

Here's the build in case you wanted to inspect it:

http://cern.ch/nacho/srcpkg/python3_d72e6e94f61758904182d107347d4fcfc5a2e92b/

[ymartin59]

@nbarrientos Thanks for testing. This invalid file naming for shared objects is a concerned as far as we have reports wheels do not load properly when trying to run application packages over it.

[nbarrientos]

Thanks for merging this! Just a follow-up question for my own education. When can we expect to have an official build of Python 3.6.10-11 for rtd1296 (aarch64)? I'm currently using my own build but I'd like to align my system to the packages in the official Synocommunity repos. Thanks again!

[ymartin59]

@nbarrientos What applications are you running over Python 3.6.10. For details please refer #3944

[nbarrientos]

I see, thanks for the info @ymartin59 :)

I only run Flexget but installed standalone using a Python venv, not via Synocommunity packages. Flexget is supported on 3.6 and 3.7 and it's "meant to work" using 3.8. Happy help testing in the future if needed. Just ping me.

For info (sorry for the off-topic) this should make upgrading Synocommunity's Flexget easier.