Refactor crossenv spksrc.wheel.mk and build libtorrent #5180

th0ma7 · 2022-03-27T14:12:38Z

Description

Demonstration of feasibility to build using crossenv only. Required a code refactoring of crossenv wheel build mostly for properly passing individual wheel build flags as ADDITIONAL_*FLAGS. Updated code may lead to allow parallel wheel building eventually...

Currently fails to build on evansport-7.0 (arvidn/libtorrent#6792)

Reference: #4153

Checklist

Build rule all-supported completed successfully
New installation of package completed successfully
Package upgrade completed successfully (Manually install the package again)
Package functionality was tested
Any needed documentation is updated/created

Type of change

hgy59 · 2022-03-27T14:17:02Z

@th0ma7 good luck...
there is already a try to update boost (#4817) (boost 1.76 yet).

th0ma7 · 2022-03-27T14:19:10Z

I was able to get boost 1.78.0 to build ok btw... (At least it looked like so)

hgy59 · 2022-03-27T17:06:25Z

@th0ma7 I have created a patch for boost with python 3.10.

Now I am failing to compile the wheel for "rencode==1.0.6" in cross/rencode
do you have any hint?

make[3]: Entering directory '/spksrc/cross/rencode'
===>  Compiling for rencode
Traceback (most recent call last):
  File "/spksrc/spk/deluge/work-x64-7.0/rencode-1.0.6/setup.py", line 53, in <module>
    ext_modules = cythonize(ext_modules)
  File "/spksrc/spk/deluge/work-x64-7.0/crossenv/build/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 970, in cythonize
    module_list, module_metadata = create_extension_list(
  File "/spksrc/spk/deluge/work-x64-7.0/crossenv/build/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 816, in create_extension_list
    for file in nonempty(sorted(extended_iglob(filepattern)), "'%s' doesn't match any files" % filepattern):
  File "/spksrc/spk/deluge/work-x64-7.0/crossenv/build/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 114, in nonempty
    raise ValueError(error_msg)
ValueError: 'rencode/rencode.pyx' doesn't match any files
make[3]: *** [../../mk/spksrc.python-module.mk:33: compile_python_module] Error 1
make[3]: Leaving directory '/spksrc/cross/rencode'
make[2]: *** [../../mk/spksrc.depend.mk:51: depend_target] Error 2
make[2]: Leaving directory '/spksrc/spk/deluge'
make[1]: *** [../../mk/spksrc.spk.mk:542: build-arch-x64-7.0] Error 1
make[1]: Leaving directory '/spksrc/spk/deluge'
make: *** [../../mk/spksrc.spk.mk:537: arch-x64-7.0] Error 2

when I install deluge in a local venv (x64 only) rencode is successfully built:

Searching for rencode
Reading https://pypi.org/simple/rencode/
Downloading https://files.pythonhosted.org/packages/3a/fb/3c03dbe4438dd596e1378b5299990b81041739658a76e0f4a301eded67f4/rencode-1.0.6.tar.gz#sha256=2586435c4ea7d45f74e26765ad33d75309de7cf47c4d762e8efabd39905c0718
Best match: rencode 1.0.6
Processing rencode-1.0.6.tar.gz
Writing /tmp/easy_install-kcq5ku3_/rencode-1.0.6/setup.cfg
Running rencode-1.0.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-kcq5ku3_/rencode-1.0.6/egg-dist-tmp-n3au9t31
rencode/rencode.c: In function ‘__Pyx_ParseOptionalKeywords’:
rencode/rencode.c:9404:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                     (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                     ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
 static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9404:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                     (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                     ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
 Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                             ^~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9404:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                     (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                     ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
 static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9404:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                     (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                     ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
 static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9404:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                     (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                     ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
 Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                             ^~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9404:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                     (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                     ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
 static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9420:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
 static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9420:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
 Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                             ^~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9420:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
 static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9420:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
 static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9420:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
 Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                             ^~~~~~~~~~~~~~~~~~~
rencode/rencode.c:9420:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
In file included from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/unicodeobject.h:1046,
                 from /spksrc/native/python310/work-native/install/usr/local/include/python3.10/Python.h:83,
                 from rencode/rencode.c:20:
/spksrc/native/python310/work-native/install/usr/local/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
 static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
zip_safe flag not set; analyzing archive contents...
rencode.__pycache__._rencode.cpython-310: module references __file__
creating /spksrc/spk/deluge/work-x64-7.0/deluge-2.0.5/env/lib/python3.10/site-packages/rencode-1.0.6-py3.10-linux-x86_64.egg
Extracting rencode-1.0.6-py3.10-linux-x86_64.egg to /spksrc/spk/deluge/work-x64-7.0/deluge-2.0.5/env/lib/python3.10/site-packages
Adding rencode 1.0.6 to easy-install.pth file

Installed /spksrc/spk/deluge/work-x64-7.0/deluge-2.0.5/env/lib/python3.10/site-packages/rencode-1.0.6-py3.10-linux-x86_64.egg

th0ma7 · 2022-03-27T22:53:53Z

@th0ma7 I have created a patch for boost with python 3.10.

Now I am failing to compile the wheel for "rencode==1.0.6" in cross/rencode do you have any hint?

I just tried it from src/requirements-crossenv.txt and it worked out of the box? I've commited the change in this PR as a proof-of-concept, in hope this helps. BTW, deluge as well compile properly.

th0ma7 · 2022-03-28T21:42:51Z

@hgy59 I got it to work boost 1.78+libtorrent using crossenv!!! Have a look at it, this might allow you to clean-up #4153
I had some pointers at arvidn/libtorrent#6788

While playing with the numerous options I ended-up also finding a small issue in the code (now fixed). Only thing left is confirming if the -std=c11 is really needed for boost.

EDIT: In turn upgrading boost as other implicationson domoticz and mkvtoolnix).. and build only worked on 64-bit archs. But still, its a step in the right direction :)

th0ma7 · 2022-04-03T19:21:49Z

@hgy59 I may have nailed it! Note that libtorrent 2.x required c++14 and as such is only compatible with DSM > 7.

I still need to complete a few round of testing as there was a major code refactoring. Will re-tests against all wheels and archs to confirm all really is ok.

hgy59 · 2022-04-04T18:12:51Z

@hgy59 I got it to work boost 1.78+libtorrent using crossenv!!! Have a look at it, this might allow you to clean-up #4153
I had some pointers at arvidn/libtorrent#6788

@th0ma7 FYI: when building for evansports we are not cross compiling, as the build system for the compiler is set to i686 (see spksrc.tc.mk: TC_CONFIGURE_ARGS := --host=$(TC_TARGET) --build=i686-pc-linux).

This was an issue when building erlang, as this requires real cross compilation. The solution was to use --build=x86_64-pc-linux when building for evansports (i.e. i686) see cross/erlang/Makefile.

th0ma7 · 2022-04-05T15:16:22Z

@hgy59 I'll have a good look at it. As libtorrent uses the resulting configuration from boost through b2 user-config.jam file I then guess boost would need to be adapted so it affects libtorrent ...

Current evansport-7.0 config is:

using gcc : 7.5.0 : /home/spksrc/master/spksrc/toolchain/syno-evansport-7.0/work/i686-pc-linux-gnu/bin/i686-pc-linux-gnu-g++ : <address-model>"32" <cfla
gs>"-I/home/spksrc/master/spksrc/toolchain/syno-evansport-7.0/work/i686-pc-linux-gnu/i686-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/master/spksrc
/cross/boost/work-evansport-7.0/install//usr/local/boost/include " <cxxflags>"-I/home/spksrc/master/spksrc/toolchain/syno-evansport-7.0/work/i686-pc-lin
ux-gnu/i686-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/master/spksrc/cross/boost/work-evansport-7.0/install//usr/local/boost/include -Wno-deprecat
ed-declarations -D_GLIBCXX_USE_C99_MATH -D_GLIBCXX_HAVE_OBSOLETE_ISNAN -D_GLIBCXX_HAVE_OBSOLETE_ISINF" <linkflags>"-L/home/spksrc/master/spksrc/toolchai
n/syno-evansport-7.0/work/i686-pc-linux-gnu/i686-pc-linux-gnu/sys-root/lib -L/home/spksrc/master/spksrc/cross/boost/work-evansport-7.0/install//usr/loca
l/boost/lib -Wl,--rpath-link,/home/spksrc/master/spksrc/cross/boost/work-evansport-7.0/install//usr/local/boost/lib -Wl,--rpath,/usr/local/boost/lib " <
link>"shared" ; 
project : requirements <cxxflags> ;

EDIT: I tried a few things but honestly I got lost and wasn't able to get anywhere close to a solution... Perhaps you'll be luckier than me... https://www.boost.org/doc/libs/1_78_0/tools/build/doc/html/index.html

th0ma7 · 2022-04-07T16:29:05Z

@hgy59 I think I've found the reason why... it simply is not supported by evansport kernel...

armv7-7.0:

syno-armv7-7.0$ grep -R IFLA_CARRIER *
Binary file work/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot/lib/libc-2.26.so matches
Binary file work/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot/lib/libc.so.6 matches
Binary file work/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot/usr/lib/libc.a matches
work/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot/usr/include/linux/if_link.h:	IFLA_CARRIER,

evansport-7.0

syno-evansport-7.0$ grep -R IFLA_CARRIER *
Binary file work/i686-pc-linux-gnu/i686-pc-linux-gnu/sys-root/lib/libc.so.6 matches
Binary file work/i686-pc-linux-gnu/i686-pc-linux-gnu/sys-root/lib/libc-2.26.so matches

So basically, libtorrent 2.0.5 will ONLY work on DSM-7 except at least evansport.

th0ma7 mentioned this pull request Mar 28, 2022

Build directly through pip? arvidn/libtorrent#6788

Closed

th0ma7 added 13 commits April 3, 2022 18:03

Demo: Try to build libtorrent and deluge using crossenv

edfa1c9

rencode: test build

39964c7

Almost having libtorrent to get to build from crossenv... almost

deac09d

Saving current state - success at building using crossenv!!!

6342103

wheel.mk: Finish playing with includes

85c3f02

python310: Finish cleaning-up for libtorrent example

794087d

eorder and 1st attempt at fixing 32-bit for libtorrent

c5d5fa5

wheel.mk: Add single quote to each --global-option=''

57a4cab

python310: Add double-quote to global-option passed

4121903

wheel.mk: Do not add single-quote as it creates an error

5c54d58

wheel.mk: Fix crossenv pip when PATH variable varies

7a85e83

boost: Aling with @hgy59 proposed changes PR SynoCommunity#5187

9a67d19

Is there finally light ahead?

06c77dc

th0ma7 force-pushed the deluge-testing branch from ff2a707 to 06c77dc Compare April 3, 2022 18:03

th0ma7 added 4 commits April 3, 2022 19:12

wheel.mk: Remove debug and fix global-option parameters

d98f6cc

python310: Mark DSM<7 as unsupported for libtorrent (c++14)

87ac846

python310: Use latest fork for rencode that does build

90e9828

python310: Re-enable libtorrent for github-action build

03d3d9e

th0ma7 added 5 commits April 3, 2022 19:32

libtorrent: Remove BOOST_ROOT as only BOOST_BUILD_PATH is needed

c8fab41

libtorrent: Address-model is now found with BOOST_BUILD_PATH

e9810ea

deluge: Mark as broken to clean-up github-action resulsts

617e77f

libtorrent: Use RC_2_0 branch instead of master

2993580

boost: ADDITIONAL_*FLAGS nice to have but not mandatory

bd2e050

wheel.mk: Move exit 1 back to while loop

cbcc28e

th0ma7 self-assigned this Apr 3, 2022

th0ma7 requested a review from hgy59 April 3, 2022 20:49

th0ma7 changed the title ~~Deluge testing using crossenv~~ Refactor crossenv code in spksrc.wheel.mk and allow building libtorrent Apr 3, 2022

th0ma7 changed the title ~~Refactor crossenv code in spksrc.wheel.mk and allow building libtorrent~~ Refactor crossenv spksrc.wheel.mk and build libtorrent Apr 3, 2022

th0ma7 added the status/needs-review label Apr 3, 2022

th0ma7 mentioned this pull request Apr 3, 2022

Fails to cross-compile due to IFLA_* being mandatory arvidn/libtorrent#6792

Closed

th0ma7 added 4 commits April 3, 2022 21:32

python310: Test all wheels

1d62239

python310: Disable all wheels and prepare for merge

38364be

python310: Forgot to disable gevent and force pure wheel pkging

f317fac

deluge: Unmark as broken for final merge

ffff5c4

th0ma7 merged commit c892f46 into SynoCommunity:master Apr 4, 2022

th0ma7 deleted the deluge-testing branch April 4, 2022 12:46

th0ma7 mentioned this pull request Apr 4, 2022

META: Python 3.10 / 3.11 package status #5043

Closed

10 tasks

th0ma7 removed the status/needs-review label Apr 5, 2022

This was referenced Apr 7, 2022

libtorrent 1.2 + boost 1.70 or libtorrent 2.0 + boost 1.78 #5202

Merged

Remove non-existant IFLA_* on non-x86_64 archs arvidn/libtorrent#6806

Merged

Remove non-existant IFLA_* on non-x86_64 archs (RC2 branch) arvidn/libtorrent#6807

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor crossenv spksrc.wheel.mk and build libtorrent #5180

Refactor crossenv spksrc.wheel.mk and build libtorrent #5180

th0ma7 commented Mar 27, 2022 •

edited

Loading

hgy59 commented Mar 27, 2022

th0ma7 commented Mar 27, 2022

hgy59 commented Mar 27, 2022

th0ma7 commented Mar 27, 2022

th0ma7 commented Mar 28, 2022 •

edited

Loading

th0ma7 commented Apr 3, 2022

hgy59 commented Apr 4, 2022

th0ma7 commented Apr 5, 2022 •

edited

Loading

th0ma7 commented Apr 7, 2022

Refactor crossenv spksrc.wheel.mk and build libtorrent #5180

Refactor crossenv spksrc.wheel.mk and build libtorrent #5180

Conversation

th0ma7 commented Mar 27, 2022 • edited Loading

Description

Checklist

Type of change

hgy59 commented Mar 27, 2022

th0ma7 commented Mar 27, 2022

hgy59 commented Mar 27, 2022

th0ma7 commented Mar 27, 2022

th0ma7 commented Mar 28, 2022 • edited Loading

th0ma7 commented Apr 3, 2022

hgy59 commented Apr 4, 2022

th0ma7 commented Apr 5, 2022 • edited Loading

th0ma7 commented Apr 7, 2022

th0ma7 commented Mar 27, 2022 •

edited

Loading

th0ma7 commented Mar 28, 2022 •

edited

Loading

th0ma7 commented Apr 5, 2022 •

edited

Loading