Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pyhon3* wheel fixes + TVH Fall 2021 update & Fix 403 Forbidden #4921

Merged
merged 33 commits into from
Oct 22, 2021

Conversation

th0ma7
Copy link
Contributor

@th0ma7 th0ma7 commented Oct 17, 2021

Motivation: Update tvheadend to latest available version and:

  1. Use -C flag at startup to manage the initial creation of the admin user account using built-in app functionality instead of playing with wizard. This allow removing wizard entirely along with POST_STRIP_TARGET.
  2. Enable logs using the -l flag
  3. Update to version from October 16th 2021 (git 1ee9c5b)
  4. Fix SPK_DEPENDS to add ffmpeg on top of python38
  5. Import tvheadend specific changes from 4797 to its own PR

IMPORTANT: While investigating I also found issues with latest python38 package (ref. #4925). Currently this PR also contains important fixes for python3* to work properly.

Linked issues: #4758 and #3707 for tvheadend and created #4925 while investigating

Checklist

  • Build rule all-supported completed successfully
  • Package upgrade completed successfully
  • New installation of package completed successfully

1. Use -C flag at startup to manage the initial creation of the
admin user account using built-in app functionality instead of
playing with wizard.  This allow removing wizard entirely along
with POST_STRIP_TARGET.
2. Enable logs using the -l flag
3. Update to version from October 16th 2021 (git 1ee9c5b)
4. Fix SPK_DEPENDS to add ffmpeg on top of python38
1. Use -C flag at startup to manage the initial creation of the
admin user account using built-in app functionality instead of
playing with wizard.  This allow removing wizard entirely along
with POST_STRIP_TARGET.
2. Enable logs using the -l flag
3. Update to version from October 16th 2021 (git 1ee9c5b)
4. Fix SPK_DEPENDS to add ffmpeg on top of python38
@th0ma7 th0ma7 linked an issue Oct 19, 2021 that may be closed by this pull request
@th0ma7 th0ma7 requested a review from hgy59 October 19, 2021 03:31
fi

build_wheel_target: $(PRE_WHEEL_TARGET)
@if [ ! -z "$(WHEELS)" ] ; then \
$(foreach e,$(shell cat $(WORK_DIR)/python-cc.mk),$(eval $(e))) \
if [ ! -z "$(CROSS_COMPILE_WHEELS)" ] ; then \
if [ -f "$(WORK_DIR)/wheelhouse/cross-requirements.txt" ]; then \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what happens here? How does the rename from requirements to cross-requirements fixes things?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its not only a rename but rather telling to cross-compile the ones in cross and "python" compile the other ones by default (e.g. pure). Again for reference #4925 (comment)

@th0ma7
Copy link
Contributor Author

th0ma7 commented Oct 21, 2021

It's missing a newline when combing the 2 requirement files. I see now that my requirement-cross.txt file that I made is missing a newline at the end of the file, so that's probably it. But it's something to be careful about.

Good catch. I'll check if I can easily force adding a new line to alleviate that.

EDIT: Added a fix for that, can you give that a shot and confirm it works as expected? Also, @Safihre besides that issue, where your packages good and fully functional applied over this branch and using python38 from here?

@th0ma7
Copy link
Contributor Author

th0ma7 commented Oct 21, 2021

@Safihre related to your previous comment:

Many of these are pure python, why are they in the cross file?

How do you figure out what's pure vs cross (other than looking for build failures)? Using that, would you have a moment to help out and revisit this PR in order to move wheels from cross to pure in affected packages when you see fit so only a minimal set of cross wheels remain?

@Safihre
Copy link
Contributor

Safihre commented Oct 21, 2021

@Safihre related to your previous comment:

Many of these are pure python, why are they in the cross file?

How do you figure out what's pure vs cross (other than looking for build failures)? Using that, would you have a moment to help out and revisit this PR in order to move wheels from cross to pure in affected packages when you see fit so only a minimal set of cross wheels remain?

I looked at the pypi.org pages for each package, if they are binary then they usually have many whl files listed for each platform.
For example: https://pypi.org/project/lxml/#files
If they are pure-python, they are usually just the xxx-py3-none-any.whl file.
For example: https://pypi.org/project/chardet/#files

@Safihre
Copy link
Contributor

Safihre commented Oct 22, 2021

I installed the python38 from this build on my DSM6 88f6281 , the log seems fine (see below).
Additionally, I installed the SABnzbd package that was build using this PR and it also installed fine and is working.

@hgy59 or @publicarray Looks good to me, maybe you have any more comments?

python38

2021/10/22 10:00:50     ===> Step postinst. USER= GROUP= SHARE_PATH=
2021/10/22 10:00:50     Begin service_postinst
2021/10/22 10:02:21     Looking in links: /volume1/@appstore/python38/share/wheelhouse
2021/10/22 10:02:21     Processing /volume1/@appstore/python38/share/wheelhouse/MarkupSafe-2.0.1-cp38-none-any.whl
2021/10/22 10:02:21     Processing /volume1/@appstore/python38/share/wheelhouse/PyYAML-5.4.1-cp38-none-any.whl
2021/10/22 10:02:21     Processing /volume1/@appstore/python38/share/wheelhouse/SQLAlchemy-1.4.25-cp38-none-any.whl
2021/10/22 10:02:21     Processing /volume1/@appstore/python38/share/wheelhouse/appdirs-1.4.4-py2.py3-none-any.whl
2021/10/22 10:02:21     Processing /volume1/@appstore/python38/share/wheelhouse/distlib-0.3.3-py3-none-any.whl
2021/10/22 10:02:22     Processing /volume1/@appstore/python38/share/wheelhouse/filelock-3.3.1-py3-none-any.whl
2021/10/22 10:02:22     Processing /volume1/@appstore/python38/share/wheelhouse/idna-3.2-py3-none-any.whl
2021/10/22 10:02:22     Processing /volume1/@appstore/python38/share/wheelhouse/immutables-0.16-cp38-none-any.whl
2021/10/22 10:02:22     Processing /volume1/@appstore/python38/share/wheelhouse/importlib_metadata-4.8.1-py3-none-any.whl
2021/10/22 10:02:22     Processing /volume1/@appstore/python38/share/wheelhouse/importlib_resources-5.2.2-py3-none-any.whl
2021/10/22 10:02:22     Processing /volume1/@appstore/python38/share/wheelhouse/ipaddress-1.0.23-py2.py3-none-any.whl
2021/10/22 10:02:22     Processing /volume1/@appstore/python38/share/wheelhouse/lxml-4.6.3-cp38-none-any.whl
2021/10/22 10:02:22     Processing /volume1/@appstore/python38/share/wheelhouse/msgpack_python-0.5.6-cp38-none-any.whl
2021/10/22 10:02:23     Processing /volume1/@appstore/python38/share/wheelhouse/olefile-0.46-py2.py3-none-any.whl
2021/10/22 10:02:23     Processing /volume1/@appstore/python38/share/wheelhouse/platformdirs-2.4.0-py3-none-any.whl
2021/10/22 10:02:23     Processing /volume1/@appstore/python38/share/wheelhouse/psutil-5.8.0-cp38-none-any.whl
2021/10/22 10:02:23     Processing /volume1/@appstore/python38/share/wheelhouse/pyOpenSSL-21.0.0-py2.py3-none-any.whl
2021/10/22 10:02:24     Processing /volume1/@appstore/python38/share/wheelhouse/pyasn1-0.4.8-py2.py3-none-any.whl
2021/10/22 10:02:25     Processing /volume1/@appstore/python38/share/wheelhouse/pycurl-7.43.0.6-cp38-none-any.whl
2021/10/22 10:02:26     Processing /volume1/@appstore/python38/share/wheelhouse/ruamel.yaml-0.16.10-py3-none-any.whl
2021/10/22 10:02:27     Processing /volume1/@appstore/python38/share/wheelhouse/six-1.16.0-py2.py3-none-any.whl
2021/10/22 10:02:27     Processing /volume1/@appstore/python38/share/wheelhouse/virtualenv-20.4.6-py2.py3-none-any.whl
2021/10/22 10:02:27     Processing /volume1/@appstore/python38/share/wheelhouse/zipp-3.6.0-py3-none-any.whl
2021/10/22 10:02:28     Processing /volume1/@appstore/python38/share/wheelhouse/zope.interface-5.4.0-cp38-none-any.whl
2021/10/22 10:02:35     Installing collected packages: zope.interface, zipp, virtualenv, SQLAlchemy, six, ruamel.yaml, PyYAML, pyOpenSSL, pycurl, pyasn1, psutil, platformdirs, olefile, msgpack-python, MarkupSafe, lxml, ipaddress, importlib-resources, importlib-metadata, immutables, idna, filelock, distlib, appdirs
2021/10/22 10:04:36     Successfully installed MarkupSafe-2.0.1 PyYAML-5.4.1 SQLAlchemy-1.4.25 appdirs-1.4.4 distlib-0.3.3 filelock-3.3.1 idna-3.2 immutables-0.16 importlib-metadata-4.8.1 importlib-resources-5.2.2 ipaddress-1.0.23 lxml-4.6.3 msgpack-python-0.5.6 olefile-0.46 platformdirs-2.4.0 psutil-5.8.0 pyOpenSSL-21.0.0 pyasn1-0.4.8 pycurl-7.43.0.6 ruamel.yaml-0.16.10 six-1.16.0 virtualenv-20.4.6 zipp-3.6.0 zope.interface-5.4.0
2021/10/22 10:04:36     WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
2021/10/22 10:04:39     Installed version: Python 3.8.12
2021/10/22 10:04:39     Installed modules:
2021/10/22 10:05:14     appdirs @ file:///volume1/%40appstore/python38/share/wheelhouse/appdirs-1.4.4-py2.py3-none-any.whl
2021/10/22 10:05:14     bcrypt==3.2.0
2021/10/22 10:05:14     cffi==1.14.6
2021/10/22 10:05:14     cryptography==3.3.2
2021/10/22 10:05:14     distlib @ file:///volume1/%40appstore/python38/share/wheelhouse/distlib-0.3.3-py3-none-any.whl
2021/10/22 10:05:14     filelock @ file:///volume1/%40appstore/python38/share/wheelhouse/filelock-3.3.1-py3-none-any.whl
2021/10/22 10:05:14     idna @ file:///volume1/%40appstore/python38/share/wheelhouse/idna-3.2-py3-none-any.whl
2021/10/22 10:05:14     immutables @ file:///volume1/%40appstore/python38/share/wheelhouse/immutables-0.16-cp38-none-any.whl
2021/10/22 10:05:14     importlib-metadata @ file:///volume1/%40appstore/python38/share/wheelhouse/importlib_metadata-4.8.1-py3-none-any.whl
2021/10/22 10:05:14     importlib-resources @ file:///volume1/%40appstore/python38/share/wheelhouse/importlib_resources-5.2.2-py3-none-any.whl
2021/10/22 10:05:14     ipaddress @ file:///volume1/%40appstore/python38/share/wheelhouse/ipaddress-1.0.23-py2.py3-none-any.whl
2021/10/22 10:05:14     lxml @ file:///volume1/%40appstore/python38/share/wheelhouse/lxml-4.6.3-cp38-none-any.whl
2021/10/22 10:05:14     MarkupSafe @ file:///volume1/%40appstore/python38/share/wheelhouse/MarkupSafe-2.0.1-cp38-none-any.whl
2021/10/22 10:05:14     msgpack-python @ file:///volume1/%40appstore/python38/share/wheelhouse/msgpack_python-0.5.6-cp38-none-any.whl
2021/10/22 10:05:14     olefile @ file:///volume1/%40appstore/python38/share/wheelhouse/olefile-0.46-py2.py3-none-any.whl
2021/10/22 10:05:14     platformdirs @ file:///volume1/%40appstore/python38/share/wheelhouse/platformdirs-2.4.0-py3-none-any.whl
2021/10/22 10:05:14     psutil @ file:///volume1/%40appstore/python38/share/wheelhouse/psutil-5.8.0-cp38-none-any.whl
2021/10/22 10:05:14     pyasn1 @ file:///volume1/%40appstore/python38/share/wheelhouse/pyasn1-0.4.8-py2.py3-none-any.whl
2021/10/22 10:05:14     pycparser==2.20
2021/10/22 10:05:14     pycrypto==2.6.1
2021/10/22 10:05:14     pycurl @ file:///volume1/%40appstore/python38/share/wheelhouse/pycurl-7.43.0.6-cp38-none-any.whl
2021/10/22 10:05:14     pyOpenSSL @ file:///volume1/%40appstore/python38/share/wheelhouse/pyOpenSSL-21.0.0-py2.py3-none-any.whl
2021/10/22 10:05:14     PyYAML @ file:///volume1/%40appstore/python38/share/wheelhouse/PyYAML-5.4.1-cp38-none-any.whl
2021/10/22 10:05:14     ruamel.yaml @ file:///volume1/%40appstore/python38/share/wheelhouse/ruamel.yaml-0.16.10-py3-none-any.whl
2021/10/22 10:05:14     six @ file:///volume1/%40appstore/python38/share/wheelhouse/six-1.16.0-py2.py3-none-any.whl
2021/10/22 10:05:14     SQLAlchemy @ file:///volume1/%40appstore/python38/share/wheelhouse/SQLAlchemy-1.4.25-cp38-none-any.whl
2021/10/22 10:05:14     virtualenv @ file:///volume1/%40appstore/python38/share/wheelhouse/virtualenv-20.4.6-py2.py3-none-any.whl
2021/10/22 10:05:14     zipp @ file:///volume1/%40appstore/python38/share/wheelhouse/zipp-3.6.0-py3-none-any.whl
2021/10/22 10:05:14     zope.interface @ file:///volume1/%40appstore/python38/share/wheelhouse/zope.interface-5.4.0-cp38-none-any.whl
2021/10/22 10:05:15     End service_postinst

sabnzbd

2021/10/22 10:15:27     Begin service_postinst
2021/10/22 10:19:01     Looking in links: /volume1/@appstore/sabnzbd/share/wheelhouse
2021/10/22 10:19:01     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/Cheetah3-3.2.6-cp38-none-any.whl
2021/10/22 10:19:01     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/CherryPy-18.6.1-py2.py3-none-any.whl
2021/10/22 10:19:01     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/babelfish-0.6.0-py3-none-any.whl
2021/10/22 10:19:01     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/chardet-4.0.0-py2.py3-none-any.whl
2021/10/22 10:19:01     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/cheroot-8.5.2-py2.py3-none-any.whl
2021/10/22 10:19:02     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/configobj-5.0.6-py3-none-any.whl
2021/10/22 10:19:02     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/feedparser-6.0.8-py3-none-any.whl
2021/10/22 10:19:03     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/guessit-3.3.1-py3-none-any.whl
2021/10/22 10:19:05     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/jaraco.classes-3.2.1-py3-none-any.whl
2021/10/22 10:19:05     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/jaraco.collections-3.4.0-py3-none-any.whl
2021/10/22 10:19:06     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/jaraco.functools-3.3.0-py3-none-any.whl
2021/10/22 10:19:06     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/jaraco.text-3.5.1-py3-none-any.whl
2021/10/22 10:19:06     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/more_itertools-8.9.0-py3-none-any.whl
2021/10/22 10:19:06     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/portend-2.7.1-py3-none-any.whl
2021/10/22 10:19:07     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/puremagic-1.10-py3-none-any.whl
2021/10/22 10:19:07     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/python_dateutil-2.8.2-py2.py3-none-any.whl
2021/10/22 10:19:07     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/pytz-2021.1-py3-none-any.whl
2021/10/22 10:19:08     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/rebulk-3.0.1-py3-none-any.whl
2021/10/22 10:19:08     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/sabyenc3-4.0.2-cp38-none-any.whl
2021/10/22 10:19:08     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/sgmllib3k-1.0.0-py3-none-any.whl
2021/10/22 10:19:09     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/tempora-4.1.1-py3-none-any.whl
2021/10/22 10:19:09     Processing /volume1/@appstore/sabnzbd/share/wheelhouse/zc.lockfile-2.0-py2.py3-none-any.whl
2021/10/22 10:19:15     Installing collected packages: zc.lockfile, tempora, sgmllib3k, sabyenc3, rebulk, pytz, python-dateutil, puremagic, portend, more-itertools, jaraco.text, jaraco.functools, jaraco.collections, jaraco.classes, guessit, feedparser, configobj, CherryPy, cheroot, Cheetah3, chardet, babelfish
2021/10/22 10:20:26     Successfully installed Cheetah3-3.2.6 CherryPy-18.6.1 babelfish-0.6.0 chardet-4.0.0 cheroot-8.5.2 configobj-5.0.6 feedparser-6.0.8 guessit-3.3.1 jaraco.classes-3.2.1 jaraco.collections-3.4.0 jaraco.functools-3.3.0 jaraco.text-3.5.1 more-itertools-8.9.0 portend-2.7.1 puremagic-1.10 python-dateutil-2.8.2 pytz-2021.1 rebulk-3.0.1 sabyenc3-4.0.2 sgmllib3k-1.0.0 tempora-4.1.1 zc.lockfile-2.0
2021/10/22 10:20:29     End service_postinst

@th0ma7
Copy link
Contributor Author

th0ma7 commented Oct 22, 2021

Thnx @Safihre this is much appreciated.
I'm thinking of two last adjustment:

  1. process the pure-python first (currently its the contrary)
  2. migrate all pure-python from cross to pure (as you pointed out that many are still in cross)

@th0ma7
Copy link
Contributor Author

th0ma7 commented Oct 22, 2021

@Safihre I've nwo pushed all the changes (with exception of homeassistant that I haven't touched). Can you give it one more shot to confirm that all looks good from your end?

@Safihre
Copy link
Contributor

Safihre commented Oct 22, 2021

@Safihre I've nwo pushed all the changes (with exception of homeassistant that I haven't touched). Can you give it one more shot to confirm that all looks good from your end?

I rebased my SABnzbd PR on this one and it works as expected.
Will have to wait for the build to complete (takes a few hours) before being able to test Python 3.8 package, I don't have my own spksrc local environment anymore.

@th0ma7
Copy link
Contributor Author

th0ma7 commented Oct 22, 2021

Great, thnx for this initial feedback.
I now wonder, why not include urllib3 by default in python38 as so many packages requires it?

@Safihre
Copy link
Contributor

Safihre commented Oct 22, 2021

urllib3

Yes, it's used by a lot of packages (13 from what I see). Even more (18) use it's big brother: requests. I always would have expected us to include those two, but maybe let's skip that for another day..
This PR is already big enough :)

@BKSteve
Copy link
Contributor

BKSteve commented Oct 22, 2021

Python 3 & 38 are very useful and important packages why not pop in urllib3 and requests now?
Is the PR really that big?

@th0ma7
Copy link
Contributor Author

th0ma7 commented Oct 22, 2021

@BKSteve good point although conceptually there is more to that:

  1. I the new changes I don't think there is any need to have cross/* python wheels anymore. I believe they can be entirely stripped out and just be using requirements-cross.txt file isntead.
  2. There probably are other wheels worth adding by default, at least in pythone38. urllib3 and requests are just the start.
  3. There is a need to stop rebuilding python for every sub-package that requires cross-compiled wheels. @ymartin59 has a proposal on that Improve binary re-use to fasten compilation and reduce package size (Conan) #3630 . @hgy59 was also mentioning of doing a similar work as what's done for ffmpeg for instance.
  4. Remove optimizations when building outside of spk/python38 in order to reduce build time

All in all there is a point where we have to get things out and wait for the right time for subsequent changes. As such I tend to agree with @Safihre , lets get this one out and fix multiple issues right away, and leave the remaining for later.

@th0ma7 th0ma7 merged commit 72f591e into SynoCommunity:master Oct 22, 2021
@th0ma7 th0ma7 deleted the tvh-update branch October 22, 2021 19:45
@Safihre
Copy link
Contributor

Safihre commented Oct 23, 2021

Tested and works as expected :)

@th0ma7 th0ma7 added status/published Published and activated (may take up to 48h until visible in DSM package manager) and removed status/needs-review labels Oct 23, 2021
@BKSteve
Copy link
Contributor

BKSteve commented Oct 24, 2021

3\. There is a need to stop rebuilding python for every sub-package that requires cross-compiled wheels.  @ymartin59 has a proposal on that

Yep, so much time spent building Python with all it's own cross and native items, it would be great if this was a tar so the whole lot could be pulled already compiled. Plug all the other bits you need for your individual package in and done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug status/published Published and activated (may take up to 48h until visible in DSM package manager)
Projects
None yet
4 participants