Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python310 minimal and always download pure-python wheels #4986

Merged
merged 43 commits into from
Dec 6, 2021

Conversation

th0ma7
Copy link
Contributor

@th0ma7 th0ma7 commented Nov 29, 2021

Motivation: Reduce overall python package size by remove entirely all pure-python wheels when ever possible. Using existing framework now allows to only build mandatory cross-compiled wheels and always download every pure-python wheels. This reduces considerably the package file and the overall package dependency. Using that approach python310 is now limited to the bare minimal wheels including NO cross-compiled wheels (although its configuration reflect what would be needed to compile all common cross-compiled wheels). As such every packages contains a requirement file including all its dependencies and includes all of its cross-compiled wheels to ensure supportability on every platforms.

Linked issues: #4981, https://github.com/SickChill/SickChill/issues/7672#issuecomment-985213900

Checklist

  • Documentation (wiki)
  • Build rule all-supported completed successfully
  • Package upgrade completed successfully
  • New installation of package completed successfully

Tested packages

  • beets - tested on: kvmx64, armv7, armv5tel
  • borgbackup - tested on: kvmx64, armv7, armv5tel
  • duplicity - tested on: kvmx64, armv7, armv5tel unable to test due to gnuPG not installable
  • flexget - tested on: kvmx64, armv7, armv5tel
  • homeassistant - tested on: kvmx64, armv7
  • mercurial - tested on: kvmx64, armv7
  • python310 - tested on: kvmx64, armv7, armv5tel
  • octoprint - tested on: kvmx64, armv7
  • rdiff-backup - tested on: kvmx64, armv7
  • sabnzbd - tested on: kvmx64, armv7
  • sickchill - tested on: kvmx64, armv7, armv5tel

@th0ma7
Copy link
Contributor Author

th0ma7 commented Nov 30, 2021

@hgy59 using discussion from other PR #4981 and the added flexibility now built into the framework with the recent changes I was able to easily achieve a solution merging what you where envisioning along a few details:

  1. python310 is now to its simplest expression e.g. almost only cross-compiled wheels are included due to being needed in noarch packages
  2. by default it now download ALL pure-python wheels. As the requirement files already allows categorizing them it was trivial to extend that to the installation methodoly
  3. I've added a standard wheel installation method that greatly simplify and standardize the code in service-setup.sh

To demonstrate this I've adapted python310 + homeassistant + beets.

This has major benefits:

  • all package are independent for the most part of the python310 default package wheels
  • python310 becomes much simpler and provide a basic implementation
  • all packages are reduced in size (as all pure-python wheels are now downloaded)

@th0ma7 th0ma7 changed the title Always download pure-python wheels [WIP] Always download pure-python wheels Dec 1, 2021
@th0ma7 th0ma7 requested a review from hgy59 December 4, 2021 17:42
@th0ma7 th0ma7 changed the title [WIP] Always download pure-python wheels python310 minimal and always download pure-python wheels Dec 4, 2021
@th0ma7
Copy link
Contributor Author

th0ma7 commented Dec 4, 2021

@hgy59 you where right, the only viable path is to have a python310 minimal and ensure all wheels are included in each individual packages. With all the testing I've done, unless a package only requires pure-ptyhon, I wouldn't recommend adding any cross-compiled wheels into python310 and always defer them back to the individual package as otherwise something always breaks for one or another package. With that, beets is not noarch anymore.

@th0ma7
Copy link
Contributor Author

th0ma7 commented Dec 4, 2021

And forgot to mention, this to me concludes the work on the framework to fix all known python wheels issues (0.0.0, none-any, etc).

As soon as this is merge I'll publish the updated pyton310 package and will start looking at migrating any remaining python3 packages (starting with a tvheadend update). Although, unless told otherwise I'd let each maintainers to release their package updated versions (e.g. homeassistant, sickshill, etc).

Was missing cryptography and cffi in order to install properly
on arm (tested on armv7)
@hgy59 hgy59 mentioned this pull request Dec 5, 2021
3 tasks
@th0ma7 th0ma7 merged commit 90f7145 into SynoCommunity:master Dec 6, 2021
@th0ma7 th0ma7 deleted the min-wheel branch December 6, 2021 01:28
@th0ma7
Copy link
Contributor Author

th0ma7 commented Dec 6, 2021

@SynoCommunity/developers in process to publish python310. Not additional changes to be expected to the framework (and thnx for your patience on this). I'll let you look into your individual packages for publishing, let me know if you need assistance.

@th0ma7 th0ma7 added status/published Published and activated (may take up to 48h until visible in DSM package manager) and removed status/to-publish labels Dec 6, 2021
publicarray pushed a commit to publicarray/spksrc that referenced this pull request Dec 19, 2021
…ty#4986)

* installer.functions: New install_python_wheels() function helper

* wheel.mk: Allow enabling pure-python compiling (default disabled)

* homeassistant: Migrate to using new python wheel install helper

* python310: Use new python wheel installation helper

* homeassistant: Move requirements back from python310

* homeassistant: Use generic wheel installer

* beet: Move requirements back from python310

* beet: Use generic wheel installer

* python310: Remove all non-mandatory wheels (e.g. basic package)

* wheel.mk: noarch package may not have any *.whl files

* installer.functions: Add a virtualenv default function

* installer.functions: Allow dependency mode for pure python wheels

* beets: Adjustments to requirements.txt

* python310: Adjustments to requirements for pillow & beets

* homeassistant: Adjustments to requirement files

* installer.functions: Accept parameters to installing wheels

* sickchill: Migrate to using function based wheel installer

* sickchill: Final adjustments to requirements

* mercurial: Update to using standard wheel functions

* borgbackup: Migrate to using standard wheel install functions

* duplicity: Use standard method to create usr/local/bin symlink

* python310: Simplify requirements to bare minimal

* borgbackup: Re-adjust to new python310 requirements

* duplicity: Adjust to new python310 requirements

* homeassistant: Adjust to new python310 requirements

* sickchill: Adjust to new python310 requirements

* duplicity: Update lxml and enforce pbr versions

* rdiff-backup: Migrate to using standard wheel install method

* python310: Reducing package wheels to bare minimal

* sickchill: Use alternate tmp at installation

* sickchill: Move back compiled wheels from python310

* borgbackup: Update packaging wheel

* beets: Move to arch-specific (i.e. no longer noarch)

* octoprint: Migrate wheels back from python310 and use default inst.

* duplicity: Migrate back wheels from python310

* homeassistant: Move back cross-compiled wheels from python310

* installer.functions: Add method to install all wheels at once

* sickchill: Rename requirements-cross.txt to crossenv

* sabnzbd: Migrate to using standard wheel install function

* flexget: Update to version 3.1.155 and use new installer funct.

* python310: Bump package version

* installer.functions: Remove alt wheel installer function

* flexget: Fix for arm user alt install dir and add missing wheels

Was missing cryptography and cffi in order to install properly
on arm (tested on armv7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status/published Published and activated (may take up to 48h until visible in DSM package manager)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant