pep517: in_process: try _in_process.pyc file if _in_process.py doesn't exist #130

commodo · 2021-09-06T09:40:07Z

This change came about to exist after this bug report:
openwrt/packages#11912

The general context is that OpenWrt does not ship Python source-code by
default. Instead, the build byte-compiles all .py files into .pyc files and
removes the .py files.
There's a wide reasoning for this, from performance to reducing the size of
these packages.

This change is kept in the Python3 build as this patch:
https://github.com/openwrt/packages/blob/master/lang/python/python3/patches-pip/001-pep517-pyc-fix.patch

This is an attempt to start a discussion, to see whether it makes sense to
remove it from the OpenWrt packages feed, and live in the pip source-base.

The original author of this patch is Jeffery To.
I am the submitter to this repo.
We're both co-maintainers of the Python3 package in the OpenWrt realm.

Signed-off-by: Jeffery To jeffery.to@gmail.com
Signed-off-by: Alexandru Ardelean ardeleanalex@gmail.com

…t exist This change came about to exist after this bug report: openwrt/packages#11912 The general context is that OpenWrt does not ship Python source-code by default. Instead, the build byte-compiles all .py files into .pyc files and removes the .py files. There's a wide reasoning for this, from performance to reducing the size of these packages. This change is kept in the Python3 build as this patch: https://github.com/openwrt/packages/blob/master/lang/python/python3/patches-pip/001-pep517-pyc-fix.patch This is an attempt to start a discussion, to see whether it makes sense to remove it from the OpenWrt packages feed, and live in the pip source-base. The original author of this patch is Jeffery To. I am the submitter to this repo. We're both co-maintainers of the Python3 package in the OpenWrt realm. Signed-off-by: Jeffery To <jeffery.to@gmail.com> Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>

Refreshed patches. And apply hack for line-endings in pep517 (from pip). Hack comment: # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them... # I actually tried this in a number of ways and the only way to fix this is to implement # a poor-man's dos2unix using cat + echo. # The issue is with the pip package; it seems that it throws in some Windows line-endings # and 'patch' won't handle them. So, we do a "dos2unix" and then patch. # We can get rid of this once this is solved upstream and in pip: # pypa/pyproject-hooks#130 Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>

Refreshed patches. And apply hack for line-endings in pep517 (from pip). Hack comment: # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them... # I actually tried this in a number of ways and the only way to fix this is to implement # a poor-man's dos2unix using sed. # The issue is with the pip package; it seems that it throws in some Windows line-endings # and 'patch' won't handle them. So, we do a "dos2unix" and then patch. # We can get rid of this once this is solved upstream and in pip: # pypa/pyproject-hooks#130 Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>

Refreshed patches. And apply hack for line-endings in pep517 (from pip). Hack comment: # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them... # I actually tried this in a number of ways and the only way to fix this is to implement # a poor-man's dos2unix using sed. # The issue is with the pip package; it seems that it throws in some Windows line-endings # and 'patch' won't handle them. So, we do a "dos2unix" and then patch. # We can get rid of this once this is solved upstream and in pip: # pypa/pyproject-hooks#130 Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com> (cherry picked from commit 26988f9)

Refreshed patches. And apply hack for line-endings in pep517 (from pip). Hack comment: # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them... # I actually tried this in a number of ways and the only way to fix this is to implement # a poor-man's dos2unix using sed. # The issue is with the pip package; it seems that it throws in some Windows line-endings # and 'patch' won't handle them. So, we do a "dos2unix" and then patch. # We can get rid of this once this is solved upstream and in pip: # pypa/pyproject-hooks#130 Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com> Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>

commodo · 2021-10-27T09:14:24Z

ping on this;

not saying this is the best solution, but i would like an idea on how to handle this better;

i'm fine to implement it

Refreshed patches. And apply hack for line-endings in pep517 (from pip). Hack comment: # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them... # I actually tried this in a number of ways and the only way to fix this is to implement # a poor-man's dos2unix using sed. # The issue is with the pip package; it seems that it throws in some Windows line-endings # and 'patch' won't handle them. So, we do a "dos2unix" and then patch. # We can get rid of this once this is solved upstream and in pip: # pypa/pyproject-hooks#130 Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>

Refreshed patches. And apply hack for line-endings in pep517 (from pip). Hack comment: # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them... # I actually tried this in a number of ways and the only way to fix this is to implement # a poor-man's dos2unix using sed. # The issue is with the pip package; it seems that it throws in some Windows line-endings # and 'patch' won't handle them. So, we do a "dos2unix" and then patch. # We can get rid of this once this is solved upstream and in pip: # pypa/pyproject-hooks#130 Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com> (cherry picked from commit 26988f9)

commodo · 2022-03-16T07:30:52Z

ping on this :)

Refreshed patches. And apply hack for line-endings in pep517 (from pip). Hack comment: # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them... # I actually tried this in a number of ways and the only way to fix this is to implement # a poor-man's dos2unix using sed. # The issue is with the pip package; it seems that it throws in some Windows line-endings # and 'patch' won't handle them. So, we do a "dos2unix" and then patch. # We can get rid of this once this is solved upstream and in pip: # pypa/pyproject-hooks#130 Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>

pfmoore · 2022-08-26T12:33:29Z

Assuming the .pyc is in the same location as the .py is wrong. You should probably use importlib.util.cache_from_source, but even with that, there could be problems - for example running pip from a metapath hook that references a sourceless pip¹ - which should be tested.

This would happen in real life when setting up an isolated build environment from a sourceless pip, for example. ↩

commodo · 2022-08-29T05:54:31Z

Assuming the .pyc is in the same location as the .py is wrong. You should probably use importlib.util.cache_from_source, but even with that, there could be problems - for example running pip from a metapath hook that references a sourceless pip1 - which should be tested.

Footnotes
1. This would happen in real life when setting up an isolated build environment from a sourceless pip, for example. [leftwards_arrow_with_hook](#user-content-fnref-1-de8c8293481778e25b7a0f5852353604)

phew, it's been a while since i opened this PR;
anyway the use-case for OpenWrt is a bit specific (with regards to .pyc files); we store them along-side with .py usually;

i'll re-review this and comeback;

pradyunsg · 2022-11-24T19:22:09Z

Closing this out since its not been updated in a while and there's some significant changes needed for this to even be considered.

Let's keep the discussion in #148 before we get back to this.

Refreshed patches. And apply hack for line-endings in pep517 (from pip). Hack comment: # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them... # I actually tried this in a number of ways and the only way to fix this is to implement # a poor-man's dos2unix using sed. # The issue is with the pip package; it seems that it throws in some Windows line-endings # and 'patch' won't handle them. So, we do a "dos2unix" and then patch. # We can get rid of this once this is solved upstream and in pip: # pypa/pyproject-hooks#130 Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com> (cherry picked from commit 26988f9)

commodo mentioned this pull request Sep 8, 2021

python3: bump version to 3.9.7 openwrt/packages#16571

Merged

commodo mentioned this pull request Sep 10, 2021

[21.02] python3: bump version to 3.9.7 openwrt/packages#16603

Merged

BubuOT mentioned this pull request Aug 26, 2022

Source-less distribution support #148

Open

pradyunsg closed this Nov 24, 2022

commodo deleted the in_process_pyc branch December 9, 2022 14:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pep517: in_process: try _in_process.pyc file if _in_process.py doesn't exist #130

pep517: in_process: try _in_process.pyc file if _in_process.py doesn't exist #130

commodo commented Sep 6, 2021

commodo commented Oct 27, 2021

commodo commented Mar 16, 2022

pfmoore commented Aug 26, 2022

commodo commented Aug 29, 2022

Footnotes

pradyunsg commented Nov 24, 2022

pep517: in_process: try _in_process.pyc file if _in_process.py doesn't exist #130

pep517: in_process: try _in_process.pyc file if _in_process.py doesn't exist #130

Conversation

commodo commented Sep 6, 2021

commodo commented Oct 27, 2021

commodo commented Mar 16, 2022

pfmoore commented Aug 26, 2022

Footnotes

commodo commented Aug 29, 2022

Footnotes

pradyunsg commented Nov 24, 2022