Implement pep 503 data-requires-python #3877

Carreau · 2016-07-26T23:44:41Z

Allows pip to understand the data-requires-python metadata
information that can be set on a simple repository. This allows pip to
ignore any release or file that would not be compatible with the current
Python version even before trying to download and install this version.

Relevant extract of pep 503 at the time of this writing.

A repository MAY include a data-requires-python attribute on a file
link. This exposes the Requires-Python metadata field, specified in PEP
345 , for the corresponding release. Where this is present, installer
tools SHOULD ignore the download when installing to a Python version
that doesn't satisfy the requirement. For example:

<a href="..." data-requires-python="&gt;=3">...</a>
In the attribute value, < and > have to be HTML encoded as &lt; and &gt;
, respectively.

This can mostly be used to, for example, mark a new sdist of a new
package version as requires-python >3.4, and not fail to install or
upgrade on users systems.

This will require extra patches to PyPI-legacy and warehouse to be
usable. Though releasing a version of pip that understand this feature
is necessary to have wide adoption at the time when these metadata get
actually published.

This will partially replace #3847 (from @takluyver ) once pypi-legacy and warehouse support exposing this metadata as well. I suppose this will be of interest for @astrofrog.

@michaelpacer and I are trying to get a the data-requires-python into pypi-legacy, though we have issue to run it locally. So we tested with a "Fake" local PyPI that just exposes static webpages that we editted by hand to include 'date-requires-python'.

This allows pip to understand the `data-requires-python` metadata information that can be set on a simple repository. This allows pip to ignore any release or file that would not be compatible with the current Python version even before trying to download and install this version. Relevant extract of pep 503 at the time of this writing. A repository MAY include a data-requires-python attribute on a file link. This exposes the Requires-Python metadata field, specified in PEP 345 , for the corresponding release. Where this is present, installer tools SHOULD ignore the download when installing to a Python version that doesn't satisfy the requirement. For example: <a href="..." data-requires-python=">=3">...</a> In the attribute value, < and > have to be HTML encoded as < and > , respectively. This can mostly be used to, for example, mark a new sdist of a new package version as requires-python >3.4, and not fail to install or upgrade on users systems. This will require extra patches to PyPI-legacy and warehouse to be usable. Though releasing a version of pip that understand this feature is necessary to have wide adoption at the time when these metadata get actually published.

xavfernandez · 2016-07-27T07:47:18Z

pip/index.py

@@ -828,7 +836,8 @@ def links(self):
                url = self.clean_link(
                    urllib_parse.urljoin(self.base_url, href)
                )
-                yield Link(url, self)
+                pyrequire = anchor.get('data-requires-python')
+                yield Link(url, self, requires_python=pyrequire)


I would put the unescape here since we are already dealing with html here

takluyver · 2016-07-27T11:50:16Z

Thanks for jumping on this; I'm keen to get it into pip as soon as possible so it can start getting to users before we rely on it.

There are still a few Travis failures in running PEP-8.

move the unescape outside of Link class. reraise using raise that is available on Python 2.6

Carreau · 2016-07-27T19:21:06Z

There are still a few Travis failures in running PEP-8.

Yep, saw that, should be fixed as well.

Thanks for jumping on this; I'm keen to get it into pip as soon as possible so it can start getting to users before we rely on it.

This is the exact reason why I did that. Thank for pushing the fixes to pep 503.

takluyver · 2016-07-27T20:13:58Z

pip/index.py

@@ -828,7 +841,8 @@ def links(self):
                url = self.clean_link(
                    urllib_parse.urljoin(self.base_url, href)
                )
-                yield Link(url, self)
+                pyrequire = unescape(anchor.get('data-requires-python'))


We need to check that the attribute is not None before passing it to unescape, because unescape doesn't like None.

xavfernandez · 2016-07-28T08:32:44Z

A few tests would also be nice.
You can create an index.html file containing a few links and pass the file path to PackageFinder as find_links.
Then check that find_all_candidates returns the expected InstallationCandidate list

Carreau · 2016-07-28T22:32:48Z

A few tests would also be nice.
You can create an index.html file containing a few links and pass the file path to PackageFinder as find_links.
Then check that find_all_candidates returns the expected InstallationCandidate list

Should be done. Waiting for test to finish, but the one I wrote seems to pass.

xavfernandez · 2016-07-29T08:04:10Z

tests/unit/test_finder.py

@@ -365,6 +366,27 @@ def test_finder_only_installs_stable_releases(data):
        assert link.url == "https://foo/bar-1.0.tar.gz"


+def test_finder_only_installs_data_require(data):
+    """
+    """


No need for empty docstring ;)

Oops, likely did not commit the description of the tests.

xavfernandez · 2016-07-29T08:08:40Z

@Carreau Thanks for your work !
A small changelog in CHANGES.txt and we should be good to go I think.
cc @dstufft for an other review ?

Carreau · 2016-07-29T16:42:37Z

Docstring fixed, and added an item to CHANGES.txt.

Carreau · 2016-08-03T04:10:14Z

Is there anything I can do ?

takluyver · 2016-08-08T15:49:00Z

This looks good to me.

Carreau · 2016-08-11T19:54:54Z

Gentle nudge to know if there is anything I can do...

pfmoore · 2016-08-11T20:30:42Z

I've checked the code and it LGTM. @xavfernandez did you want to wait for @dstufft to OK this?

xavfernandez · 2016-08-11T20:48:18Z

@pfmoore I wanted to wait for a second check but yours is fine too :)

xavfernandez · 2016-08-11T20:48:50Z

@Carreau Thanks again 👍

takluyver · 2016-08-11T20:49:33Z

Thanks for landing this :-). Now we just need to implement the server-side part.

Carreau · 2016-08-11T21:29:14Z

Thanks a lot !

I'm trying to get a fix on PyPI-legacy, but I can't figure out how to run it locally... if any of you have insights... I would appreciate help there.

takluyver · 2016-08-14T12:21:48Z

I was just about to say that maybe we should focus on making Warehouse respect it and hope that it fully takes over as PyPI soon, but it sounds like you're making some progress with getting legacy PyPI running.

Carreau · 2016-08-14T18:28:14Z

I was just about to say that maybe we should focus on making Warehouse respect it and hope that it fully takes over as PyPI soon, but it sounds like you're making some progress with getting legacy PyPI running.

Yes, I have an ugly patch running, I need to comment a bunch of code that try to access EC2, but once you get what's wrong it's not that hard to run locally.

I'll document what I'm doing, but we can also implement the simple repo on warehouse.
I'm just wondering is this is the best format for pip to use, I was wondering if "just" a json API would be better on warehouse. It's seem simpler to parse, and easier to extend... but I'm far from knowledgeable enough to make this decision.

I'll be low bandwidth until next monday so It will take me time to clean things up; you can see the branch there though.

dstufft · 2016-08-14T18:30:47Z

We will eventually replace this with a newer API, but that is a longer term effort. Right now that is out of scope of any current efforts.

Sent from my iPhone

On Aug 14, 2016, at 2:28 PM, Matthias Bussonnier notifications@github.com wrote:

I was just about to say that maybe we should focus on making Warehouse respect it and hope that it fully takes over as PyPI soon, but it sounds like you're making some progress with getting legacy PyPI running.

Yes, I have an ugly patch running, I need to comment a bunch of code that try to access EC2, but once you get what's wrong it's not that hard to run locally.

I'll document what I'm doing, but we can also implement the simple repo on warehouse.
I'm just wondering is this is the best format for pip to use, I was wondering if "just" a json API would be better on warehouse. It's seem simpler to parse, and easier to extend... but I'm far from knowledgeable enough to make this decision.

I'll be low bandwidth until next monday so It will take me time to clean things up; you can see the branch there though.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Carreau and others added 6 commits July 26, 2016 16:02

fixed debug log message

dbcefb7

Add DocString to Link.

86b80cc

Update check_requires_python and describe behavior in docstring

f7373a4

Remove get_metadata

17d830f

'fix pep8'

4c31a55

xavfernandez added this to the 8.2 milestone Jul 27, 2016

xavfernandez reviewed Jul 27, 2016
View reviewed changes

Fix pep-8, assume work on this Python or invalid specifiers.

1d10fca

move the unescape outside of Link class. reraise using raise that is available on Python 2.6

Carreau force-pushed the implement-pep-503-data-requires branch from c8433b8 to 1d10fca Compare July 27, 2016 19:18

takluyver reviewed Jul 27, 2016
View reviewed changes

Add tests

d4e22ea

xavfernandez reviewed Jul 29, 2016
View reviewed changes

Set test docstring ad add to CHANGES.txt

bfd8794

takluyver mentioned this pull request Jul 30, 2016

Add data-requires-python attribute to file links (PEP 503) python/peps#56

Merged

xavfernandez added the type: enhancement Improvements to functionality label Aug 1, 2016

astrofrog mentioned this pull request Aug 9, 2016

Specify required Python version in setup.py astropy/astropy#5227

Merged

xavfernandez merged commit b506992 into pypa:master Aug 11, 2016

Carreau deleted the implement-pep-503-data-requires branch August 11, 2016 21:27

xavfernandez mentioned this pull request Aug 12, 2016

Allow -py3 (or -py2) suffix on filenames #3847

Closed

Carreau mentioned this pull request Aug 13, 2016

Pep-503, implementation questions. pypi/legacy#505

Closed

Carreau mentioned this pull request Oct 30, 2016

Practicalities section points to nonexistant information? python3statement/python3statement.github.io#53

Open

xavfernandez mentioned this pull request Jan 25, 2017

Fix python version check #4254

Closed

This was referenced Apr 11, 2017

Setuptools pulls down incompatible version of package with develop pypa/setuptools#1009

Closed

Outreach re: ipython 6.0.0rc1 breaking builds of related projects on Python 2.7 ipython/ipython#10438

Closed

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 3, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement pep 503 data-requires-python #3877

Implement pep 503 data-requires-python #3877

Carreau commented Jul 26, 2016

xavfernandez Jul 27, 2016

Carreau Jul 27, 2016

takluyver commented Jul 27, 2016

Carreau commented Jul 27, 2016

takluyver Jul 27, 2016

Carreau Jul 28, 2016

xavfernandez commented Jul 28, 2016

Carreau commented Jul 28, 2016

xavfernandez Jul 29, 2016

Carreau Jul 29, 2016

xavfernandez commented Jul 29, 2016

Carreau commented Jul 29, 2016

Carreau commented Aug 3, 2016

takluyver commented Aug 8, 2016

Carreau commented Aug 11, 2016

pfmoore commented Aug 11, 2016

xavfernandez commented Aug 11, 2016

xavfernandez commented Aug 11, 2016

takluyver commented Aug 11, 2016

Carreau commented Aug 11, 2016

takluyver commented Aug 14, 2016

Carreau commented Aug 14, 2016

dstufft commented Aug 14, 2016

Implement pep 503 data-requires-python #3877

Implement pep 503 data-requires-python #3877

Conversation

Carreau commented Jul 26, 2016

xavfernandez Jul 27, 2016

Choose a reason for hiding this comment

Carreau Jul 27, 2016

Choose a reason for hiding this comment

takluyver commented Jul 27, 2016

Carreau commented Jul 27, 2016

takluyver Jul 27, 2016

Choose a reason for hiding this comment

Carreau Jul 28, 2016

Choose a reason for hiding this comment

xavfernandez commented Jul 28, 2016

Carreau commented Jul 28, 2016

xavfernandez Jul 29, 2016

Choose a reason for hiding this comment

Carreau Jul 29, 2016

Choose a reason for hiding this comment

xavfernandez commented Jul 29, 2016

Carreau commented Jul 29, 2016

Carreau commented Aug 3, 2016

takluyver commented Aug 8, 2016

Carreau commented Aug 11, 2016

pfmoore commented Aug 11, 2016

xavfernandez commented Aug 11, 2016

xavfernandez commented Aug 11, 2016

takluyver commented Aug 11, 2016

Carreau commented Aug 11, 2016

takluyver commented Aug 14, 2016

Carreau commented Aug 14, 2016

dstufft commented Aug 14, 2016