check if a download is successfull before deciding not to try the nex… #609

JensTimmerman · 2016-06-08T14:56:41Z

…t possible dist

jayvdb · 2016-06-08T15:19:35Z

setuptools/package_index.py


        if local_index is not None:
-            dist = dist or find(requirement, local_index)
+            dist, mylocation = dist, mylocation if dist else  find(requirement, local_index)



Two consecutive spaces here

jaraco · 2016-06-24T14:45:44Z

Can you supply a justification for this change? What behavior led you to make the change? How does the change improve the behavior? Are there any backward compatible considerations?

jaraco · 2016-06-24T14:48:06Z

setuptools/package_index.py

+                    mylocation = self.download(dist.location, tmpdir)
+                    if os.path.exists(mylocation):
+                        return dist, mylocation
+            return None, None


The presence of None, None is a bad smell to me. Better would be to return above a single value which wraps the necessary components, maybe a namedtuple('DistLocation', 'dist location').

Or maybe augment the resulting dist with a ._found_location attribute or similar.

sure, I can clean this up,
thought this would be ok, since process_index inner scan(link) function already does the same ;-)

No worries. I acknowledge - there are some old and undesirable patterns in setuptools, given its 11 year history.

JensTimmerman · 2016-06-24T15:09:16Z

the problem this fixes for me is: when I specify dependency_links as something like

 dependency_links=[
        "git+https://server1.example.com/mydependency.git@1.5.5#egg=mydependnecy-1.5.5",
        "git+ssh://server2.example.com/mydependency.git@1.5.5#egg=mydependnecy-1.5.5",
],

where server1 and server2 could e..g be a public github.com and a private github enterprise,
and public network is not available, then setuptools will currently fail to locate this dependency.
(changing the order of the links doesn't seem to matter)
However, with this change in place, after the first download fails, it tries the second download, and succeeds.

I could see more cases where it would be beneficial to try a second location if it exists, if the first download fails. Since, we have a second location, why not?

JensTimmerman · 2016-06-24T15:13:05Z

backwards compatibility: I don't believe a lot of work-flows would rely on the fact that an installation won't happen if the first tried link doesn't work, because this first tried link seems to be random. Depending on a few factors the first or second link will be tried first or last.

This does check if the download path exists, so if downloaders are downloading something that does not result in an existing path this could break their behaviour, but what is a download if it doesn't result in a valid path?

JensTimmerman · 2016-06-24T15:31:13Z

setuptools/package_index.py

-        if local_index is not None:
-            dist = dist or find(requirement, local_index)
+        if not dist and local_index is not None:
+            dist = find(requirement, local_index)


this change is not relevant to this pr, but this happens to be a nicer way to check, so it's in here.

JensTimmerman · 2016-06-24T15:36:57Z

old behaviour:

Searching for vsc-base>=2.4.16
Best match: vsc-base 2.4.16
Doing git clone from ssh://github.ugent.be/hpcugent/vsc-base.git to /tmp/easy_install-qVryB5/vsc-base.git
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Processing vsc-base.git
error: Couldn't find a setup script in /tmp/easy_install-qVryB5/vsc-base.git

new behaviour:

Searching for vsc-base>=2.4.16
Doing git clone from ssh://github.ugent.be/hpcugent/vsc-base.git to /tmp/easy_install-A9wgLC/vsc-base.git
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Doing git clone from https://github.com/hpcugent/vsc-base.git to /tmp/easy_install-A9wgLC/vsc-base.git
Best match: vsc-base 2.4.16
Processing vsc-base.git
Writing /tmp/easy_install-A9wgLC/vsc-base.git/setup.cfg
Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-A9wgLC/vsc-base.git/egg-dist-tmp-djmMaJ
...

jaraco · 2016-07-24T03:35:19Z

Can you please add an entry to the changelog (0.1 bump) to describe to the users what they should expect from the change?

…t possible dist

… check if downloads could succeed

JensTimmerman · 2016-07-25T14:15:54Z

@jaraco added changes and rebased on top of master

jayvdb reviewed Jun 8, 2016
View reviewed changes

jaraco reviewed Jun 24, 2016
View reviewed changes

JensTimmerman reviewed Jun 24, 2016
View reviewed changes

JensTimmerman force-pushed the check_downloads branch from 72673f6 to 3856953 Compare July 25, 2016 11:43

Jens Timmerman added 6 commits July 25, 2016 16:14

check if a download is successfull before deciding not to try the nex…

3fcd44b

…t possible dist

addressed remarks, None, None since a tuple is expected

1e5bb5e

don't return a tuple, add a new attribute to current returnvalue

c4ef24f

pick better names for variables

e2124c8

specify that no 'working' links were found, previous behaviour didn't…

e2af233

… check if downloads could succeed

added changes to changelog

87c7762

JensTimmerman force-pushed the check_downloads branch from 3856953 to 87c7762 Compare July 25, 2016 14:15

jaraco merged commit 1b94bba into pypa:master Jul 25, 2016

JensTimmerman deleted the check_downloads branch July 25, 2016 14:49

This was referenced Dec 13, 2016

bootstrap_eb.py requires git easybuilders/easybuild-framework#1962

Open

installing vsc-base with easy_install (from PyPI) picks up vsc-install from GitHub (rather than PyPI)? hpcugent/vsc-base#243

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

check if a download is successfull before deciding not to try the nex… #609

check if a download is successfull before deciding not to try the nex… #609

JensTimmerman commented Jun 8, 2016

jayvdb Jun 8, 2016

jaraco commented Jun 24, 2016

jaraco Jun 24, 2016

jaraco Jun 24, 2016

JensTimmerman Jun 24, 2016

jaraco Jun 24, 2016

JensTimmerman commented Jun 24, 2016

JensTimmerman commented Jun 24, 2016 •

edited

Loading

JensTimmerman Jun 24, 2016

JensTimmerman commented Jun 24, 2016

jaraco commented Jul 24, 2016

JensTimmerman commented Jul 25, 2016

check if a download is successfull before deciding not to try the nex… #609

check if a download is successfull before deciding not to try the nex… #609

Conversation

JensTimmerman commented Jun 8, 2016

jayvdb Jun 8, 2016

Choose a reason for hiding this comment

jaraco commented Jun 24, 2016

jaraco Jun 24, 2016

Choose a reason for hiding this comment

jaraco Jun 24, 2016

Choose a reason for hiding this comment

JensTimmerman Jun 24, 2016

Choose a reason for hiding this comment

jaraco Jun 24, 2016

Choose a reason for hiding this comment

JensTimmerman commented Jun 24, 2016

JensTimmerman commented Jun 24, 2016 • edited Loading

JensTimmerman Jun 24, 2016

Choose a reason for hiding this comment

JensTimmerman commented Jun 24, 2016

jaraco commented Jul 24, 2016

JensTimmerman commented Jul 25, 2016

JensTimmerman commented Jun 24, 2016 •

edited

Loading