Adjust the Fallback logic for obtaining the hashes from private indexes #5866
Conversation
| if session is None: | ||
| with closing(urllib_request.urlopen(link)) as f: | ||
| yield f | ||
| session = PipSession(cache=USER_CACHE_DIR) |
There was a problem hiding this comment.
Well, it turns out PipSession removed a lot of code :-)
|
Nice job with the caching improvement as well! |
|
so...what's the final solution? I'm using the latest version of pipenv. when I run pipenv install torch==2.0.1+cu117 --index https://download.pytorch.org/whl/cu117It still keeps locking and downloading something I don't know... pipfile |
|
@BetterWorld-Liuser The first time you lock its going to download 16 GB of wheel files from the torch index. |
|
@matteius so there is no way to install just like |
|
@BetterWorld-Liuser Once you have a valid lock file, which needs the hashes of all matching wheel files for that version, which is why it has to download approximately 16 GB, then the subsequent install just downloads the one matching wheel for your system, which should be cached by the prior lock. |
|
@matteius OK, I understand, thanks! |
The else Fallback here is not working right, for example if my source url is: url = "https://download.pytorch.org/whl/cu117" and the page contains url's like: /whl/cu117/torch-2.0.1%2Bcu117-cp310-cp310-linux_x86_64.whl So the url that gets generated is: https://download.pytorch.org/whl/cu117/whl/cu117/torch-2.0.1%2Bcu117-cp310-cp310-linux_x86_64.whl which is wrong because it repeats /whl/cu117/
The issue
Fixes #5864
Fixes #5860
Maybe Fixes #5848
The fix
Use urllib.parse urljoin which will intelligently handle this case:
Install:
I've also added better error handling for this case to avoid capturing a hash if the response is not 200. When I undo the fix, that looks like:
Latest Update is I got caching to work -- so if you download the expensive pytorch wheels in one lock phase, the subsequent lock phase is much faster! 🎉
The checklist
news/directory to describe this fix with the extension.bugfix.rst,.feature.rst,.behavior.rst,.doc.rst..vendor.rst. or.trivial.rst(this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.