Caching results of PackageInfo._pep517_metadata to cut lock time by 85%?

[sneakers-the-rat]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

On poetry 1.2.0b.

Profiled the lock routine, and it spends almost all of its time,( in this lock, 217 out of 260 total seconds) in this method:

poetry/src/poetry/inspection/info.py

Line 453 in dca0c56

def _pep517_metadata(cls, path: Path) -> PackageInfo:

which is understandable because it needs to install the package. But in my case it ends up installing the same versions of the package multiple times (eg. I have different dependencies for pandas depending on python version, and then for each version of pandas it has to solve the same dependencies for numpy depending on arch), and then across multiple locks the problem really compounds.

Shouldn't the result be the same every time you install the package? Why not cache this (and maybe offer a "clear/don't use installed metadata cache" if that causes problems for people")? I just did an extremely brutal hack of just pickling it within that function and loading it at the beginning if it was present and it seemed to work...

[abn]

@sneakers-the-rat If you have variations of a dependency all pointing at a path dependency, then I suspect the inspection is done each time. You are correct that this can be cached, however only within a single run. Caching it on disk might not be recommended as if the path dependency has a version/dependency change, we will get incorrect infomation.

Maybe a easy win here is to add @cache to the method.

[sneakers-the-rat]

It seems like it would be possible to invalidate the cache by comparing hashes, no? like have the output of this function tied to a hash of the tarfile it's made from and if that changes then invalidate? i've spent a while trying to get my head around this library but admittedly still dont' quite understand it. I can try using @cache, is that something y'all would take a PR for?

[abn]

I would say we start with the cache. The issue is that _pep517_metadata is also used with directory paths, ie. it is not always related to tar files. But also sometimes local path dependencies (eg: vcs dependencies) etc.

It would be great if we can add some testing in there too if workabe, but I suspect other cases should cover this as well.

[sneakers-the-rat]

great, I'll take a look at the existing tests, how @cache is used, and how _pep517_metadata is called and give it a shot.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.