Support poetry lockfile version 2.0

[maxrake]

Overview

The Python poetry lockfile version is going to be updated to v2.0, starting with the poetry v1.3 release. It is not clear when this release will happen. However, the release of poetry v1.2.2 happened on 10 OCT 2022 and the announcement for that release shows that the release includes backwards and forwards compatibility for the new lockfile version for both poetry version 1.2 and 1.3

Additional Details

• It appears that the main difference is that package files and hashes are no longer stored in metadata.hashes and instead are stored in package.files
  • This should not affect the way the lockfile is parsed for Phylum
  • Only version 1.x poetry lockfiles are currently supported
• Consider adding a new test fixture to account for the new lockfile version format

References:

• PR for lockfile format 2.0
• PR to add forward compatibility for lock file format 2.0, which will be used by Poetry 1.3

Acceptance Criteria

• poetry lockfile versions 1.x and 2.x are supported for parsing
• Tests are updated to include all supported poetry lockfile versions
• Documentation is updated

[cd-work]

Hm I think this also brings up the interesting point of making sure all our other parsers stay up to date. Luckily you use Python every once in a while and I'd imagine a similar "automatic" realization would happen with cargo lockfile updates, but for many of our ecosystems we don't heavily testdrive our parser on the latest package lockfiles.

Would it maybe be worth considering adding simple tests for every supported ecosystem that generates a completely new lockfile (which should ideally use the latest version then), add a package, then try to parse it?

[maxrake]

Would it maybe be worth considering adding simple tests for every supported ecosystem that generates a completely new lockfile (which should ideally use the latest version then), add a package, then try to parse it?

Great idea. It certainly helps more than waiting for new versions to break the parser.

There may still have to be a bit of a manual element involved, though, to keep up with ecosystem and package manager changes. For instance, the Python poetry v1.2.0 release added subdirectory support for git dependencies, which is a feature that an established test would not have been able to account for.

Similar features are added all the time in poetry but maybe the cadence is slower on other, more established, package managers. Still, the 80-90% use case can be accounted for with a basic lockfile generation test as described.

Maybe something like a "lockfile zoo" repository could be used...or created if one does not already exist...to host the menagerie of lockfiles across all supported ecosystems and versions. It could be public and allow for community submissions for their esoteric use cases. Maybe the various lockfiles get categorized by level of support...kind of like how Rust platforms have tiers with varying levels of test coverage and support. The zoo language and taxonomic classification aspects certainly fit well with the "Phylum" theme...

[kylewillmon]

Another potential option for testing this is to pick a few major projects in each ecosystem and download and parse their latest lockfile

[maxrake]

Another option is to use the test fixtures from the package manager's own repos. For instance, poetry has an entire directory of known good and bad lockfiles and even has one called up_to_date_lock that uses the latest v2.0 lockfile format.