Collect core package metadata on demand, live, aka. purl2meta

[pombredanne]

To depart from our batch data collection ways and provide current, focused package metadata, we should collect core package metadata live, on demand.

We should add a REST API end point that responds to a PURL and collects core metadata live, return this and store these in the PurlDB for a requested package version.

We will use Maven and JavaScript/npm as base ecosystems and do the following:

• An extended JSON REST API end-point that accepts a PURL input and return live metadata for Maven #338
• Add support for npm for on-demand package metadata collection #339

[pombredanne]

This is completed. We did update PurlDB with a new collect/ REST API end point that responds to a PURL and collects core metadata live as needed. It can also collect many versions of a package at once. We wired support for Maven (that required extensive additional work to be operational) and JavaScript/npm as ecosystems.

• We created a new Maven crawler that can handle Maven repositories without nexus indexes:
  v3.0.0...main#diff-9b8a11fb5e4f59a7bd5fdd6f0e0516a58ab9ac7054d25a439d4c779045a2d273R458
  https://github.com/nexB/purldb/blob/21c24f4a47a03c2f47a8661c23f5330a0ecf10ab/minecode/management/commands/maven_crawler.py
• And also we now consider the parent POM metadata as part of the merged metadata for a Maven package v3.0.0...main#diff-9b8a11fb5e4f59a7bd5fdd6f0e0516a58ab9ac7054d25a439d4c779045a2d273R203
• We further expanded Maven support to better track the release dates which is an important topic to determine release precedence: https://github.com/nexB/purldb/blob/21c24f4a47a03c2f47a8661c23f5330a0ecf10ab/minecode/management/commands/get_maven_release_dates.py
• We created a new API endpoint with the “CollectViewSet”. It returns the Package metadata for a PURL and if the package does not exist, we will fetch the Package metadata live and return it in the same request. We wired it for npm and Maven
  See v3.0.0...main#diff-56868aeeeef335bde38d62cfb44dc6e518ae311bc6bcdf298bb5e7bc73cd1afcR576
• We added support to fetch package versions live for many packages including Maven and npm https://github.com/nexB/purldb/blob/21c24f4a47a03c2f47a8661c23f5330a0ecf10ab/packagedb/package_managers.py
  The v3.0.0...main#diff-8aa34cd945d4b246084dd663e0613081a33aba4d144b7e89229d0050ef7edaf3R938 ImportableURI is also serving as a new queue

[pombredanne]

The simple way to test this is to go to the collect/ endpoint as in:

• https://<your purldb host>/api/collect/?purl=pkg:maven/net.java.dev.jna/jna-jpms@5.14.0

Closing this as done.