Proposal: Store full location in published metadata versions

[thomashoneyman]

Looking at the package location and ref in metadata will not be sufficient to know where a package was fetched from if a Transfer operation has been run.

If the transfer was from Git to Git then the refs may be preserved, and this will surely be the most common case. But there are other transfers that can happen, such as transferring a package within a repository by changing its subdir key. In that case, old refs will point to a repo in which the given subdir does not exist.

I think we should specify, in package metadata, the full location + ref that was used to package each version, instead of just a top-level location. The top-level location field in metadata would remain, of course, because it's possible to process a transfer without actually publishing a new version.

Alternately, if we don't really care if the published version location is fully accurate, then perhaps we don't even store this information in the first place?

[f-f]

I'm not sure I follow what this is about, but I'll note that:

• the location that we store in the metadata file is not for historical purposes, it's the location where we will fetch the next published version. In this sense, it needs to stay a top-level field, as it's about the future
• the location field that we store in the manifest file of each published version is for historical record, but it would also store the subdir - see the Location type. We are even storing the git ref in the PublishedMetadata for each version, so I'm not sure why you think we don't have enough data to know where a package is from?

[thomashoneyman]

The top level location stores the next location to fetch from — no disagreement there, it’s necessary.

i’m noting that in each published version we store only the ref, which is insufficient to tell where that version was fetched from if a transfer has taken place.

[f-f]

I see now - indeed we should either include the full location, or remove that data entirely. Maybe remove it? It's stored inside the manifest after all.

[thomashoneyman]

Yes, and we verify the manifest location against the metadata when we publish the package, so we know the manifests are correct. So I’m in favor of removal from the metadata if we can instead rely on the manifest.

On the other hand, now that transfers are not handled via updates to the manifests but rather a distinct API operation, we could go the other way and not include the location in the manifest and include it in the metadata only instead. I’m not sure what the downside would be there; it’s one less piece of data for the user to manage directly.

I suppose the primary downside is that package managers themselves may not know the package location if the user doesn’t specify it in a config file somewhere, so nothing is gained by omitting it from the manifest.

[f-f]

We should try to minimise the amount of data that we store in the metadata files, because it's main database and source of truth; all updates to it must be correct and easy to analyse, etc; the more data there is, the harder it is for us humans to spot errors and/or malicious data.
So if we can choose to omit data from it and push it to another place (the manifest in this case) then I feel we should do that.
This will hopefully be a piece of data that the user won't manage directly, but would be handled by the package manager.

[thomashoneyman]

I agree, and something as critical as the package location seems sensible to push out to the manifest for each version.

On the other hand, we do always have to store the current package location in the metadata, and we always know where we fetched a version from when we publish it, so it seems difficult to get that wrong — it doesn’t feel like risky information to keep in the metadata. After all, we already do keep it there!

In fact, it seems more likely to be wrong in the manifest. For example, you might process a transfer and then publish a new version of your package, but forget to update the location in the manifest to the new one — in this case your publish will be rejected. It could be easier to have the registry record the location because it always knows it at publish time and can verify it is right.

This will hopefully be a piece of data that the user won't manage directly, but would be handled by the package manager.

This would solve the issue with the user having to specify the location. But how would this work? Detect a Git repo and its upstream? What about non-Git locations? If this is straightforward then this would be quite nice.

[f-f]

it doesn’t feel like risky information to keep in the metadata. After all, we already do keep it there!

It's not really about the information itself being risky, more about keeping the Metadata as clutter-free as possible.

This would solve the issue with the user having to specify the location. But how would this work? Detect a Git repo and its upstream? What about non-Git locations? If this is straightforward then this would be quite nice.

In the case of Spago, I'm thinking that you'd have a location specified in your spago.yaml (I'd say that keeping this info there is pretty unavoidable, since this will also be the source of truth the the spago publish command), and when doing e.g. spago transfer ... Spago would keep the info in the config (and the manifest) up to date.

[thomashoneyman]

With this all in mind it does sound best to remove the location from the published versions metadata, only retaining the top-level location so that we know where to fetch the next version. To see the exact location for a particular version you must look at the manifest.

[thomashoneyman]

@f-f alas, we can't remove the ref field from metadata until we no longer produce legacy package sets. The package sets require the Git ref that the version points to. It's exceedingly rare that the Registry version matches the ref exactly.

With this in mind, we could:

1. Add the location to the published / unpublished metadata, OR
2. Plan to remove the ref field when we stop publishing legacy sets OR
3. Store the ref elsewhere (a file containing a map of package name -> version -> ref), and remove the ref field from metadata, and remove this other file when we stop publishing legacy sets

[f-f]

The easiest option seems (2), where we just keep things as is for the time being.

[thomashoneyman]

Closing, as we are not going to store the full location in metadata and the ref field has been omitted from the spec and is clearly marked in the source code as being slated for removal at some future date.