[FLORA-223] Progressive index import

[RaoulHC]

Proposed changes

This adds an option to import directly from index tarballs rather than needing to unpack them, and then uses the timestamp to avoid re importing anything that is older and should have already been importing. It might be that in the future we want to use the timestamp to grab only that latest part of the index tarball, and avoid checking each cabal file for its timestamp, but we'll still want to be storing the timestamp.

Needs some tests and I'm unsure of the metadata table, but could do with some review in its current form.

I also have a commit to update the freeze file which was out of sync with horizon, but this may be superseded by #386.

Contributor checklist

• My PR is related to <insert ticket number>
• I have read and understood the CONTRIBUTING guide
• I have inserted my change and a link to this PR in the CHANGELOG

[tchoutri]

@Mergifyio rebase

[mergify]

rebase

❌ Base branch update has failed

Git reported the following error:

Rebasing (1/5)
Auto-merging cabal.project.freeze
CONFLICT (content): Merge conflict in cabal.project.freeze
error: could not apply 149ec95... Update freeze file
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 149ec95... Update freeze file

err-code: 8A708

[tchoutri]

Let's have instead a table package_indexes with the following:

• package_index uuid primary key;
• repository text not null;
• timestamp timestamptz not null;

And an index on the repository field.

:-)

[RaoulHC]

Does having a repository make sense quite yet? At some point we'll want to point to a repo and then will make sense, but right now we're importing from a directory or an index file, which doesn't have a repository attached to it. Maybe it makes the most sense to default to hackage.haskell.org and perhaps have an argument to the mirror-cli for specifying the repository?

[tchoutri]

Good question, at the moment the only repo is Hackage so you can default to it (in the Haskell code, not the SQL).
A later workflow for CHaP will have to make the distinction between the two, but we're in the clear for now :)

[tchoutri]

@RaoulHC do you want to mark this PR as ready for review? Or are there other stuff you want done before the merge?

[RaoulHC]

@RaoulHC do you want to mark this PR as ready for review? Or are there other stuff you want done before the merge?

It could do with some stuff in the test suite, and maybe it wants a flag when importing for whether to update the timestamp table or not, but I think the main logic is good for review. Will try and finish off those bits this week.

[RaoulHC]

Right think this is all good for review, I've added an extra field to releases so we can check the provenance of where packages came from which helps check what we want to check up to. Will update the changelog if you're happy with this approach.

[tchoutri]

Ok overall I'm very satisfied with this PR. Only one point to resolve. :)

[tchoutri]

I would have thought a foreign key to the package_index_id would be the most appropriate

[RaoulHC]

Updated. This does mean we have to add the package index without a timestamp before we start importing releases else you get foreign key violations. We could take the approach of adding/updating the timestamp as we import each release but I think this is probably better.

[tchoutri]

Indeed it's not stupid to have package indexes that we could know about before having importing them, especially if we want to provision the database with public stuff like hackage & chap (and others in the future).

[tchoutri]

LGTM. Thank you very much, it's highly appreciated and extremely valuable.