Sometimes package files are missing in automatic update PR

[fingolfin]

This happened at least twice, see

• [liepring] Update to 2.6 #310 (comment)
• [hap] Update to 1.39 #314 (comment)

[fingolfin]

Looking at the logs for PR #314, in the download phase of the "Build GAP and packages"

64 _archives/hap-1.39.tar.gz already exists, not downloading again

Then later:

132 Extracting /home/runner/work/PackageDistro/PackageDistro/_archives/hap-1.39.tar.gz

But then looking at the "Build package" step, we see no mention of hap; it should be right next to the part where hapcryst is being "built". But it isn't. I can only guess that perhaps the hap archive was damaged and thus not properly extracted?!?!? Perhaps adding some more debug output to the parts where the archives are extracted would help solve this mystery.

Also in the various jobs for the individual packages, when looking at the "Extract GAP artifact" step (the raw logs in this case are a bit easier to search for me), it is clear that no pkg/hap-1.3.9/ directory is created.

[fingolfin]

Urgh, in the case of hap the problem is upstream: https://github.com/gap-packages/hap/releases/download/v1.39/hap-1.39.tar.gz is missing and in fact there is no release there at all.

So the download produce a 404 error:

$ wget https://github.com/gap-packages/hap/releases/download/v1.39/hap-1.39.tar.gz
--2022-04-22 01:00:15--  https://github.com/gap-packages/hap/releases/download/v1.39/hap-1.39.tar.gz
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-04-22 01:00:15 ERROR 404: Not Found.

So the real bug in this case then is why we don't detect the broken URL and go on, instead of erroring out...

[fingolfin]

Also submitted gap-packages/hap#67

[fingolfin]

Indeed we ignored the HTTP status code. Should be fixed in 5495746

[fingolfin]

OK, I missed that tools/scan_for_updates.py actually uses download() from utils.py; so that also needs to be adjusted to check the HTTP status code...

... but then how best to deal with it? We don't want to just abort scan_for_updates.py here, as it would be unfair if one broken package affected blocked updates for other non-broken packages.

So we could just skip over updates with a 404 error... but then how do we learn about such broken updates?? It'll be quite easy to miss

Another option would be to open a PR in this case anyway, with a fake ArchiveSHA256 value, and a clear message somewhere that the update actually could not be downloaded (and don't bother running test-all.yml in this case either

[fingolfin]

Actually, what really should have happened is that tools/validate_package.py should have caught this, too: after all it is supposed to inspect the downloaded tarball and among other things verify that it is a valid tarball containing a PackageInfo.g file....

And then I noticed the (in hindsight) obvious bug 🤦 which will be fixed by PR #322