Support redirects with iso_url & iso_checksum

[elreydetoda]

Overview of the Issue

While I'm experiencing other issues (bento & packer data source which aren't addressable by you I'm pretty sure), I also noticed while looking at the debug logs (PACKER_LOG=1) for my build that the QEMU build doesn't appear to follow redirects.

While I've used http urls for ISO checksums before, I attempted to use a URL that redirects to the checksum and ISO but it appeared to fail on the initial getting of the file as you can see in the image below:

I technically haven't tested if the iso_url works with a redirect, but I'm imagining it doesn't since iso_checksum doesn't.

Here are the links I am using:

file:https://mirrors.centos.org/mirrorlist?path=/9-stream/BaseOS/x86_64/iso/SHA256SUM&redirect=1&protocol=https
file:https://mirrors.centos.org/mirrorlist?path=/9-stream/BaseOS/x86_64/iso/CentOS-Stream-9-latest-x86_64-dvd1.iso&redirect=1&protocol=https

Reproduction Steps

packer build centos-stream-9-x86_64.json - using the linked to packer build file

Plugin and Packer version

Packer v1.7.9

Simplified Packer Buildfile

https://github.com/elreydetoda/bento/blob/da1b122d89dc045c1020165e7f264c0935f7d571/packer_templates/centos/centos-stream-9-x86_64.json

Operating system and Environment details

OS: Ubuntu 21.10
arch: amd64

Log Fragments and crash.log files

packer.log

[elreydetoda]

@hashicorp it looks like from the Code Owners file that there is a @hashicorp/packer team, but I don't know who I should reach out to about making sure someone at least saw this issue (not necessarily do anything about it, but just want to make sure that @hashicorp/packer isn't a dead team tag or something ). Even if it's an 👀 emoji on the post or something.

[kienanstewart]

I ran into this issue today.

OS: Debian unstable
Packer version: 1.6.6

==> qemu: Retrieving ISO                                                                                                                                                                                             
==> qemu: Trying https://cdimage.debian.org/cdimage/weekly-builds//amd64/iso-cd//debian-testing-amd64-netinst.iso                                                                                                    ==> qemu: Trying https://cdimage.debian.org/cdimage/weekly-builds//amd64/iso-cd//debian-testing-amd64-netinst.iso?checksum=sha512%3A61beb93622a597fe232c7ba785a884c8320d5f188231b2e9ae1fda5c8e8d72b8ea615467a0cb652f9
26be662a26c3ec55e137234530548c3c24c85e4bac217a8                                                                                                                                                                          qemu: weekly-builds 6.05 KiB / 6.05 KiB [======================================================================================================================================================] 100.00% 0s      
==> qemu: Checksum did not match, removing packer_cache/2f53eec7889d269209b51bd37d96e772a4951c1c.iso
==> qemu: error downloading ISO: [Checksums did not match for /tmp/getter2562255610/temp.  

To work around the issue I replaced the mirror with one found using curl -I.

Eg.

$ curl -I https://cdimage.debian.org/cdimage/weekly-builds//amd64/iso-cd//debian-testing-amd64-netinst.iso
HTTP/1.1 302 Found
Date: Sun, 11 Sep 2022 14:46:17 GMT
Server: Apache/2.4.54 (Unix)
Location: https://chuangtzu.ftp.acc.umu.se/cdimage/weekly-builds/amd64/iso-cd/debian-testing-amd64-netinst.iso
Cache-Control: max-age=300Expires: Sun, 11 Sep 2022 14:51:17 GMT
Content-Type: text/html; charset=iso-8859-1

[elreydetoda]

To work around the issue I replaced the mirror with one found using curl -I.

Ya, I have done that for my personal setup, and thank you for the recommendation 😊, but it's not a "solution".

Mainly because the mirror you typically get redirected to relative to your geographic location. So, if I plug in the address from that and I'm in the US then someone in China is going to try and reach out to a US mirror. It's going to be extremely slow, compared to if packer handled this it'll automatically choose the closest mirror.

[lbajolet-hashicorp]

Hey @elreydetoda, sorry for taking this long to address this, I'm taking a look at this behaviour.

From the looks of it, it seems the problem has its roots in the SDK (even in the go-getter project which is a direct dependency of the SDK), which leads me to wonder: have you tried this redirection for the checksum with other builders? If so, so you experience the same problem?

If my intuition is correct, the problem should be widespread. Fortunately we do maintain the whole chain here, so it's not unaddressable, but I want to make sure I'm looking at the right place, can you confirm?

I'll try to replicate it locally and test this with multiple builders in the meantime, I'll submit some patches if I can get it to work.

Thanks for signalling this problem, I hope we can get it fixed soon.