Consider providing uncompressed QEMU disk images

[invidian]

Right now Flatcar releases only offers flatcar_production_qemu_image.img.bz2 images for QEMU, which cannot be used directly with e.g. QEMU Packer builder, which possibly blocks some speed improvements to building Flatcar images (kubernetes-sigs/image-builder#924).

I'm aware that images are compressed likely to reduce the data sent over the wire, but I'm still curious if this is something which could be considered.

Alternatively, perhaps Packer QEMU builder could support decompressing the images directly perhaps (or just some generic image transformation).

[pothos]

Not sure if you can use it as in in packer - it seems you would loose the resize option (disk_size parameter) unless packer can grow the existing image. If this is not supported I recommend downloading, uncompressing and resizing the image in a Makefile step before packer is run (e.g. in https://github.com/kubernetes-sigs/image-builder/blob/master/images/capi/Makefile#L413).

[pothos]

Found it in the docs, resizing existing images is supported: disk_image (bool) - Packer defaults to building from an ISO file, this parameter controls whether the ISO URL supplied is actually a bootable QEMU image. When this value is set to true, the machine will either clone the source or use it as a backing file (if use_backing_file is true); then, it will resize the image according to disk_size and boot it.

[invidian]

Now that I think about it, served images could be compressed on the fly using e.g. gzip

[invidian]

According to my test switching from bzip to gzip won't affect number of data transferred much. It will of course affect CPU usage.

Below download, bunzip, then gzip and count bytes produced.

$ wget https://stable.release.flatcar-linux.net/amd64-usr/3033.2.4/flatcar_production_qemu_image.img.bz2 -O- | bunzip2 -d | gzip -c | wc -c | numfmt --to iec --format "%8.4f"
--2022-06-28 11:09:00--  https://stable.release.flatcar-linux.net/amd64-usr/3033.2.4/flatcar_production_qemu_image.img.bz2
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving stable.release.flatcar-linux.net (stable.release.flatcar-linux.net)... 147.75.81.135
Connecting to stable.release.flatcar-linux.net (stable.release.flatcar-linux.net)|147.75.81.135|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 357515394 (341M) [application/x-bzip2]
Saving to: ‘STDOUT’

-                                                           100%[===========================================================================================================================================>] 340.95M  6.59MB/s    in 53s

2022-06-28 11:09:54 (6.45 MB/s) - written to stdout [357515394/357515394]

357.4576M

[pothos]

I wouldn't block on this and add a step in the Makefile that downloads and unpacks (this is what packer would do anyway if it had support for compressed images).

[invidian]

BTW, it's not the first time I hit this limitation. E.g. with libvirt_volume in Terraform (https://registry.terraform.io/providers/multani/libvirt/latest/docs/resources/volume), it also supports network-based volumes, having uncompressed image available would make it much easier to use. Current situation requires exiting your domain (Terrafrom -> Makefile, Packer -> Makefile) to make it work.

It seem fixing it on Flatcar side would be most beneficial. And then bzipped images can stay as they are of course for backward compatibility.

[pothos]

What we have now as mechanism is COMPRESSION_FORMAT in the VM image generation. We can try to specify bz2,none for the qemu images to host them compressed and uncompressed.

[invidian]

What we have now as mechanism is COMPRESSION_FORMAT in the VM image generation. We can try to specify bz2,none for the qemu images to host them compressed and uncompressed.

That seem straightforward, cool.

However, we still need to make sure hosting we use supports gzip compression. Or perhaps we want to only enable gzip for uncompressed images to avoid double compression?

[pothos]

I'm not sure if we want to have on-the-fly compression, the qcow image is 850MB because it's sparse, so not that bad compared to a raw image.

[invidian]

Raw? The biggest image released right now is GKE image, which is 465 MiB from what I see. 850MB is almost double of that and the fact being a sparse image does not make a difference here I think, as all those zeroes will still be transferred I think. At least Python HTTP server does transfer all of them:

$ curl -s localhost:8000/flatcar_production_qemu_image.img | wc -c | numfmt --to iec --format "%8.4f"
883.4375M

With gzip, this can be avoided.

[pothos]

The raw (.bin) image is much larger uncompressed because it would be transfered non-sparse. The qcow2 format is internally already sparse, that's why it is only 850MB.

[sayanchowdhury]

I'm closing this issue. If you think otherwise, please free to re-open.

In my view, instead of directly implementing this in Flatcar, a better approach would be to integrate it upstream into the component responsible for downloads and decompression.

[pothos]

An idea could be to provide the .img qcow2 without wrapped compression by enabling the inline compression. This will reduce the file to half and as it's currently already only ~950 MB the result is ~410 MB which I think is ok as an additional artifact we could provide by default.
Tracked in: #1135

[pothos]

Reopened, focus is only on the .img qcow2 image as initially suggested, the .bin raw image should not be provided uncompressed, on-the-fly HTTP compression is also not considered.

[pothos]

The non-bz2 qcow2 .img will be part of the next round of releases.