-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider providing uncompressed QEMU disk images #791
Comments
Not sure if you can use it as in in packer - it seems you would loose the resize option ( |
Found it in the docs, resizing existing images is supported: disk_image (bool) - Packer defaults to building from an ISO file, this parameter controls whether the ISO URL supplied is actually a bootable QEMU image. When this value is set to true, the machine will either clone the source or use it as a backing file (if use_backing_file is true); then, it will resize the image according to disk_size and boot it. |
Now that I think about it, served images could be compressed on the fly using e.g. |
According to my test switching from bzip to gzip won't affect number of data transferred much. It will of course affect CPU usage. Below download, bunzip, then gzip and count bytes produced.
|
I wouldn't block on this and add a step in the Makefile that downloads and unpacks (this is what packer would do anyway if it had support for compressed images). |
BTW, it's not the first time I hit this limitation. E.g. with It seem fixing it on Flatcar side would be most beneficial. And then bzipped images can stay as they are of course for backward compatibility. |
What we have now as mechanism is |
That seem straightforward, cool. However, we still need to make sure hosting we use supports gzip compression. Or perhaps we want to only enable gzip for uncompressed images to avoid double compression? |
I'm not sure if we want to have on-the-fly compression, the qcow image is 850MB because it's sparse, so not that bad compared to a raw image. |
Raw? The biggest image released right now is GKE image, which is 465 MiB from what I see. 850MB is almost double of that and the fact being a sparse image does not make a difference here I think, as all those zeroes will still be transferred I think. At least Python HTTP server does transfer all of them: $ curl -s localhost:8000/flatcar_production_qemu_image.img | wc -c | numfmt --to iec --format "%8.4f"
883.4375M With gzip, this can be avoided. |
The raw (.bin) image is much larger uncompressed because it would be transfered non-sparse. The qcow2 format is internally already sparse, that's why it is only 850MB. |
I'm closing this issue. If you think otherwise, please free to re-open. In my view, instead of directly implementing this in Flatcar, a better approach would be to integrate it upstream into the component responsible for downloads and decompression. |
An idea could be to provide the .img qcow2 without wrapped compression by enabling the inline compression. This will reduce the file to half and as it's currently already only ~950 MB the result is ~410 MB which I think is ok as an additional artifact we could provide by default. |
Reopened, focus is only on the .img qcow2 image as initially suggested, the .bin raw image should not be provided uncompressed, on-the-fly HTTP compression is also not considered. |
The non-bz2 qcow2 .img will be part of the next round of releases. |
Right now Flatcar releases only offers
flatcar_production_qemu_image.img.bz2
images for QEMU, which cannot be used directly with e.g. QEMU Packer builder, which possibly blocks some speed improvements to building Flatcar images (kubernetes-sigs/image-builder#924).I'm aware that images are compressed likely to reduce the data sent over the wire, but I'm still curious if this is something which could be considered.
Alternatively, perhaps Packer QEMU builder could support decompressing the images directly perhaps (or just some generic image transformation).
The text was updated successfully, but these errors were encountered: