Make ext4 rootfs great again #3830
Conversation
rouming
requested review from
deitch,
zedi-pramodh and
eriknordmark
as code owners
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #3830 +/- ##
=======================================
Coverage 17.51% 17.51%
=======================================
Files 3 3
Lines 805 805
=======================================
Hits 141 141
Misses 629 629
Partials 35 35 ☔ View full report in Codecov by Sentry. |
rouming
changed the title
rouming
force-pushed
the
rootfs-as-ext4
branch
5 times, most recently
from
8297582 to
7d9372e
Compare
There was a problem hiding this comment.
I dislike the near-duplication of make-raw in both mkimage-raw-efi and mkverification-raw-efi, but that wasn't introduced here, so not your job to fix here. :-)
In any case, as long as this all works well and is tested, looks good for what it does. I like that it calculates size dynamically.
How does that work? |
Answering my own question, the partition size (at least in the live image) is the size of the image. That means it can't be used to update to a slightly larger image. Can't we fix that to use the same fixed size for the partitions? |
A few minor things done here: 1. /mnt folder should be created, otherwise mount fails. 2. Proper round up on 1kb. 3. Remove unused variable ROOTFS_PART_SECTORS Signed-off-by: Roman Penyaev <r.peniaev@gmail.com>
This is very good point, I forgot about upgrade. Actually without borrowing free space from the /persist we are screwed by default to upgrade to image greater than these hardcoded values. So in the future if image grows this has to be fixed by doing dangerous operations on the /persist. Now I can fix that by returning old values and taking max.
This was fixed just before your comment, here https://github.com/lf-edge/eve/compare/8ab4fde0385564de8f57a560a255a6d8bed92179..21b71376ddaa6789ee9303e52b63c737743f2c9d |
Easy to show that long alignment equation can be simplified to
a shorter 1MB alignment:
x + 2^10 - 1
------------ + 2^10 - 1 x + 2^20 - 1
2^10 = ------------
------------------------ 2^20
2^10
Take the left side and do a simplification
x + 2^10 - 1 2^10 - 1
------------ + --------
2^20 2^10
x + 2^10 - 1 + 2^10(2^10 - 1)
= -----------------------------
2^20
x + 2^10 - 1 + 2^20 - 2^10
= --------------------------
2^20
x + 2^20 - 1
= ------------
2^20
Signed-off-by: Roman Penyaev <r.peniaev@gmail.com>
! Patch covers two almost identical 'make-raw' files. The purpose of the change is to access those variables earlier from functions, which will appear in following patches. Signed-off-by: Roman Penyaev <r.peniaev@gmail.com>
! Patch covers two almost identical 'make-raw' files. Check for IMX8 files repeats several times. Introduce a separate 'imx8_exists()' function for this purpose. Note! The 'make-raw' in the verification package is a bit outdated, so IMX8 blobs are copied to the /efifs folder. Originally this was introduced in the following commit: b8f73a8 ("arm: Add support to install EVE image on eMMC for i.MX8MP devices") but was not ported to the verification 'make-raw'. This patch does that. Signed-off-by: Roman Penyaev <r.peniaev@gmail.com>
! Patch covers two almost identical 'make-raw' files.
The 'make-raw' script expects the resulting image being created by the
caller. The main problem of this approach is that the caller should
create an image in advance without knowing exactly the size of
partitions, which will be created lately. This causes a corruption of
the image with the ext4 rootfs, which is bigger than usual squashfs
and does not fit ~500Mb.
This patch fixes the problem by calculating resulting image size
by iterating over each requested partitions, e.g. 'imga' or 'imgb'
partitions are created by requesting size of the actual rootfs binary
image.
Having this calculations in place a caller does not need to predict
the resulting image size, but simply pass the '-C' argument, which
means the image file will be created according to the size of the
partitions.
Keeping the upgrade logic in mind we can't unfortunately change the
squashfs (default) rootfs partition size, so introduce minimum rootfs
size, which is equal to the default size of 512Mb.
We always use the maximum:
max(512MB, file_size $ROOTFS_IMG)
so that default squashfs size won't be change, but ext4 will grow
(and yes, unfortunately you can't upgrade from squashfs to ext4,
because ext4 partition is a way bigger)
Patch makes creation of the ext4 rootfs possible and also has a
size reduction for the installer.raw image:
before after
installer.raw 592Mb 296Mb
Signed-off-by: Roman Penyaev <r.peniaev@gmail.com>
…tion This patch removes creation of the image in the 'makeflash.sh' by passing '-C' option to the 'mkimage-raw-efi' and 'mkverification-raw-efi' in order to use dynamic image calculation. Signed-off-by: Roman Penyaev <r.peniaev@gmail.com>
In order to generate live/verification-raw/installer-raw images with the ext4 as rootfs call as the following: make ROOTFS_FORMAT=ext4 live (replace 'live' with 'verification-raw' or 'installer-raw') Also disable warning for ext4 if image size exceeds 250Mb. Signed-off-by: Roman Penyaev <r.peniaev@gmail.com>
Reflect '-C' option changes: now image size is calculted dynamically, so in order to ask mkimage-raw-efi to create an image pass '-C' only. Signed-off-by: Roman Penyaev <r.peniaev@gmail.com>
|
@eriknordmark @jsfakian Also the rootfs partition size is kept as is, i.e. 512Mb, so upgrade won't fail. This is for the imga, imgb partitions only, the installer/verificator keeps rootfs as a file on fat32, so we don't keep this max value. |
@deitch here #3842. Can we continue improving? Still a lot of room for improvements, but help of the linuxkit is required. |
|
This needs to be approved and merged before #3842 |
So this is ready to go?
For this PR? Or for next steps you want in #3842? Or for something else beyond that? |
This is ready to go. There was some important concern from @eriknordmark , I fixed that. If he does not have any objections or anyone else, this can be merged. Also would be great if @jsfakian can help me with verification image. Last time he spotted one thing, which breaks the image. This is also fixed.
The #3842 goes on top of the ext4 (this PR). Nothing more. The #3842 requires more eyes and testing and doc. |
|
Yeah, I see @eriknordmark still has changes pending. Let's let him sign off and merge it in. Hats off @rouming |
This set of patches fixes the ext4 rootfs generation. Main problem was in the image size difference between squashfs image and ext4 which leads to an incorrect GPT table. The solution is to calculate resulting image dynamically.
Patch makes creation of the ext4 rootfs possible and also has a size reduction for the installer.raw images:
Unfortunately for the rootfs we have a hardcoded partition size equal to 512Mb, which can't be changed, otherwise upgrade logic fail when we try to fit bigger rootfs into smaller one partition. This means that upgrade from squashfs image to ext4 is currently impossible. But who cares.
Now live image can be generated as following:
My motivation to generate ext4 rootfs is to have possibility to persist changes on eve while running.
Another great feature becomes possible is to "patch" resulting qemu image with changed files and not generate rootfs from scratch, which can take minutes of CPU time.
This patch changes the pkg/mkverification-raw-efi/make-raw, which almost a copy of the original one. I kindly ask @jsfakian to help me to review and to test the resulting verification images, that those work and nothing is broken because of my changes.
cc: @mikem-zed