Differing behavior for aarch64 vs x86_64 disk images #855
Comments
|
I'm inclined to say no, but I'd like to hear what others think. If we care about matching both partition numbers and partition starting offsets across architectures, we'd need to waste 124 MB on ppc64le and 128 MB on s390x. Even if we only care about matching partition numbers, s390x will be a problem. coreos-installer has code to translate the GPT to a DASD VTOC on ECKD DASDs, and the VTOC only supports three Linux partitions. We'd need to make code changes to discard the unneeded partitions (leaving one unused partition slot for user data), and then we'd still have an inconsistency on that architecture. |
Indeed. Will try to bring it up in the meeting tomorrow!
IMO the <130 MB hit is relatively cheap.
Yeah it is quite a snowflake. As you mention it's going to continue to be a snowflake regardless of what we do. I don't think that should prevent us from trying across the other architectures. If we could ignore the goal on s390x (because of the limitations you mentioned) and get some uniformity for |
Maybe not in small VMs. It's also 130 MB of additional disk I/O during each install. |
|
Without doing a deep dive into this, my intuition is to try to handle this (~= make it easy for the user) in either Butane or Ignition and not try to go to great lengths to have identical disk layout. |
|
Backing up a bit: what problem are we trying to solve? What would a solution in Butane or Ignition look like? |
|
We discussed this in the community meeting today. We didn't have nearly enough time for a complete discussion and we'll bring it up again next week. For now it seems like the sentiments range. Lot's were discussed including:
Will continue discussion next week. |
I'm approaching this from an "uneducated user" perspective. A lot of times I fit into this category when trying new things in FCOS. So let's say the scenario is: I'm a user of FCOS on AWS for my data crunching operations at Of course, the answer is "no" and everything should be fine, but I really like the principal of least surprise when we can achieve it. Sure we don't and can't offer promises that Ignition configs will work across architectures, but if they did when they can it would be nice. Another scenario: My name is Dusty and I help manage and support users who run FCOS. It's easier for me to remember that "all partitions match for every architecture but s390x" than it is to remember the differences for each one. Documentation would help, but it sure wouldn't be as easy to remember.
Not sure. I'm not really proposing any changes to Butane or Ignition I don't think. Are you asking "if we were to solve this a different way, how would that look?" |
I think we can take that to coreos/ignition#1219 right? Then this ticket is for the question of whether we add an empty partition or some other hack or so. |
Would it be sufficient to solve that with storage documentation? The partition config will look different but that shouldn't actually affect anything. Ideally users will refer to partitions by label rather than number, and if they care about the partition number, they should probably explicitly specify one.
They won't completely match anyway. For example, ppc64le has an equivalent to
Sure, I'm not proposing a detailed discussion here, just a sketch of an idea. It's not clear to me what changes could be made to Ignition or Butane that would improve the situation described here. |
It's certainly better than not having the documentation :)
You're totally right.. but, I still think that when it pops up and the partition a user created is at index Let's say I make two partitions, though. Now the first one is at
Now I'll show you how much of a hypocrite I am :) - That particular difference I don't think matters that much so I think it's fine if the label and GUID don't match exactly.. Basically they are the same part index and size and have no bearing on the new partitions i'll create. I lump it into "boot stuff".
I guess one option could be to do away with That's an extreme, though. I still think we can achieve a similar goal (less surprises for the user) with the proposed partition number aligning (and optionally offset alignment). |
|
Suggested during the meeting to add an empty |
|
We discussed this in the community meeting today. Unfortunately we ran out of time (again), but I do think we've boiled it down into 2ish options and we just need to vote at this point:
Neither of these options is wrong. There are good arguments for and against. Either way we'll get better documentation! Option
Since we've discussed this several times already we'll just hold a vote in the next meeting and try to not eat up too much meeting time with it. |
|
We discussed this in the community meeting today. Now the action item is to update the docs with the architecture specific partition layouts. |
dustymabe
added
status/decided
status/pending-action
|
Another question: Should we keep some space at the beginning to allow non-EFI aarch64 to inject their own u-boot to get a UEFI bootloader? |
|
It turns out that the Butane templates for boot device mirroring don't include partition numbers, so we get something like this: {
"label": "bios-1",
"sizeMiB": 1,
"typeGuid": "21686148-6449-6E6F-744E-656564454649"
},
{
"label": "esp-1",
"sizeMiB": 127,
"typeGuid": "C12A7328-F81F-11D2-BA4B-00A0C93EC93B"
},
{
"label": "boot-1",
"sizeMiB": 384
},
{
"label": "root-1"
}Thus, with the We could change Butane to include partition numbers in the templates, but that would break compatibility with existing configs that override the root partition size because of coreos/ignition#1219. When performing config merging, Ignition matches partitions by label unless either the parent or child config specifies the partition number, in which case it matches by number instead. So that change would cause affected configs to create two root partitions per disk with identical labels. Assuming we want the same config to work on every architecture, we'd need to add partition numbers on x86_64 too, so that architecture would be affected. Sooo the conclusion in #855 (comment) might be worth revisiting. Adding empty partitions would be less user-visible than breaking compatibility with either a) configs that override the root partition size (by adding partition numbers in Butane) or b) that use duplicate labels (by fixing coreos/ignition#1219). It feels hacky though. |
|
I'm going to assume (reading from the commit message) you didn't mean to close this issue with that commit. |
On those architectures, the root partition is placed on partition 3 instead of partition 4 due to a Butane bug that's difficult to fix: coreos/fedora-coreos-tracker#855 (comment)
We decided to try to keep the partition numbers in sync across our
x86_64, aarch64, and ppc64le platforms. Previously we skipped partition
numbers on some architectures (for example there was no sda1 by default
on aarch64) and this caused a few issues:
- user confusion
- it is a common misconception that partition numbers need to start
at one and increment
- it can be surprising if the disk layout changes when you go from
one architecture from the other and re-use the same ignition config
- tooling bugs
- the butane templates for device boot mirroring didn't include
partition numbers and ended up with inconsistent behavior on
aarch64/ppc64le versus x86_64
Keeping the partition numbers in sync will help alleviate some of these
issues.
Fixes: coreos/fedora-coreos-tracker#855
We decided to try to keep the partition numbers in sync across our
x86_64, aarch64, and ppc64le platforms. Previously we skipped partition
numbers on some architectures (for example there was no sda1 by default
on aarch64) and this caused a few issues:
- user confusion
- it is a common misconception that partition numbers need to start
at one and increment
- it can be surprising if the disk layout changes when you go from
one architecture from the other and re-use the same ignition config
- tooling bugs
- the butane templates for device boot mirroring didn't include
partition numbers and ended up with inconsistent behavior on
aarch64/ppc64le versus x86_64
Keeping the partition numbers in sync will help alleviate some of these
issues.
Fixes: coreos/fedora-coreos-tracker#855
|
Reopening pending Butane change, Butane release, and test re-enablement. |
|
Butane changes in coreos/butane#278. |
dustymabe
added
status/pending-stable-release
|
Reopening pending Butane release, and test re-enablement. |
For completeness I wanted to post this here. @bgilbert and I had a chat a while back with pwhalen to see if there is anything we could or should do now with regards to arm/uboot. The full log of the discussion is reproduced below, but the outcome is/was: |
|
The The fix for this went into stable stream release |
dustymabe
removed
the
status/pending-stable-release
|
Butane 0.14.0 includes the updated |
As of coreos/butane#278 in Butane 0.14.0, Butane RAID templates now create reserved partitions on aarch64 and ppc64le to keep partition numbers aligned across platforms. Fixes coreos/fedora-coreos-tracker#855. This reverts commit 46af3f4.
On
aarch64we don't have a 1st partition.The goal here was to keep the partition numbers the same but omitting something taking up the first partition number means we get differing behavior if you configure disks using Ignition when you compare
x86_64andaarch64.For example, the butane config from an ignition bug I just filed gives us this on x86_64:
and this on aarch64:
For consistency, should we just leave a sda1 on the aarch64 disk of the same size as the x86_64 disk? It looks like it's only
1Min size.The text was updated successfully, but these errors were encountered: