Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cgroups v2 strategy #292

Closed
bgilbert opened this issue Oct 8, 2019 · 68 comments
Closed

cgroups v2 strategy #292

bgilbert opened this issue Oct 8, 2019 · 68 comments
Labels
area/usability kind/design

Comments

@bgilbert
Copy link
Contributor

bgilbert commented Oct 8, 2019

Fedora 31 is planning to switch to cgroups v2 by default. The explicit goal is for Fedora to drive the leading edge, encouraging components that don't fully support v2 to do so. However, this may not yet be the best choice for Fedora CoreOS, which aims to provide broad and reliable container support for production environments.

We have the ability to make a different decision here than the rest of Fedora. However, if we ship the stable release with cgroups v1, we'll need to think about compatibility constraints when/if we later switch to v2.

We should define a strategy for the cgroups v2 transition.
@ajeddeloh ajeddeloh removed the meeting topics for meetings label Oct 9, 2019
@jlebon jlebon mentioned this issue Oct 18, 2019
Merged
@bgilbert
Copy link
Contributor Author

As a data point, Docker in F31 doesn't run on cgroups v2 and the problem has been rejected as a Fedora release blocker.

jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Oct 18, 2019
@jlebon
image.yaml: add systemd.unified_cgroup_hierarchy=0 karg
eab078c 
FCOS is likely to ship with CGroupsV1 to start, since most of the
containerization ecosystem isn't yet ready for V2.

Pass the `systemd.unified_cgroup_hierarchy=0` karg to enable this. This
should at least allow podman Kola tests to pass.

For more information, see:
coreos/fedora-coreos-tracker#292
@jlebon
Copy link
Member

jlebon commented Oct 18, 2019

@rhatdan We discussed this during the last community meeting and agreed that shipping with v1 for now is the safer course of action. One question we had was: is there anything we can do now to make migration to v2 easier later?

@rhatdan
Copy link

rhatdan commented Oct 18, 2019

Switching to crun by default, would help. But I guess you should shift on Fedora 32. Since we have made the change, we are witnessing a lot of movement in runc and k8s towards cgroup v2.

jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Oct 21, 2019
@jlebon
image.yaml: add systemd.unified_cgroup_hierarchy=0 karg
200b439 
FCOS is likely to ship with CGroupsV1 to start, since most of the
containerization ecosystem isn't yet ready for V2.

Pass the `systemd.unified_cgroup_hierarchy=0` karg to enable this. This
should at least allow podman Kola tests to pass.

For more information, see:
coreos/fedora-coreos-tracker#292
jlebon added a commit to coreos/fedora-coreos-config that referenced this issue Oct 21, 2019
@jlebon
image.yaml: add systemd.unified_cgroup_hierarchy=0 karg
73cac9f 
FCOS is likely to ship with CGroupsV1 to start, since most of the
containerization ecosystem isn't yet ready for V2.

Pass the `systemd.unified_cgroup_hierarchy=0` karg to enable this. This
should at least allow podman Kola tests to pass.

For more information, see:
coreos/fedora-coreos-tracker#292
@dustymabe dustymabe added the meeting topics for meetings label Nov 20, 2019
@dustymabe
Copy link
Member

dustymabe commented Nov 25, 2019 •
edited
Loading

We found an issue with the current approach specifically when it comes to handling the F30->F31 update. We'll proceed with trying the workaround described in that issue comment.

@dustymabe dustymabe removed the meeting topics for meetings label Nov 25, 2019
dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue Nov 25, 2019
@dustymabe
Revert recent promotion to testing branch
3c259fc 
We found an issue [1] with our cgroups v2 strategy [2]. We need to
revert the recent promotions (that move us to F31) and do a new release
of F30 content with a slight modification. See the proposed solution
in the issue comment.

- Revert "tree: promote changes from testing-devel at 20e1222"
    - This reverts commit 0d8e188.
- Revert "manifest.yaml: bump to Fedora 31"
    - This reverts commit c11e08f.
- Revert "manifest.yaml: adapt for new path to fedora-coreos.yaml"
    - This reverts commit 01850ff.

[1] coreos/fedora-coreos-streams#26 (comment)
[2] coreos/fedora-coreos-tracker#292
@dustymabe dustymabe mentioned this issue Nov 25, 2019
Merged
jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Nov 26, 2019
@jlebon
Add coreos-keep-cgroup-v1.service
64d41e7 
In f31, the default cgroup changed to v2. However, we've decided to stay
on v1 for the time being. Thus, we don't want older nodes upgrading to
f31 to be forced into v2.

Add a tiny service which just scans the BLS configs and injects the
`systemd.unified_cgroup_hierarchy` karg as needed.

For more information, see:
coreos/fedora-coreos-tracker#292
coreos/fedora-coreos-streams#26 (comment)
@jlebon jlebon mentioned this issue Nov 26, 2019
Merged
jlebon pushed a commit to coreos/fedora-coreos-config that referenced this issue Nov 26, 2019
@dustymabe @jlebon
Revert recent promotion to testing branch
6bd4b0d 
We found an issue [1] with our cgroups v2 strategy [2]. We need to
revert the recent promotions (that move us to F31) and do a new release
of F30 content with a slight modification. See the proposed solution
in the issue comment.

- Revert "tree: promote changes from testing-devel at 20e1222"
    - This reverts commit 0d8e188.
- Revert "manifest.yaml: bump to Fedora 31"
    - This reverts commit c11e08f.
- Revert "manifest.yaml: adapt for new path to fedora-coreos.yaml"
    - This reverts commit 01850ff.

[1] coreos/fedora-coreos-streams#26 (comment)
[2] coreos/fedora-coreos-tracker#292
jlebon added a commit to jlebon/fedora-coreos-config that referenced this issue Nov 26, 2019
@jlebon
Add coreos-keep-cgroup-v1.service
235b0f2 
In f31, the default cgroup changed to v2. However, we've decided to stay
on v1 for the time being. Thus, we don't want older nodes upgrading to
f31 to be forced into v2.

Add a tiny service which just scans the BLS configs and injects the
`systemd.unified_cgroup_hierarchy` karg as needed.

For more information, see:
coreos/fedora-coreos-tracker#292
coreos/fedora-coreos-streams#26 (comment)
jlebon added a commit to coreos/fedora-coreos-config that referenced this issue Nov 26, 2019
@jlebon
Add coreos-keep-cgroup-v1.service
b2bd629 
In f31, the default cgroup changed to v2. However, we've decided to stay
on v1 for the time being. Thus, we don't want older nodes upgrading to
f31 to be forced into v2.

Add a tiny service which just scans the BLS configs and injects the
`systemd.unified_cgroup_hierarchy` karg as needed.

For more information, see:
coreos/fedora-coreos-tracker#292
coreos/fedora-coreos-streams#26 (comment)
@dustymabe dustymabe mentioned this issue Dec 11, 2019
Closed
@dustymabe
Copy link
Member

In the meeting from a few weeks ago we said:

this ticket is mostly done but we'll try to create docs and use fcct kargs support to show how to set it to v1 or v2

I opened coreos/butane#57 for the remaining items.

@lucab
Copy link
Contributor

lucab commented Dec 16, 2019

Before stepping into docs and fcct, it is unclear to me how we plan to tackle kargs configuration at the Ignition/initramfs level.

Right now we can arrange things (via Ignition) so that the BLS is changed after the first boot, but that requires a reboot to apply (thus is doesn't affect the first boot).
Did I miss anything?

@jlebon
Copy link
Member

jlebon commented Dec 16, 2019

Before stepping into docs and fcct, it is unclear to me how we plan to tackle kargs configuration at the Ignition/initramfs level.

Yup, that's https://github.com/coreos/ignition-dracut/issues/81. The status on that is essentially:

  1. fix Support default kernel arguments ostreedev/ostree#479 (Support kargs.d directories for default kargs ostreedev/ostree#1836)
  2. get ostree working in the initrd
  3. add ignition-dracut service that detects kargs.d dropins

@jtligon
Copy link

jtligon commented Aug 20, 2020

with containerd/containerd#3726 merging, do we need to look at this again?

@dustymabe
Copy link
Member

Yeah. It needs to land in upstream releases and that needs to hit Fedora so we can pick it up, not sure if it will happen for fedora 33.

Anybody know what the kubernetes cgroups v2 status is now?

@giuseppe
Copy link

the plan is to have cgroup v2 support in Kubernetes 1.19

@miabbott
Copy link
Member

Per a message[1] to the Fedora CoreOS mailing list from @Conan-Kudo, Docker/Moby 20.10 now has cgroups v2 support.

[1] https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/MIZ3FTYH46AW3TRTZ44S5CTW2CNIRLZ4/

@LorbusChris
Copy link
Contributor

@olivierlemasle seems to be Fedora RPM maintainer.

Hi Olivier,
could you give us an ETA for the new Moby release landing in Fedora? 😇

coreosbot pushed a commit to coreos/fedora-coreos-config that referenced this issue Apr 16, 2021
@jlebon
Move next-devel to cgroupsv2 for new nodes
5d24c00 
All we need to do is to make the associated karg be specific to
`testing-devel` by uplisting it to `image.yaml`. Then `next-devel` will
naturally shed it when `image-base.yaml` gets synced over.

See: coreos/fedora-coreos-tracker#292
dghubble added a commit to poseidon/typhoon that referenced this issue Apr 17, 2021
@dghubble
Update Fedora CoreOS Kubelet for cgroups v2
d0ff1cd 
* Fedora CoreOS is beginning to switch from cgroups v1 to
cgroups v2 by default, which changes the sysfs hierarchy
* This will be needed when using a Fedora Coreos OS image
that enables cgroups v2 (`next` stream as of this writing)

Rel: coreos/fedora-coreos-tracker#292
@dghubble dghubble mentioned this issue Apr 17, 2021
Merged
dghubble added a commit to poseidon/typhoon that referenced this issue Apr 26, 2021
@dghubble
Update Fedora CoreOS Kubelet for cgroups v2
cb355c4 
* Fedora CoreOS is beginning to switch from cgroups v1 to
cgroups v2 by default, which changes the sysfs hierarchy
* This will be needed when using a Fedora Coreos OS image
that enables cgroups v2 (`next` stream as of this writing)

Rel: coreos/fedora-coreos-tracker#292
dghubble added a commit to poseidon/typhoon that referenced this issue Apr 26, 2021
@dghubble
Update Fedora CoreOS Kubelet for cgroups v2
5f87eb3 
* Fedora CoreOS is beginning to switch from cgroups v1 to
cgroups v2 by default, which changes the sysfs hierarchy
* This will be needed when using a Fedora Coreos OS image
that enables cgroups v2 (`next` stream as of this writing)

Rel: coreos/fedora-coreos-tracker#292
dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue May 25, 2021
@dustymabe
move to cgroups v2 everywhere
11a3801 
Adjust the kernel arguments so that we're now using cgroups v2
in our testing-devel (and subsequently, testing and stable) stream(s).

Context: coreos/fedora-coreos-tracker#292
@dustymabe dustymabe mentioned this issue May 25, 2021
Merged
@dustymabe
Copy link
Member

Looks like everything is in place and we've waited a sufficient amount of time since the email was sent out. Moving forward: coreos/fedora-coreos-config#1033

dustymabe added a commit to coreos/fedora-coreos-config that referenced this issue May 25, 2021
@dustymabe
move to cgroups v2 everywhere
aa16ec3 
Adjust the kernel arguments so that we're now using cgroups v2
in our testing-devel (and subsequently, testing and stable) stream(s).

Context: coreos/fedora-coreos-tracker#292
@jlebon
Copy link
Member

jlebon commented May 26, 2021

It's not very realistic, but as a sanity-check, I created a simple podman sleep container on cgroups v1, rebooted into cgroups v2, and I was able to restart the container just fine.

@dustymabe dustymabe added meeting topics for meetings and removed meeting topics for meetings labels May 26, 2021
@dustymabe
Copy link
Member

The fix for this went into testing stream release 34.20210529.2.0. Please try out the new release and report issues.

@dustymabe dustymabe added the status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. label Jun 2, 2021
khuedoan added a commit to khuedoan/homelab that referenced this issue Jun 15, 2021
@khuedoan
Use CoreOS testing stream for cgroups v2
cf74317 
coreos/fedora-coreos-tracker#292
coreos/fedora-coreos-config#1033
@dustymabe
Copy link
Member

The fix for this went into stable stream release 34.20210529.3.0.

@dustymabe dustymabe removed the status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. label Jun 23, 2021
@bgilbert bgilbert mentioned this issue Jul 7, 2021
Open
dghubble added a commit to poseidon/typhoon that referenced this issue Jul 24, 2021
@dghubble
Update Cilium for cgroups v2 support
09fcdc4 
* On Fedora CoreOS, Cilium cross-node service IP load balancing
stopped working for a time (first observable as CoreDNS pods
located on worker nodes not being able to reach the kubernetes
API service 10.3.0.1). This turned out to have two parts:
* Fedora CoreOS switched to cgroups v2 by default. In our early
testing with cgroups v2, Calico (default) was used. With the
cgroups v2 change, SELinux policy denied some eBPF operations.
Since fixed in all Fedora CoreOS channels
* Cilium requires new mounts to support cgroups v2, which are
added here

* coreos/fedora-coreos-tracker#292
* coreos/fedora-coreos-tracker#881
* cilium/cilium#16259
@dghubble dghubble mentioned this issue Jul 24, 2021
Merged
dghubble added a commit to poseidon/typhoon that referenced this issue Jul 24, 2021
@dghubble
Update Cilium for cgroups v2 support
f03045f 
* On Fedora CoreOS, Cilium cross-node service IP load balancing
stopped working for a time (first observable as CoreDNS pods
located on worker nodes not being able to reach the kubernetes
API service 10.3.0.1). This turned out to have two parts:
* Fedora CoreOS switched to cgroups v2 by default. In our early
testing with cgroups v2, Calico (default) was used. With the
cgroups v2 change, SELinux policy denied some eBPF operations.
Since fixed in all Fedora CoreOS channels
* Cilium requires new mounts to support cgroups v2, which are
added here

* coreos/fedora-coreos-tracker#292
* coreos/fedora-coreos-tracker#881
* cilium/cilium#16259
foltik pushed a commit to foltik/typhoon that referenced this issue Sep 1, 2021
@dghubble @foltik
Update Cilium for cgroups v2 support
c14628a 
* On Fedora CoreOS, Cilium cross-node service IP load balancing
stopped working for a time (first observable as CoreDNS pods
located on worker nodes not being able to reach the kubernetes
API service 10.3.0.1). This turned out to have two parts:
* Fedora CoreOS switched to cgroups v2 by default. In our early
testing with cgroups v2, Calico (default) was used. With the
cgroups v2 change, SELinux policy denied some eBPF operations.
Since fixed in all Fedora CoreOS channels
* Cilium requires new mounts to support cgroups v2, which are
added here

* coreos/fedora-coreos-tracker#292
* coreos/fedora-coreos-tracker#881
* cilium/cilium#16259
elemental-lf pushed a commit to elemental-lf/typhoon that referenced this issue Dec 11, 2021
@dghubble @elemental-lf
Update Cilium for cgroups v2 support
8ce2964 
* On Fedora CoreOS, Cilium cross-node service IP load balancing
stopped working for a time (first observable as CoreDNS pods
located on worker nodes not being able to reach the kubernetes
API service 10.3.0.1). This turned out to have two parts:
* Fedora CoreOS switched to cgroups v2 by default. In our early
testing with cgroups v2, Calico (default) was used. With the
cgroups v2 change, SELinux policy denied some eBPF operations.
Since fixed in all Fedora CoreOS channels
* Cilium requires new mounts to support cgroups v2, which are
added here

* coreos/fedora-coreos-tracker#292
* coreos/fedora-coreos-tracker#881
* cilium/cilium#16259
Snaipe pushed a commit to aristanetworks/monsoon that referenced this issue Apr 13, 2023
@dghubble
Update Fedora CoreOS Kubelet for cgroups v2
13a12fd 
* Fedora CoreOS is beginning to switch from cgroups v1 to
cgroups v2 by default, which changes the sysfs hierarchy
* This will be needed when using a Fedora Coreos OS image
that enables cgroups v2 (`next` stream as of this writing)

Rel: coreos/fedora-coreos-tracker#292
Snaipe pushed a commit to aristanetworks/monsoon that referenced this issue Apr 13, 2023
@dghubble
Update Cilium for cgroups v2 support
64d5c70 
* On Fedora CoreOS, Cilium cross-node service IP load balancing
stopped working for a time (first observable as CoreDNS pods
located on worker nodes not being able to reach the kubernetes
API service 10.3.0.1). This turned out to have two parts:
* Fedora CoreOS switched to cgroups v2 by default. In our early
testing with cgroups v2, Calico (default) was used. With the
cgroups v2 change, SELinux policy denied some eBPF operations.
Since fixed in all Fedora CoreOS channels
* Cilium requires new mounts to support cgroups v2, which are
added here

* coreos/fedora-coreos-tracker#292
* coreos/fedora-coreos-tracker#881
* cilium/cilium#16259
HuijingHei pushed a commit to HuijingHei/fedora-coreos-config that referenced this issue Oct 10, 2023
@jlebon @HuijingHei
Move next-devel to cgroupsv2 for new nodes
61cc5f2 
All we need to do is to make the associated karg be specific to
`testing-devel` by uplisting it to `image.yaml`. Then `next-devel` will
naturally shed it when `image-base.yaml` gets synced over.

See: coreos/fedora-coreos-tracker#292
HuijingHei pushed a commit to HuijingHei/fedora-coreos-config that referenced this issue Oct 10, 2023
@dustymabe @HuijingHei
move to cgroups v2 everywhere
d514086 
Adjust the kernel arguments so that we're now using cgroups v2
in our testing-devel (and subsequently, testing and stable) stream(s).

Context: coreos/fedora-coreos-tracker#292
HuijingHei pushed a commit to HuijingHei/fedora-coreos-config that referenced this issue Oct 10, 2023
@jlebon @HuijingHei
Move next-devel to cgroupsv2 for new nodes
8ce8774 
All we need to do is to make the associated karg be specific to
`testing-devel` by uplisting it to `image.yaml`. Then `next-devel` will
naturally shed it when `image-base.yaml` gets synced over.

See: coreos/fedora-coreos-tracker#292
HuijingHei pushed a commit to HuijingHei/fedora-coreos-config that referenced this issue Oct 10, 2023
@dustymabe @HuijingHei
move to cgroups v2 everywhere
a5d7130 
Adjust the kernel arguments so that we're now using cgroups v2
in our testing-devel (and subsequently, testing and stable) stream(s).

Context: coreos/fedora-coreos-tracker#292
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/usability kind/design
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests