Why did fixing CPU limit annotation break CPU limiting with cgroup v2?

[kylos101]

Bug description

We broke the gitpod.io/cpuLimit annotation when we introduced cpu limiting. We then broke cpu limiting in this PR. As a result, many workspaces were trying to use 6 cores at the same time, and this resulted in a poor experience for users.

The intent of this issue is to:

1. Examine what in this PR broke cpu limiting in gen39 and gen40, and why reverting it worked for gen34 (which used cgroup v2)
2. See if we can truly fix the CPU limit annotation, without breaking CPU limiting for cgroup v2

Steps to reproduce

Revert this revert. You should be able to have many workspaces in the cluster that use up to 6 cores of CPU.

Workspace affected

n/a

Expected behavior

1. CPU limiting should rotate CPU fairly, so it is not possible for every workspace pod to try and use 6 CPUs. For example, the 6 cores should 'rotate' from workspace to workspace - not all workspaces should be able to use 6 cores at the same time.
2. When the gitpod.io/cpuLimit annotation is applied, ws-daemon should enforce it. For example, if it is set to 1, you get 1.

Example repository

n/a

Anything else?

No response

[utam0k]

Revert this revert. You should be able to have many workspaces in the cluster that use up to 6 cores of CPU.

@kylos101 Maybe #8459 is wrong? Does this revert point #9382?

[utam0k]

Hi, @csweichel. I'd like to confirm to you that fixedLimiter of CPU limit isn't implemented, right? Do you remember them?
This feels unimplemented to me, is that correct?
https://github.com/gitpod-io/gitpod/pull/8036/files#diff-49497202e6fb01e6a2d9fe29e32980d3c4f25dcdacfffd9b41e6447895ed27a5R213

[csweichel]

@utam0k I'm not sure I follow. fixedLimiter is implemented here.

[utam0k]

@csweichel I'm sorry, I misunderstood. You are right. Thank you.

[kylos101]

Closing, as we need to circle back to cgroup v2 at a later time. In other words, IO limits do not work in cgroup v2.