I/O throttles content-init and maybe backups

[kylos101]

Bug description

Workspace start is slow when using iolimits, we think this is because content-init is being i/o limited.

We're unsure if backups are being limited.

Steps to reproduce

Try starting a workspace in a cluster where iolimit is defined in the configmap for ws-daemon.

Workspace affected

n/a

Expected behavior

Ideally we would not throttle workspaces on content-init or backup.

Example repository

n/a

Anything else?

I/O limiting was introduced here.

[utam0k]

First of all, The custom limitation of the cgroup depends on the life cycle of the pod, making it difficult to control the timing. But we have some cgroup group, container/worksapce/user.

<container-cgorup>  drwxr-xr-x 3 root      root
 └── workspace      drwxr-xr-x 5 gitpodUid gitpodGid
   └── user         drwxr-xr-x 5 gitpodUid gitpodGid

We are currently applying restrictions to container cgroups. It means all processes in the workspace are limited.

I know timing is difficult to control but thought it might be smart to separate by the process. A user-run process should be sufficient to apply the restrictions. Other content-init, etc. should belong to the workspace cgroup, and these custom cgroup restrictions should apply to the user cgroup, which should solve these problems cleanly.

[utam0k]

Relates codes.

gitpod/components/ws-daemon/pkg/iws/iws.go

Lines 372 to 379 in 70d832b

	for _, pidStr := range strings.Split(string(cgroupProcsBytes), "\n") {
	if pidStr == "" || pidStr == "0" {
	continue
	}
	if err := os.WriteFile(filepath.Join(newPath, "cgroup.procs"), []byte(pidStr), 0644); err != nil {
	log.WithError(err).Warnf("failed to move process %s to cgroup %q", pidStr, newGroup)
	}
	}

[kylos101]

@aledbf are we still throttling content-init when using iolimits?

For example, I'm unsure if it was resolved by:
#9309 or #9309

I ask to see if it is something @utam0k should focus on tomorrow or not. Otherwise his next priority is the .NET issue.

[csweichel]

I still don't see how this should be the case. io.max is written to the cgroup of the workspace pod. Content init happens in the process tree of ws-daemon which cannot be subject to those limits.

[atduarte]

Since IO limits were not working, do we still believe content-init and backups IO was throttled?

[kylos101]

Closing this. [1]

Plus, we're actively working on IO limiting here.