Skip to content

proposal: x/sys/unix: add (*CPUSet).Fill helper to enable all CPUs #75186

@kolyshkin

Description

@kolyshkin

Proposal Details

We propose to add a new Fill method to struct CPUSet, which sets all CPU bits to 1.
The implementation can look like this (see CL 698015):

// Fill adds all possible CPU bits to the set s. On Linux, [SchedSetaffinity]
// will silently ignore any invalid CPU bits in [CPUSet] so this is an
// efficient way of resetting the CPU affinity of a process.
func (s *CPUSet) Fill() {
	for i := range s {
		s[i] = ^cpuMask(0)
	}
}

Context

Some programs (such as container runtimes) want to reset their CPU
affinity if they are spawned by processes with a particular CPU
affinity. Container runtimes didn't really have to deal with this issue
until Linux 6.2 when the cpuset cgroup was changed to no longer
auto-reset CPU affinity in this case.

A naive approach to resetting your CPU affinity would be to get the
number of CPUs by looking at "/proc/stat" or "/sys/devices/system/cpu"
(note that runtime.NumCPU() actually returns the CPU affinity of the
process at startup time, which isn't useful for this purpose) and then
asking for all of those CPUs.

However, sched_setaffinity(2) will silently ignore any CPU bits set in
the provided CPUSet if they do not exist or are not enabled in the
cpuset cgroup of the process. This means that you can reset your CPU
affinity by just setting every CPU bit in CPUSet and passing it to
sched_setaffinity(2).

Unfortunately, setting every CPU bit in CPUSet with (*CPUSet).Set() is
very inefficient. If it were possible to just memset(0xFF) the CPUSet
array, users would be able to reset their CPU affinity even more
cheaply. However, Go doesn't have a memset primitive that can be used in
that way.

Obvious solutions like setting the array elements of CPUSet to (^0) do
not work because CPUSet is an array of a private newtype and so the
compiler complains if you try to use a constant like (^0) without a
cast (and we cannot use a cast because the type is private):

cannot use ^0 (untyped int constant -1) as
"golang.org/x/sys/unix".cpuMask value in assignment (overflows)

The only real alternative is to do something quite hacky like:

cpuset := unix.CPUSet{}
for i := range cpuset {
    cpuset[i]-- // underflow to 0xFF..FF
}

... which is the solution we use in runc.

It would be much nicer to have a helper that does this memset for us in
a less hacky way, since resetting CPU affinity seems like a fairly
common operation.

Ref: Linux kernel commit da019032819a ("sched: Enforce user requested affinity")

(Description and implementation by @cyphar)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions