Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CPU affinity to executed processes #1253

Merged
merged 2 commits into from
Jun 25, 2024
Merged

Add CPU affinity to executed processes #1253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2149fb5
config-linux: describe the format of cpus and mems
kolyshkin May 19, 2024
119ae42
Add CPU affinity to executed processes
kolyshkin May 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions config-linux.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,8 +395,8 @@ The following parameters can be specified to set up the controller:
* **`period`** *(uint64, OPTIONAL)* - specifies a period of time in microseconds for how regularly a cgroup's access to CPU resources should be reallocated (CFS scheduler only)
* **`realtimeRuntime`** *(int64, OPTIONAL)* - specifies a period of time in microseconds for the longest continuous period in which the tasks in a cgroup have access to CPU resources
* **`realtimePeriod`** *(uint64, OPTIONAL)* - same as **`period`** but applies to realtime scheduler only
* **`cpus`** *(string, OPTIONAL)* - list of CPUs the container will run in
* **`mems`** *(string, OPTIONAL)* - list of Memory Nodes the container will run in
* **`cpus`** *(string, OPTIONAL)* - list of CPUs the container will run on. This is a comma-separated list, with dashes to represent ranges. For example, `0-3,7` represents CPUs 0,1,2,3, and 7.
* **`mems`** *(string, OPTIONAL)* - list of memory nodes the container will run on. This is a comma-separated list, with dashes to represent ranges. For example, `0-3,7` represents memory nodes 0,1,2,3, and 7.
* **`idle`** *(int64, OPTIONAL)* - cgroups are configured with minimum weight, 0: default behavior, 1: SCHED_IDLE.

#### Example
Expand Down
17 changes: 16 additions & 1 deletion config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,17 @@ For Linux-based systems, the `process` object supports the following process-spe

* **`class`** (string, REQUIRED) specifies the I/O scheduling class. Possible values are `IOPRIO_CLASS_RT`, `IOPRIO_CLASS_BE`, and `IOPRIO_CLASS_IDLE`.
* **`priority`** (int, REQUIRED) specifies the priority level within the class. The value should be an integer ranging from 0 (highest) to 7 (lowest).
* **`execCPUAffinity`** (object, OPTIONAL) specifies CPU affinity used to execute the process.
This setting is not applicable to the container's init process.
The following properties are available:
* **`initial`** (string, OPTIONAL) is a list of CPUs a runtime parent
process to be run on initially, before the transition to container's
cgroup. This is a a comma-separated list, with dashes to represent
ranges. For example, `0-3,7` represents CPUs 0,1,2,3, and 7.
* **`final`** (string, OPTIONAL) is a list of CPUs the process will be run
on after the transition to container's cgroup. The format is the same as
for `initial`. If omitted or empty, the container's default CPU affinity,
as defined by [cpu.cpus property](./config.md#configLinuxCPUs)), is used.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the tests, if you don't call sched_setaffinity(), the kernel itself will reset it to cgroups cpuset.cpu, so, maybe here statement can be simplified, that if not specified, after transition into container namespace, default affinity depends on kernel implementation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is nicer to have a reference to another way of setting cpu affinity here. If removed, it still says the same thing:

       on after the transition to container's cgroup. The format is the same as
-      for `initial`. If omitted or empty, the container's default CPU affinity,
-      as defined by [cpu.cpus property](./config.md#configLinuxCPUs)), is used.
+      for `initial`. If omitted or empty, the container's default CPU affinity
+      applies.

so I think it's better to keep it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, ok for me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the tests, if you don't call sched_setaffinity(), the kernel itself will reset it to cgroups cpuset.cpu

@kad It seems your tests are wrong. Mine are showing this only happens when the initial (pre cgroup-move) affinity is not a subset of cgroup's affinity.

So maybe we need to change the spec to say something like

If omitted or empty, no attempt to change the CPU affinity of the process after moving it to container's cgroup is made, and its final affinity is determined by the Linux kernel.

I'd rather not go into more details here, because otherwise I'll be documenting the kernel.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my tests, it is not the case. (-p is "pause" where I'm attaching in another terminal process to cgroup with cpuset.cpus=0-10):

r640-2:/home/akanevsk # ~akanevsk/getaffinity -g -s "0-5" -g -p -g
get_affinity: 0-87
set_affinity: affinity set to 0-5
get_affinity: 0-5
get_affinity: 0-10
r640-2:/home/akanevsk # ~akanevsk/getaffinity -g -s "50-60" -g -p -g
get_affinity: 0-87
set_affinity: affinity set to 50-60
get_affinity: 50-60
get_affinity: 0-10


### <a name="configUser" />User

Expand Down Expand Up @@ -416,7 +427,11 @@ _Note: symbolic name for uid and gid, such as uname and gname respectively, are
"hard": 1024,
"soft": 1024
}
]
],
"execCPUAffinity": {
"initial": "7",
"final": "0-3,7"
}
}
```
### Example (Solaris)
Expand Down
15 changes: 14 additions & 1 deletion schema/config-schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,20 @@
}
}
}
}
},
"execCPUAffinity": {
"type": "object",
"properties": {
"initial": {
"type": "string",
"pattern": "^[0-9, -]*$"
},
"final": {
"type": "string",
"pattern": "^[0-9, -]*$"
}
}
}
}
},
"linux": {
Expand Down
8 changes: 8 additions & 0 deletions specs-go/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ type Process struct {
SelinuxLabel string `json:"selinuxLabel,omitempty" platform:"linux"`
// IOPriority contains the I/O priority settings for the cgroup.
IOPriority *LinuxIOPriority `json:"ioPriority,omitempty" platform:"linux"`
// ExecCPUAffinity specifies CPU affinity for exec processes.
ExecCPUAffinity *CPUAffinity `json:"execCPUAffinity,omitempty" platform:"linux"`
}

// LinuxCapabilities specifies the list of allowed capabilities that are kept for a process.
Expand Down Expand Up @@ -127,6 +129,12 @@ const (
IOPRIO_CLASS_IDLE IOPriorityClass = "IOPRIO_CLASS_IDLE"
)

// CPUAffinity specifies process' CPU affinity.
type CPUAffinity struct {
Initial string `json:"initial,omitempty"`
Final string `json:"final,omitempty"`
}

// Box specifies dimensions of a rectangle. Used for specifying the size of a console.
type Box struct {
// Height is the vertical dimension of a box.
Expand Down
Loading