Add CPU affinity to executed processes #1253

kolyshkin · 2024-05-18T18:01:40Z

This allows to set initial and final CPU affinity for a process being run in a container, which is needed to solve the issue described in opencontainers/runc#3922.

This is an alternative to #1252.

config.md

giuseppe

LGTM

kolyshkin · 2024-05-30T23:25:33Z

@opencontainers/runtime-spec-maintainers PTAL (I'm going to merge this this week if there are no other comments)

bartwensley

LGTM

kad · 2024-05-31T17:30:14Z

config.md

+      on after the transition to container's cgroup. The format is either the
+      same as for `initial`, or a special value `cgroup` which means container's
+      CPU affinity, as defined by [cpu.cpus property](./config.md#configLinuxCPUs)).
+      If omitted or empty, the `initial` setting is kept.


shouldn't it be the opposite? I mean "if ommitted, cpu.cpus will be in effect"?
The initial seems to allow to set affinity to cpu cores that are outside of the set which is specified by cpu.cpus.
This also trigger the question of validation of the values between cpuAffinity and cpu.cpus.

There are two questions here.

What is the default if final is not set. You are probably right that cpu.cpus makes more sense. UPDATED.

Whether to allow affinity that's outside of container's cpu.cpus. While it seems theoretically possible for the initial cpuset, I haven't checked that it works, and I don't want to add any specific constraints to the spec (or to runc/crun).

kolyshkin · 2024-06-01T19:21:45Z

@cclerget PTAL

utam0k

LGTM. It's interesting.

cclerget

LGTM

kad · 2024-06-04T08:19:54Z

@kolyshkin please, give us some time, we would like to do few experiments with @klihub, as we also observed corner cases scenarios with DPDK-style apps in the past, but we need some time to replicate that environment and see behaviour in case of invalid arguments between requests via cgroups or setaffinity.

Regarding PR in general:

naming of the fields: initial and final: would it make sense to have more consistent names with other parts of the spec that corresponds to same lifecycle stages of the container? e.g. normalize names with hook names: createRuntime/createContainer (or startContainer)? This also helps spec consumers to understand at which stage this CPU affinity is supposed to be applied in container lifecycle.
initial seems to be mandatory argument right now, even were it is applied before entering container namespace. We seen in the past that bigger problem for execution inside container namespace, not before. I think it would make more sense if both initial/final will be optional. WDYT?
Overall error handling with setaffinity(): it might be good to add clause in the description that incorrect setting of those parameters would lead to container creation failure? because if spec requests some value in those fields, but corresponding function call fails (invalid arguments/permissions/etc), we can't silently ignore such errors.

kolyshkin · 2024-06-06T19:09:56Z

@kad thanks for reviewing this!

naming of the fields: initial and final: would it make sense to have more consistent names with other parts of the spec that corresponds to same lifecycle stages of the container? e.g. normalize names with hook names: createRuntime/createContainer (or startContainer)? This also helps spec consumers to understand at which stage this CPU affinity is supposed to be applied in container lifecycle.

I think you are misunderstanding. These two are for exec, not create/start/run. The problem we're trying to solve with these is laid out at opencontainers/runc#3922.

initial seems to be mandatory argument right now, even were it is applied before entering container namespace. We seen in the past that bigger problem for execution inside container namespace, not before. I think it would make more sense if both initial/final will be optional. WDYT?

See, they (initial and final) are both part of cpuAffinity object, which itself is optional. By saying that initial is mandatory we're just saying that if cpuAffinity is specified, it should not be empty.

Overall error handling with setaffinity(): it might be good to add clause in the description that incorrect setting of those parameters would lead to container creation failure? because if spec requests some value in those fields, but corresponding function call fails (invalid arguments/permissions/etc), we can't silently ignore such errors.

Again, this is about exec not create/start/run. Unfortunately, this distinction (between starting container's PID 1 and executing an arbitrary process in a running container) is not laid out very well in the runtime-spec.

I think it should be up to implementation what to do it case of errors. IOW the spec should not be way too rigid. Maybe some implementation could have a flag saying "ignore this" or "ignore error from that" or "if this won't work, fall back to that" etc. If the spec is too rigid, such hacks render an implementation incompatible, which I guess is not good.

kolyshkin · 2024-06-11T17:55:36Z

I guess I am merging this later today. @kad if you have something, please open a followup PR.

kolyshkin · 2024-06-11T18:11:25Z

We also need to add a flag into runtime features.

kolyshkin · 2024-06-11T18:58:20Z

We also need to add a flag into runtime features.

I couldn't find a good place to add such a flag, so maybe the upper-level runtime (cri-o, containerd) can check the features.ociVersionMax field instead? WDYT @haircommander

haircommander · 2024-06-11T19:00:26Z

if the feature piece isn't sanctified in the runtime spec that's fine, as long as runtimes that support it agree on the name of the feature

kad · 2024-06-11T19:10:29Z

sorry for delay, missed notification from previous reply :(

@kad thanks for reviewing this!

naming of the fields: initial and final: would it make sense to have more consistent names with other parts of the spec that corresponds to same lifecycle stages of the container? e.g. normalize names with hook names: createRuntime/createContainer (or startContainer)? This also helps spec consumers to understand at which stage this CPU affinity is supposed to be applied in container lifecycle.

I think you are misunderstanding. These two are for exec, not create/start/run. The problem we're trying to solve with these is laid out at opencontainers/runc#3922.

well, then it is even worse with naming, I'd say. Description says "specifies CPU affinity used to execute the process." but doesn't mention that it is not applicable for normal container entry point process.

Furthermore, the "exec" is not really part of the runtime spec since 2016 or so. Yes, runtimes are implementing it, and it can be used, but it is not covered by spec officially. So, any fields in the spec is originally referring to standard operations create/start/stop/status/...

if we want this field only be respected on exec, we have two choices: a) explicitly call it with exec prefix in name or something similar, b) having field still named more generically as now, but inside initial/final rename to something more closely named about exec.

initial seems to be mandatory argument right now, even were it is applied before entering container namespace. We seen in the past that bigger problem for execution inside container namespace, not before. I think it would make more sense if both initial/final will be optional. WDYT?

See, they (initial and final) are both part of cpuAffinity object, which itself is optional. By saying that initial is mandatory we're just saying that if cpuAffinity is specified, it should not be empty.

it is not really a problem to have empty struct, effect should be the same as it is not specified at all.

One thing which I'm asking about 'initial' be optional, is because sched_cpuaffinity is reset once process is attached to cgroup with cpuset. So, practically initial makes sense only if you want to execute something outside container, but once you're entering container namespace, the sched_cpuaffinity() needed to be called again.

So, my line of thinking is like this:

if cpuAffinity struct not set at all, don't call sched_setaffinity at all.
if set:
- if initial is set -> set in createRuntime stage. fail on error.
- if initial not set or empty -> don't call sched_setaffinity()
- if final is set -> set in createContainer, fail on error. (this will validate disjoint sets with cpu.cpus).
- if final not set or empty, default to cgroup cpu.cpus, but only if inital was set (probably not needed, as kernel will reset it anyway to match cgroup)

with both of those variants being optional, it would allow to do affinity only for "infrastructure" operations before entering into container, only after entering container, or both.

And if we do it linked with other container lifecycle points like createRuntime, createRuntime, etc... it will give more flexibility to set affinity for all scenarios.

Overall error handling with setaffinity(): it might be good to add clause in the description that incorrect setting of those parameters would lead to container creation failure? because if spec requests some value in those fields, but corresponding function call fails (invalid arguments/permissions/etc), we can't silently ignore such errors.

Again, this is about exec not create/start/run. Unfortunately, this distinction (between starting container's PID 1 and executing an arbitrary process in a running container) is not laid out very well in the runtime-spec.

well, exec was part of the spec at some point, but then it was removed from it. So, as I wrote above, fields present in spec are assumed to be used at all container lifecycle points, unless really explicitly called out.

I think it should be up to implementation what to do it case of errors. IOW the spec should not be way too rigid. Maybe some implementation could have a flag saying "ignore this" or "ignore error from that" or "if this won't work, fall back to that" etc. If the spec is too rigid, such hacks render an implementation incompatible, which I guess is not good.

Regarding implementation: if sched_cpuaffinity() is called with disjoint set of cores compared to cpuset.cpus, it would return EINVAL. for the rest, it is AND masked with cpuset.cpus.... So, in worst case scenarios the requested affinity might not be in place at all (if error is ignored) or failure (if error not ignored), to "best" case where affinity would be set to subset from what was actually requested....

Also, s/in/on/g. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin · 2024-06-11T20:10:26Z

@kad thanks, changed like this:

diff --git a/config.md b/config.md
index eab6eec..b9b5573 100644
--- a/config.md
+++ b/config.md
@@ -340,9 +340,10 @@ For Linux-based systems, the `process` object supports the following process-spe
 
     * **`class`** (string, REQUIRED) specifies the I/O scheduling class. Possible values are `IOPRIO_CLASS_RT`, `IOPRIO_CLASS_BE`, and `IOPRIO_CLASS_IDLE`.
     * **`priority`** (int, REQUIRED) specifies the priority level within the class. The value should be an integer ranging from 0 (highest) to 7 (lowest).
-* **`cpuAffinity`** (object, OPTIONAL) specifies CPU affinity used to execute the process.
+* **`execCPUAffinity`** (object, OPTIONAL) specifies CPU affinity used to execute the process.
+    This setting is not applicable to the container's init process.
     The following properties are available:

-    * **`initial`** (string, REQUIRED) is a list of CPUs a runtime parent
+    * **`initial`** (string, OPTIONAL) is a list of CPUs a runtime parent
       process to be run on initially, before the transition to container's
       cgroup. This is a a comma-separated list, with dashes to represent
       ranges. For example, `0-3,7` represents CPUs 0,1,2,3, and 7.
@@ -427,7 +427,7 @@ _Note: symbolic name for uid and gid, such as uname and gname respectively, are
             "soft": 1024
         }
     ],
-    "cpuAffinity": {
+    "execCPUAffinity": {
         "initial": "7",
         "final": "0-3,7"
     }
diff --git a/schema/config-schema.json b/schema/config-schema.json
index 39cc71e..cb74342 100644
--- a/schema/config-schema.json
+++ b/schema/config-schema.json
@@ -221,11 +221,8 @@
                         }
                     }
                 },
-                "cpuAffinity": {
+                "execCPUAffinity": {
                     "type": "object",
-                    "required": [
-                        "initial"
-                    ],
                     "properties": {
                         "initial": {
                             "type": "string",
diff --git a/specs-go/config.go b/specs-go/config.go
index 00bf946..290b6dc 100644
--- a/specs-go/config.go
+++ b/specs-go/config.go
@@ -94,8 +94,8 @@ type Process struct {
        SelinuxLabel string `json:"selinuxLabel,omitempty" platform:"linux"`
        // IOPriority contains the I/O priority settings for the cgroup.
        IOPriority *LinuxIOPriority `json:"ioPriority,omitempty" platform:"linux"`
-       // CPUAffinity specifies CPU affinity for the process.
-       CPUAffinity *CPUAffinity `json:"cpuAffinity,omitempty" platform:"linux"`
+       // ExecCPUAffinity specifies CPU affinity for exec processes.
+       ExecCPUAffinity *CPUAffinity `json:"execCPUAffinity,omitempty" platform:"linux"`
 }
 
 // LinuxCapabilities specifies the list of allowed capabilities that are kept for a process.

kad

thanks for update. looks better now. I'm still not convinced about names initial and final, maybe something more descriptive can be done like execRuntime / execContainer for consistency with hooks?
But for that I'll let others to chime in.

specs-go/config.go

kad · 2024-06-11T21:49:39Z

config.md

+    * **`final`** (string, OPTIONAL) is a list of CPUs the process will be run
+      on after the transition to container's cgroup. The format is the same as
+      for `initial`. If omitted or empty, the container's default CPU affinity,
+      as defined by [cpu.cpus property](./config.md#configLinuxCPUs)), is used.


Based on the tests, if you don't call sched_setaffinity(), the kernel itself will reset it to cgroups cpuset.cpu, so, maybe here statement can be simplified, that if not specified, after transition into container namespace, default affinity depends on kernel implementation?

I think it is nicer to have a reference to another way of setting cpu affinity here. If removed, it still says the same thing:

on after the transition to container's cgroup. The format is the same as - for `initial`. If omitted or empty, the container's default CPU affinity, - as defined by [cpu.cpus property](./config.md#configLinuxCPUs)), is used. + for `initial`. If omitted or empty, the container's default CPU affinity + applies.

so I think it's better to keep it.

well, ok for me.

Based on the tests, if you don't call sched_setaffinity(), the kernel itself will reset it to cgroups cpuset.cpu

@kad It seems your tests are wrong. Mine are showing this only happens when the initial (pre cgroup-move) affinity is not a subset of cgroup's affinity.

So maybe we need to change the spec to say something like

If omitted or empty, no attempt to change the CPU affinity of the process after moving it to container's cgroup is made, and its final affinity is determined by the Linux kernel.

I'd rather not go into more details here, because otherwise I'll be documenting the kernel.

In my tests, it is not the case. (-p is "pause" where I'm attaching in another terminal process to cgroup with cpuset.cpus=0-10):

r640-2:/home/akanevsk # ~akanevsk/getaffinity -g -s "0-5" -g -p -g get_affinity: 0-87 set_affinity: affinity set to 0-5 get_affinity: 0-5 get_affinity: 0-10 r640-2:/home/akanevsk # ~akanevsk/getaffinity -g -s "50-60" -g -p -g get_affinity: 0-87 set_affinity: affinity set to 50-60 get_affinity: 50-60 get_affinity: 0-10

This allows to set initial and final CPU affinity for a process being run in a container, which is needed to solve the issue described in [1]. [1] opencontainers/runc#3922 Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

As per - opencontainers/runtime-spec#1253 - opencontainers/runtime-spec#1261 Add some tests (alas it's impossible to test initial CPU affinity without adding debug logging). Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin requested review from AkihiroSuda, crosbymichael, cyphar, dqminh, giuseppe, hqhq, mrunalp, thaJeztah, tianon and utam0k as code owners May 18, 2024 18:01

AkihiroSuda reviewed May 18, 2024

View reviewed changes

config.md Outdated Show resolved Hide resolved

kolyshkin force-pushed the exec-aff branch 2 times, most recently from 8179d10 to cb918d2 Compare May 19, 2024 22:27

kolyshkin requested a review from AkihiroSuda May 19, 2024 22:27

AkihiroSuda approved these changes May 19, 2024

View reviewed changes

AkihiroSuda modified the milestones: v1.3.0 (tentative), v1.2.1 (tentative) May 20, 2024

giuseppe reviewed May 20, 2024

View reviewed changes

config.md Outdated Show resolved Hide resolved

giuseppe approved these changes May 21, 2024

View reviewed changes

This comment was marked as duplicate.

Sign in to view

utam0k mentioned this pull request May 21, 2024

Add CPU affinity to executed processes youki-dev/youki#2795

Open

kolyshkin mentioned this pull request May 22, 2024

Set temporary single CPU affinity before cgroup cpuset transition. opencontainers/runc#3923

Merged

lifubang mentioned this pull request May 22, 2024

Revert "Set temporary single CPU affinity before cgroup cpuset transition" opencontainers/runc#4283

Merged

bartwensley mentioned this pull request May 28, 2024

config: add annotation for exec isolated CPU affinity #1252

Closed

tianon approved these changes May 31, 2024

View reviewed changes

bartwensley approved these changes May 31, 2024

View reviewed changes

kad reviewed May 31, 2024

View reviewed changes

keisku mentioned this pull request Jun 1, 2024

Add CPU affinity to executed processes youki-dev/oci-spec-rs#168

Closed

utam0k approved these changes Jun 2, 2024

View reviewed changes

kolyshkin force-pushed the exec-aff branch from cb918d2 to 9444658 Compare June 2, 2024 18:41

kolyshkin requested a review from kad June 2, 2024 18:42

cclerget approved these changes Jun 3, 2024

View reviewed changes

config-linux: describe the format of cpus and mems

2149fb5

Also, s/in/on/g. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin force-pushed the exec-aff branch from 9444658 to 2a18cd8 Compare June 11, 2024 20:10

kad reviewed Jun 11, 2024

View reviewed changes

Add CPU affinity to executed processes

119ae42

This allows to set initial and final CPU affinity for a process being run in a container, which is needed to solve the issue described in [1]. [1] opencontainers/runc#3922 Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin force-pushed the exec-aff branch from 2a18cd8 to 119ae42 Compare June 11, 2024 23:49

utam0k mentioned this pull request Jun 21, 2024

Add support for windows CPU affinity #1258

Open

kolyshkin merged commit 7017384 into opencontainers:main Jun 25, 2024
3 checks passed

kolyshkin mentioned this pull request Jul 3, 2024

Issue joining cgroups cpuset with kernel scheduler task "random" distribution opencontainers/runc#3922

Open

h-vetinari mentioned this pull request Jul 8, 2024

Blockers for v1.2.0 opencontainers/runc#4114

Closed

21 tasks

kolyshkin mentioned this pull request Jul 17, 2024

config: simplify final CPU affinity rule #1261

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CPU affinity to executed processes #1253

Add CPU affinity to executed processes #1253

kolyshkin commented May 18, 2024

giuseppe left a comment

This comment was marked as duplicate.

kolyshkin commented May 30, 2024

bartwensley left a comment

kad May 31, 2024

kolyshkin Jun 2, 2024

kolyshkin Jun 2, 2024

kolyshkin commented Jun 1, 2024

utam0k left a comment

cclerget left a comment

kad commented Jun 4, 2024

kolyshkin commented Jun 6, 2024

kolyshkin commented Jun 11, 2024

kolyshkin commented Jun 11, 2024

kolyshkin commented Jun 11, 2024 •

edited

Loading

haircommander commented Jun 11, 2024

kad commented Jun 11, 2024 •

edited

Loading

kolyshkin commented Jun 11, 2024

kad left a comment

kad Jun 11, 2024

kolyshkin Jun 11, 2024

kad Jun 12, 2024

kolyshkin Jul 17, 2024

kad Jul 17, 2024

Add CPU affinity to executed processes #1253

Add CPU affinity to executed processes #1253

Conversation

kolyshkin commented May 18, 2024

giuseppe left a comment

Choose a reason for hiding this comment

This comment was marked as duplicate.

kolyshkin commented May 30, 2024

bartwensley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kolyshkin commented Jun 1, 2024

utam0k left a comment

Choose a reason for hiding this comment

cclerget left a comment

Choose a reason for hiding this comment

kad commented Jun 4, 2024

kolyshkin commented Jun 6, 2024

kolyshkin commented Jun 11, 2024

kolyshkin commented Jun 11, 2024

kolyshkin commented Jun 11, 2024 • edited Loading

haircommander commented Jun 11, 2024

kad commented Jun 11, 2024 • edited Loading

kolyshkin commented Jun 11, 2024

kad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kolyshkin commented Jun 11, 2024 •

edited

Loading

kad commented Jun 11, 2024 •

edited

Loading