drivers/raw_exec: enable setting cgroup override values

[shoenig]

This PR enables configuration of cgroup override values on the raw_exec
task driver. WARNING: setting cgroup override values eliminates any
guarantee Nomad can make about resource availability for any task on
the client node.

For cgroup v2 systems, set a single unified cgroup path using cgroup_v2_override.
The path may be either absolute or relative to the cgroup root.

config {
  cgroup_v2_override = "custom.slice/app.scope"
}

or

config {
  cgroup_v2_override = "/sys/fs/cgroup/custom.slice/app.scope"
}

For cgroup v1 systems, set a per-controller path for each controller using
cgroup_v1_override. The path(s) may be either absolute or relative to
the controller root.

config {
  cgroup_v1_override = {
    "pids": "custom/app",
    "cpuset": "custom/app",
  }
}

or

config {
  cgroup_v1_override = {
    "pids": "/sys/fs/cgroup/pids/custom/app",
    "cpuset": "/sys/fs/cgroup/cpuset/custom/app",
  }
}

[shoenig]

Example (cgroups v1):

job "sleep" {
  group "group" {
    task "task" {
      driver = "raw_exec"
      config {
        command = "sleep"
        args    = ["infinity"]
        cgroup_v1_override = {
          "pids" : "custom/app",
          "cpuset" : "custom/app",
        }
      }
    }
  }
}

Ensure our custom cgroup under the pids controller exists.

➜ sudo mkdir -p /sys/fs/cgroup/pids/custom/app

Ensure our custom cgroup under the cpuset controller exists and has cpus and mems available.

➜ sudo mkdir -p /sys/fs/cgroup/cpuset/custom/app
➜ echo "0-3" | sudo tee /sys/fs/cgroup/cpuset/custom/cpuset.cpus
➜ echo "0-3" | sudo tee /sys/fs/cgroup/cpuset/custom/app/cpuset.cpus
➜ echo "0" | sudo tee /sys/fs/cgroup/cpuset/custom/cpuset.mems
➜ echo "0" | sudo tee /sys/fs/cgroup/cpuset/custom/app/cpuset.mems

Run job.

Our custom cgroups are in use.

➜ cat /sys/fs/cgroup/pids/custom/app/cgroup.procs
135047
135063
➜ cat /sys/fs/cgroup/cpuset/custom/app/cgroup.procs
135047
135063

[shoenig]

Example (cgroups v2)

job "sleep" {
  group "group" {
    task "task" {
      driver = "raw_exec"
      config {
        command            = "sleep"
        args               = ["infinity"]
        cgroup_v2_override = "custom.slice/app.scope"
      }
    }
  }
}

Ensure our custom unified cgroup exists.

➜ sudo mkdir -p /sys/fs/cgroup/custom.slice/app.scope

Run job.

Ensure our custom cgroup is in use.

➜ pcat custom.slice/app.scope/cgroup.procs
124226

[schmichael]

Just to confirm: this doesn't appear to create cgroups if they don't exist or remove cgroups on text exit, correct?

[schmichael]

Don't we need to check the return value here or risk a task running without being added to its intended cgroup?

[shoenig]

Making sure task does not start if cgroup override does not exist

Cgroups v1

2024-04-29T14:07:26Z  Driver Failure   failed to launch command with executor: rpc error: code = Unknown desc = unable to configure cgroups: unable to write to custom cgroup: open /sys/fs/cgroup/pids/invalid/app/cgroup.procs: no such file or directory
2024-04-29T14:07:26Z  Task Setup       Building Task Directory
2024-04-29T14:07:26Z  Received         Task received by client

Cgroups v2

2024-04-29T14:09:17Z  Driver Failure  failed to launch command with executor: rpc error: code = Unknown desc = unable to configure cgroups: no such file or directory
2024-04-29T14:09:17Z  Task Setup      Building Task Directory
2024-04-29T14:09:17Z  Received        Task received by client

[shoenig]

Making sure config allows for only one of v1 or v2 override being set

config {
  command         = "sleep"
  args            = ["infinity"]
  cgroup_v1_override = {
    "pids": "custom/app",
    "cpuset": "custom/app",
  }
  cgroup_v2_override = "custom.slice/app.scope"
}

2024-04-29T14:19:04Z  Driver Failure   only one of cgroups_v1_override and cgroups_v2_override may be set
2024-04-29T14:19:04Z  Task Setup       Building Task Directory
2024-04-29T14:19:04Z  Received         Task received by client

[schmichael]

Is it safe to ignore this error?

[shoenig]

Historically we had to ignore these errors because people keep running Nomad as non-root or on systems where only some of the cgroup controllers are enabled. But they still want raw_exec to Just Work for them.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.