fix cpuset cgroup manager initialization

[shoenig]

While tracking down another bug, I got really confused by log messages that you'll only see on the first run of a Nomad agent on a new system, caused by some spaghetti code in how we bootstrap the cpuset manager on the client.

(the error content is from a branch)

    2022-08-22T21:30:21.284-0500 [WARN]  client: failed to lookup cpuset from cgroup parent, and not set as reservable_cores: parent=nomad.slice error="openat2 /sys/fs/cgroup/nomad.slice/cpuset.cpus.effective: no such file or directory"
    2022-08-22T21:30:21.284-0500 [DEBUG] client.cpuset.v2: initializing with: cores=[]
    2022-08-22T21:30:21.300-0500 [DEBUG] client.cpuset.v2: establish cgroup hierarchy: parent=nomad.slice

This block will attempt to retrieve the effective cpuset (if reservable cores are not set) - before actually ensuring the parent cgroup for nomad exists. This means on first run, we get the warning message above and an empty set of usable cores.

// Ensure cgroups are created on linux platform
if runtime.GOOS == "linux" && c.cpusetManager != nil {
	// use the client configuration for reservable_cores if set
	cores := conf.ReservableCores
	if len(cores) == 0 {
		// otherwise lookup the effective cores from the parent cgroup
		cores, err = cgutil.GetCPUsFromCgroup(conf.CgroupParent)
		if err != nil {
			c.logger.Warn("failed to lookup cpuset from cgroup parent, and not set as reservable_cores", "parent", conf.CgroupParent)
			// will continue with a disabled cpuset manager
		}
	}
	if cpuErr := c.cpusetManager.Init(cores); cpuErr != nil {
		// If the client cannot initialize the cgroup then reserved cores will not be reported and the cpuset manager
		// will be disabled. this is common when running in dev mode under a non-root user for example.
		c.logger.Warn("failed to initialize cpuset cgroup subsystem, cpuset management disabled", "error", cpuErr)
		c.cpusetManager = new(cgutil.NoopCpusetManager)
	}
}

The empty set of useable cores means tasks configured to run will not have a value set in their cpuset.cpus cgroup.

cgroup/nomad.slice/594f8fba-c4e1-0e6a-38c3-b056db697bfa.py1.scope 
➜ cat cpuset.cpus


cgroup/nomad.slice/594f8fba-c4e1-0e6a-38c3-b056db697bfa.py1.scope

Any restart of the Nomad agent afterwords and the bug is gone - the parent cgroup will exist and the initialization will work as intended.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.