GetCpuLoad Failed on cgroup v2 environment

[BSWANG]

How to reproduce it?

Start cadvisor with -enable_load_reader on the cgroup v2 environment.

Will see following error message during housekeeping:

W0719 17:10:36.691918 3600017 container.go:589] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee447a3b_5409_4bef_afd2_64b7f7835849.slice": failed to get load stat for "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee447a3b_5409_4bef_afd2_64b7f7835849.slice" - path "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee447a3b_5409_4bef_afd2_64b7f7835849.slice", error netlink request failed with error invalid argument

What I found after investigate:

The cadvisor get cpu load info by send CGROUPSTATS_CMD_GET request though netlink message:

cadvisor/utils/cpuload/netlink/netlink.go

Lines 128 to 132 in d6b0ddb

	func prepareCmdMessage(id uint16, cfd uintptr) (msg netlinkMessage) {
	buf := bytes.NewBuffer([]byte{})
	addAttribute(buf, unix.CGROUPSTATS_CMD_ATTR_FD, uint32(cfd), 4)
	return prepareMessage(id, unix.CGROUPSTATS_CMD_GET, buf.Bytes())
	}

The kernel process the get request in cgroupstats_user_cmd :

https://github.com/torvalds/linux/blob/master/kernel/taskstats.c#L407

static int cgroupstats_user_cmd(struct sk_buff *skb, struct genl_info *info)

And build the result in cgroupstats_build:
https://github.com/torvalds/linux/blob/5c1ee569660d4a205dced9cb4d0306b907fb7599/kernel/cgroup/cgroup-v1.c#L699

int cgroupstats_build(struct cgroupstats *stats, struct dentry *dentry)
{
……
	/* it should be kernfs_node belonging to cgroupfs and is a directory */
	if (dentry->d_sb->s_type != &cgroup_fs_type || !kn ||
	    kernfs_type(kn) != KERNFS_DIR)
		return -EINVAL;  // which cause EINVAL code return 

The cgroup_fs_type is type of cgroup v1 not cgroup v2. So cgroupstats_build function return EINVAL on path type judge statement.

Same issue on kernel community: https://lore.kernel.org/all/20200910055207.87702-1-zhouchengming@bytedance.com/T/#r50c826a171045e42d0b40a552e0d4d1b2a2bab4d

How to resolve it?

Maybe we can obtain the cpu stat from cpu.pressure file when psi is enabled, not from the CGROUPSTATS_CMD_GET netlink api?

[BSWANG]

kubernetes/enhancements#2254

[ShadowJonathan]

So does this issue effectively mean that cpuLoad is not working on systems with cgroup v2?

[ShadowJonathan]

PSI is discussed in #3052