Something went wrong with the 'timeout_ms' field in the 'struct sched_ext_ops' structure

[Faione]

I'am interest in sched_ext and try it on debian on qemu, with sched_ext kernel(sched_ext/d93dc).Some schedulers works fine but BPF code that utilizes the timeout_ms field generates 'Failed to attach struct_ops' errors. This occurs both on scx_qmap and scx_rustland scheduler, built from scx version 0.1.7,.

$ ./scx_qmap
scx_qmap.c:88 [scx panic]: Argument list too long
Failed to attach struct_op

------

$ ./scx_rustland
Error: Failed to attach struct ops

Caused by:
    bpf call "libbpf_rs::map::Map::attach_struct_ops::{{closure}}" returned NULL

schedulers work fine after removing timeout_ms field

$ ./test/scx_qmap 
enq=0, dsp=0, delta=0, reenq=0, deq=0, core=0
enq=80, dsp=80, delta=0, reenq=0, deq=0, core=0
enq=156, dsp=156, delta=0, reenq=0, deq=0, core=0
^CEXIT: BPF scheduler unregistered

--- 

$ ./test/scx_rustland 
06:24:54 [INFO] RustLand scheduler attached
06:24:55 [INFO] vruntime=105489434
06:24:55 [INFO]   tasks=7
06:24:55 [INFO]   nr_user_dispatches=18 nr_kernel_dispatches=144
06:24:55 [INFO]   nr_cancel_dispatches=0 nr_bounce_dispatches=0
06:24:55 [INFO]   nr_waiting=1 [nr_queued=1 + nr_scheduled=0]
06:24:55 [INFO]   nr_failed_dispatches=0 nr_sched_congested=0 nr_page_faults=0 [OK]
06:24:55 [INFO] time slice = 20000 us
06:24:55 [INFO] slice boost = 200

Same results on newest sched_ext kernel

Is there any way to solve this problem?

[htejun]

Can you try one of the packaged kernels (for arch or ubuntu)? This might be hitting a known limitation in BPF struct_ops which is being addressed.

[Faione]

I tired v6.7.5-scx1 and v6.8.0-rc4-scx2, which are mentioned in the PKGBUILD of scx-packaging-arch, but still encountered a similar situation as above.

I also tired sched_ext-release-v6.6 . It seems things have changed, but I believe it should be an issue with version mismatch.

$ ./test/scx_rustland 
libbpf: extern (func ksym) 'scx_bpf_dispatch_cancel': not found in kernel or module BTFs
libbpf: failed to load object 'bpf_bpf'
libbpf: failed to load BPF skeleton 'bpf_bpf': -22
Error: Failed to load BPF program

Caused by:
    Invalid argument (os error 22)

[htejun]

Can you please try a packaged kernel and schedulers? Just so that we have a better idea on where the problem is - ie. does a packaged kernel + self-built schedulers work? How about self-built kernel + packaged scheduler?

[Faione]

Thanks a lot. I am currently testing on my Arch Linux and waiting for the results.

[htejun]

That's curious tho. It'd be interesting to find out what went wrong. If you still have a copy of the scheduler binaries from the previous unsuccessful installations, can you copy them over to the arch and see whether it works there?

[Faione]

Yes, although everything workes fine in my Arch Linux with packaged kernel, if I change to my own compiled kernel, the previous error occurs again, so it can be determined that the problem is with my kernel configuration.

I was using a kernel configuration optimized for virtualization scenarios, with sched_ext related options enabled. I checked all the configurations and finally found that it was caused by CONFIG_HZ. Mine is 100 instead of 300, and then in the following code where sched_ext attaches the eBPF scheduler, it will cause an error.

SCX_WATCHDOG_MAX_TIMEOUT	= 30 * HZ,

...

case offsetof(struct sched_ext_ops, timeout_ms):
	if (*(u32 *)(udata + moff) > SCX_WATCHDOG_MAX_TIMEOUT)
		return -E2BIG;
	ops->timeout_ms = *(u32 *)(udata + moff);
	return 1;

In scx_qmap and scx_rustland, the timeout_ms feild is hard-coded as 5000, which is greater than 3000 (30 * 100). Changing timeout_ms to 3000U(for CONFIG_HZ_100) solved the problem. I was now able to experience sched_ext. Thanks for your assistance.

[htejun]

Thanks for tracking that down. That's definitely a bug in sched_ext. Will fix.

[htejun]

sched-ext/sched_ext#147 should fix it. Thanks for reporting and debugging the issue. Closing.