Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scx_lavd: add LAVD (Latency-criticality Aware Virtual Deadline) scheduler #192

Merged
merged 2 commits into from
Mar 18, 2024

Conversation

multics69
Copy link
Contributor

Provide a deadline-based scheduler focused on latency-critical tasks inspired by gaming workloads. scx_lavd quantifies how much a task is latency-critical and leverages the task's latency-criticality information in making various scheduling decisions (e.g., task's execution deadline, time slice, etc). It consists of rust and BPF parts. The rust part simply loads the BPF scheduler and conducts the chores (e.g., printing statistics, etc). The BPF part makes all the scheduling decisions.

Changwoo Min added 2 commits March 16, 2024 10:31
…cheduler

scx_lavd is a BPF scheduler that implements an LAVD (Latency-criticality
Aware Virtual Deadline) scheduling algorithm. While LAVD is new and
still evolving, its core ideas are 1) measuring how much a task is
latency critical and 2) leveraging the task's latency-criticality
information in making various scheduling decisions (e.g., task's
deadline, time slice, etc.). As the name implies, LAVD is based on the
foundation of deadline scheduling. This scheduler consists of the BPF
part and the rust part. The BPF part makes all the scheduling decisions;
the rust part loads the BPF code and conducts other chores (e.g.,
printing sampled scheduling decisions).
Copy link
Contributor

@arighi arighi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks good to me. I've done some quick tests and I haven't seen any obvious issue. I left some inline comments.

Moreover, you may want to add a Signed-off-by line to your commits (all the other commits have a SoB line) and there are also some little coding style issues here and there, but nothing relevant.

For the Rust part you may want to run rustfmt scheds/rust/scx_lavd/src/main.rs to automatically fix all the formatting issues.

scheds/rust/scx_lavd/src/main.rs Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
scheds/rust/scx_lavd/src/main.rs Show resolved Hide resolved
scheds/rust/scx_lavd/src/main.rs Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
Copy link
Contributor

@htejun htejun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I scanned through the code. Overall, looks great to me. Thanks for the contribution and please feel free to land whenever you are ready. We can iterate in the tree.

Copy link
Contributor

@htejun htejun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I actually went through the code this time. I really like the simplicity of it. The heuristics are easy to understand and conceptually concise. Please feel free to land anytime. Some overall thoughts:

  • It looks like it would be relatively straight-forward to encapsulate the LAVD scheduling algorithm into a library so that it can be used by other schedulers to manage their in-domain scheduling.
  • The mixture of raw nice values, 0-40 priority values and weights are a bit cumbersome. I suppose the intention was to ease table-mapping across different terms but that's not that difficult to do with well-defined linear weight values. The use of static_prio also makes it difficult to use it for flattened cgroup hierarchical scheduling as the flattened weight values need more than 40 levels of distinction. I think it should be possible to unify the values used to standardized weights without meaningfully affecting actual operation.
  • There is no preemption. Was it that it wasn't necessary given the slices are fairly short and interactive tasks are prioritized heavily already? Or would it be interesting to explore that in the future?

scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
scheds/rust/scx_lavd/src/bpf/main.bpf.c Show resolved Hide resolved
@SoulHarsh007
Copy link

👋 I am not sure if this is the right place to report this, but I find the following kernel oops when using scx_lavd: https://paste.soulharsh007.dev/p/43a4476.log

Kernel Version: 6.8.1-2-cachyos

@multics69
Copy link
Contributor Author

Thank you for the report. If possible, can you provide some information when/how the bug happened? For example, shutdowning a laptop while listening a audio, etc.

👋 I am not sure if this is the right place to report this, but I find the following kernel oops when using scx_lavd: https://paste.soulharsh007.dev/p/43a4476.log

Kernel Version: 6.8.1-2-cachyos

@Byte-Lab
Copy link
Contributor

Thanks for the report @SoulHarsh007. This crash would be due to a bug in the core framework, so I'm going to go ahead and merge this as it's an issue we'll have to fix over in https://github.com/sched-ext/sched_ext. I'll create an issue over there (tagging you in it) so we can track accordingly.

@Byte-Lab Byte-Lab merged commit 80986e4 into sched-ext:main Mar 18, 2024
1 check passed
@Byte-Lab
Copy link
Contributor

@multics69 hope it's OK that I merged and to address the feedback in a subsequent PR -- some folks were asking about playing with this and I felt it might be easier to just merge it and iterate in tree given that it had been open for a few days.

@multics69
Copy link
Contributor Author

Sure, no problem. Thanks for taking care of that bug!

htejun added a commit that referenced this pull request Mar 19, 2024
scx_lavd: addressed comments from PR #192
@multics69 multics69 deleted the scx_lavd branch March 20, 2024 06:52
@shelterx
Copy link

Not sure if this is the correct place since it's merged, but I got this:

sched_ext: BPF scheduler "lavd" errored, disabling
sched_ext: runnable task stall (kworker/15:2[20480] failed to run for 5.206s)
   scx_watchdog_workfn+0x12d/0x1c0
   process_scheduled_works+0x1e4/0x420
   worker_thread+0x2ff/0x620
   kthread+0x1b3/0x250
   ret_from_fork+0x34/0x40
   ret_from_fork_asm+0x11/0x20
hrtimer: interrupt took 13489 ns

Also I had huge audio lag/issues in game when this scheduler was running.

@multics69
Copy link
Contributor Author

Thank you for reporting the issue. Would you mind providing a bit more detail which might help me to reproduce the problem? For example, CPU model, the played game, other programs ran, etc.

Not sure if this is the correct place since it's merged, but I got this:

sched_ext: BPF scheduler "lavd" errored, disabling
sched_ext: runnable task stall (kworker/15:2[20480] failed to run for 5.206s)
   scx_watchdog_workfn+0x12d/0x1c0
   process_scheduled_works+0x1e4/0x420
   worker_thread+0x2ff/0x620
   kthread+0x1b3/0x250
   ret_from_fork+0x34/0x40
   ret_from_fork_asm+0x11/0x20
hrtimer: interrupt took 13489 ns

Also I had huge audio lag/issues in game when this scheduler was running.

@shelterx
Copy link

@multics69
Some more info, CPU is Intel 9900KS, nothing else was running.
Another crash with more output, the game is AC Valhalla.

07:17:48 [INFO] scx_lavd scheduler is initialized
07:17:48 [INFO] scx_lavd scheduler starts running.

DEBUG DUMP
================================================================================

kworker/u32:0[2494] triggered exit kind 1026:
  runnable task stall (audio_client_ma[2660] failed to run for 6.928s)

Backtrace:
  process_scheduled_works+0x238/0x420
  worker_thread+0x37f/0x670
  kthread+0x2dd/0x320
  ret_from_fork+0x34/0x40
  ret_from_fork_asm+0x11/0x20

Runqueue states
---------------

CPU 0   : nr_run=1 flags=0x0 cpu_rel=0 ops_qseq=369019 pnt_seq=1198790
          curr=ACValhalla.exe[2590] class=ext_sched_class

 *R ACValhalla.exe[2590] -2ms
      scx_state/flags=3/0xd dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=(n/a)
      cpus=ffff


CPU 1   : nr_run=1 flags=0x0 cpu_rel=0 ops_qseq=328046 pnt_seq=840722
          curr=kworker/u32:4[181] class=ext_sched_class

 *R kworker/u32:4[181] +0ms
      scx_state/flags=3/0xd dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=(n/a)
      cpus=ffff


CPU 2   : nr_run=1 flags=0x0 cpu_rel=0 ops_qseq=356138 pnt_seq=952088
          curr=ACValhalla.exe[2586] class=ext_sched_class

 *S ACValhalla.exe[2586] +0ms
      scx_state/flags=3/0xd dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=(n/a)
      cpus=ffff


CPU 4   : nr_run=1 flags=0x0 cpu_rel=0 ops_qseq=337482 pnt_seq=899483
          curr=swapper/4[0] class=idle_sched_class

  R ACValhalla.exe[2583] +0ms
      scx_state/flags=3/0x9 dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=0x8000000000000002
      cpus=ffff

    __se_sys_futex_waitv+0x2f7/0x3b0
    do_syscall_64+0x8e/0x170
    entry_SYSCALL_64_after_hwframe+0x46/0x4e

CPU 5   : nr_run=1 flags=0x0 cpu_rel=0 ops_qseq=218990 pnt_seq=518331
          curr=swapper/5[0] class=idle_sched_class

  R ACValhalla.exe[2585] +0ms
      scx_state/flags=3/0x9 dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=0x8000000000000002
      cpus=ffff

    __se_sys_futex_waitv+0x2f7/0x3b0
    do_syscall_64+0x8e/0x170
    entry_SYSCALL_64_after_hwframe+0x46/0x4e

CPU 9   : nr_run=1 flags=0x0 cpu_rel=0 ops_qseq=459582 pnt_seq=1259272
          curr=kworker/u32:0[2494] class=ext_sched_class

 *R kworker/u32:0[2494] +0ms
      scx_state/flags=3/0xd dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=(n/a)
      cpus=ffff

    irq_work_run+0x65/0x190
    __sysvec_irq_work+0x1c/0xc0
    sysvec_irq_work+0x69/0x80
    asm_sysvec_irq_work+0x1a/0x20
    scx_watchdog_workfn+0x18c/0x2a0
    process_scheduled_works+0x238/0x420
    worker_thread+0x37f/0x670
    kthread+0x2dd/0x320
    ret_from_fork+0x34/0x40
    ret_from_fork_asm+0x11/0x20

CPU 10  : nr_run=3 flags=0x0 cpu_rel=0 ops_qseq=430700 pnt_seq=1089487
          curr=vkd3d-swapchain[2683] class=ext_sched_class

 *R vkd3d-swapchain[2683] -3ms
      scx_state/flags=3/0x5 dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=(n/a)
      cpus=ffff


  R audio_client_ma[2660] -6928ms
      scx_state/flags=3/0x9 dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=0x8000000000000002
      cpus=ffff

    __x64_sys_poll+0xfe/0x1a0
    do_syscall_64+0x8e/0x170
    entry_SYSCALL_64_after_hwframe+0x46/0x4e

  R kworker/10:3[602] -6740ms
      scx_state/flags=3/0x9 dsq_flags=0x1 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=0x0
      cpus=0400

    kthread+0x2dd/0x320
    ret_from_fork+0x34/0x40
    ret_from_fork_asm+0x11/0x20

CPU 12  : nr_run=1 flags=0x0 cpu_rel=0 ops_qseq=228768 pnt_seq=557851
          curr=ACValhalla.exe[2584] class=ext_sched_class

 *R ACValhalla.exe[2584] +0ms
      scx_state/flags=3/0xd dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=(n/a)
      cpus=ffff


================================================================================

Error: EXIT: runnable task stall (audio_client_ma[2660] failed to run for 6.928s)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants