Will the performance of io_uring be better than that of spdk under the same traffic model in the latest kernel? #1153
Replies: 6 comments 11 replies
-
|
That job should not be 500% CPU, it should be 1 SQPOLL thread probably running 100% of the time, and then a userspace thread running anywhere from 0..100% of the time. In other words, a max load of 2. Here's running that job on a fast drive: and everything else mostly idle, which is as expected, and it's doing about 1.5M IOPS here. If you're seeing 500% CPU usage, which I'm assuming is 5 things running all the time, then you have something else running. In general, for peak performance, you don't want sqpoll. You either want hipri=1, which is polled IO, or just plain IRQ driven (don't set anything but engine=io_uring and direct=1). For hipri, you want to configure your nvme drive(s) with polled queues, using the nvme parameter poll_queues. If you look in dmesg, you should see something like: if configured correctly, the above shows nvme7 has 16 poll queues. This is important, for polled IO. Outside of that, using fixedbufs=1 will help, and so will registerfiles=1. The former is the more important of the two. |
Beta Was this translation helpful? Give feedback.
-
|
I directly tested the io_uring, and the traffic model is as described earlier. The CPU situation is shown in the following figure: [root@localhost ~]# pstree -p 28797 [root@localhost ~]# dmesg |grep queues Performance reaches 57W iops |
Beta Was this translation helpful? Give feedback.
-
|
When my traffic model is not configured with sqthread_poll=1, the performance drops to 48W iops and the CPU usage is as follows: 37847 root 20 0 577260 22628 1288 R 323.3 0.0 2:10.85 fio [root@localhost wxh_shell_copy]# fio io_uring.conf [root@localhost wxh_shell_copy]# cat io_uring.conf [job1] |
Beta Was this translation helpful? Give feedback.
-
|
As described above, when I do not configure sqthread_poll=1, the CPU becomes 323. Therefore, according to previous tests, when configuring sqthread_poll, 200% of the CPU will be used, and the remaining CPU will be 323%, which should be Fio occupying 0-100%. I am not sure what CPU is being used. |
Beta Was this translation helpful? Give feedback.
-
|
I don't know what 'W' means in terms of IOPS. I think you may have a crappy nvme device with a low queue depth. Can you try and paste the output of: as the only explanation here would be that the io depth exceeds the device depth. |
Beta Was this translation helpful? Give feedback.
-
|
[root@localhost ~]# grep . /sys/block/nvme1n1/queue/* |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I didn't express myself clearly. My W represents 0000, for example, 48W iops represents 480000 iops. |
Beta Was this translation helpful? Give feedback.
-
Please don't invent suffixes, the industry norm here would be to use 48K IOPS to mean 48000 IOPS. |
Beta Was this translation helpful? Give feedback.
-
|
What kernel are you using? |
Beta Was this translation helpful? Give feedback.
-
|
5.10.0-136.74.0.154.oe2203sp1.x86_64 |
Beta Was this translation helpful? Give feedback.
-
Please update to something that isn't ancient, this discussion isn't going to be fruitful if you're running something that is several years old. |
Beta Was this translation helpful? Give feedback.
-
|
I see that the latest kernel version of Euler system 5.10 is this one. I'm not sure what you mean by updating to the latest kernel version. What do you mean by upgrading to the latest kernel version? |
Beta Was this translation helpful? Give feedback.
-
|
5.10 is from 2020, and particularly io_uring has improved a lot since then. We can't support anything that isn't at least 6.x in terms of kernel version. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, I understand. Thank you. I will directly upgrade to a kernel version 6. x or higher for verification. Thank you very much. |
Beta Was this translation helpful? Give feedback.
-
|
I conducted an experiment and when I set iodepth=1, the corresponding CPU drops. Therefore, I speculate that the extra threads should be used for kernel or user mode processing to complete io, which is also known as polling to complete io threads? But I'm a bit unsure how to set the number of threads? |
Beta Was this translation helpful? Give feedback.
-
|
I conducted another experiment and wrote a demo program to call the disk write operation of the library interface. I configured the params. flags=IORING SETUP IOPOLL | IORING SETUP SQPOLL | IORING SETUP SQ OFF; Io_uring-register_iowq-aff (&ring1, Sizeof (mask),&mask, Io_uring-register_iowq_max_workers (&ring1,&threadnum), when I set fd=open (disk_filename, O_RDWR| O_DIRECT)), At this point, the CPU usage is less than 300%, which is in line with expectations (SQ occupies one thread, Worker occupies one thread, one demo thread) But when the current surface conditions remain unchanged, I set fd=open (disk_filename, O_RDWR）， At this point, the larger the iodepth, the more threads there are, The demo thread and the SQ thread each occupy one constant, and I am not sure what type of thread the remaining threads are. When I changed the configuration to the following fd=open (disk_filename, O-RDWR), params. flags=IORING-SETUP-SQPOLL | IORING-SETUP-SQ-OFF, io_uring-register_iowq-aff (&ring1, Sizeof (mask),&mask, The number of threads displayed in io_uring-register_iowq_max_workers (&ring1,&threadnum) is still as expected, even if the iodepth increases, The CPU ratio is also correct. In summary, I guess which threads change with iodepth should be in user mode (program queries CQE in CQ queue) or kernel mode (SQE ->CQE) used to process threads that complete io. |
Beta Was this translation helpful? Give feedback.
-
|
Re-open if necessary once you're on a more recent kernel. |
Beta Was this translation helpful? Give feedback.
-
I tested io_uring and spdk using the same traffic model and found that the performance of io_uring was slightly higher than that of spdk. However, when testing io_uring, the CPU usage reached nearly 500%, although I only set one job。
The traffic model is as follows：
[global]
ioengine=io_uring
sqthread_poll=1
direct=1
time_based
group_reporting
bs=4k
rw=randwrite
rwmixread=70
numjobs=1
iodepth=64
runtime=1800
ramp_time=60
[job1]
filename=/dev/nvme1n1
name=drive1
Beta Was this translation helpful? Give feedback.
All reactions