Why is the performance of my lio iscsi gateway so poor?

[yangxg89]

Hi, I have a three nodes(kernel 4.16.3, updated by yum) cluster that has 19 OSDs without ssd. I test the performance of lio gateway with fio and compare with the tgt gateway. The result is that:
1.test ceph image directly in rbd ioengine:
4M,write:398MB/s, 4M,randwrite:402MB/s, 4M,read:603MB/s, 4M,randread:709MB/s
1M,write:227MB/s, 1M,randwrite:213MB/s, 1M,read:377MB/s, 1M,randread:646MB/s
2.test lio gateway in libaio ioengine:
4M,write:45MB/s, 4M,randwrite:42MB/s, 4M,read:128MB/s, 4M,randread:95MB/s
1M,write:45MB/s, 1M,randwrite:37MB/s, 1M,read:118MB/s, 1M,randread:73MB/s
3.test tgt gateway in libaio ioengine:
4M,write:106MB/s, 4M,randwrite:117MB/s, 4M,read:267MB/s, 4M,randread:411MB/s
1M,write:106MB/s, 1M,randwrite:94MB/s, 1M,read:285MB/s, 1M,randread:545MB/s
I also test suse iscsi gateway(https://www.suse.com/documentation/ses-4/book_storage_admin/data/ceph_iscsi_install.html), the performance of its write is better than tgt and its read is a little worse than tgt.
The performance of lio gateway is the worst. Is there any optimization methods?
Thanks!

[mikechristie]

Check out the issues described here:

open-iscsi/tcmu-runner#359

The patch mentioned is merged in the upstream tcmu-runner repo master branch. You need to manually config the session/cmd_sn queue depth settings to work around the open-iscsi issue.