take longer time got io response when exec a process in container on clh with dax enabled

[lifupan]

Description of problem

It will take more than 10 seconds got io response on cloud hypervisor with dax enabled.

reproduce steps:

1: run a pod with crictl runp;
2: create a container in this pod;
3. start the container;
4: exec a process in the container with tty enabled;
It would take more than 10 seconds for the io response.

After disabled the virtiofs dax cache mode, it would worked well.

Expected result

(replace this text with an explanation of what you thought would happen)

Actual result

(replace this text with details of what actually happened)

[lifupan]

Hi @sboeuf ,have you ever met this issue?

[sboeuf]

@lifupan no I haven't run into this issue. Did you compare the same use case with QEMU/virtio-fs?
Also, make sure the cache_size is large enough if you don't want to get bad performances out of virtio-fs. Usually 8G should be fine.

[lifupan]

@sboeuf I had tried qemu, and it worked well with cache size of 1024M.
You can try with the latest kata static release https://github.com/kata-containers/runtime/releases/tag/1.10.0-rc0 with docker.

I had tired it but got error:

$ sudo docker exec -ti d4076cc5eb04 sh
OCI runtime exec failed: context deadline exceeded: unknown

[jcvenegas]

I have seen similar behaivor, by time to time it takes a long time to reponse, but I was able to reproduce it, but also I see some times a container takes a long time to boot. I have ignore this for now as we are looking for another rece condition that seems to happen at stop ( but I what I found is that the agent stop responding) I am not sure if this is related with this, do you have a consistent reproducible way get more debug information @lifupan ?

[lifupan]

@jcvenegas
I had even reproduced this bad performance on a bare cloud-hypervisor env without kata involved as below:
1)following the steps in https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/fs.md to boot a clear linux in cloud-hypervisor share a container image rootfs dir using virtiofs from host to cloud-hypervisor guest os.

2. login the guest os and then mount the shared container rootfs on to a directory with dax enabled;
3. chroot into the mounted directory with running a bash command;
4. you could find that it would take a longer time get the bash's response.

[sboeuf]

@lifupan could you give this a new try based on the latest cloud-hypervisor commit on master. I updated the virtio-fs support through cloud-hypervisor/cloud-hypervisor#557 and I ran some fio testing showing it was performing very well.