Sandbox creating is slower when use_vsock enabled

[BetaXOi]

Description of problem

Sandbox createing is slower when use vsock than virtio-serial.

Set 'enable_tracing = true' and create 10 kata containers with and without 'use_vsock=true', record the time of 'docker run' and opentracing span.
for i in seq 10; do time docker run --rm --runtime kata-runtime-test -tid busybox sh; sleep 1; done

Expected result

The time of creating shoud be similar.

Actual result

The time of using vsock is slower 2 second almost than virtio-serial.

[BetaXOi]

The reason is 'connect' be invoked befor vsock is ready, and the timeout is 2 second in vsock kernel module.
#define VSOCK_DEFAULT_CONNECT_TIMEOUT (2 * HZ)

current sequence

[egernst]

/cc @amshinde ! Good find @BetaXOi

[Pennyzct]

Hi guys

Are x86_64 still suffering this issue???
We've encountered the same problem on aarch64. The whole boot-up time with vsock enabled is affected by VSOCK_DEFAULT_CONNECT_TIMEOUT.

Reading the discussion under #1918 and https://lists.nongnu.org/archive/html/qemu-devel/2019-12/msg02225.html. we're trying to reverse vsock connection to avoid the time out???
Referring the @stefanha stefan's suggestion here:

This can be done efficiently as follows:
1. kata-runtime listens on a vsock port
2. kata-agent-port=PORT is added to the kernel command-line options
3. kata-agent parses the port number and connects to the host

He also pointed out other ways to avoid this 2s time out:

Userspace APIs to avoid the 2 second wait already exist:

1. The SO_VM_SOCKETS_CONNECT_TIMEOUT socket option controls the connect
   timeout for this socket.

2. Non-blocking connect allows the userspace process to do other things
   while a connection attempt is being made.

So, what's the progress here? ;)
cc @bergwolf @lifupan @gnawux @amshinde @devimc @grahamwhaley @jodh-intel @egernst @justin-he @jongwu

[stefanha]

I still think the direction of the connection should be reversed and the agent should be launched with a kata-agent-port=PORT parameter.

Unfortunately I am currently busy with other work so I will not be able to implement this change.

[Pennyzct]

@stefanha okay, I'll try to follow your thinking to do the implementation. ;)

[stefano-garzarella]

@justin-he sent a patch for Linux vhost-vsock to fix this issue and now it is merged in the Linus' tree.
It will be released with Linux 5.7-rc5.

The patch should fix this issue, but I agree with @stefanha about the direction of the connection.

[stefanha]

The issue with the kernel patch is that it leaves kata-runtime with a loop that consumes 100% CPU when attempting to connect before kata-agent has created the socket and put it into the listen state.

[justin-he]

Besides, kata should not be 100% sure that host kernel must contain https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0b841030625cde5f784dd62aec72d6a766faae70. Hence the proposal from @stefanha (reverse connection) still make sense IMO.

[Pennyzct]

Hi @stefanha
I got a little confused here, even in my method of reversing connection, I just use it to do a quick handshake to avoid old kernel's 2s blocking issue.
I let runtime launch a goroutine to accept one connection from agent, to do a quick handshake, but, still, at the same time, the main routine needs to wait/block until the connection arrived, then we could proceed.
cc @lifupan

[stefanha]

On Wed, May 6, 2020 at 5:02 AM Penny ***@***.***> wrote: I got a little confused here, even in my method of reversing connection, I just use it to do a quick handshake to avoid old kernel's 2s blocking issue. I let runtime launch a goroutine to accept one connection from agent, to do a quick handshake, but, still, at the same time, the main routine needs to wait/block until the connection arrived, then we could proceed.

kata-runtime establishes the connection in vendor/github.com/kata-containers/agent/protocols/client/client.go. virtcontainers/kata_agent.go calls NewAgentClient() to connect to kata-agent. The code modified so that a vsock listen socket is created before QEMU is launched and the port number is passed to kata-agent on the guest kernel command-line. A goroutine isn't necessary when creating the socket but the socket must be put into the listen state so that kata-agent can connect to it. When kata-runtime reaches NewAgentClient() the vsock dialFunc() can use vsock.Accept() instead of vsock.Dial().

[Pennyzct]

@stefanha thanks for the explaining.
I misunderstood what you said before. you want a total reverse connection, let runtime be server, not agent anymore.
my pr is just a quick handshake started by kata-agent to walk around the 2s timeout, but still agent will be server.
I'll close to issue a new one. ;)

[stefanha]

On Wed, May 6, 2020 at 6:25 AM Penny ***@***.***> wrote: I misunderstood what you said before. you want a total reverse connection, let runtime be server, not agent anymore. my pr is just a quick handshake started by kata-agent to walk around the 2s timeout, but still agent will be server.

Okay, sorry it wasn't clear before. I think the issue with trying to do a quick handshake is that there is a race condition between closing the handshake socket and opening the permanent agent socket. Another process could use the port number in the meantime.

[bergwolf]

Let alone the incompatibility of reversing the server/client roles, It would be another attack surface for runtime to listen for guest connections.

What about setting a short connecting timeout and busy loop connecting on the host(runtime) side? That would be much easier to implement and doesn't break existing compatibility.

[stefanha]

On Wed, May 6, 2020 at 7:53 AM Peng Tao ***@***.***> wrote: Let alone the incompatibility of reversing the server/client roles, It would be another attack surface for runtime to listen for guest connections.

Can you be more specific about how this is a new attack surface? The vsock code in the host kernel is already being exercised by every sandbox. The only difference is that the accept(2) code path is currently not being exercised - but in exchange we stop exercising the connect(2) code path when reversing the connection. Is there a significant difference here? When the socket is accepted, kata-runtime should use getpeername(2) to verify that the remote CID matches the expected sandbox VM. This prevents the attack you mentioned. BTW the reverse of this problem also exists right now: if an attacker takes over an unprivileged host process they can run a busy loop that attempts to connect(2) to vsock ports and can win the race with kata-runtime's connect(2). That would give them access to sandboxes that are booting up. The reverse connection is better in this respect: kata-runtime can verify that the expected CID has connected, while there is no socket API that allows kata-agent to verify that it's really kata-runtime that is connecting. So overall I see the security of these approaches as about the same. The difference is that the reverse connection avoid unnecessary sleeps and busy loops.

[Pennyzct]

Hi @stefanha @bergwolf
Here is one thing about reversing the whole connection i couldn't figure out:
Requests always come from kata-runtime to agent at uncertain time, like the hotplug issue.
So I couldn't figure a way in agent to avoid to set up a server to deal with all sorts of requests.

That's also part of reason of why I just reverse the connection in the first handshake in my earlier commits.

The whole kernel boot up with systemd will maybe cost aound 300ms?? In order to avoid this
300ms total busy loop, how about using non-block mode and poll a while??

[BetaXOi]

The short lifetime of the runtime determines that it is not suitable for use as a server.

I have implemented the sample code for reverse connection before, but the code is really smell and dirty, so we did not use it inside our project.
BetaXOi@187ccb9
BetaXOi/agent@a111362

Another solution is to do busy waiting in runtime and implement connect timeout control, which is the solution we are currently using.
BetaXOi@6cd6780

[stefanha]

On Wed, May 6, 2020 at 9:53 AM BetaXOi ***@***.***> wrote: The short lifetime of the runtime determines that it is not suitable for use as a server.

Good point! Using the reverse vsock connection all the time requires a proxy process (unless you are using the v2 Shim interface?). I think a hybrid approach is best: kata-agent can use the reverse connection for launch only and also accept incoming connections for cases where kata-runtime is invoked later. This eliminates busy waiting (if spinning) or startup latency (if sleeping). I have implemented the sample code for reverse connection before, but the

code is really smell and dirty, so we did not use it inside our project. ***@***.*** <BetaXOi@187ccb9> ***@***.*** <BetaXOi/agent@a111362> Another solution is to do busy waiting in runtime and implement connect timeout control, which is the solution we are currently using. ***@***.*** <BetaXOi@6cd6780>

What do you think about the hybrid solution?