use_vsock=true gives context deadline exceeded error with ubuntu bionic kernel #1203

jodh-intel · 2019-01-31T15:29:09Z

Setting use_vsock=true gives the following error with 1.5.0 (packaged or static):

$ sudo docker run -ti --runtime kata-runtime busybox sh
docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: context deadline exceeded: unknown.

The text was updated successfully, but these errors were encountered:

jodh-intel · 2019-01-31T15:31:56Z

I'm struggling to see how this happened given we have explicit vsock jenkins slaves in our CI:

https://github.com/kata-containers/ci/tree/master/jenkins/jobs

It's possible though that something broke at the packaging stage? We must get the following setup asap to avoid any chance of that:

CI for package testing ci#10

grahamwhaley · 2019-01-31T15:52:13Z

But, then it seems to work for me. I did a fresh F28 server install today with devicemapper and only installed the kata 1.5 static tarball, and I have just run a bunch of kata-fc tests, and tried a kata-qemu as well... Here is my kata-env from the qemu:

$ /opt/kata/bin/kata-qemu kata-env

[Meta]
  Version = "1.0.20"

[Runtime]
  Debug = false
  Trace = false
  DisableGuestSeccomp = true
  DisableNewNetNs = false
  Path = "/opt/kata/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.5.0"
    Commit = "5f7fcd773089a615b776862f92217e987f06df0a"
    OCI = "1.0.1-dev"
  [Runtime.Config]
    Path = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.2(kata-static)\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/opt/kata/bin/qemu-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = false
  UseVSock = true

[Image]
  Path = "/opt/kata/share/kata-containers/kata-containers-image_clearlinux_1.5.0_agent_a581aebf473.img"

[Kernel]
  Path = "/opt/kata/share/kata-containers/vmlinuz-4.14.67-22"
  Parameters = "init=/usr/lib/systemd/systemd systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket"

[Initrd]
  Path = ""

[Proxy]
  Type = "noProxy"
  Version = ""
  Path = ""
  Debug = false

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.5.0-efbf3bb25065ce89099630b753d218cdc678e758"
  Path = "/opt/kata/libexec/kata-containers/kata-shim"
  Debug = false

[Agent]
  Type = "kata"

[Host]
  Kernel = "4.16.3-301.fc28.x86_64"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = true
  [Host.Distro]
    Name = "Fedora"
    Version = "28"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz"

[Netmon]
  Version = "kata-netmon version 1.5.0"
  Path = "/opt/kata/libexec/kata-containers/kata-netmon"
  Debug = false
  Enable = false

and

$ docker info --format "{{json .Runtimes}}"
{"kata-fc":{"path":"/opt/kata/bin/kata-fc"},"kata-qemu":{"path":"/opt/kata/bin/kata-qemu"},"runc":{"path":"docker-runc"}}
$ docker run --rm -ti --runtime=kata-qemu busybox sh
/ # uname -a
Linux 6f41cd0e986a 4.14.67 #1 SMP Tue Jan 22 23:49:04 CST 2019 x86_64 GNU/Linux
/ # exit
$ uname -a
Linux skull 4.16.3-301.fc28.x86_64 #1 SMP Mon Apr 23 21:59:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Let me know if you want more info, or for me to try to run something else..

egernst · 2019-01-31T16:02:18Z

What distro?

Baremetal, or?

jodh-intel · 2019-01-31T16:05:03Z

I'm on Ubuntu 18.04 baremetal.

jodh-intel · 2019-01-31T16:07:26Z

Host kernel 4.15.0-45-generic.

sboeuf · 2019-01-31T16:09:51Z

same here, 4.15.0-44-generic with Ubuntu 18.04 baremetal

grahamwhaley · 2019-01-31T16:14:54Z

For more ref then, it looks like the jenkins CI vsocks jobs run on Fedora.

jodh-intel · 2019-01-31T16:21:36Z

@devimc is also on 18.04.

jodh-intel · 2019-01-31T16:50:47Z

F29 works. Ubuntu 18.04 doesn't:

distro	host kernel version	vsock works?
Ubuntu 18.04	`4.15.0-45-generic`	no
Ubuntu 18.04	`4.15.0-44-generic`	no

devimc · 2019-01-31T17:08:24Z

@jodh-intel

distro	host kernel version	vsock works?
Ubuntu 18.04	`4.18.0-1007-azure`	yes
Ubuntu 18.04	`4.18.0-14-generic`	no

jodh-intel · 2019-01-31T19:00:12Z

F29:

distro	host kernel version	vsock works?
Fedora 29	`4.18.16-300`	yes

jodh-intel · 2019-01-31T19:59:26Z

The problem first appeared in 4.15.0-44-generic for bionic:

distro	host kernel version	vsock works?
Ubuntu 18.04	`4.15.0-45-generic`	no
Ubuntu 18.04	`4.15.0-44-generic`	no
Ubuntu 18.04	`4.15.0-43-generic`	yes
Ubuntu 18.04	`4.15.0-39-generic`	yes

jodh-intel · 2019-02-01T09:49:55Z

This appears to be an Ubuntu-specific kernel bug. To take this further will require raising the issue on the Ubuntu kernel team on launchpad.net, installing an upstream kernel and tring to recreate with that, plus a few other bits I think. I'm not in a position to do that currently, so throwing it out to the team to help with this one...

I've removed the P2 label as this is not a Kata issue and only affects a single distro as far as we know at this point.

jodh-intel · 2019-02-01T13:39:56Z

Looks like the issue may relate to a fix for CVE-2018-14625:

https://launchpad.net/ubuntu/+source/linux/4.15.0-44.47

CVE-2018-14625
- vhost/vsock: fix use-after-free in network stack callers

devimc · 2019-02-01T13:49:51Z

nice hunting @jodh-intel

sboeuf · 2019-02-01T17:11:00Z

Thanks for doing this @jodh-intel.
@egernst any further action needed? Maybe @devimc could get in touch with Canonical folks about this.
We need to make sure this will be fixed in next generic kernels.

devimc · 2019-02-01T17:53:34Z

May be Kamal from Canonical can give us some clues about this issue

cc @kamalmostafa

jodh-intel · 2019-02-04T11:11:51Z

Adding the security label as it looks like it might apply.

pmorjan · 2019-02-04T15:36:34Z

A fix has been committed to the kernel tree:
https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/commit/?h=master-next&id=90865e0a59e9474da046e91ac14c2cc0c22acfc9

sboeuf · 2019-02-04T15:41:41Z

Good! Thanks for the heads up @pmorjan

@pmorjan btw, do you have a list of all the Ubuntu kernels affected by this issue? It'd be nice for us if we could list those as not supporting Kata+vsock to our users.

pmorjan · 2019-02-04T16:27:37Z

@sboeuf
I don't have that list. From my observations 4.15.0-44-generic #47-Ubuntu is the first kernel that's failing. But it's not limitted to Ubuntu. I've seen this on other distros runnig 4.19 and 4.20 kernels.
E.g. current Fedora 28/29 and Arch Linux are also affected.
The fix is in 5.0rc3 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7fbe078c37aba3088359c9256c1a1d0c3e39ee81

I created a simple test application to run on the host that can trigger the bug:
https://gist.github.com/pmorjan/9f49c667045f8c0abaf9be3a35f888ef

jodh-intel · 2019-02-04T16:47:50Z

launchpad bug: https://bugs.launchpad.net/bugs/1813934.

mcastelino · 2019-02-06T23:15:42Z

Folks can we flag this higher. I wasted a lot of time today till I realized this was the issue.

chavafg · 2019-02-07T00:00:22Z

We actually started to faced this issue on the vsocks job.
E.g:

19:35:57 Run kata-runtime: nginx:1.14: 
19:35:58 1dd544a859e7098ca5f4abf9c827492404fafe89673e4c83f883505315b0ac5a
19:36:14 docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: context deadline exceeded: unknown.
19:36:14 Checking 18 containers have all relevant components
19:36:14 pgrep: no matching criteria specified
19:36:14 Try `pgrep --help' for more information.
19:36:14 Wrong number of shims running (18 != 17) - stopping
19:36:14 Wrong number of netmons running (18 != 17) - stopping
19:36:15 Wrong number of 'runtime list' containers running (18 != 17) - stopping
19:36:15 Wrong number of pods in /var/lib/vc/sbs (18 != 17) - stopping)

From http://jenkins.katacontainers.io/job/kata-containers-tests-fedora-vsocks-PR/821/console

mcastelino · 2019-02-07T00:06:47Z

I updated my system to the latest available mainline kernel on ubuntu and I still see this

$uname -r
4.20.7-042007-generic

jodh-intel · 2019-02-07T09:38:03Z

The launchpad bug shows Ubuntu plan to update their kernel on February 25th. I haven't checked the other distros.

@grahamwhaley, @chavafg - could you add something about vsock tests failing on the dashboard maybe (http://jenkins.katacontainers.io/view/CI%20Status/)? fwics, I think the only CI distro version affected is going to be Fedora since Ubuntu 16.04 and Centos 7 have much older kernels.

grahamwhaley · 2019-02-07T10:14:45Z

I would add it to the status page (anybody with Jenkins admin rights can do it btw..), but, looking at the Jenkins fedora vsock build job trend ... it seems we have started passing pretty steadily?
http://jenkins.katacontainers.io/job/kata-containers-tests-fedora-vsocks-PR/buildTimeTrend

So, I'm going to hold off and wait for some input from @chavafg

chavafg · 2019-02-07T15:12:44Z

@grahamwhaley you are right, I'll add it if it happens again.

egernst · 2019-02-07T16:45:51Z

From looking at the actual patch upstream; I think we can mitigate by just masking the CID to the lower 32 bits (or using u32). I think this is a viable work-around, as we already have a collision check anyway. Can someone try this?

/cc @devimc

devimc · 2019-02-07T16:57:45Z

the problem is s390x, see #960 (review)

cc @alicefr

jodh-intel · 2019-02-07T18:06:44Z

One workaround is to use the last known working kernel. For example on Ubuntu 18.04:

$ sudo apt-get -y install linux-image-4.15.0-43-generic

Then reboot into that kernel (or tweak grub to select it by default until Canonical push out a kernel that fixes the issue).

devimc · 2019-02-07T19:14:47Z

I tried changing everything to uint32 and that didn't work, also I tried with a simple client-server test [1] that uses uint32 for CIDs and I got the same result

2019/02/07 19:06:51 receive: creating file "cpuinfo.txt" for output
2019/02/07 19:06:51 receive: opening listener: 1024
2019/02/07 19:06:51 receive: listening: host(2):1024
2019/02/07 19:06:51 vscp: receive: failed to accept: resource temporarily unavailable

[1] - https://github.com/mdlayher/vsock/tree/master/cmd/vscp

egernst · 2019-02-07T23:20:45Z

Thanks @devimc .

It sounds like "go to last known working kernel" or "wait for updated kernel" are our two options, then.

caoruidong · 2019-07-22T12:13:15Z

Update to 4.15.0-55-generic and works

jodh-intel added the high-priority Very urgent issue (resolve quickly) label Jan 31, 2019

jodh-intel mentioned this issue Jan 31, 2019

CI for package testing kata-containers/ci#10

Closed

jodh-intel changed the title ~~use_vsock=true gives context deadline exceeded error~~ use_vsock=true gives context deadline exceeded error with ubuntu bionic kernel Feb 1, 2019

jodh-intel removed the high-priority Very urgent issue (resolve quickly) label Feb 1, 2019

jodh-intel added the security Potential or actual security issue label Feb 4, 2019

grahamwhaley mentioned this issue Feb 8, 2019

Not possible to run a container with Firecracker #1222

Closed

GabyCT mentioned this issue Feb 12, 2019

network: Remove grpc AddInterface and RemoveInterface api kata-containers/agent#457

Merged

egernst mentioned this issue Mar 5, 2019

vsock - context deadline exceed on create command. #1319

Closed

devimc mentioned this issue Mar 15, 2019

Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = context deadline exceeded kata-containers/kata-containers#28

Closed

chavafg mentioned this issue Mar 15, 2019

Firecracker not working #1373

Closed

Pennyzct mentioned this issue Aug 6, 2019

virtcontainers: Create the kata agent vsock earlier #1014

Closed

alicefr mentioned this issue Aug 8, 2019

vsock: Kata fails to start with use_vsock = true on ppc64le #1943

Closed

jodh-intel closed this as completed Apr 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use_vsock=true gives context deadline exceeded error with ubuntu bionic kernel #1203

use_vsock=true gives context deadline exceeded error with ubuntu bionic kernel #1203

jodh-intel commented Jan 31, 2019

jodh-intel commented Jan 31, 2019

grahamwhaley commented Jan 31, 2019

egernst commented Jan 31, 2019

jodh-intel commented Jan 31, 2019

jodh-intel commented Jan 31, 2019

sboeuf commented Jan 31, 2019

grahamwhaley commented Jan 31, 2019

jodh-intel commented Jan 31, 2019

jodh-intel commented Jan 31, 2019 •

edited

Loading

devimc commented Jan 31, 2019 •

edited

Loading

jodh-intel commented Jan 31, 2019

jodh-intel commented Jan 31, 2019

jodh-intel commented Feb 1, 2019

jodh-intel commented Feb 1, 2019

devimc commented Feb 1, 2019

sboeuf commented Feb 1, 2019

devimc commented Feb 1, 2019

jodh-intel commented Feb 4, 2019

pmorjan commented Feb 4, 2019

sboeuf commented Feb 4, 2019 •

edited

Loading

pmorjan commented Feb 4, 2019

jodh-intel commented Feb 4, 2019

mcastelino commented Feb 6, 2019

chavafg commented Feb 7, 2019

mcastelino commented Feb 7, 2019

jodh-intel commented Feb 7, 2019

grahamwhaley commented Feb 7, 2019

chavafg commented Feb 7, 2019

egernst commented Feb 7, 2019

devimc commented Feb 7, 2019

jodh-intel commented Feb 7, 2019

devimc commented Feb 7, 2019

egernst commented Feb 7, 2019

caoruidong commented Jul 22, 2019

use_vsock=true gives context deadline exceeded error with ubuntu bionic kernel #1203

use_vsock=true gives context deadline exceeded error with ubuntu bionic kernel #1203

Comments

jodh-intel commented Jan 31, 2019

jodh-intel commented Jan 31, 2019

grahamwhaley commented Jan 31, 2019

egernst commented Jan 31, 2019

jodh-intel commented Jan 31, 2019

jodh-intel commented Jan 31, 2019

sboeuf commented Jan 31, 2019

grahamwhaley commented Jan 31, 2019

jodh-intel commented Jan 31, 2019

jodh-intel commented Jan 31, 2019 • edited Loading

devimc commented Jan 31, 2019 • edited Loading

jodh-intel commented Jan 31, 2019

jodh-intel commented Jan 31, 2019

jodh-intel commented Feb 1, 2019

jodh-intel commented Feb 1, 2019

devimc commented Feb 1, 2019

sboeuf commented Feb 1, 2019

devimc commented Feb 1, 2019

jodh-intel commented Feb 4, 2019

pmorjan commented Feb 4, 2019

sboeuf commented Feb 4, 2019 • edited Loading

pmorjan commented Feb 4, 2019

jodh-intel commented Feb 4, 2019

mcastelino commented Feb 6, 2019

chavafg commented Feb 7, 2019

mcastelino commented Feb 7, 2019

jodh-intel commented Feb 7, 2019

grahamwhaley commented Feb 7, 2019

chavafg commented Feb 7, 2019

egernst commented Feb 7, 2019

devimc commented Feb 7, 2019

jodh-intel commented Feb 7, 2019

devimc commented Feb 7, 2019

egernst commented Feb 7, 2019

caoruidong commented Jul 22, 2019

jodh-intel commented Jan 31, 2019 •

edited

Loading

devimc commented Jan 31, 2019 •

edited

Loading

sboeuf commented Feb 4, 2019 •

edited

Loading