Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

vsock: Kata fails to start with use_vsock = true on ppc64le #1943

Closed
nitkon opened this issue Aug 7, 2019 · 33 comments
Closed

vsock: Kata fails to start with use_vsock = true on ppc64le #1943

nitkon opened this issue Aug 7, 2019 · 33 comments
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.

Comments

@nitkon
Copy link
Contributor

nitkon commented Aug 7, 2019

Description of problem

Set use_vsock = true in configuration.toml and enabled +CONFIG_BINFMT_MISC=y ; +CONFIG_VSOCKETS=y in the kata kernel.

root@soe13:/home/nitesh# docker run -ti --runtime kata-runtime busybox sh
<hangs>

Expected result

Kata container starts without any issue

Actual result

Fails to start. Hangs/no response after starting the container.

journalctl -f

Aug 07 21:50:48 soe13 kata-runtime[77350]: time="2019-08-07T21:50:48.262428432+05:30" level=debug msg="Restricting maxmem to 32GB as Qemu Version < 2.10, Assumption: Kernel Version >= 4.11" arch=ppc64le command=create container=eb639d5a3585dbc77d2f7ec8e18f49ba123b145f845bfcd77cd29a8f3daac2f5 name=kata-runtime pid=77350 source=virtcontainers subsystem=qemu
Aug 07 21:50:48 soe13 kata-runtime[77350]: time="2019-08-07T21:50:48.262512883+05:30" level=debug msg="agent: Using vsock VM socket endpoint" arch=ppc64le command=create container=eb639d5a3585dbc77d2f7ec8e18f49ba123b145f845bfcd77cd29a8f3daac2f5 name=kata-runtime pid=77350 source=virtcontainers subsystem=kata_agent
Aug 07 21:50:48 soe13 kata-runtime[77350]: time="2019-08-07T21:50:48.262555357+05:30" level=debug msg="Could not retrieve anything from storage" arch=ppc64le command=create container=eb639d5a3585dbc77d2f7ec8e18f49ba123b145f845bfcd77cd29a8f3daac2f5 name=kata-runtime pid=77350 source=virtcontainers subsystem=kata_agent
Aug 07 21:50:48 soe13 kata-runtime[77350]: time="2019-08-07T21:50:48.262609682+05:30" level=warning msg="load sandbox devices failed" arch=ppc64le command=create container=eb639d5a3585dbc77d2f7ec8e18f49ba123b145f845bfcd77cd29a8f3daac2f5 error="open /run/vc/sbs/eb639d5a3585dbc77d2f7ec8e18f49ba123b145f845bfcd77cd29a8f3daac2f5/devices.json: no such file or directory" name=kata-runtime pid=77350 sandbox=eb639d5a3585dbc77d2f7ec8e18f49ba123b145f845bfcd77cd29a8f3daac2f5 sandboxid=eb639d5a3585dbc77d2f7ec8e18f49ba123b145f845bfcd77cd29a8f3daac2f5 source=virtcontainers subsystem=sandbox

System:

root@soe13:~# cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS

Output from: kata-collect-data.sh

kata-collect-data.log

@nitkon nitkon added bug Incorrect behaviour needs-review Needs to be assessed by the team. labels Aug 7, 2019
@nitkon
Copy link
Contributor Author

nitkon commented Aug 7, 2019

@nitkon
Copy link
Contributor Author

nitkon commented Aug 7, 2019

Not sure if this is the issue but from here #947
On ppc64le:

root@soe13:~# go run test64.go 
err:ioctl: 1 vsockFd=3 cid=4 
root@soe13:~# go run test32.go 
err:ioctl: 1 vsockFd=3 cid=4 

@alicefr
Copy link

alicefr commented Aug 8, 2019

On which kernel are you running? Have a look to this #1203 . It's not an endianess bug in any case.

@alicefr
Copy link

alicefr commented Aug 8, 2019

I saw it from your kata-collect-data. I guess you're using a buggy kernel. Could you try to upgrade it? As suggested in #1203 (comment)

@nitkon
Copy link
Contributor Author

nitkon commented Aug 8, 2019

@alicefr: I tried with an updated kernel and the results are the same

root@kata1:/home# go run test32.go 
err:ioctl: 1 vsockFd=3 cid=4 
root@kata1:/home# go run test64.go 
err:ioctl: 1 vsockFd=3 cid=4 
root@kata1:/home# uname -a
Linux kata1 5.0.7-050007-generic #201904052141 SMP Fri Apr 5 22:30:57 UTC 2019 ppc64le GNU/Linux

@nitkon
Copy link
Contributor Author

nitkon commented Aug 12, 2019

Looks like kata-runtime cannot get a unique context id: cc @devimc

Aug 12 11:09:32 soe13 kata-runtime[21842]: time="2019-08-12T11:09:32.734992035+05:30" level=warning msg="load sandbox devices failed" arch=ppc64le command=create container=e690a354ea3a5e5c8b6a334f3febceb0337b863342280638191d1d1eb7741645 error="open /run/vc/sbs/e690a354ea3a5e5c8b6a334f3febceb0337b863342280638191d1d1eb7741645/devices.json: no such file or directory" name=kata-runtime pid=21842 sandbox=e690a354ea3a5e5c8b6a334f3febceb0337b863342280638191d1d1eb7741645 sandboxid=e690a354ea3a5e5c8b6a334f3febceb0337b863342280638191d1d1eb7741645 source=virtcontainers subsystem=sandbox
Aug 12 12:04:09 soe13 kata-runtime[10005]: time="2019-08-12T12:04:09.07404988+05:30" level=error msg="Could not get a unique context ID for the vsock" arch=ppc64le command=create container=6f4d5f856bf473df8467587638116ad15fec2fae203d905b979c0eba2c633fec name=kata-runtime pid=10005 source=runtime
Aug 12 12:07:56 soe13 kata-runtime[12251]: time="2019-08-12T12:07:56.97810119+05:30" level=error msg="Could not get a unique context ID for the vsock" arch=ppc64le command=create container=90c013383aeb630f02d749d08faf320c2546bf4a5e9808577942c8af303ea1e1 name=kata-runtime pid=12251 source=runtime
Aug 12 12:25:02 soe13 kata-runtime[21842]: time="2019-08-12T12:25:02.457973621+05:30" level=error msg="Could not get a unique context ID for the vsock" arch=ppc64le command=create container=e690a354ea3a5e5c8b6a334f3febceb0337b863342280638191d1d1eb7741645 name=kata-runtime pid=21842 source=runtime

@amshinde
Copy link
Member

@nitkon Do you see any other details? Does this help in getting some more context of the error :
#1954 ?

Maybe you can try this on an older version of Kata, where it was known to work. cc @alicefr

@alicefr
Copy link

alicefr commented Aug 13, 2019

I think it's important to understand if it's a kata or kernel bug. We (s390x) also need to check if vsock is correctly working (cc @jschintag ). But as far as I got from @nitkon he has another problem than the bug reported at #1203 (comment) , right? You couldn't set the CID.

@ganeshmaharaj
Copy link
Contributor

@alicefr Our quick test across all the other architectures seem to show that it works fine with Kata. This issue came up when we tried to move vsock creation to an earlier time on Kata. #1014 (comment) has the info the hang on the host. The code in question that we think is hanging is

func Ioctl(fd uintptr, request, data uintptr) error {
which sort of pointed to a native issue and nothing with Kata. Hope i am seeing it right, but do correct me if i am missing something here.

@alicefr
Copy link

alicefr commented Aug 13, 2019

So as far as I understand he's getting a operation not permitted (errno=1) when he tried to set the CID with ioctl call (from #1943 (comment)). The best would be to have a look into the kernel code and check why it gives back this error. My test is simply replicating what kata is doing at

if err = ioctlFunc(vsockFd.Fd(), ioctlVhostVsockSetGuestCid, uintptr(unsafe.Pointer(&cid))); err == nil {

@ganeshmaharaj
Copy link
Contributor

Woah!! Seems like this has been open for a while now and you have chased down & closed quite a bit of these. Thank you very much. 💯 I am now hopefully all caught up and educated. Will wait and see what the kernel debug shows. that ought to help us on next steps.

@nitkon
Copy link
Contributor Author

nitkon commented Aug 13, 2019

@amshinde @alicefr : ioctl:1 error like we had seen earlier. But this error take an hour or so to log in the journalctl, after devices.json: no such file or directory". It hangs at devices.json: no such file or directory" for a long time before I kill it.

docker run -ti --runtime kata-runtime busybox sh
<hangs>
t
Aug 13 16:45:43 soe13 kata-runtime[140197]: time="2019-08-13T16:45:43.690393451+05:30" level=debug msg="Could not retrieve anything from storage" arch=ppc64le command=create container=dd3d2a4fa97b556a513cc1f7d0b41113e7165488e69381cb811fbe1c4340f8da name=kata-runtime pid=140197 source=virtcontainers subsystem=kata_agent
Aug 13 16:45:43 soe13 kata-runtime[140197]: time="2019-08-13T16:45:43.690451555+05:30" level=warning msg="load sandbox devices failed" arch=ppc64le command=create container=dd3d2a4fa97b556a513cc1f7d0b41113e7165488e69381cb811fbe1c4340f8da error="open /run/vc/sbs/dd3d2a4fa97b556a513cc1f7d0b41113e7165488e69381cb811fbe1c4340f8da/devices.json: no such file or directory" name=kata-runtime pid=140197 sandbox=dd3d2a4fa97b556a513cc1f7d0b41113e7165488e69381cb811fbe1c4340f8da sandboxid=dd3d2a4fa97b556a513cc1f7d0b41113e7165488e69381cb811fbe1c4340f8da source=virtcontainers subsystem=sandbox
Aug 13 17:42:34 soe13 kata-runtime[129541]: time="2019-08-13T17:42:34.874104169+05:30" level=error msg="Could not get a unique context ID for the vsock : ioctl: 1" arch=ppc64le command=create container=bd004f93c15aa472e3b69f72d6e5f2c18dec693effa77ba8260ab2e7f28e25cd name=kata-runtime pid=129541 source=runtime
Aug 13 18:00:00 soe13 kata-runtime[140197]: time="2019-08-13T18:00:00.994033133+05:30" level=error msg="Could not get a unique context ID for the vsock : ioctl: 1" arch=ppc64le command=create container=dd3d2a4fa97b556a513cc1f7d0b41113e7165488e69381cb811fbe1c4340f8da name=kata-runtime pid=140197 source=runtime

@alicefr
Copy link

alicefr commented Aug 13, 2019

can you try to run kata standalone with --log=/dev/stdout and paste the output?

@nitkon
Copy link
Contributor Author

nitkon commented Aug 13, 2019

@alicefr : Thnx for inputs. Here is the output:

root@soe13:/home/nitesh# kata-runtime --log=/dev/stdout run --bundle "$bundle" foo
INFO[0000] loaded configuration                          arch=ppc64le command=run file=/etc/kata-containers/configuration.toml format=TOML name=kata-runtime pid=119766 source=katautils
INFO[0000] vsock supported                               arch=ppc64le command=run name=kata-runtime pid=119766 source=katautils
DEBU[0000]                                               arch=ppc64le command=run default-kernel-parameters= name=kata-runtime pid=119766 source=katautils
INFO[0000] VSOCK supported, configure to not use proxy   arch=ppc64le command=run name=kata-runtime pid=119766 source=katautils
INFO[0000]                                               arch=ppc64le arguments="\"run --bundle /tmp/bundle foo\"" command=run commit=49c3f4536a41386e35a571786f3798a851719681 name=kata-runtime pid=119766 source=runtime version=1.9.0-alpha0
DEBU[0000] converting /tmp/bundle/config.json            arch=ppc64le command=run container=foo name=kata-runtime pid=119766 source=virtcontainers subsystem=oci
DEBU[0000] container rootfs: /tmp/bundle/rootfs          arch=ppc64le command=run container=foo name=kata-runtime pid=119766 source=virtcontainers subsystem=oci
INFO[0000] shm-size detected: 67108864                   arch=ppc64le command=run container=foo name=kata-runtime pid=119766 source=virtcontainers subsystem=oci
INFO[0000] create netns                                  arch=ppc64le command=run container=foo name=kata-runtime netns=/var/run/netns/cni-658d2abe-d4dc-edf5-18be-4571086a2025 pid=119766 source=katautils
DEBU[0000] New filesystem store backend for /var/lib/vc/sbs/foo  arch=ppc64le backend=filesystem command=run container=foo name=kata-runtime path=/var/lib/vc/sbs/foo pid=119766 source=virtcontainers/store subsystem=store
DEBU[0000] Creating root directory                       arch=ppc64le backend=filesystem command=run container=foo name=kata-runtime path=/var/lib/vc/sbs/foo pid=119766 source=virtcontainers/store subsystem=store
DEBU[0000] Creating raw directory                        arch=ppc64le backend=filesystem command=run container=foo name=kata-runtime path=/var/lib/vc/sbs/foo/raw pid=119766 source=virtcontainers/store subsystem=store
DEBU[0000] New filesystem store backend for /run/vc/sbs/foo  arch=ppc64le backend=filesystem command=run container=foo name=kata-runtime path=/run/vc/sbs/foo pid=119766 source=virtcontainers/store subsystem=store
DEBU[0000] Creating root directory                       arch=ppc64le backend=filesystem command=run container=foo name=kata-runtime path=/run/vc/sbs/foo pid=119766 source=virtcontainers/store subsystem=store
DEBU[0000] Creating raw directory                        arch=ppc64le backend=filesystem command=run container=foo name=kata-runtime path=/run/vc/sbs/foo/raw pid=119766 source=virtcontainers/store subsystem=store
DEBU[0000] Creating bridges                              arch=ppc64le command=run container=foo name=kata-runtime pid=119766 source=virtcontainers subsystem=qemu
DEBU[0000] Creating UUID                                 arch=ppc64le command=run container=foo name=kata-runtime pid=119766 source=virtcontainers subsystem=qemu
INFO[0000] Unable to know if the system is running inside a VM  arch=ppc64le command=run container=foo name=kata-runtime pid=119766 source=virtcontainers
DEBU[0000] Disable nesting environment checks            arch=ppc64le command=run container=foo inside-vm=false name=kata-runtime pid=119766 source=virtcontainers subsystem=qemu
DEBU[0000] Restricting maxmem to 32GB as Qemu Version < 2.10, Assumption: Kernel Version >= 4.11  arch=ppc64le command=run container=foo name=kata-runtime pid=119766 source=virtcontainers subsystem=qemu
DEBU[0000] agent: Using vsock VM socket endpoint         arch=ppc64le command=run container=foo name=kata-runtime pid=119766 source=virtcontainers subsystem=kata_agent
DEBU[0000] Could not retrieve anything from storage      arch=ppc64le command=run container=foo name=kata-runtime pid=119766 source=virtcontainers subsystem=kata_agent
WARN[0000] load sandbox devices failed                   arch=ppc64le command=run container=foo error="open /run/vc/sbs/foo/devices.json: no such file or directory" name=kata-runtime pid=119766 sandbox=foo sandboxid=foo source=virtcontainers subsystem=sandbox

<hangs>
root@soe13:~# cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
root@soe13:~# uname -a
Linux soe13 5.0.0-20-generic #21~18.04.1-Ubuntu SMP Thu Jun 27 04:04:19 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
root@soe13:~# lsmod | grep vhost_vsock
vhost_vsock            15884  1
vmw_vsock_virtio_transport_common    32218  1 vhost_vsock
vsock                  39438  2 vmw_vsock_virtio_transport_common,vhost_vsock
vhost                  49369  2 vhost_vsock,vhost_net

root@soe13:~# lscpu
Architecture:         ppc64le
Byte Order:           Little Endian
CPU(s):               160
On-line CPU(s) list:  0,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152
Off-line CPU(s) list: 1-7,9-15,17-23,25-31,33-39,41-47,49-55,57-63,65-71,73-79,81-87,89-95,97-103,105-111,113-119,121-127,129-135,137-143,145-151,153-159
Thread(s) per core:   1
Core(s) per socket:   5
Socket(s):            4
NUMA node(s):         4
Model:                2.1 (pvr 004b 0201)
Model name:           POWER8E (raw), altivec supported
CPU max MHz:          3690.0000
CPU min MHz:          2061.0000
L1d cache:            64K
L1i cache:            32K
L2 cache:             512K
L3 cache:             8192K
NUMA node0 CPU(s):    0,8,16,24,32
NUMA node1 CPU(s):    40,48,56,64,72
NUMA node16 CPU(s):   80,88,96,104,112
NUMA node17 CPU(s):   120,128,136,144,152

@nitkon
Copy link
Contributor Author

nitkon commented Aug 13, 2019

Update: After an hour or so,

ERRO[4413] Could not get a unique context ID for the vsock : ioctl: 1  arch=ppc64le command=run container=foo name=kata-runtime pid=119766 source=runtime
Could not get a unique context ID for the vsock : ioctl: 1

@grahamwhaley
Copy link
Contributor

/cc @stefanha just in case something springs to mind..

@alicefr
Copy link

alicefr commented Aug 14, 2019

@nitkon it sounds to me that after 1 hour there's a timeout and you get back this error. Can you try this small test program in C. It makes the same test as the test64/32.go program

#include <stdio.h>
#include <linux/vhost.h>
#include <errno.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdint.h>

int main() {
	uint64_t cid = 0x3;
	int rc;
	int fd;

	if ((fd = open("/dev/vhost-vsock", O_RDWR, 0666)) < 0) {
		printf("Error in getting fd errno=%d\n", errno);
	}

	rc =  ioctl(fd, VHOST_VSOCK_SET_GUEST_CID, (void*) &cid);

	if (rc != 0){
		printf("FAIL:VHOST_VSOCK_SET_GUEST_CID=%lx fd=%d cid=%lx ERRNO=%d\n",VHOST_VSOCK_SET_GUEST_CID, fd, cid, errno);
		close(fd);
		return 1;
	}
	
	close(fd);
	printf("SUCCESS:fd=%d cid=%lx \n", fd, cid);
	return 0;
}

If it fails I'd suggest that you report it to the Power KVM team

@nitkon
Copy link
Contributor Author

nitkon commented Aug 14, 2019

Thanks, @alicefr. The program returned success.

root@soe13:/home/nitesh# gcc testC.c -o testC
root@soe13:/home/nitesh# ./testC
SUCCESS:fd=3 cid=3 

@alicefr
Copy link

alicefr commented Aug 14, 2019

now I don't get it anymore. I expected this test to fail

@nitkon
Copy link
Contributor Author

nitkon commented Aug 14, 2019

@alicefr : Changed the VHOST_VSOCK_SET_GUEST_CID to 0x8008AF60 and it worked.
I guess this is the change needed in here for ppc64le.

root@soe13:/home/nitesh# gcc testC.c -o testC
root@soe13:/home/nitesh# ./testC 
VHOST_VSOCK_SET_GUEST_CID = 2148052832
SUCCESS:fd=3 cid=3 
root@soe13:/home/nitesh# go run test64.go 
SUCCESS

@grahamwhaley
Copy link
Contributor

I'm having some memories around vsock CIDs and a sign extension ... bug? - @devimc @dagrh @stefanha - do you remember??

@grahamwhaley
Copy link
Contributor

I think this is what I am remembering: https://lkml.org/lkml/2019/1/8/300

@stefanha
Copy link

@grahamwhaley nitkon found that the VHOST_VSOCK_SET_GUEST_CID ioctl constant is incorrect on ppc64le. This makes sense because the kernel code path for that ioctl never returns EPERM, so the code must be executing a different ioctl. Needs to be fixed runtime/virtcontainers/utils/utils_linux.go.

@nitkon
Copy link
Contributor Author

nitkon commented Aug 14, 2019

After fixing the const ioctlVhostVsockSetGuestCid = 0x8008AF60 in the kata-runtime, I am getting the following error now, which I am looking into. Once I fix all the issues, I will send the PR for change in runtime/virtcontainers/utils/utils_linux.go. and I guess I need to enable vsock parameters in the kernel config file as well.

msg="Failed to check if grpc server is working: context deadline exceeded" arch=ppc64le command=create container=0bd50ad2f361858beeb275cbbfdec6948a8061d8aab2d7761383b5635fde8e0a name=kata-runtime pid=13373 source=runtime

More debug logs: https://paste.fedoraproject.org/paste/6wxqscai8ZgJM45xnap1FQ

@nitkon
Copy link
Contributor Author

nitkon commented Aug 14, 2019

Hey @pmorjan @stefanha. FYI The test application that is mentioned here, errors out with connection timed out on Ubuntu 18.04, 5.x kernel-generic ppc64le.

root@soe13:/home/nitesh# go run app.go 
cid          : 0xfff001
message      : connection timed out
errno        : 0x6e (110)
root@soe13:/home/nitesh# uname -a
Linux soe13 5.0.0-20-generic #21~18.04.1-Ubuntu SMP Thu Jun 27 04:04:19 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

Looks like @Pennyzct had hit it too and kernel upgrade had fixed the issue. Not in my case though.

@nitkon
Copy link
Contributor Author

nitkon commented Aug 16, 2019

Possible reason for context deadline exceeded/ VM shutting down from kata-runtime logs post this PR Merge.

Aug 16 12:04:25 soe13 kata-runtime[29120]: time="2019-08-16T12:04:25.734807462+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 name=kata-runtime pid=29120 sandbox=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 source=virtcontainers subsystem=kata_agent vmconsole="calling  vsock_init_tables+0x0/0x58 @ 1"
Aug 16 12:04:25 soe13 kata-runtime[29120]: time="2019-08-16T12:04:25.734878922+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 name=kata-runtime pid=29120 sandbox=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 source=virtcontainers subsystem=kata_agent vmconsole="initcall vsock_init_tables+0x0/0x58 returned 0 after 6 usecs"
Aug 16 12:04:25 soe13 kata-runtime[29120]: time="2019-08-16T12:04:25.734968494+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 name=kata-runtime pid=29120 sandbox=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 source=virtcontainers subsystem=kata_agent vmconsole="calling  vsock_diag_init+0x0/0x34 @ 1"
Aug 16 12:04:25 soe13 kata-runtime[29120]: time="2019-08-16T12:04:25.735042091+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 name=kata-runtime pid=29120 sandbox=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 source=virtcontainers subsystem=kata_agent vmconsole="initcall vsock_diag_init+0x0/0x34 returned 0 after 0 usecs"
Aug 16 12:04:25 soe13 kata-runtime[29120]: time="2019-08-16T12:04:25.735126287+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 name=kata-runtime pid=29120 sandbox=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 source=virtcontainers subsystem=kata_agent vmconsole="calling  virtio_vsock_init+0x0/0xb8 @ 1"
Aug 16 12:04:25 soe13 kata-runtime[29120]: time="2019-08-16T12:04:25.73519767+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 name=kata-runtime pid=29120 sandbox=73129d8b9d55ace3081b361cb8d4bf277c33ceb127fefe03be7d5bf1033550f7 source=virtcontainers subsystem=kata_agent vmconsole="initcall virtio_vsock_init+0x0/0xb8 returned -16 after 13 usecs"

source=virtcontainers subsystem=kata_agent vmconsole="initcall virtio_vsock_init+0x0/0xb8 returned -16 after 13 usecs

Coming from here?

@nitkon
Copy link
Contributor Author

nitkon commented Aug 16, 2019

I rebuilt the kernel with the following config change from upstream kernel config for 4.19.

--- a/kernel/configs/powerpc_kata_kvm_4.19.x
+++ b/kernel/configs/powerpc_kata_kvm_4.19.x
@@ -646,7 +646,7 @@ CONFIG_COMPAT_BINFMT_ELF=y
 CONFIG_ELFCORE=y
 CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
 CONFIG_BINFMT_SCRIPT=y
-# CONFIG_BINFMT_MISC is not set
+CONFIG_BINFMT_MISC=y
 CONFIG_COREDUMP=y
 
 #
@@ -885,7 +885,9 @@ CONFIG_NET_SCH_FIFO=y
 CONFIG_DNS_RESOLVER=y
 # CONFIG_BATMAN_ADV is not set
 # CONFIG_OPENVSWITCH is not set
-# CONFIG_VSOCKETS is not set
+CONFIG_VSOCKETS=y
+CONFIG_VIRTIO_VSOCKETS=y
+CONFIG_VIRTIO_VSOCKETS_COMMON=y
 # CONFIG_NETLINK_DIAG is not set
 # CONFIG_MPLS is not set
 # CONFIG_NET_NSH is not set

But no luck. Getting the same error as mentioned above.

@nitkon
Copy link
Contributor Author

nitkon commented Aug 19, 2019

With standalone qemu command, vsock works on Power :

 /home/nitesh/qemu/ppc64-softmmu/qemu-system-ppc64 --nographic --enable-kvm -vga none -machine pseries -m 16G -device virtio-blk-pci,drive=rootdisk -drive file=/home/nitesh/disk/guest.img,if=none,cache=none,id=rootdisk,format=qcow2 -monitor telnet:127.0.0.1:1234,server,nowait -net nic,model=virtio -net user -device nec-usb-xhci  -device vhost-vsock-pci,guest-cid=3

Guest:

root@soe13-master-node:/# ./nc-vsock -l 1234
Connection from cid 2 port 2220281641...
hi
hello

Host:

oot@soe13:/# ./nc-vsock 3 1234
hi
hello

Where nc-vsock is this

However fails with Kata. Here are the runtime & qemu logs.
Posting all the logs for completeness:

root@soe13:/home/nitesh/split_patch/src/github.com/kata-containers/runtime# cat logs | grep "vsock"
Aug 19 10:42:09 soe13 kata-runtime[27227]: time="2019-08-19T10:42:09.040665243+05:30" level=info msg="vsock supported" arch=ppc64le command=create name=kata-runtime pid=27227 source=katautils
Aug 19 10:42:09 soe13 kata-runtime[27227]: time="2019-08-19T10:42:09.154655675+05:30" level=debug msg="agent: Using vsock VM socket endpoint" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 source=virtcontainers subsystem=kata_agent
Aug 19 10:42:09 soe13 kata-runtime[27227]: time="2019-08-19T10:42:09.220108434+05:30" level=info msg="launching /home/nitesh/qemu/ppc64-softmmu/qemu-system-ppc64 with: [-name sandbox-46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 -uuid c4e7ad21-72e3-46ad-91eb-d6069472df53 -machine pseries,accel=kvm,usb=off,cap-cfpc=broken,cap-sbbc=broken,cap-ibs=broken -cpu host -qmp unix:/run/vc/vm/46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=33280M -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-282308228,guest-cid=282308228,romfile= -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=4,fds=5 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-modern=false,mq=on,vectors=4,romfile= -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/src/linux-4.19.67/vmlinux -initrd /usr/share/kata-containers/kata-containers-initrd-2019-08-14-22:49:58.507190303+0530-88f8216 -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 debug panic=1 nr_cpus=20 agent.use_vsock=true agent.log=debug agent.log=debug initcall_debug -pidfile /run/vc/vm/46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81/pid -D /run/vc/vm/46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81/qemu.log -smp 1,cores=1,threads=1,sockets=20,maxcpus=20]" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 source=virtcontainers subsystem=qmp
Aug 19 10:42:09 soe13 kata-runtime[27227]: time="2019-08-19T10:42:09.39101525+05:30" level=info msg="proxy started" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 proxy-pid=27275 proxy-url="vsock://282308228:1024" sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent
Aug 19 10:42:09 soe13 kata-runtime[27227]: time="2019-08-19T10:42:09.391076493+05:30" level=info msg="New client" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 proxy=27275 source=virtcontainers subsystem=kata_agent url="vsock://282308228:1024"
Aug 19 10:42:13 soe13 kata-runtime[27227]: time="2019-08-19T10:42:13.160855557+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="command line: tsc=reliable no_timer_check rcupdate.rcu_expedited=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 debug panic=1 nr_cpus=20 agent.use_vsock=true agent.log=debug agent.log=debug initcall_debug"
Aug 19 10:42:17 soe13 kata-runtime[27227]: time="2019-08-19T10:42:17.697260656+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="calling  vsockmon_register+0x0/0x34 @ 1"
Aug 19 10:42:17 soe13 kata-runtime[27227]: time="2019-08-19T10:42:17.697367987+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="initcall vsockmon_register+0x0/0x34 returned 0 after 0 usecs"
Aug 19 10:42:17 soe13 kata-runtime[27227]: time="2019-08-19T10:42:17.736606231+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="calling  vhost_vsock_init+0x0/0x50 @ 1"
Aug 19 10:42:17 soe13 kata-runtime[27227]: time="2019-08-19T10:42:17.736768174+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="initcall vhost_vsock_init+0x0/0x50 returned 0 after 229 usecs"
Aug 19 10:42:17 soe13 kata-runtime[27227]: time="2019-08-19T10:42:17.745114488+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="calling  vsock_init_tables+0x0/0x58 @ 1"
Aug 19 10:42:17 soe13 kata-runtime[27227]: time="2019-08-19T10:42:17.745228404+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="initcall vsock_init_tables+0x0/0x58 returned 0 after 6 usecs"
Aug 19 10:42:17 soe13 kata-runtime[27227]: time="2019-08-19T10:42:17.745309658+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="calling  vsock_diag_init+0x0/0x34 @ 1"
Aug 19 10:42:17 soe13 kata-runtime[27227]: time="2019-08-19T10:42:17.745408953+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="initcall vsock_diag_init+0x0/0x34 returned 0 after 0 usecs"
Aug 19 10:42:17 soe13 kata-runtime[27227]: time="2019-08-19T10:42:17.745491987+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="calling  virtio_vsock_init+0x0/0xb0 @ 1"
Aug 19 10:42:17 soe13 kata-runtime[27227]: time="2019-08-19T10:42:17.745614076+05:30" level=debug msg="reading guest console" arch=ppc64le command=create container=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 name=kata-runtime pid=27227 sandbox=46361cb1200168e8a81f992a196304abb6847ea3d4b443659f8d81b0c282fd81 source=virtcontainers subsystem=kata_agent vmconsole="initcall virtio_vsock_init+0x0/0xb0 returned -16 after 14 usecs"

Complete logs:
runtime&qemu.log
kata-collect-data1.log

@jodh-intel
Copy link
Contributor

@nitkon - have you tried kernel tracing (ftrace?) to see if that gives any more clues as to why you get the error?

@nitkon
Copy link
Contributor Author

nitkon commented Aug 19, 2019

@jodh-intel : I will try kernel tracing with ftrace. Meanwhile, I tried what was mentioned here and got ...

modprobe (1861) used greatest stack depth: 12320 bytes left^M
modprobe (1861) used greatest stack depth: 12320 bytes left^M
Key type big_key registered^M
Key type big_key registered^M
console [netcon0] enabled^M
console [netcon0] enabled^M
netconsole: network logging started^M
netconsole: network logging started^M
rtc-generic rtc-generic: setting system clock to 2019-08-19 09:00:51 UTC (1566205251)^M
rtc-generic rtc-generic: setting system clock to 2019-08-19 09:00:51 UTC (1566205251)^M
Freeing unused kernel memory: 2716K^M
Freeing unused kernel memory: 2716K^M
This architecture does not have kernel memory protection.^M
This architecture does not have kernel memory protection.^M
Run /init as init process^M
Run /init as init process^M
announce: name="kata-agent",pid="1",source="agent",version="1.9.0-alpha0-a78e8cfda627cc350dc9d9ca9b969ebb642030c3",device-handlers="scsi,nvdimm,mmioblk,blk,blk-ccw",storage-handlers="scsi,ephemeral,local,9p,virtio-fs,blk,blk-ccw,mmioblk",system-memory="2042420 kB"^M
announce: pid="1",source="agent",name="kata-agent",storage-handlers="virtio-fs,blk,blk-ccw,mmioblk,scsi,ephemeral,local,9p",system-memory="2042420 kB",version="1.9.0-alpha0-a78e8cfda627cc350dc9d9ca9b969ebb642030c3",device-handlers="mmioblk,blk,blk-ccw,scsi,nvdimm"^M

Does this mean the VM boots fine but the agent, which is out init process, fails as soon as it starts executing? Also note that nvdimm/pmem is yet not enabled in qemu on Power. (if it matters)

@jodh-intel
Copy link
Contributor

Hi @nitkon - I don't think that shows enough detail to answer your question. The agent (PID 1) clearly starts as you see the announce message but it's unclear if the agent hangs / blocks or dies. You could try enabling developer mode since that will allow the agent to coredump, which although you cannot inspect via the debug console (as you can only run with agent-as-init) may mean you see a kernel log message detecting the crash.

@nitkon
Copy link
Contributor Author

nitkon commented Aug 21, 2019

Finally got it working with the following changes to kernel config and runtime... phew!

root@soe13:/home/nitesh/split_patch/src/github.com/kata-containers/runtime# cat /etc/kata-containers/configuration.toml | grep "vsock"
# If true and vsocks are supported, use vsocks to communicate directly
use_vsock = true
root@soe13:/home/nitesh/split_patch/src/github.com/kata-containers/runtime# /home/nitesh/qemu/ppc64-softmmu/qemu-system-ppc64 --version
QEMU emulator version 4.0.94 (v4.1.0-rc4-dirty)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
root@soe13:/home/nitesh/split_patch/src/github.com/kata-containers/runtime# git diff
diff --git a/virtcontainers/utils/utils_linux.go b/virtcontainers/utils/utils_linux.go
index efc109e..1501a02 100644
--- a/virtcontainers/utils/utils_linux.go
+++ b/virtcontainers/utils/utils_linux.go
@@ -18,7 +18,7 @@ import (
 
 // from <linux/vhost.h>
 // VHOST_VSOCK_SET_GUEST_CID = _IOW(VHOST_VIRTIO, 0x60, __u64)
-const ioctlVhostVsockSetGuestCid = 0x4008AF60
+const ioctlVhostVsockSetGuestCid = 0x8008AF60
 
 var ioctlFunc = Ioctl
 
root@soe13:/home/nitesh/split_patch/src/github.com/kata-containers/runtime# cd ../packaging/
root@soe13:/home/nitesh/split_patch/src/github.com/kata-containers/packaging# git diff
diff --git a/kernel/configs/powerpc_kata_kvm_4.19.x b/kernel/configs/powerpc_kata_kvm_4.19.x
index 12b2c0f..5084fdd 100644
--- a/kernel/configs/powerpc_kata_kvm_4.19.x
+++ b/kernel/configs/powerpc_kata_kvm_4.19.x
@@ -1,14 +1,15 @@
 #
 # Automatically generated file; DO NOT EDIT.
-# Linux/powerpc 4.19.10 Kernel Configuration
+# Linux/powerpc 4.19.67 Kernel Configuration
 #
 
 #
-# Compiler: gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
+# Compiler: gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
 #
 CONFIG_CC_IS_GCC=y
 CONFIG_GCC_VERSION=70400
 CONFIG_CLANG_VERSION=0
+CONFIG_CC_HAS_ASM_GOTO=y
 CONFIG_IRQ_WORK=y
 CONFIG_BUILDTIME_EXTABLE_SORT=y
 
@@ -483,6 +484,7 @@ CONFIG_ARCH_RANDOM=y
 CONFIG_VIRTUALIZATION=y
 # CONFIG_KVM_BOOK3S_64 is not set
 # CONFIG_VHOST_NET is not set
+# CONFIG_VHOST_VSOCK is not set
 # CONFIG_VHOST_CROSS_ENDIAN_LEGACY is not set
 CONFIG_HAVE_LIVEPATCH=y
 
@@ -646,7 +648,7 @@ CONFIG_COMPAT_BINFMT_ELF=y
 CONFIG_ELFCORE=y
 CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
 CONFIG_BINFMT_SCRIPT=y
-# CONFIG_BINFMT_MISC is not set
+CONFIG_BINFMT_MISC=y
 CONFIG_COREDUMP=y
 
 #
@@ -885,7 +887,10 @@ CONFIG_NET_SCH_FIFO=y
 CONFIG_DNS_RESOLVER=y
 # CONFIG_BATMAN_ADV is not set
 # CONFIG_OPENVSWITCH is not set
-# CONFIG_VSOCKETS is not set
+CONFIG_VSOCKETS=y
+CONFIG_VSOCKETS_DIAG=y
+CONFIG_VIRTIO_VSOCKETS=y
+CONFIG_VIRTIO_VSOCKETS_COMMON=y
 # CONFIG_NETLINK_DIAG is not set
 # CONFIG_MPLS is not set
 # CONFIG_NET_NSH is not set
@@ -1350,6 +1355,7 @@ CONFIG_NET_CORE=y
 # CONFIG_MACVLAN is not set
 # CONFIG_IPVLAN is not set
 # CONFIG_VXLAN is not set
+# CONFIG_GTP is not set
 # CONFIG_MACSEC is not set
 CONFIG_NETCONSOLE=y
 CONFIG_NETPOLL=y
@@ -1578,7 +1584,7 @@ CONFIG_SWPHY=y
 #
 # CONFIG_AMD_PHY is not set
 # CONFIG_AQUANTIA_PHY is not set
-# CONFIG_ASIX_PHY is not set
+# CONFIG_AX88796B_PHY is not set
 # CONFIG_AT803X_PHY is not set
 # CONFIG_BCM7XXX_PHY is not set
 # CONFIG_BCM87XX_PHY is not set
@@ -1766,6 +1772,7 @@ CONFIG_LEGACY_PTY_COUNT=256
 # CONFIG_N_GSM is not set
 # CONFIG_TRACE_SINK is not set
 # CONFIG_PPC_EPAPR_HV_BYTECHAN is not set
+CONFIG_LDISC_AUTOLOAD=y
 CONFIG_DEVMEM=y
 # CONFIG_DEVKMEM is not set
 
@@ -1825,7 +1832,6 @@ CONFIG_HW_RANDOM_TIMERIOMEM=y
 CONFIG_HW_RANDOM_VIRTIO=y
 CONFIG_HW_RANDOM_PSERIES=y
 CONFIG_HW_RANDOM_POWERNV=y
-# CONFIG_R3964 is not set
 # CONFIG_APPLICOM is not set
 
 #

root@soe13:/home/nitesh/split_patch/src/github.com/kata-containers/packaging# docker run -it --runtime kata-runtime fedora /bin/bash
[root@a3030697bd75 /]# uname -a
Linux a3030697bd75 4.19.67 #4 SMP Wed Aug 21 13:24:16 IST 2019 ppc64le ppc64le ppc64le GNU/Linux
exit
root@soe13:/home/nitesh/split_patch/src/github.com/kata-containers/packaging# uname -a
Linux soe13 5.0.0-20-generic #21~18.04.1-Ubuntu SMP Thu Jun 27 04:04:19 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

@bpradipt
Copy link
Contributor

This is stale issue and already fixed upstream. Closing this

egernst pushed a commit to egernst/runtime that referenced this issue Feb 9, 2021

Verified

This commit was signed with the committer’s verified signature.
gortiz Gonzalo Ortiz Jaureguizar
This updates grpc-go vendor package to v1.11.3 release, to fix server.Stop()
handling so that server.Serve() does not wait blindly.

Full commit list:
d11072e (tag: v1.11.3) Change version to 1.11.3
d06e756 clientconn: add support for unix network in DialContext. (kata-containers#1883)
452c2a7 Change version to 1.11.3-dev
d89cded (tag: v1.11.2) Change version to 1.11.2
98ac976 server: add grpc.Method function for extracting method from context (kata-containers#1961)
0f5fa28 Change version to 1.11.2-dev
1e2570b (tag: v1.11.1) Change version to 1.11.1
d28faca client: Fix race when using both client-side default CallOptions and per-call CallOptions (kata-containers#1948)
48b7669 Change version to 1.11.1-dev
afc05b9 (tag: v1.11.0) Change version to 1.11.0
f2620c3 resolver: keep full unparsed target string if scheme in parsed target is not registered (kata-containers#1943)
9d2250f status: rename Status to GRPCStatus to avoid name conflicts (kata-containers#1944)
2756956 status: Allow external packages to produce status-compatible errors (kata-containers#1927)
0ff1b76 routeguide: reimplement distance calculation
dfbefc6 service reflection can lookup enum, enum val, oneof, and field symbols (kata-containers#1910)
32d9ffa Documentation: Fix broken link in rpc-errors.md (kata-containers#1935)
d5126f9 Correct Go 1.6 support policy (kata-containers#1934)
5415d18 Add documentation and example of adding details to errors (kata-containers#1915)
57640c0 Allow storing alternate transport.ServerStream implementations in context (kata-containers#1904)
031ee13 Fix Test: Update the deadline since small deadlines are prone to flakes on Travis. (kata-containers#1932)
2249df6 gzip: Add ability to set compression level (kata-containers#1891)
8124abf credentials/alts: Remove the enable_untrusted_alts flag (kata-containers#1931)
b96718f metadata: Fix bug where AppendToOutgoingContext could modify another context's metadata (kata-containers#1930)
738eb6b fix minor typos and remove grpc.Codec related code in TestInterceptorCanAccessCallOptions (kata-containers#1929)
211a7b7 credentials/alts: Update ALTS "New" APIs (kata-containers#1921)
fa28bef client: export types implementing CallOptions for access by interceptors (kata-containers#1902)
ec9275b travis: add Go 1.10 and run vet there instead of 1.9 (kata-containers#1913)
13975c0 stream: split per-attempt data from clientStream (kata-containers#1900)
2c2d834 stats: add BeginTime to stats.End (kata-containers#1907)
3a9e1ba Reset ping strike counter right before sending out data. (kata-containers#1905)
90dca43 resolver: always fall back to default resolver when target does not follow URI scheme (kata-containers#1889)
9aba044 server: Convert all non-status errors to codes.Unknown (kata-containers#1881)
efcc755 credentials/alts: change ALTS protos to match the golden version (kata-containers#1908)
0843fd0 credentials/alts: fix infinite recursion bug [in custom error type] (kata-containers#1906)
207e276 Fix test race: Atomically access minConnecTimout in testing environment. (kata-containers#1897)
3ae2a61 interop: Add use_alts flag to client and server binaries (kata-containers#1896)
5190b06 ALTS: Simplify "New" APIs (kata-containers#1895)
7c5299d Fix flaky test: TestCloseConnectionWhenServerPrefaceNotReceived (kata-containers#1870)
f0a1202 examples: Replace context.Background with context.WithTimeout (kata-containers#1877)
a1de3b2 alts: Change ALTS proto package name (kata-containers#1886)
2e7e633 Add ALTS code (kata-containers#1865)
583a630 Expunge error codes that shouldn't be returned from library (kata-containers#1875)
2759199 Small spelling fixes (unknow -> unknown) (kata-containers#1868)
12da026 clientconn: fix a typo in GetMethodConfig documentation (kata-containers#1867)
dfa1834 Change version to 1.11.0-dev (kata-containers#1863)
46fd263 benchmarks: add flag to benchmain to use bufconn instead of network (kata-containers#1837)
3926816 addrConn: Report underlying connection error in RPC error (kata-containers#1855)
445b728 Fix data race in TestServerGoAwayPendingRPC (kata-containers#1862)
e014063 addrConn: keep retrying even on non-temporary errors (kata-containers#1856)
484b3eb transport: fix race causing flow control discrepancy when sending messages over server limit (kata-containers#1859)
6c48c7f interop test: Expect io.EOF from stream.Send() (kata-containers#1858)
08d6261 metadata: provide AppendToOutgoingContext interface (kata-containers#1794)
d50734d Add status.Convert convenience function (kata-containers#1848)
365770f streams: Stop cleaning up after orphaned streams (kata-containers#1854)
7646b53 transport: support stats.Handler in serverHandlerTransport (kata-containers#1840)
104054a Fix connection drain error message (kata-containers#1844)
d09ec43 Implement unary functionality using streams (kata-containers#1835)
37346e3 Revert "Add WithResolverUserOptions for custom resolver build options" (kata-containers#1839)
424e3e9 Stream: do not cancel ctx created with service config timeout (kata-containers#1838)
f9628db Fix lint error and typo (kata-containers#1843)
0bd008f stats: Fix bug causing trailers-only responses to be reported as headers (kata-containers#1817)
5769e02 transport: remove unnecessary rstReceived (kata-containers#1834)
0848a09 transport: remove redundant check of stream state in Write (kata-containers#1833)
c22018a client: send RST_STREAM on client-side errors to prevent server from blocking (kata-containers#1823)
82e9f61 Use keyed fields for struct initializers (kata-containers#1829)
5ba054b encoding: Introduce new method for registering and choosing codecs (kata-containers#1813)
4f7a2c7 compare atomic and mutex performance in case of contention. (kata-containers#1788)
b71aced transport: Fix a data race when headers are received while the stream is being closed (kata-containers#1814)
46bef23 Write should fail when the stream was done but context wasn't cancelled. (kata-containers#1792)
10598f3 Explain target format in DialContext's documentation (kata-containers#1785)
08b7bd3 gzip: add Name const to avoid typos in usage (kata-containers#1804)
8b02d69 remove .please-update (kata-containers#1800)
1cd2346 Documentation: update broken wire.html link in metadata package. (kata-containers#1791)
6913ad5 Document that all errors from RPCs are status errors (kata-containers#1782)
8a8ac82 update const order (kata-containers#1770)
e975017 Don't set reconnect parameters when the server has already responded. (kata-containers#1779)
7aea499 credentials: return Unavailable instead of Internal for per-RPC creds errors (kata-containers#1776)
c998149 Avoid copying headers/trailers in unary RPCs unless requested by CallOptions (kata-containers#1775)
8246210 Update version to 1.10.0-dev (kata-containers#1777)
17c6e90 compare atomic and mutex performance for incrementing/storing one variable (kata-containers#1757)
65c901e Fix flakey test. (kata-containers#1771)
7f2472b grpclb: Remove duplicate init() (kata-containers#1764)
09fc336 server: fix bug preventing Serve from exiting when Listener is closed (kata-containers#1765)
035eb47 Fix TestGracefulStop flakiness (kata-containers#1767)
2720857 server: fix race between GracefulStop and new incoming connections (kata-containers#1745)
0547980 Notify parent ClientConn to re-resolve in grpclb (kata-containers#1699)
e6549e6 Add dial option to set balancer (kata-containers#1697)
6610f9a Fix test: Data race while resetting global var. (kata-containers#1748)
f4b5237 status: add Code convenience function (kata-containers#1754)
47bddd7 vet: run golint on _string files (kata-containers#1749)
45088c2 examples: fix concurrent map accesses in route_guide server (kata-containers#1752)
4e393e0 grpc: fix deprecation comments to conform to standard (kata-containers#1691)
0b24825 Adjust keepalive paramenters in the test such that scheduling delays don't cause false failures too often. (kata-containers#1730)
f9390a7 fix typo (kata-containers#1746)
6ef45d3 fix stats flaky test (kata-containers#1740)
98b17f2 relocate check for shutdown in ac.tearDown() (kata-containers#1723)
5ff10c3 fix flaky TestPickfirstOneAddressRemoval (kata-containers#1731)
2625f03 bufconn: allow readers to receive data after writers close (kata-containers#1739)
b0e0950 After sending second goaway close conn if idle. (kata-containers#1736)
b8cf13e Make sure all goroutines have ended before restoring global vars. (kata-containers#1732)
4742c42 client: fix race between server response and stream context cancellation (kata-containers#1729)
8fba5fc In gracefull stop close server transport only after flushing status of the last stream. (kata-containers#1734)
d1fc8fa Deflake tests that rely on Stop() then Dial() not reconnecting (kata-containers#1728)
dba60db Switch balancer to grpclb when at least one address is grpclb address (kata-containers#1692)
ca1b23b Update CONTRIBUTING.md to CNCF CLA
2941ee1 codes: Add UnmarshalJSON support to Code type (kata-containers#1720)
ec61302 naming: Fix build constraints for go1.6 and go1.7 (kata-containers#1718)
b8191e5 remove stringer and go generate (kata-containers#1715)
ff1be3f Add WithResolverUserOptions for custom resolver build options (kata-containers#1711)
580defa Fix grpc basics link in route_guide example (kata-containers#1713)
b7dc71e Optimize codes.String() method using a switch instead of a slice of indexes (kata-containers#1712)
1fc873d Disable ccBalancerWrapper when it is closed (kata-containers#1698)
bf35f1b Refactor roundrobin to support custom picker (kata-containers#1707)
4308342 Change parseTimeout to not handle non-second durations (kata-containers#1706)
be07790 make load balancing policy name string case-insensitive (kata-containers#1708)
cd563b8 protoCodec: avoid buffer allocations if proto.Marshaler/Unmarshaler (kata-containers#1689)
61c6740 Add comments to ClientConn/SubConn interfaces to indicate new methods may be added (kata-containers#1680)
ddbb27e client: backoff before reconnecting if an HTTP2 server preface was not received (kata-containers#1648)
a4bf341 use the request context with net/http handler (kata-containers#1696)
c6b4608 transport: fix race sending RPC status that could lead to a panic (kata-containers#1687)
00383af Fix misleading default resolver scheme comments (kata-containers#1703)
a62701e Eliminate data race in ccBalancerWrapper (kata-containers#1688)
1e1a47f Re-resolve target when one connection becomes TransientFailure (kata-containers#1679)
2ef021f New grpclb implementation (kata-containers#1558)
10873b3 Fix panics on balancer and resolver updates (kata-containers#1684)
646f701 Change version to 1.9.0-dev (kata-containers#1682)

Fixes: kata-containers#307

Signed-off-by: Peng Tao <bergwolf@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.
Projects
None yet
Development

No branches or pull requests

8 participants