FC: Update Firecracker to v0.20.0

[Pennyzct]

The new release for Firecracker is v0.20.0.
There are a few related updates:

• Change logger options type from Value to Vec to prevent potential unwrap on None panics. (PR FC: extract error info from firecracker built-in log and metrics scheme #2073)
• VSock Connection Handshake. (FC: Add new vsock connection handshake agent#706)

Fixes: #2378

Signed-off-by: Penny Zheng penny.zheng@arm.com

[Pennyzct]

It depends on kata-containers/agent#706. And When it landed, we need to update vendor/ to officially include the changes.

[grahamwhaley]

lgtm
So, how do we co-ordinate the update of this, the vendor, and the agent connection check? :-)

[Pennyzct]

Hi @grahamwhaley
FWIT, The correct order should be the agent vsock handshake landed firstly, then we could do vendor update (I'll add another commit here to do it) and FC version bump. ;).

[devimc]

@Pennyzct add Depends-on: github.com/kata-containers/agent#706 in your commit message if you want to test both changes

[Pennyzct]

/test-fc

[Pennyzct]

Hi @devimc
only I included changes in github.com/kata-container/agent#706 under vendor/(it's only for debugging), you could finally see that PR working.

[Pennyzct]

/test-fc

[Pennyzct]

/test-fc

[Pennyzct]

/test-fc

[Pennyzct]

Getting error:

not ok 15 ctr partial line logging
# (in test file ctr.bats, line 540)
#   `[ "$status" -eq 0 ]' failed
# time="2020-01-10T06:10:23Z" level=error msg="File descriptor 3 (pipe:[3154948]) leaked on 
pvdisplay invocation. Parent PID 4437: /tmp/jenkins/workspace/kata-con\nFile descriptor 6 
(/dev/mapper/control) leaked on pvdisplay invocation. Parent PID 4437: 
/tmp/jenkins/workspace/kata-con\n  Failed to find physical volume \"/dev/sdb\".\n" error="exit status 5"

Restart to see if it's reproducible.
Hey guys
@chavafg @grahamwhaley @jodh-intel @devimc @bergwolf @gnawux @lifupan
I've done some digging, and looks like that this error was reported before by @egernst.
I once encountered similar error in one of my arm local machine, and it's because that even if the /dev/sdb entry was there, the real physical volume was never mounted.
Should we need to get into this FC CI machine to lsblk see if it really exists?? 🤯

[Pennyzct]

@teawater

[chavafg]

hi @Pennyzct,
The machine has an sdb device (otherwise, all other cri-o tests would've failed), but that line always appears when there is an error running a cri-o test. The real issue with the test can be seen below:

# time="2020-01-10 06:11:14.831287468Z" level=debug msg="response error: container create failed: Failed to check if grpc server is working: context deadline exceeded\n" file="v1alpha2/api.pb.go:7618" id=e5dc1494-904c-409b-8b04-52529e76c7c9
# time="2020-01-10T06:11:14Z" level=fatal msg="run pod sandbox failed: rpc error: code = Unknown desc = container create failed: Failed to check if grpc server is working: context deadline exceeded\n"
# time="2020-01-10 06:11:14.844093026Z" level=debug msg="request: &ListContainersRequest{Filter:&ContainerFilter{Id:,State:&ContainerStateValue{State:CONTAINER_RUNNING,},PodSandboxId:,LabelSelector:map[string]string{},},}" file="v1alpha2/api.pb.go:7780" id=a4ebbf78-d5f9-46cb-ace5-c59c4d9dd4a2

taken from http://jenkins.katacontainers.io/job/kata-containers-runtime-ubuntu-1804-PR-firecracker/1434/console

I see that the newest CI execution shows that another test failed:

Checking 12 containers have all relevant components
Run kata-runtime: nginx:1.17.0-alpine: 
fa8a9d3a448d25fa1d54932d255368bb1e9ff60a140aa2aaf820fd1f2f961026
docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: context deadline exceeded: unknown.
Checking 13 containers have all relevant components
Wrong number of shims running (13 != 12) - stopping
Wrong number of firecracker running (13 != 12) - stopping
Wrong number of 'runtime list' containers running (13 != 12) - stopping
Wrong number of pods in /run/vc/sbs (13 != 12) - stopping)

seems like same grpc issue on both tests, but I am not sure if this is related with your changes.

[Pennyzct]

@chavafg Hi~
Thanks for the checking and explanation~~~
I've encountered this grpc connection error several times, mostly in this parallel container test:
See one in my another fc-irrelevant PR #2376
http://jenkins.katacontainers.io/job/kata-containers-runtime-ubuntu-1804-PR-firecracker/1427/

docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: context deadline exceeded: unknown.
Checking 4 containers have all relevant components
Wrong number of shims running (4 != 3) - stopping
Wrong number of firecracker running (4 != 3) - stopping
Wrong number of 'runtime list' containers running (4 != 3) - stopping
Wrong number of pods in /run/vc/sbs (4 != 3) - stopping)

[chavafg]

Seems like we are having this failure randomly. I see most jobs of the firecracker CI are constantly failing. Any idea @devimc ?

[devimc]

@chavafg no idea, I think @teawater has played with vsock implementation recently

[WeiZhang555]

@chavafg @devimc The FC job is continually failing randomly, I've encountered the same problem: #2239

[jcvenegas]

@Pennyzct could you upgrade your client code to use the latest changes ? we have reverted some changes and now looks more stable the CI

[Pennyzct]

Hi~ @jcvenegas
I've deleted one DEBUG commit and re-based the code to the latest. ;).

[Pennyzct]

/test

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@bd7d310). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #2379   +/-   ##
=========================================
  Coverage          ?   50.62%           
=========================================
  Files             ?      112           
  Lines             ?    16274           
  Branches          ?        0           
=========================================
  Hits              ?     8239           
  Misses            ?     7020           
  Partials          ?     1015

[Pennyzct]

/test-fc

[Pennyzct]

/test

[Pennyzct]

/test-fc

[Pennyzct]

/test

[jcvenegas]

@Pennyzct I see fc passing now is it ready to merge? cloud hypervisor will fail that until we bump it as well and depends on this PR. we can bump after merge this PR or include the change in this PR, c8741b0

[jcvenegas]

#2440 includes the bump from here (and clh) and point to latest agent client code, both ARM CI and cloud hypervisor are running initrd job seems that is broken

[Pennyzct]

Hi~ @jcvenegas
It's ready for merging.;). ARM CI's failure is irrelevant. I've already raised an issue to elaborate, see kata-containers/ci#245

[jcvenegas]

@Pennyzct woops we need rease

[jcvenegas]

Or merge 2440

[Pennyzct]

Hi~ @jcvenegas no problem, I'll do the rebase asap.;).

[Pennyzct]

/test-fc

[jcvenegas]

/test-ubuntu