Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

agent hangs using kernel 4.14.36+ #278

Closed
jcvenegas opened this issue Jun 26, 2018 · 3 comments
Closed

agent hangs using kernel 4.14.36+ #278

jcvenegas opened this issue Jun 26, 2018 · 3 comments
Assignees

Comments

@jcvenegas
Copy link
Member

Description of problem

When we try to use kata with the latest kata 4.14.x. The container creation hangs at create command.

Expected result

The agent can be created with a new kernel

Actual result

The kata containers master fails to create kata. This does not happend to kata 1.0.0


Looking to the cause of the issue grpc request CreateSandbox never finish. This is due to the agent in early versions create a nspath using rand golang module https://github.com/kata-containers/agent/blob/master/namespace.go#L52.

rand is failing not returning due to the low-entropy. This is related to https://github.com/kata-containers/agent/blob/master/namespace.go#L52 as pointed in linuxkit/linuxkit#3032.

For this issue I think we should split in two:

  • Remove random dependency at container creation. The ns-paths are only created once so we can change it for a known path.
  • Add a test to make sure rand (/dev/random) works in a container.
  • Improve rand inside a container (use VirtIORNG or rngd if needed).
level=info msg="[\x1b[0;32m  OK  \x1b[0m] Started Kata Containers Agent.\n" name=kata-proxy pid=27048 source=agent
 level=info msg="[\x1b[0;32m  OK  \x1b[0m] Reached target Kata Containers Agent Target.\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:42.964796872Z\" level=info msg=announce device-handlers=\"blk,scsi\" name=kata-agent pid=101 source=agent storage-handlers=\"ephemeral,9p,blk,scsi\" system-memory=\"2045296 kB\" version=1.0.0-f06eb20527f01097f9185a65a6f77e1ed5d2d360\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:42.986904769Z\" level=info msg=\"Enable grpc tracing\" name=kata-agent pid=101 source=agent\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:43.007601883Z\" level=info msg=\"agent grpc server starts\" name=kata-agent pid=101 source=agent\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:43.012611116Z\" level=debug msg=\"New serial channel event\" events=4 name=kata-agent pid=101 source=agent\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:43.068168302Z\" level=debug msg=\"new request\" name=kata-agent pid=101 req=\"interface:<device:\\\"eth0\\\" name:\\\"eth0\\\" IPAddresses:<address:\\\"172.17.0.2\\\" mask:\\\"16\\\" > mtu:1500 hwAddr:\\\"02:42:ac:11:00:02\\\" > \" request=/grpc.AgentService/UpdateInterface source=agent\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:43.073116342Z\" level=info msg=\"Getting interface from MAC address\" interface-name=eth0 mac-address=\"02:42:ac:11:00:02\" name=kata-agent pid=101 source=agent\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:43.103680142Z\" level=info msg=\"Link found\" interface-name=eth0 link=\"&{LinkAttrs:{Index:2 MTU:1500 TxQLen:1000 Name:eth0 HardwareAddr:02:42:ac:11:00:02 Flags:broadcast|multicast RawFlags:4098 ParentIndex:0 MasterIndex:0 Namespace:<nil> Alias: Statistics:0xc4202045d4 Promisc:0 Xdp:0xc4200371c0 EncapType:ether Protinfo:<nil> OperState:down NetNsID:0 NumTxQueues:0 NumRxQueues:0}}\" mac-address=\"02:42:ac:11:00:02\" name=kata-agent pid=101 source=agent\n" name=kata-proxy pid=27048 source=agent
 level=info msg="[    1.340945] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:43.107904525Z\" level=debug msg=\"request end\" duration=38.233392ms name=kata-agent pid=101 request=/grpc.AgentService/UpdateInterface resp=\"device:\\\"eth0\\\" name:\\\"eth0\\\" IPAddresses:<address:\\\"172.17.0.2\\\" mask:\\\"16\\\" > mtu:1500 hwAddr:\\\"02:42:ac:11:00:02\\\" \" source=agent\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:43.120917838Z\" level=debug msg=\"new request\" name=kata-agent pid=101 req=\"routes:<Routes:<gateway:\\\"172.17.0.1\\\" device:\\\"eth0\\\" > Routes:<dest:\\\"172.17.0.0/16\\\" device:\\\"eth0\\\" source:\\\"172.17.0.2\\\" scope:253 > > \" request=/grpc.AgentService/UpdateRoutes source=agent\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:43.126118417Z\" level=debug msg=\"request end\" duration=4.258152ms name=kata-agent pid=101 request=/grpc.AgentService/UpdateRoutes resp=\"Routes:<gateway:\\\"172.17.0.1\\\" device:\\\"eth0\\\" > Routes:<dest:\\\"172.17.0.0/16\\\" device:\\\"eth0\\\" source:\\\"172.17.0.2\\\" scope:253 > \" source=agent\n" name=kata-proxy pid=27048 source=agent
 level=info msg="time=\"2018-06-26T15:43:43.131865383Z\" level=debug msg=\"new request\" name=kata-agent pid=101 req=\"hostname:\\\"8d61a7c4633f\\\" storages:<driver:\\\"9p\\\" source:\\\"kataShared\\\" fstype:\\\"9p\\\" options:\\\"trans=virtio,version=9p2000.L\\\" options:\\\"nodev\\\" options:\\\"msize=8192\\\" mount_point:\\\"/run/kata-containers/shared/containers/\\\" > storages:<driver:\\\"ephemeral\\\" source:\\\"shm\\\" fstype:\\\"tmpfs\\\" options:\\\"noexec\\\" options:\\\"nosuid\\\" options:\\\"nodev\\\" options:\\\"mode=1777\\\" options:\\\"size=67108864\\\" mount_point:\\\"/run/kata-containers/sandbox/shm\\\" > \" request=/grpc.AgentService/CreateSandbox source=agent\n" name=kata-proxy pid=27048 source=agent

@jcvenegas jcvenegas self-assigned this Jun 26, 2018
@sboeuf
Copy link

sboeuf commented Jun 26, 2018

@jcvenegas I agree with 1 and 2, but about 3, I thought this was not working (adding virtio-rng-pci). Did you manage to get it working, or do you mention this as an open question ?

jcvenegas added a commit to jcvenegas/agent that referenced this issue Jun 26, 2018
Using random char generation at early boot may slow the process due to
low entropy.

Also from kernel 4.14.36 rand call hangs due to low-entropy. Lets
remove decency on rand at early sandbox creation.

Fixes: kata-containers#278

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
@amshinde
Copy link
Member

@jcvenegas Sounds good. I had added the randomness in case more than one namespace needed to be created of the same type. But that is not required as of now.

@jcvenegas
Copy link
Member Author

@sboeuf just as open question
@amshinde good , lets handle later if is needed to create another namespace of the same type.

jcvenegas added a commit to jcvenegas/agent that referenced this issue Jun 26, 2018
Using random char generation at early boot may slow the process due to
low entropy.

Also from kernel 4.14.36 rand call hangs due to low-entropy. Lets
remove dependency on rand at early sandbox creation.

Fixes: kata-containers#278

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/agent that referenced this issue Jun 26, 2018
Using random char generation at early boot may slow the process due to
low entropy.

Also from kernel 4.14.36 rand call hangs due to low-entropy. Lets
remove dependency on rand at early sandbox creation.

Depends-on:github.com/kata-containers/runtime#436

Fixes: kata-containers#278

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/agent that referenced this issue Jun 26, 2018
Using random char generation at early boot may slow the process due to
low entropy.

Also from kernel 4.14.36 rand call hangs due to low-entropy. Lets
remove dependency on rand at early sandbox creation.

Depends-on: github.com/kata-containers/runtime#436

Fixes: kata-containers#278

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/agent that referenced this issue Jun 26, 2018
Using random char generation at early boot may slow the process due to
low entropy.

Also from kernel 4.14.36 rand call hangs due to low-entropy. Lets
remove dependency on rand at early sandbox creation.

Depends-on: github.com/kata-containers/runtime#436
Depends-on: github.com/kata-containers/tests#443

Fixes: kata-containers#278

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jcvenegas added a commit to jcvenegas/agent that referenced this issue Jun 26, 2018
Using random char generation at early boot may slow the process due to
low entropy.

Also from kernel 4.14.36 rand call hangs due to low-entropy. Lets
remove dependency on rand at early sandbox creation.

Fixes: kata-containers#278

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
jshachm pushed a commit to jshachm/agent that referenced this issue Nov 22, 2018
Using random char generation at early boot may slow the process due to
low entropy.

Also from kernel 4.14.36 rand call hangs due to low-entropy. Lets
remove dependency on rand at early sandbox creation.

Fixes: kata-containers#278

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants