Agent fails with capability and mount errors

[jodh-intel]

$ sudo docker run -ti --runtime kata-runtime busybox sh            
docker: Error response from daemon: OCI runtime create failed: rpc error: code = Internal desc = Could not resolve symlink for source                                                          
"sh*APATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin*HOSTNAME=ac0fb89e4cce*  
TERM=xterm2/:�                                 
        CAP_CHOWN                              
CAP_DAC_OVERRIDE                               

CAP_FSETID                                     

CAP_FOWNER                                     
        CAP_MKNOD                              

CAP_NET_RAW                                    

CAP_SETGID                                     

CAP_SETUID                                     

CAP_SETFCAP                                    

CAP_SETPCAP                                    
CAP_NET_BIND_SERVICE                           
CAP_SYS_CHROOT                                 
CAP_KILL                                       
CAP_AUDIT_WRITE CAP_CHOWNCAP_DAC_OVERRIDE      
CAP_FSETID                                     
CAP_FOWNER      CAP_MKNOD                      
                         CAP_NET_RAW           
CAP_SETGID                                     
CAP_SETUID                                     
          CAP_SETFCAP                          
                     CAP_SETPCAPCAP_NET_BIND_SERVICECAP_SYS_CHROOCAP_KILLCAP_AUDIT_WRITE        CAP_CHOWNCAP_DAC_OVERRIDE                                                                      
CAP_FSETID                                     
CAP_FOWNER      CAP_MKNOD                      
                         CAP_NET_RAW           
CAP_SETGID                                     
CAP_SETUID                                     
          CAP_SETFCAP                          
                     CAP_SETPCAPCAP_NET_BIND_SERVICECAP_SYS_CHROOCAP_KILLCAP_AUDIT_WRITE"       CAP_CHOWN"CAP_DAC_OVERRIDE"                                                                    
CAP_FSETID"                                    
CAP_FOWNER"     CAP_MKNOD"                     
                          CAP_NET_RAW"         
CAP_SETGID"                                    
CAP_SETUID"                                    
           CAP_SETFCAP"                        
                       CAP_SETPCAP"CAP_NET_BIND_SERVICE"CAP_SYS_CHROOTCAP_KILL"CAP_AUDIT_WRITERdocker-default: unknown. 
$ ls -l /usr/share/kata-containers/kata-containers.img             
lrwxrwxrwx 1 root root 58 Mar  2 13:48 /usr/share/kata-containers/kata-containers.img -> kata-containers-2018-03-02-13:46:48.634266943+0000-ec738d4 

It looks like #133 is the culprit.

[jodh-intel]

Hi @sboeuf - could you take a look please?

[sboeuf]

@jodh-intel I think you need your runtime to integrate the latest virtcontainers PR containers/virtcontainers#626. Also, please modify your configuration.toml so that you define virtio-blk instead of virtio-scsi as the driver to handle your block devices.
I will raise a PR to revendor virtcontainers into the runtime.

[jodh-intel]

Hi @sboeuf - thanks for the info. This is breaking the kata CI fwiw.

/cc @chavafg.

[jodh-intel]

Related: clearcontainers/runtime#1044.

[jodh-intel]

Still blocked on the following:

• CI: use virtio-blk device driver tests#110
• vendor: Update virtcontainers vendoring clearcontainers/runtime#1044

[egernst]

@jodh-intel - I think with the latest round of re-vendoring we should no longer see this issue. Can you confirm?

[jodh-intel]

Yep - we've just seen...

"OCI runtime create failed: rpc error: code = Internal desc = Could not resolve symlink for source 9p: unknown."

This was caused by an agent gRPC protocol change that modified the ordering of the gRPC
Storage message members. The problem being that the runtime (virtcontainers) version of the gRPC protocol didn't match the agents (master) version. Specifically, the error was caused by the agent attempting to mount a 9p FS with a blank mountpoint - in fact due to the change in ordering, the mountpoint was in the mount options.

I've raised #171 to give us a little more of a hint if we hit this again.

[devimc]

@jodh-intel I think this issue was fixed, isn't it?

[jodh-intel]

Yep - closing...