CRI validation test

[yujuhong]

This was brought in the @kubernetes/sig-node meeting previously. I think having a CRI validation test is necessary eventually to qualify a new CRI shim and help CRI shim developers. It would be good to have such a test sooner than later.

Basically, we want a test framework and a set of tests to validate a CRI (grpc) server alone. For starters, we can just run tests locally on a machine where the grpc server runs. The test only needs to know the socket to talk to, and can perform the basic operations such as creating/deleting sandboxes/containers. Ideally, no runtime-specific bits should be included in the tests.

The CRI validation test should replace/subsume the existing runtime validation test (formerly known as automatic docker validation test: kubernetes/enhancements#57).

@kubernetes/sig-node, since we have multiple ongoing projects implementing CRI shims, this should be a sig-node community effort. If someone already has testing bits in their repository, you're welcome to donate such tests and/or harness. Volunteers to work on the CRI validation test is also welcome.

/cc @dchen1107 @Random-Liu @euank @yifan-gu @mrunalp

[mrunalp]

We have added a testing framework to cri-o using bats that is working quite well for us so far. We would be glad to share those tests. PTAL https://github.com/kubernetes-incubator/cri-o/blob/master/test/ctr.bats as an example.

[feiskyer]

Frakti has setup a framework for cri validation: https://github.com/kubernetes/frakti/tree/master/test/e2e, it reuses same e2e framework from kubernetes.

[Random-Liu]

Frakti has setup a framework for cri validation: https://github.com/kubernetes/frakti/tree/master/test/e2e, it reuses same e2e framework from kubernetes.

What test are you running? :)

[feiskyer]

@Random-Liu Currently sandbox and image APIs in CRI are tested. Container apis will be added later.

[resouer]

+1 for https://github.com/kubernetes/frakti/tree/master/test/e2e which is using k8s e2e framework CCed zjuers who created them in frakti @xlgao-zju

[feiskyer]

@xlgao-zju will work on this.

cc/ @resouer @yujuhong

[Random-Liu]

@feiskyer @resouer Nice.
@xlgao-zju Thanks for your help!

[xlgao-zju]

I will try my best to get this job done. :)

[feiskyer]

The feature has been moved to cri-tools, and kubernetes/enhancements#292.

[feiskyer]

Closing it here.

[feiskyer]

/close