Automate Docker Validation Process

[aronchick]

EDITED by @dchen1107 on Sept 23, 2016

Description

Currently, Docker (https://www.docker.com/) is the default container runtime used by Kubernetes. Kubernetes release quality is significantly affected by Docker release quality.

The container technology evolves very rapidly, and so does Docker. Kubernetes can’t stick with a specific Docker version because

• Customers might ask for new features in new Docker release.
• Each Docker release solves some known issues and enhances the overall reliability and performance.

Docker releases per month, and before each release there are also several pre-releases. To ensure Kubernetes compatible with new Docker release, every quarter we had to invest at least 2-week engineer time to manually validate Docker release quality. For example, for 1.3 Kubernetes release, Kubernetes engineer manually validated

• Docker 1.9: Validate docker v1.9.0-rc1 release  kubernetes#16110
• Docker 1.10: Validate Docker 1.10 kubernetes#19720
• Docker 1.11: Validate Docker v1.11 kubernetes#23397

but we still couldn't cover all Docker (pre)release.

Thus, we developed the automated docker validation framework. It is a framework automatically validates newest Docker (pre)release against Kubernetes HEAD everyday, the validation includes functionality validation and performance validation.

Progress Tracker

• Before Alpha: proposed at Automated docker validation tests kubernetes#22765
  • Write and maintain draft quality doc: https://docs.google.com/document/d/1eZHAPG74GR9UkDOdp2GYdOs6_rPaIAS3BS7mtKJ9rj4/edit
    • Identify shepherd: @dchen1107
    • Identify secondary/backup contact point. @liangchenye, @wonderfly
• Before Beta
  • Initial docker micro benchmark: https://github.com/kubernetes/contrib/tree/master/docker-micro-benchmark
• Before Stable
  • Write (code + tests + docs) then get them merged.
    • Test infrastructure work on k8s side:
      • Prep for continuous Docker validation test kubernetes#26813
      • Node E2E: Prep for continuous Docker validation node e2e test kubernetes#28213
      • Change the docker validation node e2e test to use gci-canary-test kubernetes#28927
      • Change SETUP_NODE to True for node e2e docker validation test. kubernetes#29435
    • Jenkins support:
      • Add an e2e job for continuous Docker validation test-infra#113
      • Rework GCI e2e jobs test-infra#173
      • Update the GCI image family used for daily builds test-infra#280
      • Add continuous node e2e docker validation test. test-infra#283
      • Add node e2e version printer for docker validation test. test-infra#304
    • Docker specific test:
      • WIP: add docker validation test kubernetes#25471 [Pending]
    • Runtime Conformance Test:
      • E2etest container conformance test kubernetes#20415
      • image operation conformance test kubernetes#22933
      • Add pod status/ready/restartCount conformance test kubernetes#24191
      • refactor conformance test kubernetes#24329
    • Runtime Performance Test:
      • Add density (batch pods creation latency and resource) and resource performance tests to `test-e2e-node' built for Linux only kubernetes#30026
      • TODO: enable performance test on docker validation infrastructure
• Docs
  • Write user docs and get them merged in.
  • Publish the link to access the test result: https://k8s-testgrid.appspot.com/docker

[idvoretskyi]

cc @kubernetes/sig-node

[timothysc]

Same comment on this one regarding full description.

[dchen1107]

• Built the infrastructure to run node e2e tests against nightly docker build on GCI. Now we test docker nightly builds and generate the reports.

TODO:
Need to enable performance tests to validate docker nightly build too.

[philips]

@dchen1107 Do you have a link to these test runs people should watch?

[dchen1107]

I just updated the initial issue description to reflect the current progress on the project. cc/ @philips @aronchick

There are more to be done, especially on docker specific tests side. But I decided to punt on those since we are working on CRI, and those tests can be very easy to implement against the new API.

[janetkuo]

@Random-Liu @dchen1107 Are the docs ready? Please update the docs in https://github.com/kubernetes/kubernetes.github.io, and then add PR numbers and check the docs box in the issue description

[Random-Liu]

Status Update: The test result will be exposed in kubernetes/test-infra#523.

[Random-Liu]

The validation result is shown in testgrid now. https://k8s-testgrid.appspot.com/docker

[jaredbhatti]

Ping on docs. Can you give an update and/or a docs PR?

[dchen1107]

Status updates:

• Docker performance validation test are done, but only limiting to CREATE operation. DELETE, LIST, INSPECT operations will be added later.
• The test results can be accessed publicly at: https://k8s-testgrid.appspot.com/docker

[jaredbhatti]

@dchen1107 @aronchick @Random-Liu Another ping for docs. Any PRs you can point me to?

[dchen1107]

For validation test framework, I don't think we need any doc except release notes mentioning which docker version being qualified.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close