Skip to content
This repository was archived by the owner on Dec 5, 2017. It is now read-only.

Run e2e tests against mesos/docker #305

Closed
6 tasks done
karlkfi opened this issue May 28, 2015 · 14 comments
Closed
6 tasks done

Run e2e tests against mesos/docker #305

karlkfi opened this issue May 28, 2015 · 14 comments

Comments

@karlkfi
Copy link
Contributor

karlkfi commented May 28, 2015

I want to be able to use kubernetes' e2e tests (make test_e2e) with KUBERNETES_PROVIDER=mesos/docker.

  • Add KUBERNETES_PROVIDER=mesos/docker
    • Add compose scripts to k8s/cluster/mesos/docker
    • Create k8s/cluster/mesos/docker/util.sh that defines required functions
    • Run kube-up & kube-down scripts
  • Run e2e tests against mesos/docker cluster
  • Add e2e tests against mesos/docker cluster to CI

PRs:

@jdef jdef added the tests label May 28, 2015
@karlkfi karlkfi mentioned this issue May 28, 2015
7 tasks
@karlkfi karlkfi self-assigned this May 28, 2015
@karlkfi
Copy link
Contributor Author

karlkfi commented Jun 1, 2015

Note: -check_version_skew=false is required to run the k8s e2e tests against k8sm, because the server was forked from 0.14.2 and still lists that when you hit /version.

go run hack/e2e.go -v -check_version_skew=false --test

@jdef
Copy link

jdef commented Jun 1, 2015

which binaries are being used for e2e tests? if they're generated from the
upstream_k8sm branch then I'm confused as to how that version number is
being used unless we totally borked something in upstream port.

On Mon, Jun 1, 2015 at 2:41 PM, Karl Isenberg notifications@github.com
wrote:

Note: -check_version_skew=false is required to run the k8s e2e tests
against k8sm, because the server was forked from 0.14.2 and still lists
that when you hit /version.

go run hack/e2e.go -v -check_version_skew=false --test


Reply to this email directly or view it on GitHub
#305 (comment)
.

@karlkfi
Copy link
Contributor Author

karlkfi commented Jun 1, 2015

I regenerated the server binaries from upstream_k8sm and get the following:

Client Version: version.Info{Major:"0", Minor:"17.1+", GitVersion:"v0.17.1-dev", GitCommit:"c05b9d50f5d812d86b8dba966eef8ade97d5ca80", GitTreeState:"dirty"}
Server Version: version.Info{Major:"0", Minor:"14.2", GitVersion:"v0.14.2", GitCommit:"", GitTreeState:"not a git tree"}

@karlkfi
Copy link
Contributor Author

karlkfi commented Jun 1, 2015

Ok, I was building the wrong code....
However, -check_version_skew=false is still required because the GitCommit & GitTreeState don't match.

@karlkfi
Copy link
Contributor Author

karlkfi commented Jun 2, 2015

Interesting pods test failure. It doesn't seem flakey, and it doesn't fail on vanilla Kubernetes.

Pods
  should *not* be restarted with a docker exec "cat /tmp/health" liveness probe
  /Users/karl/workspace/k8sm/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:477
[BeforeEach] Pods
  /Users/karl/workspace/k8sm/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:140
>>> testContext.KubeConfig: /Users/karl/.kube/config
[It] should *not* be restarted with a docker exec "cat /tmp/health" liveness probe
  /Users/karl/workspace/k8sm/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:477
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces/e2e-test-9620eb94-08c2-11e5-af69-24a074f20656
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces
STEP: Creating pod liveness-exec in namespace e2e-test-9620eb94-08c2-11e5-af69-24a074f20656
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces/e2e-test-9620eb94-08c2-11e5-af69-24a074f20656/pods
INFO: Waiting up to 5m0s for pod liveness-exec status to be !pending
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces/e2e-test-9620eb94-08c2-11e5-af69-24a074f20656/pods/liveness-exec
INFO: Waiting for pod 'liveness-exec' in namespace 'e2e-test-9620eb94-08c2-11e5-af69-24a074f20656' status to be '"!pending"' (found phase: '"Pending"', readiness: false) (4.65285ms)
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces/e2e-test-9620eb94-08c2-11e5-af69-24a074f20656/pods/liveness-exec
INFO: Saw pod 'liveness-exec' in namespace 'e2e-test-9620eb94-08c2-11e5-af69-24a074f20656' out of pending state (found '"Running"')
STEP: Started pod liveness-exec in namespace e2e-test-9620eb94-08c2-11e5-af69-24a074f20656
STEP: checking the pod's current state and verifying that restartCount is present
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces/e2e-test-9620eb94-08c2-11e5-af69-24a074f20656/pods/liveness-exec
STEP: Initial restart count of pod liveness-exec is 0
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces/e2e-test-9620eb94-08c2-11e5-af69-24a074f20656/pods/liveness-exec
STEP: Restart count of pod liveness-exec in namespace e2e-test-9620eb94-08c2-11e5-af69-24a074f20656 is now 0
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces/e2e-test-9620eb94-08c2-11e5-af69-24a074f20656/pods/liveness-exec
STEP: Restart count of pod liveness-exec in namespace e2e-test-9620eb94-08c2-11e5-af69-24a074f20656 is now 0
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces/e2e-test-9620eb94-08c2-11e5-af69-24a074f20656/pods/liveness-exec
STEP: Restart count of pod liveness-exec in namespace e2e-test-9620eb94-08c2-11e5-af69-24a074f20656 is now 0
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces/e2e-test-9620eb94-08c2-11e5-af69-24a074f20656/pods/liveness-exec
STEP: Restart count of pod liveness-exec in namespace e2e-test-9620eb94-08c2-11e5-af69-24a074f20656 is now 1
STEP: Restart count of pod liveness-exec in namespace e2e-test-9620eb94-08c2-11e5-af69-24a074f20656 increased from 0 to 1 during the test
STEP: deleting the pod
Request URL: http://192.168.59.103:8888/api/v1beta3/namespaces/e2e-test-9620eb94-08c2-11e5-af69-24a074f20656/pods/liveness-exec

• Failure [25.085 seconds]
Pods
/Users/karl/workspace/k8sm/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:679
  should *not* be restarted with a docker exec "cat /tmp/health" liveness probe [It]
  /Users/karl/workspace/k8sm/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:477

  pod liveness-exec in namespace e2e-test-9620eb94-08c2-11e5-af69-24a074f20656 - expected restarts: false, found restarts: true

  /Users/karl/workspace/k8sm/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:92

Note: Request URL: spam is a shim i put in the kubectrl client to print what endpoints it's hitting.

@karlkfi karlkfi changed the title Run e2e tests against docker-compose-mesos Run e2e tests against mesos/docker-compose Jun 2, 2015
@karlkfi
Copy link
Contributor Author

karlkfi commented Jun 3, 2015

The ports allocated to the mesos slaves need to be exposed through docker, via docker-compose. Unfortunately, port ranges aren't supported by docker-compose yet. docker/compose#1102

Until that's fixed, the ports have to be manually generated. So I made a little script to help: https://gist.github.com/karlkfi/8b4777bea3e9fe37c3b1

@karlkfi
Copy link
Contributor Author

karlkfi commented Jun 4, 2015

After patching to work around kubernetes/kubernetes#9155 the e2e tests still fail fairly badly:

Summarizing 20 Failures:

[Fail] EmptyDir volumes [It] should support r/w on tmpfs
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:633

[Fail] EmptyDir volumes [It] volume on tmpfs should have the correct mode
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:633

[Fail] Density [It] [Performance suite] should allow starting 30 pods per node
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/density.go:142

[Fail] Density [It] [Performance suite] should allow starting 3 pods per node
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/density.go:142

[Fail] Cadvisor [It] should be healthy on every node.
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/cadvisor.go:77

[Fail] PreStop [It] should call prestop when killing a pod
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pre_stop.go:141

[Fail] Services [It] should release the load balancer when Type goes from LoadBalancer -> NodePort
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:559

[Fail] Secrets [It] should be consumable from pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:599

[Fail] Kubectl client Update Demo [It] should create and stop a replication controller
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:528

[Fail] DNS [It] should provide DNS for headless services
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/dns.go:197

[Fail] Old networking [It] should function for intra-pod communication
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/networking.go:349

[Fail] New networking [It] should function for intra-pod communication
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/networking.go:136

[Timeout...] Services [It] should serve a basic endpoint from pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:158

[Fail] Kubectl client Update Demo [It] should scale a replication controller
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:528

[Fail] Services [It] should be able to create a functioning NodePort service
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:1049

[Fail] Services [It] should serve multiport endpoints from pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:913

[Fail] Services [It] should be able to change the type and nodeport settings of a service
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:1049

[Fail] Pods [It] should *not* be restarted with a docker exec "cat /tmp/health" liveness probe
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:92

[Fail] ServiceAccounts [It] should mount an API token into pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service_accounts.go:58

[Fail] DNS [It] should provide DNS for the cluster
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/dns.go:56

Ran 61 of 70 Specs in 4192.570 seconds
FAIL! -- 41 Passed | 20 Failed | 2 Pending | 7 Skipped --- FAIL: TestE2E (4192.64s)
FAIL

Ginkgo ran 1 suite in 1h9m54.301268192s

@jdef
Copy link

jdef commented Jun 4, 2015

Is there an easy way to just run one of these e2e tests instead of the
whole suite?

On Wed, Jun 3, 2015 at 11:12 PM, Karl Isenberg notifications@github.com
wrote:

After patching to work around kubernetes/kubernetes#9155
kubernetes/kubernetes#9155 the e2e
tests still fail fairly badly:

Summarizing 20 Failures:

[Fail] EmptyDir volumes [It] should support r/w on tmpfs
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:633

[Fail] EmptyDir volumes [It] volume on tmpfs should have the correct mode
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:633

[Fail] Density [It] [Performance suite] should allow starting 30 pods per node
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/density.go:142

[Fail] Density [It] [Performance suite] should allow starting 3 pods per node
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/density.go:142

[Fail] Cadvisor [It] should be healthy on every node.
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/cadvisor.go:77

[Fail] PreStop [It] should call prestop when killing a pod
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pre_stop.go:141

[Fail] Services [It] should release the load balancer when Type goes from LoadBalancer -> NodePort
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:559

[Fail] Secrets [It] should be consumable from pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:599

[Fail] Kubectl client Update Demo [It] should create and stop a replication controller
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:528

[Fail] DNS [It] should provide DNS for headless services
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/dns.go:197

[Fail] Old networking [It] should function for intra-pod communication
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/networking.go:349

[Fail] New networking [It] should function for intra-pod communication
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/networking.go:136

[Timeout...] Services [It] should serve a basic endpoint from pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:158

[Fail] Kubectl client Update Demo [It] should scale a replication controller
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:528

[Fail] Services [It] should be able to create a functioning NodePort service
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:1049

[Fail] Services [It] should serve multiport endpoints from pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:913

[Fail] Services [It] should be able to change the type and nodeport settings of a service
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:1049

[Fail] Pods [It] should not be restarted with a docker exec "cat /tmp/health" liveness probe
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:92

[Fail] ServiceAccounts [It] should mount an API token into pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service_accounts.go:58

[Fail] DNS [It] should provide DNS for the cluster
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/dns.go:56

Ran 61 of 70 Specs in 4192.570 seconds
FAIL! -- 41 Passed | 20 Failed | 2 Pending | 7 Skipped --- FAIL: TestE2E (4192.64s)
FAIL

Ginkgo ran 1 suite in 1h9m54.301268192s


Reply to this email directly or view it on GitHub
#305 (comment)
.

@karlkfi
Copy link
Contributor Author

karlkfi commented Jun 4, 2015

Ginkgo has a "focus" feature that does pattern matching of the tests by name.

For example, the following runs just the Density tests:

go run hack/e2e.go --v --test --check_version_skew=false --test_args="--ginkgo.focus=Density.*"

@sttts
Copy link

sttts commented Jun 19, 2015

@karlkfi karlkfi changed the title Run e2e tests against mesos/docker-compose Run e2e tests against mesos/docker Jun 24, 2015
@karlkfi
Copy link
Contributor Author

karlkfi commented Jun 24, 2015

e2e tests will first be run with mesos/docker: kubernetes/kubernetes#10049

@karlkfi karlkfi added this to the Cluster Support: Docker milestone Jun 26, 2015
@sttts
Copy link

sttts commented Jun 29, 2015

I have run the e2e tests without Mesos using a docker-compose setup from #383:

[Fail] ServiceAccounts [It] should mount an API token into pods 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service_accounts.go:58

[Fail] Addon update [BeforeEach] should propagate add-on file changes 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/addon_update.go:204

[Fail] PreStop [It] should call prestop when killing a pod 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pre_stop.go:141

[Fail] DNS [It] should provide DNS for services 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/dns.go:139

[Fail] Services [It] should release the load balancer when Type goes from LoadBalancer -> NodePort 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:530

[Fail] Nodes Resize [BeforeEach] should be able to delete nodes. 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:386

[Fail] Nodes Network [BeforeEach] Uncontactable nodes, have their pods recreated by a replication controller, and can host new pods after rejoining. 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:475

[Fail] DNS [It] should provide DNS for the cluster 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/dns.go:139

[Fail] Cadvisor [It] should be healthy on every node. 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/cadvisor.go:77

[Fail] Proxy version v1 [It] should proxy to cadvisor 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/proxy.go:84

[Fail] Proxy version v1beta3 [It] should proxy to cadvisor 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/proxy.go:84

[Fail] Services [It] should be able to change the type and nodeport settings of a service 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:1033

[Fail] Services [It] should be able to create a functioning NodePort service 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:1033

[Fail] Nodes Resize [BeforeEach] should be able to add nodes. 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:386

[Fail] Networking [It] should function for intra-pod communication 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/networking.go:207

@karlkfi
Copy link
Contributor Author

karlkfi commented Aug 24, 2015

e2e tests (with conformance test switch and ginkgo focus option) are in CI, pointing at master and all PRs (raw and merged). They are not running continuously now, but can be made to do so, as soon as tests are sufficiently green that the tests don't take 5 hours to run...

@karlkfi karlkfi closed this as completed Aug 24, 2015
@karlkfi
Copy link
Contributor Author

karlkfi commented Aug 24, 2015

Updated E2E test build in TeamCity to run GINKGO_FOCUS=Guestbook.application by default, which runs in under 5m. Clear the parameter to run the full E2E suite.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants