[Question]How to execute kubetest

[OlegLoewen]

I'm confused which parameters need to be used for kubetest.

Currently, I'm testing multiple providers (azure, openstack, gce, aws, alicloud) as follows:

I turn up the cluster of desired provider by myself.
Then I run kubetest always in this manner (but different ginkgo focus):

kubetest --provider local --deployment local --test --check-version-skew false --test_args="--ginkgo.skip=Alpha|Experimental|Disruptive|Flaky|\[Driver --ginkgo.focus=\[sig-storage"

This execution runs fine as long as I only run conformance tests. However, currently I'm trying to investigate test coverage SIG groups one by one and facing some issues. Here are a few questions regarding the kubetest usage:

Kubetest results in Error in ./hack/ginkgo-e2e.sh:143

Often kubetest crashes with the following error message:

!!! Error in ./hack/ginkgo-e2e.sh:143
  Error in ./hack/ginkgo-e2e.sh:143. '"${ginkgo}" "${ginkgo_args[@]:+${ginkgo_args[@]}}" "${e2e_test}" -- "${auth_config[@]:+${auth_config[@]}}" --ginkgo.flakeAttempts="${FLAKE_ATTEMPTS}" --host="${KUBE_MASTER_URL}" --provider="${KUBERNETES_PROVIDER}" --gce-project="${PROJECT:-}" --gce-zone="${ZONE:-}" --gce-region="${REGION:-}" --gce-multizone="${MULTIZONE:-false}" --gke-cluster="${CLUSTER_NAME:-}" --kube-master="${KUBE_MASTER:-}" --cluster-tag="${CLUSTER_ID:-}" --cloud-config-file="${CLOUD_CONFIG:-}" --repo-root="${KUBE_ROOT}" --node-instance-group="${NODE_INSTANCE_GROUP:-}" --prefix="${KUBE_GCE_INSTANCE_PREFIX:-e2e}" --network="${KUBE_GCE_NETWORK:-${KUBE_GKE_NETWORK:-e2e}}" --node-tag="${NODE_TAG:-}" --master-tag="${MASTER_TAG:-}" --cluster-monitoring-mode="${KUBE_ENABLE_CLUSTER_MONITORING:-standalone}" --prometheus-monitoring="${KUBE_ENABLE_PROMETHEUS_MONITORING:-false}" --dns-domain="${KUBE_DNS_DOMAIN:-cluster.local}" ${KUBE_CONTAINER_RUNTIME:+"--container-runtime=${KUBE_CONTAINER_RUNTIME}"} ${MASTER_OS_DISTRIBUTION:+"--master-os-distro=${MASTER_OS_DISTRIBUTION}"} ${NODE_OS_DISTRIBUTION:+"--node-os-distro=${NODE_OS_DISTRIBUTION}"} ${NUM_NODES:+"--num-nodes=${NUM_NODES}"} ${E2E_REPORT_DIR:+"--report-dir=${E2E_REPORT_DIR}"} ${E2E_REPORT_PREFIX:+"--report-prefix=${E2E_REPORT_PREFIX}"} "${@:-}"' exited with status 1
Call stack:
  1: ./hack/ginkgo-e2e.sh:143 main(...)
Exiting with status 1

This error message is not very helpful, as I find. I pretty sure this error is a result of individual e2e tests. However,it comes to this crash no junit01.xml file is generated, which is crucial for my e2e test evaluations.

• Is there a way to find out which e2e test is causing this error?
• How to get more error details?
• Is there a way to avoid this? Eventually by executing each test with an individual kubtest call?

What impact does the provider parameter have?

Currently, I'm using --provider local for all clusters (azue, openstack, gce, aws).

• What does this parameter do?
• Do I need to state here the provider of the cluster?
  • There is no openstack provider, right?

What impact does the deployment parameter have?

• If I turn up the cluster by myself, do I still need to provide this parameter?
  • If so, which value?

Is there a way to reduce the CPU load?

Before I run kubetest I'm always running:

make WHAT=test/e2e/e2e.test
make WHAT=cmd/kubectl
make WHAT=vendor/github.com/onsi/ginkgo/ginkgo

Especially make WHAT=test/e2e/e2e.test takes a while. Is there a binary (environment specific) version somewhere located which can be downloaded?

Kind Regards,
Oleg

[OlegLoewen]

/area kubetest

[BenTheElder]

you should consider --provider=skeleton instead, the local provider is actually a reference to hack/local-up-cluster.sh in kubernetes

/assign @krzyzacy

[krzyzacy]

haven't tried local deployer myself... so...
/assign @dims

[OlegLoewen]

you should consider --provider=skeleton instead, the local provider is actually a reference to hack/local-up-cluster.sh in kubernetes

/assign @krzyzacy

@BenTheElder Can you elaborate how this is behaving differently?

[BenTheElder]

see EG this discussion: kubernetes/kubernetes#73402 (comment)

all of the tooling is long aware of "local" and makes assumptions that this means hack/local-up-cluster.sh. skeleton should not have any special assumptions anywhere AFAIK.

[BenTheElder]

(I cannot remember what all of the assumptions are, but I know there are at least some around log dumping, which kubetest does, along with e2e.test)

[OlegLoewen]

Any idea on how to avoid the Error in ./hack/ginkgo-e2e.sh:143error? I mean some individual e2e tests should not break the whole test run, right? My idea is to run the tests several times a day and do some evaluation on them regarding duration, confidence etc.. But if I run kubtest with ginkgo.focus=k8s.io e.g. and some individual tests abort my test run with Error in ./hack/ginkgo-e2e.sh:143error, then I have no junit XML file as result, which is crucial for me.

So my question is, how to fix or ignore the Error in ./hack/ginkgo-e2e.sh:143 error, so that the whole test run is not aborted?

[BenTheElder]

At ./hack/ginkgo-e2e.sh:143 we're to the point of just calling out to ginkgo / e2e.test. Perhaps there is a bug in ginkgo or the e2e tests.

I would make sure to set:

• export KUBERNETES_CONFORMANCE_TEST="y" (some kubetest integration may set it themselves, but not all)
• --provider=skeleton
  Annoyingly, the e2e tests try to do special provider setup / logic without those.

Otherwise, kubetest can't do much about that error, ginkgo is supposed to write the junit output and ./hack/ginkgo-e2e.sh:143 is ginkgo.

[BenTheElder]

(side note: would love to eliminate those scripts, but there's a lot to unpack there, and AFAICT it wouldn't solve your current problem 😞)

[OlegLoewen]

If I explore e2e tests that are not flagged with [Conformance], should I still use export KUBERNETES_CONFORMANCE_TEST="y"?

[BenTheElder]

currently I am, the "kind" deployer always does this -- so far I've only found the bash scripts to read this, the e2e tests should not be.

…

On Mon, Feb 4, 2019 at 11:23 PM Oleg Loewen ***@***.***> wrote: If I explore e2e tests that are not flagged with [Conformance], should I still use export KUBERNETES_CONFORMANCE_TEST="y"? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11038 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4Bq8PWlC0qAC47aKYs4O1XMMQzS7huks5vKTFdgaJpZM4aaBUd> .

[krzyzacy]

/unassign

[OlegLoewen]

If I run kubtest --ginkgo-parallel, do I need to skip [Serial] e2e tests?
If this is the case, than can I just append the e2e.log and junit files auf serial and parallel runs and push them for testgrid? Or would these files then have an unaproriate format?

[BenTheElder]

If I run kubtest --ginkgo-parallel, do I need to skip [Serial] e2e tests?

yes, technically you don't have to but they will be more likely to fail.

If this is the case, than can I just append the e2e.log and junit files auf serial and parallel runs and push them for testgrid? Or would these files then have an unaproriate format?

testgrid can read many junit files, however it think the tests that don't execute in one run will be marked skipped so it will get a bit weird with both passed and skipped entries.

testgrid doesn't care about the log file, that's just for humans to understand the failures

[dims]

this is off to the weeds :)

/close

[k8s-ci-robot]

@dims: Closing this issue.

In response to this:

this is off to the weeds :)

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.