Adding new release-test presubmit job to test helm chart installations of service controllers #85
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RedbackThomson
suggested changes
Aug 11, 2021
Contributor
RedbackThomson
left a comment
RedbackThomson
left a comment
There was a problem hiding this comment.
Thank you for removing the unnecessary code generator parts of this script that otherwise blocked me from enabling it. Have some inline comments in helm-test.sh
Contributor
|
Have some nits, otherwise happy with it. |
Contributor
RedbackThomson
left a comment
RedbackThomson
left a comment
There was a problem hiding this comment.
Just have some nits, otherwise happy with everything!
jaypipes
reviewed
Aug 16, 2021
Co-authored-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
Collaborator
|
/approve |
Collaborator
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: A-Hilaly, jaypipes, vijtrip2 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
RedbackThomson
pushed a commit
to RedbackThomson/ack-test-infra
that referenced
this pull request
Aug 16, 2021
…s of service controllers (aws-controllers-k8s#85) Issues: aws-controllers-k8s/community#863 aws-controllers-k8s/community#885 Description: Creating a new presubmit job "<service-name>-release-test" , which will 1) install the helm chart from service controller repository 2) Validate that installation was successful 3) Validate that service controller started successfully 4) Move installation scripts from community repo to the test-infra repo. aws-controllers-k8s/code-generator#153 aws-controllers-k8s/community#897 **Why a new job? Why not just add this test to existing e2e test?** I think it will good to keep these separate because it provides more clarity to user which part of tests are failing. If both tests are part of same prowjob, service teams will need to dive into prowjob logs to figure out what failed. Having separate jobs will provide more visibility into where to debug faster. Also kind-e2e and release-test are testing two separate things. kind-e2e tests the controller source code while release-test validates the helm artifacts. In addition to this PR, I will also add following functionality in future revision. As part of post submit release job, validate that helm chart values.yaml refers to same image tag as the release semver. ------------------------------------- ``` Following is the output from tests ➜ test-infra git:(helm-test) ✗ make kind-test checking AWS credentials ... ok. loading the images into the cluster ... ok. =============================================================================================================================================================================================================================================checking AWS credentials ... ok. test-helm.sh] Starting Helm Artifacts Test test-helm.sh] installing the Helm Chart ack-ecr-controller in namespace ack-system-test-helm ... ok. test-helm.sh] waiting 10 seconds for ecr controller to start ... ok test-helm.sh] ACK ecr controller pod name is ack-system-test-helm/ack-ecr-controller-84c7fcf88-zhlcl test-helm.sh] Verifying that pod status is in Running state ... ok. test-helm.sh] Verifying that there are no ERROR in controller logs ... ok. test-helm.sh] uninstalling the Helm Chart ack-ecr-controller in namespace ack-system-test-helm ... ok. test-helm.sh] removing ecr crds installed by Helm ... ok. test-helm.sh] deleting ack-system-test-helm namespace ... ok. test-helm.sh] Helm Artifacts Test Finsihed Successfully =============================================================================================================================================================================================================================================loading CRD manifests for ecr into the cluster ... ok. loading common manifests into the cluster ... ok. loading RBAC manifests for ecr into the cluster ... ok. loading service controller Deployment for ecr into the cluster ...ok. generating AWS temporary credentials and adding to env vars map ... ok. =============================================================================================================================================================================================================================================To poke around your test cluster manually: export KUBECONFIG=/Users/vijat/Documents/gocode/src/github.com/aws-controllers-k8s/test-infra/scripts/../build/tmp-ack-test-f908cfd1-58e4e70a/kubeconfig kubectl get pods -A =============================================================================================================================================================================================================================================Building e2e test container for ecr Running e2e test container sha256:c2a817f42aca4ae6ecbf85855b0e43c4007e1c73bc198aa5cb23a6739bd7d87c running python tests locally... INFO:root:Wrote bootstrap to /ecr-controller/tests/e2e/bootstrap.yaml ============================================================================================================ test session starts ============================================================================================================ platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 rootdir: /ecr-controller/tests/e2e plugins: forked-1.3.0, xdist-2.2.0 [gw0] Python 3.8.10 (default, May 4 2021, 19:07:02) -- [GCC 10.2.1 20201203] [gw1] Python 3.8.10 (default, May 4 2021, 19:07:02) -- [GCC 10.2.1 20201203] [gw2] Python 3.8.10 (default, May 4 2021, 19:07:02) -- [GCC 10.2.1 20201203] gw0 [1] / gw1 [1] / gw2 [1] scheduling tests via LoadScheduling tests/test_repository.py::TestRepository::test_smoke [gw0] [100%] PASSED tests/test_repository.py::TestRepository::test_smoke ============================================================================================================= warnings summary ============================================================================================================== bootstrap_resources.py:23 bootstrap_resources.py:23 bootstrap_resources.py:23 /ecr-controller/tests/e2e/bootstrap_resources.py:23: PytestCollectionWarning: cannot collect test class 'TestBootstrapResources' because it has a __init__ constructor (from: tests/test_repository.py) class TestBootstrapResources: -- Docs: https://docs.pytest.org/en/stable/warnings.html ====================================================================================================== 1 passed, 3 warnings in 49.45s ======================================================================================================= To resume test with the same cluster use: " TMP_DIR=/Users/vijat/Documents/gocode/src/github.com/aws-controllers-k8s/test-infra/scripts/../build/tmp-ack-test-f908cfd1-58e4e70a AWS_SERVICE_DOCKER_IMG=aws-controllers-k8s:ecr-14e3dad-dirty " ``` ----------------------------------------------------- ``` ➜ test-infra git:(helm-test) ✗ export ENABLE_E2E_TESTS=false ➜ test-infra git:(helm-test) ✗ make kind-test checking AWS credentials ... ok. loading the images into the cluster ... ok. =============================================================================================================================================================================================================================================checking AWS credentials ... ok. test-helm.sh] Starting Helm Artifacts Test test-helm.sh] installing the Helm Chart ack-ecr-controller in namespace ack-system-test-helm ... ok. test-helm.sh] waiting 10 seconds for ecr controller to start ... ok test-helm.sh] ACK ecr controller pod name is ack-system-test-helm/ack-ecr-controller-84c7fcf88-xjzzn test-helm.sh] Verifying that pod status is in Running state ... ok. test-helm.sh] Verifying that there are no ERROR in controller logs ... ok. test-helm.sh] uninstalling the Helm Chart ack-ecr-controller in namespace ack-system-test-helm ... ok. test-helm.sh] removing ecr crds installed by Helm ... ok. test-helm.sh] deleting ack-system-test-helm namespace ... ok. test-helm.sh] Helm Artifacts Test Finsihed Successfully =============================================================================================================================================================================================================================================Skipping e2e tests because ENABLE_E2E_TESTS is not true. Current value is : false ``` -------------------------------------------- ``` ➜ test-infra git:(helm-test) ✗ export ENABLE_E2E_TESTS=true ➜ test-infra git:(helm-test) ✗ export ENABLE_HELM_CHART_TEST=false ➜ test-infra git:(helm-test) ✗ make kind-test checking AWS credentials ... ok. loading the images into the cluster ... ok. Skipping Helm Chart test because ENABLE_HELM_CHART_TEST is not true. Current value is : false loading CRD manifests for ecr into the cluster ... ok. loading common manifests into the cluster ... ok. loading RBAC manifests for ecr into the cluster ... ok. loading service controller Deployment for ecr into the cluster ...ok. generating AWS temporary credentials and adding to env vars map ... ok. =============================================================================================================================================================================================================================================To poke around your test cluster manually: export KUBECONFIG=/Users/vijat/Documents/gocode/src/github.com/aws-controllers-k8s/test-infra/scripts/../build/tmp-ack-test-f908cfd1-58e4e70a/kubeconfig kubectl get pods -A =============================================================================================================================================================================================================================================Building e2e test container for ecr Running e2e test container sha256:c2a817f42aca4ae6ecbf85855b0e43c4007e1c73bc198aa5cb23a6739bd7d87c running python tests locally... INFO:root:Wrote bootstrap to /ecr-controller/tests/e2e/bootstrap.yaml ============================================================================================================ test session starts ============================================================================================================ platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 rootdir: /ecr-controller/tests/e2e plugins: forked-1.3.0, xdist-2.2.0 [gw0] Python 3.8.10 (default, May 4 2021, 19:07:02) -- [GCC 10.2.1 20201203] [gw1] Python 3.8.10 (default, May 4 2021, 19:07:02) -- [GCC 10.2.1 20201203] [gw2] Python 3.8.10 (default, May 4 2021, 19:07:02) -- [GCC 10.2.1 20201203] gw0 [1] / gw1 [1] / gw2 [1] scheduling tests via LoadScheduling tests/test_repository.py::TestRepository::test_smoke [gw0] [100%] PASSED tests/test_repository.py::TestRepository::test_smoke ============================================================================================================= warnings summary ============================================================================================================== bootstrap_resources.py:23 bootstrap_resources.py:23 bootstrap_resources.py:23 /ecr-controller/tests/e2e/bootstrap_resources.py:23: PytestCollectionWarning: cannot collect test class 'TestBootstrapResources' because it has a __init__ constructor (from: tests/test_repository.py) class TestBootstrapResources: -- Docs: https://docs.pytest.org/en/stable/warnings.html ====================================================================================================== 1 passed, 3 warnings in 50.52s ======================================================================================================= To resume test with the same cluster use: " TMP_DIR=/Users/vijat/Documents/gocode/src/github.com/aws-controllers-k8s/test-infra/scripts/../build/tmp-ack-test-f908cfd1-58e4e70a AWS_SERVICE_DOCKER_IMG=aws-controllers-k8s:ecr-14e3dad-dirty " ``` By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issues:
aws-controllers-k8s/community#863
aws-controllers-k8s/community#885
Description:
Creating a new presubmit job "-release-test" , which will
Why a new job? Why not just add this test to existing e2e test?
I think it will good to keep these separate because it provides more clarity to user which part of tests are failing. If both tests are part of same prowjob, service teams will need to dive into prowjob logs to figure out what failed. Having separate jobs will provide more visibility into where to debug faster.
Also kind-e2e and release-test are testing two separate things. kind-e2e tests the controller source code while release-test validates the helm artifacts.
In addition to this PR, I will also add following functionality in future revision.
As part of post submit release job, validate that helm chart values.yaml refers to same image tag as the release semver.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.