feat(integ-runner): test update path when running tests #19915
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR adds an "update path" workflow to the integration test workflow. When a change is being made to an existing integration test we want to be able to ensure that the change not only works for _new_ stacks, but also with _existing_ stacks. In order to accomplish this, the runner will first deploy the existing snapshot (i.e. `cdk deploy --app /path/to/snapshot`) and then perform a stack update by deploying the integration test. The update path can be disabled by passing the `--disable-update-path` command line option. This PR adds a couple of pieces of functionality in order to enable testing the "update path". 1. The runner will attempt to checkout the snapshot from the origin before running the test. This is to try and make sure that you are actually testing an update of the existing snapshot and not an intermediate snapshot (e.g. if you have been iterating locally). 2. When the runner performs the snapshot diff, it will check for any destructive changes and return that information as a warning to the user. 3. If any destructive changes are found, the runner will record that information as trace metadata in the `manifest.json` file. This will enable us to create automation that checks for this as part of the PR workflow
|
|
| * This assumes that there is an 'origin'. `git remote show origin` returns a list of | ||
| * all branches and we then search for one that starts with `HEAD branch: ` | ||
| */ | ||
| private checkoutSnapshot(): void { |
There was a problem hiding this comment.
What happens when the snapshot hasn't been added to git yet?
There was a problem hiding this comment.
hmm good question. Currently it will throw a warning saying it couldn't checkout the snapshot directory. I think it might be best to just add some info to the README about iterating on new tests, i.e. if you are working on a new test you should run with --disable-update-workflow.
If it's a new test (snapshot hasn't been added to git) you shouldn't be using the update workflow. And I don't think there is a good way for us to automatically figure this out since you could have commited the snapshot and then run the test.
There was a problem hiding this comment.
Added section to readme.
otaviomacedo
approved these changes
Apr 14, 2022
|
Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
|
Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
StevePotter
pushed a commit
to StevePotter/aws-cdk
that referenced
this pull request
Apr 27, 2022
This PR adds an "update path" workflow to the integration test workflow. When a change is being made to an existing integration test we want to be able to ensure that the change not only works for _new_ stacks, but also with _existing_ stacks. In order to accomplish this, the runner will first deploy the existing snapshot (i.e. `cdk deploy --app /path/to/snapshot`) and then perform a stack update by deploying the integration test. The update path can be disabled by passing the `--disable-update-path` command line option. This PR adds a couple of pieces of functionality in order to enable testing the "update path". 1. The runner will attempt to checkout the snapshot from the origin before running the test. This is to try and make sure that you are actually testing an update of the existing snapshot and not an intermediate snapshot (e.g. if you have been iterating locally). 2. When the runner performs the snapshot diff, it will check for any destructive changes and return that information as a warning to the user. 3. If any destructive changes are found, the runner will record that information as trace metadata in the `manifest.json` file. This will enable us to create automation that checks for this as part of the PR workflow Added logic to include the `allowDestroy` option in items 2 & 3 above. If a resource type is included in the `allowDestroy` list, then it will not be reported. ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/master/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds an "update path" workflow to the integration test
workflow. When a change is being made to an existing integration test we
want to be able to ensure that the change not only works for new
stacks, but also with existing stacks. In order to accomplish this,
the runner will first deploy the existing snapshot (i.e.
cdk deploy --app /path/to/snapshot) and then perform a stack update by deployingthe integration test.
The update path can be disabled by passing the
--disable-update-pathcommand line option.
This PR adds a couple of pieces of functionality in order to enable
testing the "update path".
before running the test. This is to try and make sure that you are
actually testing an update of the existing snapshot and not an
intermediate snapshot (e.g. if you have been iterating locally).
destructive changes and return that information as a warning to the
user.
information as trace metadata in the
manifest.jsonfile. This willenable us to create automation that checks for this as part of the PR
workflow
Added logic to include the
allowDestroyoption in items 2 & 3 above.If a resource type is included in the
allowDestroylist, thenit will not be reported.
All Submissions:
Adding new Unconventional Dependencies:
New Features
yarn integto deploy the infrastructure and generate the snapshot (i.e.yarn integwithout--dry-run)?By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license