Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(integ-runner): integ-runner enhancements #19865

Merged
merged 4 commits into from
Apr 12, 2022
Merged

feat(integ-runner): integ-runner enhancements #19865

merged 4 commits into from
Apr 12, 2022

Conversation

corymhall
Copy link
Contributor

@corymhall corymhall commented Apr 11, 2022

This PR adds a couple of enhancements to the integ-runner

  1. Add --profiles option which allows you to provide a list of
    AWS profiles to use. This will be used in combination with the
    --parallel-regions option to run the integration tests in parallel
    across multiple accounts + regions.
  2. Adds --from-file option that allows you to store a list of tests to
    run (or exclude) in a file, rather than passing test names as
    arguments.
  3. Adds --exclude option, which when provided will excluded the tests
    provided as arguments
  4. Adds some additional metrics. Now each test will show the total time
    in the logs. Also, if --verbose option is provided a summary will
    be printed which will contain metrics for the entire test.
  --- Integration test metrics ---
Profile mb-dev + Region us-east-2 total time: 349.866
  test/integ.bundling.js: 58.75
  test/integ.lambda-insights-mapping.js: 61.012
  test/integ.assets.lit.js: 61.121
  test/integ.lambda.docker.js: 81.936
  test/integ.log-retention.js: 87.047
Profile mb-dev + Region us-west-2 total time: 359.775
  test/integ.layer-version.lit.js: 72.61
  test/integ.autoscaling.lit.js: 82.82
  test/integ.current-version.js: 84.982
  test/integ.lambda.js: 119.363
Profile mb-dev + Region us-east-1 total time: 378.713
  test/integ.assets.file.js: 67.915
  test/integ.runtime.inlinecode.js: 68.211

There are also a couple of small refactors.

  1. Split up workers.test.ts into snapshot-worker.test.ts and integ-worker.test.ts
  2. Refactored the extract_workers functions. There used to be two functions snapshotTestBatch and integTestBatch which called singleThreadedXXX functions. Since each request is essentially single threaded I've moved the singleThreadedXXX functions into extract_worker and renamed them to be integTestWorker and snapshotTestWorker

All Submissions:

Adding new Unconventional Dependencies:

  • This PR adds new unconventional dependencies following the process described here

New Features

  • Have you added the new feature to an integration test?
    • Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

This PR adds a couple of enhancements to the `integ-runner`

1. Add `--profiles` option which allows you to provide a list of
   AWS profiles to use. This will be used in combination with the
   `--parallel-regions` option to run the integration tests in parallel
   across multiple accounts + regions.
2. Adds `--from-file` option that allows you to store a list of tests to
   run (or exclude) in a file, rather than passing test names as
   arguments.
3. Adds `--exclude` option, which when provided will excluded the tests
   provided as arguments
4. Adds some additional metrics. Now each test will show the total time
   in the logs. Also, if `--verbose` option is provided a summary will
   be printed which will contain metrics for the entire test.
```bash
  --- Integration test metrics ---
Profile mb-dev + Region us-east-2 total time: 349.866
  test/integ.bundling.js: 58.75
  test/integ.lambda-insights-mapping.js: 61.012
  test/integ.assets.lit.js: 61.121
  test/integ.lambda.docker.js: 81.936
  test/integ.log-retention.js: 87.047
Profile mb-dev + Region us-west-2 total time: 359.775
  test/integ.layer-version.lit.js: 72.61
  test/integ.autoscaling.lit.js: 82.82
  test/integ.current-version.js: 84.982
  test/integ.lambda.js: 119.363
Profile mb-dev + Region us-east-1 total time: 378.713
  test/integ.assets.file.js: 67.915
  test/integ.runtime.inlinecode.js: 68.211
```
@gitpod-io
Copy link

gitpod-io bot commented Apr 11, 2022

@aws-cdk-automation aws-cdk-automation requested a review from a team April 11, 2022 17:33
@github-actions github-actions bot added the p2 label Apr 11, 2022
@corymhall corymhall requested a review from otaviomacedo April 11, 2022 17:33
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Apr 11, 2022
@corymhall corymhall requested a review from otaviomacedo April 12, 2022 15:02
@mergify
Copy link
Contributor

mergify bot commented Apr 12, 2022

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: 174751a
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify
Copy link
Contributor

mergify bot commented Apr 12, 2022

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit 697fdbe into aws:master Apr 12, 2022
StevePotter pushed a commit to StevePotter/aws-cdk that referenced this pull request Apr 27, 2022
This PR adds a couple of enhancements to the `integ-runner`

1. Add `--profiles` option which allows you to provide a list of
   AWS profiles to use. This will be used in combination with the
   `--parallel-regions` option to run the integration tests in parallel
   across multiple accounts + regions.
2. Adds `--from-file` option that allows you to store a list of tests to
   run (or exclude) in a file, rather than passing test names as
   arguments.
3. Adds `--exclude` option, which when provided will excluded the tests
   provided as arguments
4. Adds some additional metrics. Now each test will show the total time
   in the logs. Also, if `--verbose` option is provided a summary will
   be printed which will contain metrics for the entire test.
```bash
  --- Integration test metrics ---
Profile mb-dev + Region us-east-2 total time: 349.866
  test/integ.bundling.js: 58.75
  test/integ.lambda-insights-mapping.js: 61.012
  test/integ.assets.lit.js: 61.121
  test/integ.lambda.docker.js: 81.936
  test/integ.log-retention.js: 87.047
Profile mb-dev + Region us-west-2 total time: 359.775
  test/integ.layer-version.lit.js: 72.61
  test/integ.autoscaling.lit.js: 82.82
  test/integ.current-version.js: 84.982
  test/integ.lambda.js: 119.363
Profile mb-dev + Region us-east-1 total time: 378.713
  test/integ.assets.file.js: 67.915
  test/integ.runtime.inlinecode.js: 68.211
```

There are also a couple of small refactors.
1. Split up `workers.test.ts` into `snapshot-worker.test.ts` and `integ-worker.test.ts`
2. Refactored the `extract_workers` functions. There used to be two functions `snapshotTestBatch` and `integTestBatch` which called `singleThreadedXXX` functions. Since each request is essentially single threaded I've moved the `singleThreadedXXX` functions into `extract_worker` and renamed them to be `integTestWorker` and `snapshotTestWorker`

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/master/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS. p2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants