Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Enabling S3 bucket share - COPY #846

Closed
wants to merge 23 commits into from
Closed

Test Enabling S3 bucket share - COPY #846

wants to merge 23 commits into from

Conversation

dlpzx
Copy link
Contributor

@dlpzx dlpzx commented Oct 31, 2023

Feature or Bugfix

  • Feature
  • Bugfix
  • Refactoring

Detail

Dummy PR for reference. Steps needed for #823 to be merged:

  1. merge v2m1m0 into feature branch. Resolve conflict in dataset_service.py. Here is an screenshot of the resulting code (middle code).
    image

  2. revert overwrites in backend/dataall/modules/dataset_sharing/__init__.py

  3. revert overwrites in backend/dataall/modules/dataset_sharing/aws/kms_client.py

  4. revert overwrites in backend/dataall/modules/dataset_sharing/db/share_object_repositories.py

  5. revert overwrites in backend/dataall/modules/dataset_sharing/services/share_managers/lf_share_manager.py

  6. revert overwrites in backend/dataall/modules/dataset_sharing/aws/glue_client.py

  7. revert overwrites in backend/dataall/modules/dataset_sharing/services/share_processors/lf_process_cross_account_share.py

  8. revert overwrites in backend/dataall/modules/dataset_sharing/services/share_processors/lf_process_cross_account_share.py

  9. revert overwrites in backend/dataall/modules/datasets/api/dataset/input_types.py

  10. revert overwrites in backend/dataall/modules/datasets/cdk/pivot_role_datasets_policy.py

  11. revert overwrites in backend/dataall/modules/datasets/services/dataset_service.py

  12. revert overwrites in backend/dataall/modules/datasets/tasks/dataset_subscription_task.py

  13. revert overwrites in tests/modules/datasets/tasks/test_lf_share_manager.py

  14. fix migration script references

I separated each point in a different commit:
image

Relates

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

  • Does this PR introduce or modify any input fields or queries - this includes
    fetching data from storage outside the application (e.g. a database, an S3 bucket)?
    • Is the input sanitized?
    • What precautions are you taking before deserializing the data you consume?
    • Is injection prevented by parametrizing queries?
    • Have you ensured no eval or similar functions are used?
  • Does this PR introduce any functionality or component that requires authorization?
    • How have you ensured it respects the existing AuthN/AuthZ mechanisms?
    • Are you logging failed auth attempts?
  • Are you using or adding any cryptographic features?
    • Do you use a standard proven implementations?
    • Are the used keys controlled by the customer? Where are they stored?
  • Are you introducing any new policies/roles/users?
    • Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

dlpzx and others added 23 commits October 30, 2023 18:21
# Conflicts:
#	deploy/stacks/backend_stack.py
#	deploy/stacks/backend_stage.py
#	deploy/stacks/lambda_api.py
#	deploy/stacks/pipeline.py
#	template_cdk.json
### Feature or Bugfix
- Feature

### Detail
- read KMS keys with an alias prefixed by the environment resource
prefix
- read KMS keys imported in imported datasets
- restrict pivot role policies to the KMS keys created by data.all and
those imported in the imported datasets
- move kms client from data_sharing to base as it is used in
environments and datasets

### Relates
- #580

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

This PR restricts the IAM policies of the pivot role, following the
least privilege permissions principle

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
### Feature or Bugfix
- Bugfix

### Detail
- Make `hosted_zone_id` optional, code update

### Relates
- #797 

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)? N/A
  - Is the input sanitized? N/A
- What precautions are you taking before deserializing the data you
consume? N/A
  - Is injection prevented by parametrizing queries? N/A
  - Have you ensured no `eval` or similar functions are used? N/A
- Does this PR introduce any functionality or component that requires
authorization? N/A
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
N/A
  - Are you logging failed auth attempts? N/A
- Are you using or adding any cryptographic features? N/A
  - Do you use a standard proven implementations? N/A
- Are the used keys controlled by the customer? Where are they stored?
N/A
- Are you introducing any new policies/roles/users? N/A
  - Have you used the least-privilege principle? How? N/A

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license. YES

### Description

Make `hosted_zone_id` optional and provide `HostedZoneId` and `DNSName`
in CloudFormation Stack Output, so users can create their own [Route53
AliasTarget](https://docs.aws.amazon.com/Route53/latest/APIReference/API_AliasTarget.html).

Following validation checks in
`ecs_patterns.ApplicationLoadBalancedFargateService` were considered:
* `frontend_alternate_domain` and `userguide_alternate_domain` have to
be `None` when the `hosted_zone` is `None`, see checks in
[multiple-target-groups-service-base.ts#L463](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/network-multiple-target-groups-service-base.ts#L463),
or else a `A Route53 hosted domain zone name is required to configure
the specified domain name` error is raised
* for a HTTPS ALB listener, only the `certificate` is ultimately
required, and not the `domainName` or `domainZone`, as per evaluation
logic in
[application-load-balanced-service-base.ts#L509](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/application-load-balanced-service-base.ts#L509)
### Feature or Bugfix
- Bugfix

### Detail
- Clean up prints and show better exception message when custom_domain
is not provided for SES

### Relates
- v2.1.0

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
# Conflicts:
#	backend/dataall/modules/datasets/services/dataset_service.py
@dlpzx
Copy link
Contributor Author

dlpzx commented Nov 6, 2023

Closing in favor of #823

@dlpzx dlpzx closed this Nov 6, 2023
@dlpzx dlpzx deleted the anu-s3-copy branch November 8, 2023 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants