Kms explosion fix #882

anushka-singh · 2023-11-17T04:47:14Z

Feature or Bugfix

Bugfix

Detail

DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name.
We want to collapse these SIDs into one SID.
Access point and Bucket share will have different SIDs in KMS policy.
Use role ARN instead of role ID.
NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though.
NOTE: This is to be merged after bucket share PR is merged.

Testing

Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt.

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

Does this PR introduce or modify any input fields or queries - this includes
fetching data from storage outside the application (e.g. a database, an S3 bucket)?
- Is the input sanitized?
- What precautions are you taking before deserializing the data you consume?
- Is injection prevented by parametrizing queries?
- Have you ensured no eval or similar functions are used?
Does this PR introduce any functionality or component that requires authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
- Are you logging failed auth attempts?
Are you using or adding any cryptographic features?
- Do you use a standard proven implementations?
- Are the used keys controlled by the customer? Where are they stored?
Are you introducing any new policies/roles/users?
- Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…ata-dot-all#748) ### Feature or Bugfix  - Feature Enhancement ### Detail - Adding additional error messages for KMS Key lookup when importing a new dataset - 1 Error message to determine if the KMS Key Alias Exists - 1 Error message to determine if the PivotRole has permissions to describe the KMS Key ### Relates - data-dot-all#712 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

### Feature or Bugfix  - NA ### Detail - Get latest code in `main` to `v2m1m0` branch to keep in sync ### Relates - NA ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). NA ``` - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? ``` By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com>

### Feature or Bugfix  - Enahncement / Bugfix ### Detail - When creating an environment and specifying default Env IAM Role we assume it is of the structure `arn:aws:iam::ACCOUNT:role/NAME_SPECIFIED` - This does not work when there is a service path in the role arn such as with SSO: `arn:aws:iam::ACCOUNT:role/sso/NAME_SPECIFIED` - Causes issues when importing an IAM Role for an invited Team in an environment and/or with dataset sharing - This PR takes in the full IAM role ARN when importing the IAM role in order to correctly determine the role name ### Relates - [data-dot-all#695 ](data-dot-all#695) ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

### Feature or Bugfix  - Enhancement / Bugfix ### Detail - Ensure the names passed for OpenSearch Domain and OpenSearch Serverless Collection, Access Policy, Security Policy, and VPC Endpoint all follow naming conventions required by the service, meaning - The name must start with a lowercase letter - Must be between 3 and 28 characters - Valid characters are a-z (lowercase only), 0-9, and - (hyphen). ### Relates - data-dot-all#540 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: dlpzx <dlpzx@amazon.com>

# Conflicts: # deploy/app.py

### Feature or Bugfix Update ### Detail ### Relates See data-dot-all#655: > In Nov 27, 2023 the Lambda runtime node14 and Python3.7 will be deprecated! Checked all lambdas that explicitly set the runtime engine: only cognito httpheader redirection lambda used node14. All lambdas use python3.8 and node16 or node18. For cdk dependencies: upgraded to a newest `aws-cdk-lib` `v2.99.0` just in case if python3.7 is hardcoded somewhere inside of 2.78.0 (shouldn't be) ### Testing: - [x] uploaded the changes to my isengard account - [x] deployment is green - [x] could access app page, userguide page, and userguide from the app page. ### Security `N/A` - upgraded to a newer version of node js By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

### Feature or Bugfix - Feature ### Detail The guiding principle is that: 1. dataset IAM role is the role accessing data 2. pivot role is the role used by the central account to perform SDK calls in the environment account In this PR we - Replace pivot role by dataset role in dataset Lake Formation registration - Use pivot role to trigger upload files feature and create folder feature, but use the dataset IAM role to perform the putObject operations-> removes the need for read and `putObject` permissions. for the pivot role - Redefine pivot role CDK stack to manage S3 buckets (bucket policies) for only the datasets S3 buckets that have been created or imported in the environment. - implement IAM policy utils to handle the new dynamic policies. We need to verify that the created policy statements do not exceed the maximum policy size. In addition we replace the previous "divide in chunks of 10 statements" by a function that divides in chunks based on the size of the policy statements. This way we optimize the policy size, which helps us in reducing the number of managed policies attached to the pivot role. --> it can be re-used in other "chunkenization" of policies - We did not implement force update of environments (pivot role nested stack) with new datasets added because it is already forced in `backend/dataall/modules/datasets/services/dataset_service.py` ### Backwards compatibility Testing Pre-update setup: - 1 environment (auto-created pivot role) - 2 datasets in environment, 1 created, 1 imported: with tables and folders - Run profiling jobs in tables Update with the branch changes: - [X] CICD pipeline runs successfully - [X] Click update environment on environment -> successfully updated policy of pivot role with imported datasets in policy. Reduction of policies - [X] Click update datasets --> registration in Lake formation updated to dataset role - [X] Update files works - [X] Create folder works - [X] Crawler and profiling jobs work ### Relates - data-dot-all#580 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Are you introducing any new policies/roles/users? `Yes` - Have you used the least-privilege principle? How? `In this PR we restrict the permissions of the pivot role, a super role that handles SDK calls in the environment accounts. Instead of granting permissions to all S3 buckets, we restrict it to data.all handled S3 buckets only` By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…eation (data-dot-all#781) ### Feature or Bugfix - Feature - Bugfix ### Detail The different alternatives considered are discussed in data-dot-all#556 This PR introduces a new query `listValidEnvironments` that replaces the query `listEnvironments` for certain operations. `listEnvironments` - lists all environments independently of their CloudFormation stack statys with a lot of additional details `listValidEnvironments` - lists only "CloudFormation" stable and successful environments. Retrieves only basic info about the environment. Operations such as opening a share request or creation a Dataset/Notebook/etc require the selection of an environment. The environment options are now retrieved from `listValidEnvironments` ensuring that only valid environments are selectable. Moreover, this query is more light and does not need to query and obtain as many fields as the original `listEnvironments`, improving the efficiency of the code. ### Relates - data-dot-all#556 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

### Feature or Bugfix  - Feature ### Detail Allows user to configure a session timeout . Today data.all by default sets the refresh token to 30 days but with this change it becomes configurable ### Relates data-dot-all#421 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Manjula <manjula.kasturi@gmail.com>

### Feature or Bugfix - Feature - Bugfix ### Detail As explained in the [semgrep docs](https://semgrep.dev/docs/cheat-sheets/python-command-injection/#1b-shelltrue): "Functions from the subprocess module have the shell argument for specifying if the command should be executed through the shell. Using shell=True is dangerous because it propagates current shell settings and variables. This means that variables, glob patterns, and other special shell features in the command string are processed before the command is run, making it much easier for a malicious actor to execute commands. The subprocess module allows you to start new processes, connect to their input/output/error pipes, and obtain their return codes. Methods such as Popen, run, call, check_call, check_output are intended for running commands provided as an argument ('args'). Allowing user input in a command that is passed as an argument to one of these methods can create an opportunity for a command injection vulnerability." In our case the risk is not exposed as no user input is directly taken into the subprocess commands. Nevertheless we should strive for the highest standards on security and this PR works on replacing all the `shell=True` executions in the data.all code. In this PR: - when possible we have set `shell=False` - in cases where the command was too complex a `CommandSanitizer` ensures that the input arguments are strings following the regex=`[a-zA-Z0-9-_]` Testing: - [X] local testing - deployment of any stack (`backend/dataall/base/cdkproxy/cdk_cli_wrapper.py`) - [X] local testing - deployment of cdk pipeline stack (`backend/dataall/modules/datapipelines/cdk/datapipelines_cdk_pipeline.py`) - [X] local testing - deployment of codepipeline pipeline stack (`backend/dataall/modules/datapipelines/cdk/datapipelines_pipeline.py`) - [ ] AWS testing - deployment of data.all - [ ] AWS testing - deployment of any stack (`backend/dataall/base/cdkproxy/cdk_cli_wrapper.py`) - [ ] AWS testing - deployment of cdk pipeline stack (`backend/dataall/modules/datapipelines/cdk/datapipelines_cdk_pipeline.py`) - [ ] AWS testing - deployment of codepipeline pipeline stack (`backend/dataall/modules/datapipelines/cdk/datapipelines_pipeline.py`) ### Relates - data-dot-all#738 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? ---> 🆗 This is exactly what this PR is trying to do - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…uester (data-dot-all#793) ### Feature or Bugfix - Bugfix ### Detail - Allowing to submit a share when you are both an approver and a requester ### Security **DOES NOT APPLY** Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com>

### Feature or Bugfix - Feature ### Detail Adding a redirect to the share UI once a share object is created. Additionally updating the breadcrumb message to more clearly indicate that a "Draft share request is created" rather than suggesting that the share has actually been sent to the data owners team. ### Relates N/A ### Security N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com>

### Feature or Bugfix Fix data-dot-all#792: Fix: condition when there are no public subnets ---------

### Feature or Bugfix - Feature ### Detail - Removing unused variable in local graphql server pointing to a fixed AWS region ### Relates N/A ### Security N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com>

### Feature or Bugfix - Feature ### Detail - For a dataset to make sense all the tables within a dataset should have their location pointing to the same place as the dataset S3 bucket. However it is possible that a database can have tables which do not point to the same bucket which is perfectly legal in LakeFormation. Therefore we propose that data.all automatically only lists tables that have the same S3 bucket location as the dataset. This will solve a problem for Yahoo where we want to import a database that contains many tables with different buckets. Additionally Catalog UI should also only list prefiltered tables. ### Testing - Tested this in local env. I was able to create and share datasets even after pre-filtering process takes place. - Will send separate PR for unit testing. ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com>

### Feature or Bugfix  - Bugfix ### Detail - Fix method to detect if other share objects exist on the environment before cleaning up environment-level shared resources (i.e. RAM invitation and PivotRole permissions) - Originally, if TeamA in EnvA had 2 shares approved and succeeded on DatasetB and TeamA rejects 1 of the pre-existing shares, the method `other_approved_share_object_exists` was returning `False`and deleting necessary permissions for the other existing Share - Also disables the other existing shares ability to Revoke the still existing share since pivotRole no longer has permissions - Also fixes the removal of dataall QS Group permissions if there are still existing shares to EnvA ### Security NA ``` Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? ``` By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com>

### Feature or Bugfix - Feature ### Detail Whenever a share request is created and transitions from states ( approved, revoked, etc ) a notification is created. This notification is displayed on the bell icon on the UI . We want such a similar notification to be sent to the dataset owner, requester, etc via email Please take a look at Github Issue 734 For more details - data-dot-all#734 ### Relates - data-dot-all#734 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? No - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? No - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? No - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? Yes - Have you used the least-privilege principle? How? --> **Permission granted for SES:sendEmail to Lambda on resources - (Ses identity and configuration set ) , Also created KMS and SNS for SES setup to handle email bounces . Used least privleged and restricted access on both whenever required. ** By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com>

### Feature or Bugfix - Feature ### Detail - Adding frontend support for all feature flags defined in config.json with a new util method isFeatureEnabled - Adding a new flag **preview_data** in the datasets module to control whether previewing data is allowed - Adding a new flag **glue_crawler** in the datasets module to control whether running glue crawler is allowed - Updating environment features to be hidden or visible based on whether the module is active. Adding a new util isAnyFeatureModuleEnabled to check whether to render the entire feature box. ### Relates N/A ### Security Not relevant By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com>

@TejasRGitHub

### Feature or Bugfix - Refactoring ### Detail As a rule of thumb, we encourage customization of `modules` while changes in `core` should be avoided when possible. `notifications` is a component initially in core which is only used by `dataset_sharing`. To facilitate customization of the `notifications` module and also to clearly see its dependencies we have: - Moved `notifications` code from core to modules as it is a reusable component that is not needed by any core component. - Moved dataset_sharing references inside dataset_sharing module and left `notifications` independent from any other module (done mostly in data-dot-all#734, so credits to @TejasRGitHub) - Added depends_on in the dataset_sharing module to load notifications if the data_sharing module is imported. - Modified frontend navigation bar to make it conditional of the notifications module - Added migration script to modify the notification type column - Fix tests from data-dot-all#734, some references on the payload of the notification tasks were wrong - Small fixes to SES stack: added account in KMS policy and email_id as input ### [WIP] Testing Local testing - [ ] loading of notifications with datasets enabled - [ ] ... AWS testing - [ ] CICD pipeline succeds ### Other remarks Not for this PR, but as a general note, we should clean up deprecated ECS tasks ### Relates - data-dot-all#785 - data-dot-all#734 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). `N/A` just refactoring By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

# Conflicts: # deploy/stacks/backend_stack.py # deploy/stacks/backend_stage.py # deploy/stacks/lambda_api.py # deploy/stacks/pipeline.py # template_cdk.json

### Feature or Bugfix - Feature ### Detail - read KMS keys with an alias prefixed by the environment resource prefix - read KMS keys imported in imported datasets - restrict pivot role policies to the KMS keys created by data.all and those imported in the imported datasets - move kms client from data_sharing to base as it is used in environments and datasets ### Relates - data-dot-all#580 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). This PR restricts the IAM policies of the pivot role, following the least privilege permissions principle - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

### Feature or Bugfix - Bugfix ### Detail - Make `hosted_zone_id` optional, code update ### Relates - data-dot-all#797 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? N/A - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. YES ### Description Make `hosted_zone_id` optional and provide `HostedZoneId` and `DNSName` in CloudFormation Stack Output, so users can create their own [Route53 AliasTarget](https://docs.aws.amazon.com/Route53/latest/APIReference/API_AliasTarget.html). Following validation checks in `ecs_patterns.ApplicationLoadBalancedFargateService` were considered: * `frontend_alternate_domain` and `userguide_alternate_domain` have to be `None` when the `hosted_zone` is `None`, see checks in [multiple-target-groups-service-base.ts#L463](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/network-multiple-target-groups-service-base.ts#L463), or else a `A Route53 hosted domain zone name is required to configure the specified domain name` error is raised * for a HTTPS ALB listener, only the `certificate` is ultimately required, and not the `domainName` or `domainZone`, as per evaluation logic in [application-load-balanced-service-base.ts#L509](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/application-load-balanced-service-base.ts#L509)

### Feature or Bugfix - Bugfix ### Detail - Clean up prints and show better exception message when custom_domain is not provided for SES ### Relates - v2.1.0 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

# Conflicts: # deploy/stacks/backend_stack.py # deploy/stacks/backend_stage.py # deploy/stacks/lambda_api.py # deploy/stacks/pipeline.py # template_cdk.json

### Feature or Bugfix - Feature ### Detail - read KMS keys with an alias prefixed by the environment resource prefix - read KMS keys imported in imported datasets - restrict pivot role policies to the KMS keys created by data.all and those imported in the imported datasets - move kms client from data_sharing to base as it is used in environments and datasets ### Relates - data-dot-all#580 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). This PR restricts the IAM policies of the pivot role, following the least privilege permissions principle - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

### Feature or Bugfix - Bugfix ### Detail - Make `hosted_zone_id` optional, code update ### Relates - data-dot-all#797 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? N/A - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. YES ### Description Make `hosted_zone_id` optional and provide `HostedZoneId` and `DNSName` in CloudFormation Stack Output, so users can create their own [Route53 AliasTarget](https://docs.aws.amazon.com/Route53/latest/APIReference/API_AliasTarget.html). Following validation checks in `ecs_patterns.ApplicationLoadBalancedFargateService` were considered: * `frontend_alternate_domain` and `userguide_alternate_domain` have to be `None` when the `hosted_zone` is `None`, see checks in [multiple-target-groups-service-base.ts#L463](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/network-multiple-target-groups-service-base.ts#L463), or else a `A Route53 hosted domain zone name is required to configure the specified domain name` error is raised * for a HTTPS ALB listener, only the `certificate` is ultimately required, and not the `domainName` or `domainZone`, as per evaluation logic in [application-load-balanced-service-base.ts#L509](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/application-load-balanced-service-base.ts#L509)

### Feature or Bugfix - Bugfix ### Detail - Clean up prints and show better exception message when custom_domain is not provided for SES ### Relates - v2.1.0 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…into bucket_share

…nto bucket_share_anushka

dlpzx · 2023-11-21T09:56:49Z

Hi @anushka-singh looking at the PR, it includes 2 main changes: (1) the compression of multiple statements into one and (2) the replacement of role id by role arns in the KMS policy.

The reason for (1) is very clear, you want to optimize the policy size, but it is not clear to me why you want to change the role ids. I am not saying it is wrong, I just want to better understand your motivation. The big advantage of role_arns is that they are human-readable, while role_ids need to be fetched. The advantage of role_ids is that they are unique. In case of role deletion, the deleted-role and the new-role will have the same role_arn but different role_ids. This is used to ensure that access is granted only to the "correct" role

backend/dataall/modules/dataset_sharing/services/share_managers/s3_bucket_share_manager.py

...end/dataall/modules/dataset_sharing/services/share_managers/s3_access_point_share_manager.py

tests/modules/datasets/tasks/test_s3_access_point_share_manager.py

dlpzx · 2023-11-21T10:59:53Z

Very few comments, nice job @anushka-singh. I am testing the changes in AWS:

CICD updates successfully
Approve Bucket share and Access point share in Dataset with SSE-S3 encryption
Approve Bucket share and Access point share in Dataset with KMS encryption
Revoke Bucket share and Access point share in Dataset with SSE-S3 encryption
Revoke Bucket share and Access point share in Dataset with KMS encryption --> Error

Notes:

Regarding "NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though." I have seen that the policy does not delete the statements for the roleId, instead it adds the new DataAll-Access-Point-KMS-Decrypt and DataAll-Bucket-KMS-Decrypt statements

Errors:
Error 1
KMS policy in access point revoke still contains DataAll-Access-Point-KMS-Decrypt statements

Line 212 still evaluates to True because there is an existing shared bucket. So even if there are no folders shared the function delete_dataset_bucket_key_policy won't be called

anushka-singh · 2023-11-21T20:41:26Z

Hi @anushka-singh looking at the PR, it includes 2 main changes: (1) the compression of multiple statements into one and (2) the replacement of role id by role arns in the KMS policy.

The reason for (1) is very clear, you want to optimize the policy size, but it is not clear to me why you want to change the role ids. I am not saying it is wrong, I just want to better understand your motivation. The big advantage of role_arns is that they are human-readable, while role_ids need to be fetched. The advantage of role_ids is that they are unique. In case of role deletion, the deleted-role and the new-role will have the same role_arn but different role_ids. This is used to ensure that access is granted only to the "correct" role

Yes the reason is human readability. Users complained that they couldnt easily see who the role belonged to. If they transitioned from old kms policy where you manually managed it to a new one, theyd want to see all new roles added by data all and compare with old ones they had already.
The other thing is if youre troubleshooting or checking why access is not working, having role arns helps because you can instantly identify which role name is missing and proceed with solving the problem. Otherwise we would spend extra time to figure how to convert name to id etc.
These were the reasons that prompted us to move to role arn.

anushka-singh · 2023-11-21T20:41:43Z

Very few comments, nice job @anushka-singh. I am testing the changes in AWS:

CICD updates successfully

Approve Bucket share and Access point share in Dataset with SSE-S3 encryption

Approve Bucket share and Access point share in Dataset with KMS encryption

Revoke Bucket share and Access point share in Dataset with SSE-S3 encryption

Revoke Bucket share and Access point share in Dataset with KMS encryption --> Error

Notes:

Regarding "NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though." I have seen that the policy does not delete the statements for the roleId, instead it adds the new DataAll-Access-Point-KMS-Decrypt and DataAll-Bucket-KMS-Decrypt statements

Errors: Error 1 KMS policy in access point revoke still contains DataAll-Access-Point-KMS-Decrypt statements

Line 212 still evaluates to True because there is an existing shared bucket. So even if there are no folders shared the function delete_dataset_bucket_key_policy won't be called

Addressed this comment in next revision, please take a look!

dlpzx · 2023-11-23T07:43:05Z

This time I am testing locally:

Approve Bucket share and Access point share in Dataset with SSE-S3 encryption---> Error KMS key not found! Explained in review of dataall/modules/dataset_sharing/services/share_managers/s3_access_point_share_manager.py
Approve Bucket share and Access point share in Dataset with KMS encryption
Revoke Bucket share and Access point share in Dataset with SSE-S3 encryption
Revoke Bucket share and Access point share in Dataset with KMS encryption

I tried all possible combinations when approving and revoking: only Bucket, only folder, both at the same time. With 2 different shares. So I also tested addition/deletion of one role in the KMS policy and the complete clean-up when both shares are revoked.

...end/dataall/modules/dataset_sharing/services/share_managers/s3_access_point_share_manager.py

dlpzx · 2023-11-28T14:04:43Z

Hi @anushka-singh! The changes look good! I will do a final round of testing in AWS and approve

anushka-singh · 2023-11-28T15:00:58Z

Hi @anushka-singh! The changes look good! I will do a final round of testing in AWS and approve

Thank you! Looking forward to it.

dlpzx · 2023-11-28T16:38:06Z

Testing in AWS:

CICD successfully runs
Approve Bucket share and Access point share in Dataset with SSE-S3 encryption ----> Error!
Approve Bucket share and Access point share in Dataset with KMS encryption
Revoke Bucket share and Access point share in Dataset with SSE-S3 encryption
Revoke Bucket share and Access point share in Dataset with KMS encryption

I tried all possible combinations when approving and revoking: only Bucket, only folder, both at the same time. With 2 different shares and consumption roles. So I also tested addition/deletion of one role in the KMS policy and the complete clean-up when both shares are revoked.

dlpzx

I tested the small fix locally one final time and all functionalities worked as expected (KMS and SSE-S3 Datasets sharing). Approving now

@anushka-singh

* Bigdata867 3 (data-dot-all#24) * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * trajopadhye- BIGDATA-756 -> Added Tests for Task D and E * trajopadhye - BIGDATA-756 Corrected file data_sharing_service.py to address revokedStateSM for revoked items * trajopadhye- BIGDATA-756 - Slight correction in comments * trajopadhye- BIGDATA-756 Correction on Share Status for revoke share tests * Addresed changes from the review of PR * [BIGDATA-625] Implement bucket share processor (data-dot-all#21) * Implement bucket share processor * Fix Revoke UI sharetype * BIGDATA-612 - push source from SD container to CodeCommit. Initial Makefile and SD yaml configuration. * Remove synth * Add force push * Add default cdk.context.json * Add param for branchname * Comments. * Fix email address * Add instance specific cdk.context.json * BIGDATA-612 - truncate the cfn encryption policy prefix so that together with branch name, it will fit within 32 char limit. * Update screwdriver.yaml * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) (data-dot-all#90) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#92) * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#91) * Fix screwdriver yaml for new EMR template step. (data-dot-all#116) * Bigdata 1397 mvp 3 stagingdeploy 20231129 (data-dot-all#178) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bigdata 1397 mvp 3 stagingdeploy 20231129 1 (data-dot-all#180) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env * Bugfix --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bugfix (data-dot-all#181) * Bugfix * Bugfix * [Data 409] Athenz Certs Domain and User Pool Domain Changes (data-dot-all#221) (data-dot-all#222) * trajopadhye | DATA-409- Code changes for Athenz certs domain and user pool domain * [Data-413] GA stagingdeploy 20231228 - Fix for email notifications with Athenz. Auto-create Pivot Role (data-dot-all#224) * trajopadhye | DATA-412 - Added Athenz configs and Ports in AWS Worker lambda and enabling Auto Create Pivot Role * DATA-416 - Fix while migrating from manual pivot role to auto created (data-dot-all#230) (data-dot-all#233) * trajopadhye | DATA-416 - Fix for environment updates when using auto pivot role. Changing the way KMS keys are specified in env role * [Data 447] ga stagingdeploy 20240116 (data-dot-all#244) * [Data-446] Fix for consumption role not showing up * [Data 415] Dataset import fix for circular dependency error + local dev setup fixes (data-dot-all#243) * DATA-428 - Local env fixes * Data 448 ga stagingdeploy 20240117 (data-dot-all#246) * trajopadhye | DATA-440 - Adding else if to sync glue tabls in RDS * Data 461 ga deploy 20240125 (data-dot-all#258) * DATA-404 - Add git fetch --all to the CodeCommit repo sync * DATA-420 - Switch from Cognito to Okta on Prod (data-dot-all#254) DATA-420 - Switch from Cognito to Okta on Prod * DATA-455: Shares stuck in progress when AWS does not have root access on KMS key (data-dot-all#256) * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <tejas.rajopadhye@yahooinc.com> * Data 466 ga stagingdeploy 20240126 (data-dot-all#263) * trajoadhye | DATA-456 - Removing Lake Formation SLR (data-dot-all#260) * Data-405-Adding max 30 sec delay * Synching Release notes from Staging to y-branch-2-0 (data-dot-all#262) * [Data 484] stagingdeploy 20240206 (data-dot-all#275) * fix: adding cdk synth for checkov scans (data-dot-all#264) * [DATA-452] - Adding Dataset description in shares view (data-dot-all#273) * Added Release note for DATA-481, DATA-452, DATA-480 * Syncing Release notes (data-dot-all#274) --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> * [Data 607] staging deploy email notification fix (data-dot-all#302) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * DATA - 586 - Adding confidentiality values for custom confidentiality * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * [Data 611] Disable topics dropdown (data-dot-all#304) * Disabling topics dropdown (data-dot-all#303) * [Data 619] Stagingdeploy env permission fix (data-dot-all#307) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#300) * Email notification fix + confidentiality levels config (data-dot-all#298) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * Adding confidentiality values for custom confidentiality * Adding confidentiality configs to config.json.PROD * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) * Disabling topics dropdown (data-dot-all#303) * DATA-619 - Fix permission for GET_ORGANIZATION when users are in _data teams (data-dot-all#306) * Cherry pick for issue with GET_ORG permission after 2.3 release --------- Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> * [Data 631] Staging deploy (data-dot-all#310) * [Data 629] worksheet fix for GET_ENVIRONMENT permission (data-dot-all#309) * Data690 stagingdeploy 20240425 (data-dot-all#319) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Update release notes * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Update makefile (data-dot-all#320) * Data690 stagingdeploy 20240425 2 (data-dot-all#321) * Update makefile * Reverting nodejs 16 upgrade * Reverting nodejs 16 upgrade * Data690 stagingdeploy 20240425 3 (data-dot-all#323) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Reverting nodejs 16 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 4 (data-dot-all#325) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * Reverting nodejs 18 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 5 (data-dot-all#329) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * DATA-680 - Switch node to version 17 in the Screwdriver makefile (data-dot-all#326) * bugfix (data-dot-all#328) * Remove nodejs upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * bugfix (data-dot-all#331) * Data743 stagingdeploy (data-dot-all#351) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743: Update verifier task schedule to run nightly (data-dot-all#350) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743 stagingdeploy (data-dot-all#353) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * [Data 767] staging deploy (data-dot-all#358) * Bugfix: timeout error when listing Consumption Roles (data-dot-all#1303) - Bugfix - as GraphQL resolvers are 'lazy', for ShareRequest Modal window we simply don't fetch the managedPolicy property -- no timeout - managed policies are fetched, when consumption role is selected from dropdown - data-dot-all#1288 Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * Updated Release notes --------- Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> --------- Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * data712 * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Restore yarn file * Restore yarn file * Update config --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> Co-authored-by: Mohit Arora <marora@yahooinc.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Raj Chopde <rchopde@yahooinc.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk>

@anushka-singh

* Bigdata867 3 (data-dot-all#24) * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * trajopadhye- BIGDATA-756 -> Added Tests for Task D and E * trajopadhye - BIGDATA-756 Corrected file data_sharing_service.py to address revokedStateSM for revoked items * trajopadhye- BIGDATA-756 - Slight correction in comments * trajopadhye- BIGDATA-756 Correction on Share Status for revoke share tests * Addresed changes from the review of PR * [BIGDATA-625] Implement bucket share processor (data-dot-all#21) * Implement bucket share processor * Fix Revoke UI sharetype * BIGDATA-612 - push source from SD container to CodeCommit. Initial Makefile and SD yaml configuration. * Remove synth * Add force push * Add default cdk.context.json * Add param for branchname * Comments. * Fix email address * Add instance specific cdk.context.json * BIGDATA-612 - truncate the cfn encryption policy prefix so that together with branch name, it will fit within 32 char limit. * Update screwdriver.yaml * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) (data-dot-all#90) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#92) * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#91) * Fix screwdriver yaml for new EMR template step. (data-dot-all#116) * Bigdata 1397 mvp 3 stagingdeploy 20231129 (data-dot-all#178) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bigdata 1397 mvp 3 stagingdeploy 20231129 1 (data-dot-all#180) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env * Bugfix --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bugfix (data-dot-all#181) * Bugfix * Bugfix * [Data 409] Athenz Certs Domain and User Pool Domain Changes (data-dot-all#221) (data-dot-all#222) * trajopadhye | DATA-409- Code changes for Athenz certs domain and user pool domain * [Data-413] GA stagingdeploy 20231228 - Fix for email notifications with Athenz. Auto-create Pivot Role (data-dot-all#224) * trajopadhye | DATA-412 - Added Athenz configs and Ports in AWS Worker lambda and enabling Auto Create Pivot Role * DATA-416 - Fix while migrating from manual pivot role to auto created (data-dot-all#230) (data-dot-all#233) * trajopadhye | DATA-416 - Fix for environment updates when using auto pivot role. Changing the way KMS keys are specified in env role * [Data 447] ga stagingdeploy 20240116 (data-dot-all#244) * [Data-446] Fix for consumption role not showing up * [Data 415] Dataset import fix for circular dependency error + local dev setup fixes (data-dot-all#243) * DATA-428 - Local env fixes * Data 448 ga stagingdeploy 20240117 (data-dot-all#246) * trajopadhye | DATA-440 - Adding else if to sync glue tabls in RDS * Data 461 ga deploy 20240125 (data-dot-all#258) * DATA-404 - Add git fetch --all to the CodeCommit repo sync * DATA-420 - Switch from Cognito to Okta on Prod (data-dot-all#254) DATA-420 - Switch from Cognito to Okta on Prod * DATA-455: Shares stuck in progress when AWS does not have root access on KMS key (data-dot-all#256) * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <tejas.rajopadhye@yahooinc.com> * Data 466 ga stagingdeploy 20240126 (data-dot-all#263) * trajoadhye | DATA-456 - Removing Lake Formation SLR (data-dot-all#260) * Data-405-Adding max 30 sec delay * Synching Release notes from Staging to y-branch-2-0 (data-dot-all#262) * [Data 484] stagingdeploy 20240206 (data-dot-all#275) * fix: adding cdk synth for checkov scans (data-dot-all#264) * [DATA-452] - Adding Dataset description in shares view (data-dot-all#273) * Added Release note for DATA-481, DATA-452, DATA-480 * Syncing Release notes (data-dot-all#274) --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> * [Data 607] staging deploy email notification fix (data-dot-all#302) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * DATA - 586 - Adding confidentiality values for custom confidentiality * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * [Data 611] Disable topics dropdown (data-dot-all#304) * Disabling topics dropdown (data-dot-all#303) * [Data 619] Stagingdeploy env permission fix (data-dot-all#307) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#300) * Email notification fix + confidentiality levels config (data-dot-all#298) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * Adding confidentiality values for custom confidentiality * Adding confidentiality configs to config.json.PROD * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) * Disabling topics dropdown (data-dot-all#303) * DATA-619 - Fix permission for GET_ORGANIZATION when users are in _data teams (data-dot-all#306) * Cherry pick for issue with GET_ORG permission after 2.3 release --------- Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> * [Data 631] Staging deploy (data-dot-all#310) * [Data 629] worksheet fix for GET_ENVIRONMENT permission (data-dot-all#309) * Data690 stagingdeploy 20240425 (data-dot-all#319) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Update release notes * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Update makefile (data-dot-all#320) * Data690 stagingdeploy 20240425 2 (data-dot-all#321) * Update makefile * Reverting nodejs 16 upgrade * Reverting nodejs 16 upgrade * Data690 stagingdeploy 20240425 3 (data-dot-all#323) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Reverting nodejs 16 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 4 (data-dot-all#325) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * Reverting nodejs 18 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 5 (data-dot-all#329) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * DATA-680 - Switch node to version 17 in the Screwdriver makefile (data-dot-all#326) * bugfix (data-dot-all#328) * Remove nodejs upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * bugfix (data-dot-all#331) * Data743 stagingdeploy (data-dot-all#351) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743: Update verifier task schedule to run nightly (data-dot-all#350) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743 stagingdeploy (data-dot-all#353) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * [Data 767] staging deploy (data-dot-all#358) * Bugfix: timeout error when listing Consumption Roles (data-dot-all#1303) - Bugfix - as GraphQL resolvers are 'lazy', for ShareRequest Modal window we simply don't fetch the managedPolicy property -- no timeout - managed policies are fetched, when consumption role is selected from dropdown - data-dot-all#1288 Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * Updated Release notes --------- Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> --------- Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * data712 * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Restore yarn file * Restore yarn file * Update config * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> Co-authored-by: Mohit Arora <marora@yahooinc.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Raj Chopde <rchopde@yahooinc.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk>

@anushka-singh

* Bigdata867 3 (data-dot-all#24) * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * trajopadhye- BIGDATA-756 -> Added Tests for Task D and E * trajopadhye - BIGDATA-756 Corrected file data_sharing_service.py to address revokedStateSM for revoked items * trajopadhye- BIGDATA-756 - Slight correction in comments * trajopadhye- BIGDATA-756 Correction on Share Status for revoke share tests * Addresed changes from the review of PR * [BIGDATA-625] Implement bucket share processor (data-dot-all#21) * Implement bucket share processor * Fix Revoke UI sharetype * BIGDATA-612 - push source from SD container to CodeCommit. Initial Makefile and SD yaml configuration. * Remove synth * Add force push * Add default cdk.context.json * Add param for branchname * Comments. * Fix email address * Add instance specific cdk.context.json * BIGDATA-612 - truncate the cfn encryption policy prefix so that together with branch name, it will fit within 32 char limit. * Update screwdriver.yaml * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) (data-dot-all#90) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#92) * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#91) * Fix screwdriver yaml for new EMR template step. (data-dot-all#116) * Bigdata 1397 mvp 3 stagingdeploy 20231129 (data-dot-all#178) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bigdata 1397 mvp 3 stagingdeploy 20231129 1 (data-dot-all#180) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env * Bugfix --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bugfix (data-dot-all#181) * Bugfix * Bugfix * [Data 409] Athenz Certs Domain and User Pool Domain Changes (data-dot-all#221) (data-dot-all#222) * trajopadhye | DATA-409- Code changes for Athenz certs domain and user pool domain * [Data-413] GA stagingdeploy 20231228 - Fix for email notifications with Athenz. Auto-create Pivot Role (data-dot-all#224) * trajopadhye | DATA-412 - Added Athenz configs and Ports in AWS Worker lambda and enabling Auto Create Pivot Role * DATA-416 - Fix while migrating from manual pivot role to auto created (data-dot-all#230) (data-dot-all#233) * trajopadhye | DATA-416 - Fix for environment updates when using auto pivot role. Changing the way KMS keys are specified in env role * [Data 447] ga stagingdeploy 20240116 (data-dot-all#244) * [Data-446] Fix for consumption role not showing up * [Data 415] Dataset import fix for circular dependency error + local dev setup fixes (data-dot-all#243) * DATA-428 - Local env fixes * Data 448 ga stagingdeploy 20240117 (data-dot-all#246) * trajopadhye | DATA-440 - Adding else if to sync glue tabls in RDS * Data 461 ga deploy 20240125 (data-dot-all#258) * DATA-404 - Add git fetch --all to the CodeCommit repo sync * DATA-420 - Switch from Cognito to Okta on Prod (data-dot-all#254) DATA-420 - Switch from Cognito to Okta on Prod * DATA-455: Shares stuck in progress when AWS does not have root access on KMS key (data-dot-all#256) * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <tejas.rajopadhye@yahooinc.com> * Data 466 ga stagingdeploy 20240126 (data-dot-all#263) * trajoadhye | DATA-456 - Removing Lake Formation SLR (data-dot-all#260) * Data-405-Adding max 30 sec delay * Synching Release notes from Staging to y-branch-2-0 (data-dot-all#262) * [Data 484] stagingdeploy 20240206 (data-dot-all#275) * fix: adding cdk synth for checkov scans (data-dot-all#264) * [DATA-452] - Adding Dataset description in shares view (data-dot-all#273) * Added Release note for DATA-481, DATA-452, DATA-480 * Syncing Release notes (data-dot-all#274) --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> * [Data 607] staging deploy email notification fix (data-dot-all#302) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * DATA - 586 - Adding confidentiality values for custom confidentiality * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * [Data 611] Disable topics dropdown (data-dot-all#304) * Disabling topics dropdown (data-dot-all#303) * [Data 619] Stagingdeploy env permission fix (data-dot-all#307) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#300) * Email notification fix + confidentiality levels config (data-dot-all#298) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * Adding confidentiality values for custom confidentiality * Adding confidentiality configs to config.json.PROD * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) * Disabling topics dropdown (data-dot-all#303) * DATA-619 - Fix permission for GET_ORGANIZATION when users are in _data teams (data-dot-all#306) * Cherry pick for issue with GET_ORG permission after 2.3 release --------- Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> * [Data 631] Staging deploy (data-dot-all#310) * [Data 629] worksheet fix for GET_ENVIRONMENT permission (data-dot-all#309) * Data690 stagingdeploy 20240425 (data-dot-all#319) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Update release notes * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Update makefile (data-dot-all#320) * Data690 stagingdeploy 20240425 2 (data-dot-all#321) * Update makefile * Reverting nodejs 16 upgrade * Reverting nodejs 16 upgrade * Data690 stagingdeploy 20240425 3 (data-dot-all#323) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Reverting nodejs 16 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 4 (data-dot-all#325) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * Reverting nodejs 18 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 5 (data-dot-all#329) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * DATA-680 - Switch node to version 17 in the Screwdriver makefile (data-dot-all#326) * bugfix (data-dot-all#328) * Remove nodejs upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * bugfix (data-dot-all#331) * Data743 stagingdeploy (data-dot-all#351) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743: Update verifier task schedule to run nightly (data-dot-all#350) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743 stagingdeploy (data-dot-all#353) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * [Data 767] staging deploy (data-dot-all#358) * Bugfix: timeout error when listing Consumption Roles (data-dot-all#1303) - Bugfix - as GraphQL resolvers are 'lazy', for ShareRequest Modal window we simply don't fetch the managedPolicy property -- no timeout - managed policies are fetched, when consumption role is selected from dropdown - data-dot-all#1288 Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * Updated Release notes --------- Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> --------- Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * data712 * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Restore yarn file * Restore yarn file * Update config * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: update import * Data712: update import --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> Co-authored-by: Mohit Arora <marora@yahooinc.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Raj Chopde <rchopde@yahooinc.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk>

noah-paige and others added 30 commits September 15, 2023 08:18

Merge branch 'main' into v2m1m0

16c7026

# Conflicts: # deploy/app.py

Fix: condition when there are no public subnets (data-dot-all#794)

f448613

### Feature or Bugfix Fix data-dot-all#792: Fix: condition when there are no public subnets ---------

Merge branch 'main' into v2m1m0

8ad760b

# Conflicts: # deploy/stacks/backend_stack.py # deploy/stacks/backend_stage.py # deploy/stacks/lambda_api.py # deploy/stacks/pipeline.py # template_cdk.json

Merge branch 'main' into v2m1m0

6d3c016

# Conflicts: # deploy/stacks/backend_stack.py # deploy/stacks/backend_stage.py # deploy/stacks/lambda_api.py # deploy/stacks/pipeline.py # template_cdk.json

feat: Enabling S3 bucket share

5fb7cf8

feat: Enabling S3 bucket share

cf9afc1

Merge branch 'v2m1m0' of https://github.com/anushka-singh/aws-dataall …

ddf8623

…into bucket_share

anushka-singh added 7 commits November 16, 2023 00:07

feat: Enabling S3 bucket share

7ab6427

feat: Enabling S3 bucket share - Addressing comments on PR

e251fcc

feat: Enabling S3 bucket share - Addressing comments on PR

e8bfb4b

Merge branch 'main' of https://github.com/anushka-singh/aws-dataall i…

2ff67bc

…nto bucket_share_anushka

Update share.js

3254260

Update index.js

eb8bf3d

bugfix: Fixing KMS policy explosion

deb2dd4

anushka-singh force-pushed the kms_explosion_fix branch from 27fbd8b to deb2dd4 Compare November 20, 2023 18:58

Merge branch 'main' into kms_explosion_fix

7665a5e

dlpzx reviewed Nov 21, 2023

View reviewed changes

backend/dataall/modules/dataset_sharing/services/share_managers/s3_bucket_share_manager.py Outdated Show resolved Hide resolved

dlpzx reviewed Nov 21, 2023

View reviewed changes

...end/dataall/modules/dataset_sharing/services/share_managers/s3_access_point_share_manager.py Show resolved Hide resolved

dlpzx reviewed Nov 21, 2023

View reviewed changes

tests/modules/datasets/tasks/test_s3_access_point_share_manager.py Show resolved Hide resolved

dlpzx reviewed Nov 21, 2023

View reviewed changes

tests/modules/datasets/tasks/test_s3_access_point_share_manager.py Outdated Show resolved Hide resolved

bugfix: Fixing KMS policy explosion: Addressing PR comments

bb34b24

dlpzx reviewed Nov 23, 2023

View reviewed changes

...end/dataall/modules/dataset_sharing/services/share_managers/s3_access_point_share_manager.py Show resolved Hide resolved

bugfix: Fixing KMS policy explosion: Addressing PR comments

d4486ff

bugfix: Fixing KMS policy explosion: Addressing PR comments

6be9b02

dlpzx approved these changes Nov 29, 2023

View reviewed changes

dlpzx merged commit d1d0e15 into data-dot-all:main Nov 29, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kms explosion fix #882

Kms explosion fix #882

anushka-singh commented Nov 17, 2023 •

edited

Loading

dlpzx commented Nov 21, 2023 •

edited

Loading

dlpzx commented Nov 21, 2023 •

edited

Loading

anushka-singh commented Nov 21, 2023

anushka-singh commented Nov 21, 2023

dlpzx commented Nov 23, 2023 •

edited

Loading

dlpzx commented Nov 28, 2023

anushka-singh commented Nov 28, 2023

dlpzx commented Nov 28, 2023

dlpzx left a comment

Kms explosion fix #882

Kms explosion fix #882

Conversation

anushka-singh commented Nov 17, 2023 • edited Loading

Feature or Bugfix

Detail

Testing

Security

dlpzx commented Nov 21, 2023 • edited Loading

dlpzx commented Nov 21, 2023 • edited Loading

anushka-singh commented Nov 21, 2023

anushka-singh commented Nov 21, 2023

dlpzx commented Nov 23, 2023 • edited Loading

dlpzx commented Nov 28, 2023

anushka-singh commented Nov 28, 2023

dlpzx commented Nov 28, 2023

dlpzx left a comment

Choose a reason for hiding this comment

anushka-singh commented Nov 17, 2023 •

edited

Loading

dlpzx commented Nov 21, 2023 •

edited

Loading

dlpzx commented Nov 21, 2023 •

edited

Loading

dlpzx commented Nov 23, 2023 •

edited

Loading