-
Notifications
You must be signed in to change notification settings - Fork 82
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bugfix 932: Investigate why some shares did not go to failed state, but remained stuck or in-progress #933
Commits on Sep 15, 2023
-
Add Additional Error Messages for KMS Key lookup on imported dataset (d…
…ata-dot-all#748) ### Feature or Bugfix <!-- please choose --> - Feature Enhancement ### Detail - Adding additional error messages for KMS Key lookup when importing a new dataset - 1 Error message to determine if the KMS Key Alias Exists - 1 Error message to determine if the PivotRole has permissions to describe the KMS Key ### Relates - data-dot-all#712 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for ad4ab1f - Browse repository at this point
Copy the full SHA ad4ab1fView commit details
Commits on Sep 19, 2023
-
Get Latest in main to v2m1m0 (data-dot-all#771)
### Feature or Bugfix <!-- please choose --> - NA ### Detail - Get latest code in `main` to `v2m1m0` branch to keep in sync ### Relates - NA ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). NA ``` - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? ``` By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for dbbef3c - Browse repository at this point
Copy the full SHA dbbef3cView commit details
Commits on Sep 26, 2023
-
Handle Environment Import of IAM service roles (data-dot-all#749)
### Feature or Bugfix <!-- please choose --> - Enahncement / Bugfix ### Detail - When creating an environment and specifying default Env IAM Role we assume it is of the structure `arn:aws:iam::ACCOUNT:role/NAME_SPECIFIED` - This does not work when there is a service path in the role arn such as with SSO: `arn:aws:iam::ACCOUNT:role/sso/NAME_SPECIFIED` - Causes issues when importing an IAM Role for an invited Team in an environment and/or with dataset sharing - This PR takes in the full IAM role ARN when importing the IAM role in order to correctly determine the role name ### Relates - [data-dot-all#695 ](data-dot-all#695) ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for d096160 - Browse repository at this point
Copy the full SHA d096160View commit details
Commits on Oct 5, 2023
-
Build Compliant Names for Opensearch Resources (data-dot-all#750)
### Feature or Bugfix <!-- please choose --> - Enhancement / Bugfix ### Detail - Ensure the names passed for OpenSearch Domain and OpenSearch Serverless Collection, Access Policy, Security Policy, and VPC Endpoint all follow naming conventions required by the service, meaning - The name must start with a lowercase letter - Must be between 3 and 28 characters - Valid characters are a-z (lowercase only), 0-9, and - (hyphen). ### Relates - data-dot-all#540 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: dlpzx <dlpzx@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for a53434f - Browse repository at this point
Copy the full SHA a53434fView commit details
Commits on Oct 10, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 16c7026 - Browse repository at this point
Copy the full SHA 16c7026View commit details -
Update Lambda runtime (data-dot-all#782)
### Feature or Bugfix Update ### Detail ### Relates See data-dot-all#655: > In Nov 27, 2023 the Lambda runtime node14 and Python3.7 will be deprecated! Checked all lambdas that explicitly set the runtime engine: only cognito httpheader redirection lambda used node14. All lambdas use python3.8 and node16 or node18. For cdk dependencies: upgraded to a newest `aws-cdk-lib` `v2.99.0` just in case if python3.7 is hardcoded somewhere inside of 2.78.0 (shouldn't be) ### Testing: - [x] uploaded the changes to my isengard account - [x] deployment is green - [x] could access app page, userguide page, and userguide from the app page. ### Security `N/A` - upgraded to a newer version of node js By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for c61ba15 - Browse repository at this point
Copy the full SHA c61ba15View commit details
Commits on Oct 12, 2023
-
Feat: limit pivot role S3 permissions (data-dot-all#780)
### Feature or Bugfix - Feature ### Detail The guiding principle is that: 1. dataset IAM role is the role accessing data 2. pivot role is the role used by the central account to perform SDK calls in the environment account In this PR we - Replace pivot role by dataset role in dataset Lake Formation registration - Use pivot role to trigger upload files feature and create folder feature, but use the dataset IAM role to perform the putObject operations-> removes the need for read and `putObject` permissions. for the pivot role - Redefine pivot role CDK stack to manage S3 buckets (bucket policies) for only the datasets S3 buckets that have been created or imported in the environment. - implement IAM policy utils to handle the new dynamic policies. We need to verify that the created policy statements do not exceed the maximum policy size. In addition we replace the previous "divide in chunks of 10 statements" by a function that divides in chunks based on the size of the policy statements. This way we optimize the policy size, which helps us in reducing the number of managed policies attached to the pivot role. --> it can be re-used in other "chunkenization" of policies - We did not implement force update of environments (pivot role nested stack) with new datasets added because it is already forced in `backend/dataall/modules/datasets/services/dataset_service.py` ### Backwards compatibility Testing Pre-update setup: - 1 environment (auto-created pivot role) - 2 datasets in environment, 1 created, 1 imported: with tables and folders - Run profiling jobs in tables Update with the branch changes: - [X] CICD pipeline runs successfully - [X] Click update environment on environment -> successfully updated policy of pivot role with imported datasets in policy. Reduction of policies - [X] Click update datasets --> registration in Lake formation updated to dataset role - [X] Update files works - [X] Create folder works - [X] Crawler and profiling jobs work ### Relates - data-dot-all#580 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Are you introducing any new policies/roles/users? `Yes` - Have you used the least-privilege principle? How? `In this PR we restrict the permissions of the pivot role, a super role that handles SDK calls in the environment accounts. Instead of granting permissions to all S3 buckets, we restrict it to data.all handled S3 buckets only` By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for f84250e - Browse repository at this point
Copy the full SHA f84250eView commit details -
Fix: ensure valid environments for share request and other objects cr…
…eation (data-dot-all#781) ### Feature or Bugfix - Feature - Bugfix ### Detail The different alternatives considered are discussed in data-dot-all#556 This PR introduces a new query `listValidEnvironments` that replaces the query `listEnvironments` for certain operations. `listEnvironments` - lists all environments independently of their CloudFormation stack statys with a lot of additional details `listValidEnvironments` - lists only "CloudFormation" stable and successful environments. Retrieves only basic info about the environment. Operations such as opening a share request or creation a Dataset/Notebook/etc require the selection of an environment. The environment options are now retrieved from `listValidEnvironments` ensuring that only valid environments are selectable. Moreover, this query is more light and does not need to query and obtain as many fields as the original `listEnvironments`, improving the efficiency of the code. ### Relates - data-dot-all#556 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for 7d9122d - Browse repository at this point
Copy the full SHA 7d9122dView commit details
Commits on Oct 13, 2023
-
Adding configurable session timeout to IDP (data-dot-all#786)
### Feature or Bugfix <!-- please choose --> - Feature ### Detail Allows user to configure a session timeout . Today data.all by default sets the refresh token to 30 days but with this change it becomes configurable ### Relates data-dot-all#421 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Manjula <manjula.kasturi@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 1801cf1 - Browse repository at this point
Copy the full SHA 1801cf1View commit details
Commits on Oct 16, 2023
-
Fix: shell true semgrep (data-dot-all#760)
### Feature or Bugfix - Feature - Bugfix ### Detail As explained in the [semgrep docs](https://semgrep.dev/docs/cheat-sheets/python-command-injection/#1b-shelltrue): "Functions from the subprocess module have the shell argument for specifying if the command should be executed through the shell. Using shell=True is dangerous because it propagates current shell settings and variables. This means that variables, glob patterns, and other special shell features in the command string are processed before the command is run, making it much easier for a malicious actor to execute commands. The subprocess module allows you to start new processes, connect to their input/output/error pipes, and obtain their return codes. Methods such as Popen, run, call, check_call, check_output are intended for running commands provided as an argument ('args'). Allowing user input in a command that is passed as an argument to one of these methods can create an opportunity for a command injection vulnerability." In our case the risk is not exposed as no user input is directly taken into the subprocess commands. Nevertheless we should strive for the highest standards on security and this PR works on replacing all the `shell=True` executions in the data.all code. In this PR: - when possible we have set `shell=False` - in cases where the command was too complex a `CommandSanitizer` ensures that the input arguments are strings following the regex=`[a-zA-Z0-9-_]` Testing: - [X] local testing - deployment of any stack (`backend/dataall/base/cdkproxy/cdk_cli_wrapper.py`) - [X] local testing - deployment of cdk pipeline stack (`backend/dataall/modules/datapipelines/cdk/datapipelines_cdk_pipeline.py`) - [X] local testing - deployment of codepipeline pipeline stack (`backend/dataall/modules/datapipelines/cdk/datapipelines_pipeline.py`) - [ ] AWS testing - deployment of data.all - [ ] AWS testing - deployment of any stack (`backend/dataall/base/cdkproxy/cdk_cli_wrapper.py`) - [ ] AWS testing - deployment of cdk pipeline stack (`backend/dataall/modules/datapipelines/cdk/datapipelines_cdk_pipeline.py`) - [ ] AWS testing - deployment of codepipeline pipeline stack (`backend/dataall/modules/datapipelines/cdk/datapipelines_pipeline.py`) ### Relates - data-dot-all#738 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? ---> 🆗 This is exactly what this PR is trying to do - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for 599fc1a - Browse repository at this point
Copy the full SHA 599fc1aView commit details -
Fix: allow to submit a share when you are both and approver and a req…
…uester (data-dot-all#793) ### Feature or Bugfix - Bugfix ### Detail - Allowing to submit a share when you are both an approver and a requester ### Security **DOES NOT APPLY** Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com>
Configuration menu - View commit details
-
Copy full SHA for b356bf2 - Browse repository at this point
Copy the full SHA b356bf2View commit details -
feat: redirect upon creating a share request (data-dot-all#799)
### Feature or Bugfix - Feature ### Detail Adding a redirect to the share UI once a share object is created. Additionally updating the breadcrumb message to more clearly indicate that a "Draft share request is created" rather than suggesting that the share has actually been sent to the data owners team. ### Relates N/A ### Security N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com>
Configuration menu - View commit details
-
Copy full SHA for 793a078 - Browse repository at this point
Copy the full SHA 793a078View commit details
Commits on Oct 18, 2023
-
Fix: condition when there are no public subnets (data-dot-all#794)
### Feature or Bugfix Fix data-dot-all#792: Fix: condition when there are no public subnets ---------
Configuration menu - View commit details
-
Copy full SHA for f448613 - Browse repository at this point
Copy the full SHA f448613View commit details -
feat: removing unused variable (data-dot-all#815)
### Feature or Bugfix - Feature ### Detail - Removing unused variable in local graphql server pointing to a fixed AWS region ### Relates N/A ### Security N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com>
Configuration menu - View commit details
-
Copy full SHA for 66b9a08 - Browse repository at this point
Copy the full SHA 66b9a08View commit details -
feat: Handle Pre-filtering of tables (data-dot-all#811)
### Feature or Bugfix - Feature ### Detail - For a dataset to make sense all the tables within a dataset should have their location pointing to the same place as the dataset S3 bucket. However it is possible that a database can have tables which do not point to the same bucket which is perfectly legal in LakeFormation. Therefore we propose that data.all automatically only lists tables that have the same S3 bucket location as the dataset. This will solve a problem for Yahoo where we want to import a database that contains many tables with different buckets. Additionally Catalog UI should also only list prefiltered tables. ### Testing - Tested this in local env. I was able to create and share datasets even after pre-filtering process takes place. - Will send separate PR for unit testing. ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com>
Configuration menu - View commit details
-
Copy full SHA for c833c26 - Browse repository at this point
Copy the full SHA c833c26View commit details -
Fix Check other share exists before clean up (data-dot-all#769)
### Feature or Bugfix <!-- please choose --> - Bugfix ### Detail - Fix method to detect if other share objects exist on the environment before cleaning up environment-level shared resources (i.e. RAM invitation and PivotRole permissions) - Originally, if TeamA in EnvA had 2 shares approved and succeeded on DatasetB and TeamA rejects 1 of the pre-existing shares, the method `other_approved_share_object_exists` was returning `False`and deleting necessary permissions for the other existing Share - Also disables the other existing shares ability to Revoke the still existing share since pivotRole no longer has permissions - Also fixes the removal of dataall QS Group permissions if there are still existing shares to EnvA ### Security NA ``` Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? ``` By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 6cc564e - Browse repository at this point
Copy the full SHA 6cc564eView commit details
Commits on Oct 20, 2023
-
Email Notification on Share Workflow - Issue - 734 (data-dot-all#818)
### Feature or Bugfix - Feature ### Detail Whenever a share request is created and transitions from states ( approved, revoked, etc ) a notification is created. This notification is displayed on the bell icon on the UI . We want such a similar notification to be sent to the dataset owner, requester, etc via email Please take a look at Github Issue 734 For more details - data-dot-all#734 ### Relates - data-dot-all#734 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? No - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? No - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? No - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? Yes - Have you used the least-privilege principle? How? --> **Permission granted for SES:sendEmail to Lambda on resources - (Ses identity and configuration set ) , Also created KMS and SNS for SES setup to handle email bounces . Used least privleged and restricted access on both whenever required. ** By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com>
Configuration menu - View commit details
-
Copy full SHA for 8b7b82e - Browse repository at this point
Copy the full SHA 8b7b82eView commit details
Commits on Oct 25, 2023
-
feat: adding frontend and backend feature flags (data-dot-all#817)
### Feature or Bugfix - Feature ### Detail - Adding frontend support for all feature flags defined in config.json with a new util method isFeatureEnabled - Adding a new flag **preview_data** in the datasets module to control whether previewing data is allowed - Adding a new flag **glue_crawler** in the datasets module to control whether running glue crawler is allowed - Updating environment features to be hidden or visible based on whether the module is active. Adding a new util isAnyFeatureModuleEnabled to check whether to render the entire feature box. ### Relates N/A ### Security Not relevant By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com>
Configuration menu - View commit details
-
Copy full SHA for 48c32e5 - Browse repository at this point
Copy the full SHA 48c32e5View commit details
Commits on Oct 26, 2023
-
Feat: Refactor notifications from core to modules (data-dot-all#822)
### Feature or Bugfix - Refactoring ### Detail As a rule of thumb, we encourage customization of `modules` while changes in `core` should be avoided when possible. `notifications` is a component initially in core which is only used by `dataset_sharing`. To facilitate customization of the `notifications` module and also to clearly see its dependencies we have: - Moved `notifications` code from core to modules as it is a reusable component that is not needed by any core component. - Moved dataset_sharing references inside dataset_sharing module and left `notifications` independent from any other module (done mostly in data-dot-all#734, so credits to @TejasRGitHub) - Added depends_on in the dataset_sharing module to load notifications if the data_sharing module is imported. - Modified frontend navigation bar to make it conditional of the notifications module - Added migration script to modify the notification type column - Fix tests from data-dot-all#734, some references on the payload of the notification tasks were wrong - Small fixes to SES stack: added account in KMS policy and email_id as input ### [WIP] Testing Local testing - [ ] loading of notifications with datasets enabled - [ ] ... AWS testing - [ ] CICD pipeline succeds ### Other remarks Not for this PR, but as a general note, we should clean up deprecated ECS tasks ### Relates - data-dot-all#785 - data-dot-all#734 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). `N/A` just refactoring By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for 6d727e9 - Browse repository at this point
Copy the full SHA 6d727e9View commit details
Commits on Oct 27, 2023
-
Merge branch 'main' into v2m1m0
# Conflicts: # deploy/stacks/backend_stack.py # deploy/stacks/backend_stage.py # deploy/stacks/lambda_api.py # deploy/stacks/pipeline.py # template_cdk.json
Configuration menu - View commit details
-
Copy full SHA for 8ad760b - Browse repository at this point
Copy the full SHA 8ad760bView commit details -
Feat: pivot role limit kms (data-dot-all#830)
### Feature or Bugfix - Feature ### Detail - read KMS keys with an alias prefixed by the environment resource prefix - read KMS keys imported in imported datasets - restrict pivot role policies to the KMS keys created by data.all and those imported in the imported datasets - move kms client from data_sharing to base as it is used in environments and datasets ### Relates - data-dot-all#580 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). This PR restricts the IAM policies of the pivot role, following the least privilege permissions principle - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for 3f100b4 - Browse repository at this point
Copy the full SHA 3f100b4View commit details -
Make hosted_zone_id optional, code update (data-dot-all#812)
### Feature or Bugfix - Bugfix ### Detail - Make `hosted_zone_id` optional, code update ### Relates - data-dot-all#797 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? N/A - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. YES ### Description Make `hosted_zone_id` optional and provide `HostedZoneId` and `DNSName` in CloudFormation Stack Output, so users can create their own [Route53 AliasTarget](https://docs.aws.amazon.com/Route53/latest/APIReference/API_AliasTarget.html). Following validation checks in `ecs_patterns.ApplicationLoadBalancedFargateService` were considered: * `frontend_alternate_domain` and `userguide_alternate_domain` have to be `None` when the `hosted_zone` is `None`, see checks in [multiple-target-groups-service-base.ts#L463](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/network-multiple-target-groups-service-base.ts#L463), or else a `A Route53 hosted domain zone name is required to configure the specified domain name` error is raised * for a HTTPS ALB listener, only the `certificate` is ultimately required, and not the `domainName` or `domainZone`, as per evaluation logic in [application-load-balanced-service-base.ts#L509](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/application-load-balanced-service-base.ts#L509)
Configuration menu - View commit details
-
Copy full SHA for fb7b61b - Browse repository at this point
Copy the full SHA fb7b61bView commit details
Commits on Oct 30, 2023
-
Clean-up for v2.1 (data-dot-all#843)
### Feature or Bugfix - Bugfix ### Detail - Clean up prints and show better exception message when custom_domain is not provided for SES ### Relates - v2.1.0 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for b51da2c - Browse repository at this point
Copy the full SHA b51da2cView commit details -
Merge branch 'main' into v2m1m0
# Conflicts: # deploy/stacks/backend_stack.py # deploy/stacks/backend_stage.py # deploy/stacks/lambda_api.py # deploy/stacks/pipeline.py # template_cdk.json
Configuration menu - View commit details
-
Copy full SHA for 6d3c016 - Browse repository at this point
Copy the full SHA 6d3c016View commit details -
Feat: pivot role limit kms (data-dot-all#830)
### Feature or Bugfix - Feature ### Detail - read KMS keys with an alias prefixed by the environment resource prefix - read KMS keys imported in imported datasets - restrict pivot role policies to the KMS keys created by data.all and those imported in the imported datasets - move kms client from data_sharing to base as it is used in environments and datasets ### Relates - data-dot-all#580 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). This PR restricts the IAM policies of the pivot role, following the least privilege permissions principle - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for 7912a24 - Browse repository at this point
Copy the full SHA 7912a24View commit details -
Make hosted_zone_id optional, code update (data-dot-all#812)
### Feature or Bugfix - Bugfix ### Detail - Make `hosted_zone_id` optional, code update ### Relates - data-dot-all#797 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? N/A - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. YES ### Description Make `hosted_zone_id` optional and provide `HostedZoneId` and `DNSName` in CloudFormation Stack Output, so users can create their own [Route53 AliasTarget](https://docs.aws.amazon.com/Route53/latest/APIReference/API_AliasTarget.html). Following validation checks in `ecs_patterns.ApplicationLoadBalancedFargateService` were considered: * `frontend_alternate_domain` and `userguide_alternate_domain` have to be `None` when the `hosted_zone` is `None`, see checks in [multiple-target-groups-service-base.ts#L463](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/network-multiple-target-groups-service-base.ts#L463), or else a `A Route53 hosted domain zone name is required to configure the specified domain name` error is raised * for a HTTPS ALB listener, only the `certificate` is ultimately required, and not the `domainName` or `domainZone`, as per evaluation logic in [application-load-balanced-service-base.ts#L509](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/application-load-balanced-service-base.ts#L509)
Configuration menu - View commit details
-
Copy full SHA for 55c579b - Browse repository at this point
Copy the full SHA 55c579bView commit details -
Clean-up for v2.1 (data-dot-all#843)
### Feature or Bugfix - Bugfix ### Detail - Clean up prints and show better exception message when custom_domain is not provided for SES ### Relates - v2.1.0 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Configuration menu - View commit details
-
Copy full SHA for 92d4324 - Browse repository at this point
Copy the full SHA 92d4324View commit details
Commits on Oct 31, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 5fb7cf8 - Browse repository at this point
Copy the full SHA 5fb7cf8View commit details -
Configuration menu - View commit details
-
Copy full SHA for cf9afc1 - Browse repository at this point
Copy the full SHA cf9afc1View commit details -
Merge branch 'v2m1m0' of https://github.com/anushka-singh/aws-dataall …
…into bucket_share
Configuration menu - View commit details
-
Copy full SHA for ddf8623 - Browse repository at this point
Copy the full SHA ddf8623View commit details -
fix: adding missing pivot role permission to get key policy (data-dot…
…-all#845) ### Feature or Bugfix - Bugfix ### Detail - getKeyPolicy permission is required by share manager. I'm not sure if it is required in 2.0 but it is definitely required for S3 bucket policy sharing. Though I suspect it should be needed for OS version to as for access points to work pivotRole needs to update KMS key policy right? Without this permission sharing manager fails to get the policy and fails. The only workaround is if you manually add the pivotRole to the KMS key policy. ### Relates N/A ### Security This change expands the pivot role with a new permission to get key policy. This is still following least permission principle. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com>
Configuration menu - View commit details
-
Copy full SHA for b54860d - Browse repository at this point
Copy the full SHA b54860dView commit details -
Merge branch 'v2m1m0' into anu-s3-copy
# Conflicts: # backend/dataall/modules/datasets/services/dataset_service.py
Configuration menu - View commit details
-
Copy full SHA for a05e548 - Browse repository at this point
Copy the full SHA a05e548View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1365e92 - Browse repository at this point
Copy the full SHA 1365e92View commit details -
Configuration menu - View commit details
-
Copy full SHA for bbcfbd5 - Browse repository at this point
Copy the full SHA bbcfbd5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e8cdf1 - Browse repository at this point
Copy the full SHA 9e8cdf1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5d90797 - Browse repository at this point
Copy the full SHA 5d90797View commit details -
Configuration menu - View commit details
-
Copy full SHA for 94be491 - Browse repository at this point
Copy the full SHA 94be491View commit details -
Configuration menu - View commit details
-
Copy full SHA for cff577f - Browse repository at this point
Copy the full SHA cff577fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5ff80fb - Browse repository at this point
Copy the full SHA 5ff80fbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3383166 - Browse repository at this point
Copy the full SHA 3383166View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7ed96af - Browse repository at this point
Copy the full SHA 7ed96afView commit details -
Configuration menu - View commit details
-
Copy full SHA for c051896 - Browse repository at this point
Copy the full SHA c051896View commit details -
Configuration menu - View commit details
-
Copy full SHA for f5d62d7 - Browse repository at this point
Copy the full SHA f5d62d7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3783a95 - Browse repository at this point
Copy the full SHA 3783a95View commit details -
Configuration menu - View commit details
-
Copy full SHA for dacba14 - Browse repository at this point
Copy the full SHA dacba14View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3b404cd - Browse repository at this point
Copy the full SHA 3b404cdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5d0fe68 - Browse repository at this point
Copy the full SHA 5d0fe68View commit details
Commits on Nov 2, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 158925a - Browse repository at this point
Copy the full SHA 158925aView commit details
Commits on Nov 3, 2023
-
Configuration menu - View commit details
-
Copy full SHA for d112a21 - Browse repository at this point
Copy the full SHA d112a21View commit details
Commits on Nov 8, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 06edb53 - Browse repository at this point
Copy the full SHA 06edb53View commit details
Commits on Nov 10, 2023
-
Configuration menu - View commit details
-
Copy full SHA for f43003c - Browse repository at this point
Copy the full SHA f43003cView commit details
Commits on Nov 15, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 4516f4d - Browse repository at this point
Copy the full SHA 4516f4dView commit details
Commits on Nov 16, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 0f2faf7 - Browse repository at this point
Copy the full SHA 0f2faf7View commit details -
Merge branch 'main' into bucket_share_anushka
# Conflicts: # backend/dataall/modules/datasets/api/table/resolvers.py # backend/migrations/versions/4f3c1d84a628_modify_notifications_column_types.py # deploy/stacks/backend_stack.py # deploy/stacks/ses_stack.py # frontend/src/modules/Environments/views/EnvironmentCreateForm.js # template_cdk.json
Configuration menu - View commit details
-
Copy full SHA for 9b0ab34 - Browse repository at this point
Copy the full SHA 9b0ab34View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7ab6427 - Browse repository at this point
Copy the full SHA 7ab6427View commit details -
Configuration menu - View commit details
-
Copy full SHA for e251fcc - Browse repository at this point
Copy the full SHA e251fccView commit details -
Configuration menu - View commit details
-
Copy full SHA for e8bfb4b - Browse repository at this point
Copy the full SHA e8bfb4bView commit details -
Merge branch 'main' of https://github.com/anushka-singh/aws-dataall i…
…nto bucket_share_anushka
Configuration menu - View commit details
-
Copy full SHA for 2ff67bc - Browse repository at this point
Copy the full SHA 2ff67bcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3254260 - Browse repository at this point
Copy the full SHA 3254260View commit details -
Configuration menu - View commit details
-
Copy full SHA for eb8bf3d - Browse repository at this point
Copy the full SHA eb8bf3dView commit details
Commits on Dec 18, 2023
-
Configuration menu - View commit details
-
Copy full SHA for a06838f - Browse repository at this point
Copy the full SHA a06838fView commit details
Commits on Dec 19, 2023
-
Bugfix#932: Investigate why some shares did not go to failed state, b…
…ut remained stuck or in-progress
Configuration menu - View commit details
-
Copy full SHA for bed3d51 - Browse repository at this point
Copy the full SHA bed3d51View commit details
Commits on Dec 27, 2023
-
Bugfix#932: Investigate why some shares did not go to failed state, b…
…ut remained stuck or in-progress
Configuration menu - View commit details
-
Copy full SHA for 35b1730 - Browse repository at this point
Copy the full SHA 35b1730View commit details
Commits on Jan 3, 2024
-
Bugfix#932: Investigate why some shares did not go to failed state, b…
…ut remained stuck or in-progress
Configuration menu - View commit details
-
Copy full SHA for e9debef - Browse repository at this point
Copy the full SHA e9debefView commit details -
Bugfix#932: Investigate why some shares did not go to failed state, b…
…ut remained stuck or in-progress
Configuration menu - View commit details
-
Copy full SHA for 2501cac - Browse repository at this point
Copy the full SHA 2501cacView commit details -
Bugfix#932: Investigate why some shares did not go to failed state, b…
…ut remained stuck or in-progress
Configuration menu - View commit details
-
Copy full SHA for 530c098 - Browse repository at this point
Copy the full SHA 530c098View commit details -
Bugfix#932: Investigate why some shares did not go to failed state, b…
…ut remained stuck or in-progress
Configuration menu - View commit details
-
Copy full SHA for e55f2cd - Browse repository at this point
Copy the full SHA e55f2cdView commit details -
Bugfix#932: Investigate why some shares did not go to failed state, b…
…ut remained stuck or in-progress
Configuration menu - View commit details
-
Copy full SHA for 102f6fb - Browse repository at this point
Copy the full SHA 102f6fbView commit details
Commits on Jan 4, 2024
-
Bugfix#932: Investigate why some shares did not go to failed state, b…
…ut remained stuck or in-progress
Configuration menu - View commit details
-
Copy full SHA for b233396 - Browse repository at this point
Copy the full SHA b233396View commit details