Skip to content

Commit

Permalink
Squashed commit of the following
Browse files Browse the repository at this point in the history
commit 22a6f6ef 
Author: Noah Paige <noahpaig@amazon.com> 
Date: Mon Jul 08 2024 11:28:07 GMT-0400 (Eastern Daylight Time) 

    Add integ tests


commit 4fb7d653 
Author: Noah Paige <noahpaig@amazon.com> 
Date: Mon Jul 08 2024 11:26:36 GMT-0400 (Eastern Daylight Time) 

    Merge env test changes


commit 4cf42e8 
Author: Petros Kalos <kalosp@amazon.com> 
Date: Fri Jul 05 2024 08:19:34 GMT-0400 (Eastern Daylight Time) 

    improve docs


commit 65f930a 
Author: Petros Kalos <kalosp@amazon.com> 
Date: Fri Jul 05 2024 08:10:56 GMT-0400 (Eastern Daylight Time) 

    fix failures


commit 170b7ce 
Author: Petros Kalos <kalosp@amazon.com> 
Date: Wed Jul 03 2024 10:52:20 GMT-0400 (Eastern Daylight Time) 

    add group/consumption_role invite/remove tests


commit ba77d69 
Author: dlpzx <71252798+dlpzx@users.noreply.github.com> 
Date: Wed Jul 03 2024 06:51:47 GMT-0400 (Eastern Daylight Time) 

    Rename alias for env_vars kms key in cognito lambdas FE and BE (#1385)

### Feature or Bugfix
- Bugfix

### Detail
For the case in which we deploy FE and BE in us-east-1 the new lambda
env_key alias is the same one for TriggerFunctionCognitoUrlsConfig in FE
and for TriggerFunctionCognitoConfig in BE, which results in a failure
of the CICD in the FE stack because the alias already exists.

This PR changes the name of both aliases to avoid this conflict. It also
adds envname to avoid issues with other deployment environments/tooling
account in the future

### Relates
- <URL or Ticket>

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit e5923a9 
Author: dlpzx <71252798+dlpzx@users.noreply.github.com> 
Date: Wed Jul 03 2024 04:27:11 GMT-0400 (Eastern Daylight Time) 

    Fix lambda_env_key out of scope for vpc-facing cognito setup (#1384)

### Feature or Bugfix
- Bugfix

### Detail
The KMS key for the Lambda environment variables in the Cognito IdP
stack was defined inside an if-clause for internet facing frontend.
Outside of that if, for vpc-facing architecture the kms key does not
exist and the CICD pipeline fails. This PRs move the creation of the KMS
key outside of the if.

### Relates

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit 3ccacfc 
Author: Noah Paige <69586985+noah-paige@users.noreply.github.com> 
Date: Mon Jul 01 2024 13:56:58 GMT-0400 (Eastern Daylight Time) 

    Add delete docs not found when re indexing in catalog task (#1365)

### Feature or Bugfix
<!-- please choose -->
- Feature

### Detail
- Add logic to Catalog Indexer Task to Delete Docs No Longer in RDS
- TODO: Add Ability to Re-index Catalog Items via Dataall Admin UI

### Relates
- #1078

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit e2817a1 
Author: Noah Paige <69586985+noah-paige@users.noreply.github.com> 
Date: Mon Jul 01 2024 05:14:07 GMT-0400 (Eastern Daylight Time) 

    Fix/glossary status (#1373)

### Feature or Bugfix
<!-- please choose -->
- Bugfix


### Detail
- Add back `status` to Glossary GQL Object for GQL Operations
(getGlossary, listGlossaries)
- Fix  `listOrganizationGroupPermissions` enforce non null on FE


### Relates


### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit c3c58bd 
Author: Petros Kalos <kalosp@amazon.com> 
Date: Fri Jun 28 2024 06:55:42 GMT-0400 (Eastern Daylight Time) 

    add environment tests (#1371)

### Feature or Bugfix
Feature

### Detail
* add list_environment tests
* add test for updating an environment (via update_stack)
* generalise the polling functions for stacks

### Relates
#1220 

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit e913d48 
Author: dlpzx <71252798+dlpzx@users.noreply.github.com> 
Date: Fri Jun 28 2024 04:15:49 GMT-0400 (Eastern Daylight Time) 

    Add search (Autocomplete) in miscellaneous dropdowns (#1367)

### Feature or Bugfix
- Feature

### Detail
Autocomplete for environments and teams in the following frontend views
as requested in #1012. In this case the views required custom dropdowns.

❗ I used `noOptionsText` whenever it was necessary instead of checking
groupOptions lenght >0
- [x] DatasetEditForm.js -> ❗ I kept the stewards field as `freesolo` -
what that means is that users CAN specify options that are not on the
list. I would like the reviewer to confirm this is what we want. At the
end stewardship is a delegation of permissions, it makes sense that
delegation happens to other teams. Also changed DatasetCreateForm
- [X] RequestDashboardAccessModal.js - already implemented, minor
changes
- [X] EnvironmentTeamInviteForm.js - already implemented, minor changes.
-> Kept `freesolo` because invited teams might not be the user teams.
Same reason why there is no check for groupOptions == 0, if there are no
options there is still the free text option.
- [X] EnvironmentRoleAddForm.js
- [X] NetworkCreateModal.js 

### Relates
- #1012 

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit ee71d7b 
Author: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> 
Date: Thu Jun 27 2024 14:08:27 GMT-0400 (Eastern Daylight Time) 

    [Gh 1301] Enhancement Feature - Bulk share reapply on dataset  (#1363)

### Feature or Bugfix
- Feature


### Detail

- Adds feature to reapply shares in bulk for a dataset. 
- Also contains bugfix for AWS worker lambda errors 

### Relates
- #1301
- #1364

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)? N/A
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization? N/A
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features? N/A
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users? N/A
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

---------

Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com>

commit 27f1ad7 
Author: Noah Paige <69586985+noah-paige@users.noreply.github.com> 
Date: Thu Jun 27 2024 13:18:32 GMT-0400 (Eastern Daylight Time) 

    Convert Dataset Lock Mechanism to Generic Resource Lock (#1338)

### Feature or Bugfix
<!-- please choose -->
- Feature
- Bugfix
- Refactoring

### Detail
- Convert Dataset Lock Mechanism to Generic Resource Lock
- Extend locking to Share principals (i.e. EnvironmentGroup and
Consumption Roles)

- Making locking a generic component not tied to datasets


### Relates
- #1093 

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

---------

Co-authored-by: dlpzx <dlpzx@amazon.com>

commit e3b8658 
Author: Petros Kalos <kalosp@amazon.com> 
Date: Thu Jun 27 2024 12:50:59 GMT-0400 (Eastern Daylight Time) 

    ignore ruff change in blame (#1372)

### Feature or Bugfix
<!-- please choose -->
- Feature
- Bugfix
- Refactoring

### Detail
- <feature1 or bug1>
- <feature2 or bug2>

### Relates
- <URL or Ticket>

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit 2e80de4 
Author: dlpzx <71252798+dlpzx@users.noreply.github.com> 
Date: Thu Jun 27 2024 10:59:18 GMT-0400 (Eastern Daylight Time) 

    Generic shares_base module and specific s3_datasets_shares module - part 11 (renaming and cleaning up s3_shares) (#1359)

### Feature or Bugfix
- Refactoring

### Detail
As explained in the design for #1123 and #1283 we are trying to
implement generic `datasets_base` and `shares_base` modules that can be
used by any type of datasets and by any type of shareable object in a
generic way.

This is one of the last PRs focused on renaming files and cleaning-up
the s3_datasets_shares module. The first step is a consolidation of the
file and classes names in the services to clearly refer to s3_shares:
- `services.managed_share_policy_service.SharePolicyService` --->
`services.s3_share_managed_policy_service.S3SharePolicyService`
- `services.dataset_sharing_alarm_service.DatasetSharingAlarmService`
--> `services.s3_share_alarm_service.S3ShareAlarmService`
- `services.managed_share_policy_service.SharePolicyService` -->
`services.s3_share_managed_policy_service.S3SharePolicyService`

👀 The main refactoring happens in what is used to be
`services.dataset_sharing_service`.
- The part that implements the `DatasetServiceInterface` has been moved
to `services/s3_share_dataset_service.py` as the `S3ShareDatasetService`
- The part used in the resolvers and by other methods has been renamed
as `services.s3_share_service.py` and the methods for the folder/table
permissions are also added to the S3ShareService (from
share_item_service)

Lastly, there is one method previously in share_item_service that has
been moved to the GlueClient directly as
`get_glue_database_from_catalog`.


### Relates
- #1283 
- #1123 
- #955 

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit 1c09015 
Author: Noah Paige <69586985+noah-paige@users.noreply.github.com> 
Date: Thu Jun 27 2024 04:16:14 GMT-0400 (Eastern Daylight Time) 

    fix listOrganizationGroupPermissions (#1369)

### Feature or Bugfix
<!-- please choose -->
- Bugfix


### Detail
- Fix listOrganizationGroupPermissions


### Relates
- <URL or Ticket>

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit 976ec6b 
Author: dlpzx <71252798+dlpzx@users.noreply.github.com> 
Date: Thu Jun 27 2024 04:13:14 GMT-0400 (Eastern Daylight Time) 

    Add search (Autocomplete) in create pipelines (#1368)

### Feature or Bugfix
- Feature

### Detail
Autocomplete for environments and teams in the following frontend views
as requested in #1012.
This PR implements it for createPipelines

### Relates
- #1012 

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit 6c909a3 
Author: Noah Paige <69586985+noah-paige@users.noreply.github.com> 
Date: Wed Jun 26 2024 11:18:04 GMT-0400 (Eastern Daylight Time) 

    fix migration to not rely on OrganizationService or RequestContext (#1361)

### Feature or Bugfix
<!-- please choose -->
- Bugfix

### Detail
- Ensure migration script does not need RequestContext - otherwise fails
in migration trigger lambda as context info not set / available


### Relates
- #1306

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit 90835fb 
Author: Anushka Singh <anushka.singh.2511@gmail.com> 
Date: Wed Jun 26 2024 11:17:22 GMT-0400 (Eastern Daylight Time) 

    Issue1248: Persistent Email Reminders (#1354)

### Feature or Bugfix
- Feature


### Detail
- When a share request is initiated and remains pending for an extended
period, dataset producers will receive automated email reminders at
predefined intervals. These reminders will prompt producers to either
approve or extend the share request, thereby preventing delays in
accessing datasets.

Attaching screenshots for emails:

<img width="1336" alt="Screenshot 2024-06-20 at 5 34 31 PM"
src="https://github.com/data-dot-all/dataall/assets/26413731/d7be28c3-5c98-4146-92b1-295e136137a3">

<img width="1322" alt="Screenshot 2024-06-20 at 5 34 52 PM"
src="https://github.com/data-dot-all/dataall/assets/26413731/047556e8-59ee-4ebf-b8a7-c0a6684e2a63">


- Email will be sent every Monday at 9am UTC. Schedule can be changed in
cron expression in container.py

### Relates
- #1248

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com>
Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com>
Co-authored-by: Mohit Arora <marora@yahooinc.com>
Co-authored-by: rbernota <rbernota@yahooinc.com>
Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com>
Co-authored-by: Raj Chopde <rchopde@yahooinc.com>
Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com>
Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: jaidisido <jaidisido@gmail.com>
Co-authored-by: dlpzx <dlpzx@amazon.com>
Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com>
Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com>
Co-authored-by: MK <manjula_kasturi@hotmail.com>
Co-authored-by: Manjula <manjula.kasturi@gmail.com>
Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com>
Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com>
Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com>
Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com>
Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com>
Co-authored-by: Sofia Sazonova <sofia-s@304.ru>
Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk>

commit e477bdf 
Author: Noah Paige <69586985+noah-paige@users.noreply.github.com> 
Date: Wed Jun 26 2024 10:39:09 GMT-0400 (Eastern Daylight Time) 

    Enforce non null on GQL query string if non null defined (#1362)

### Feature or Bugfix
<!-- please choose -->
- Bugfix


### Detail
- Add `String!` to ensure non null input argument on FE if defined as
such on backend GQL operation for `listS3DatasetsSharedWithEnvGroup`


### Relates

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit d6b59b3 
Author: Noah Paige <69586985+noah-paige@users.noreply.github.com> 
Date: Wed Jun 26 2024 08:48:52 GMT-0400 (Eastern Daylight Time) 

    Fix Init Share Base (#1360)

### Feature or Bugfix
<!-- please choose -->
- Bugfix

### Detail
- Need to register processors in init for s3 dataset shares API module


### Relates

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

commit bd3698c 
Author: Petros Kalos <kalosp@amazon.com> 
Date: Wed Jun 26 2024 05:19:14 GMT-0400 (Eastern Daylight Time) 

    split cognito urls setup and cognito user creation (#1366)

### Feature or Bugfix
- Bugfix
### Details
For more details about the issue read #1353 
In this PR we are solving the problem by splitting the configuration of
Cognito in 2.
* First part (cognito_users_config.py) is setting up the required groups
and users and runs after UserPool deployment
* Second part (cognito_urls_config.py) is setting up Cognito's
callback/logout urls and runs after the CloudFront deployment

We chose to split the functionality because we need to have the
users/groups setup for the integration tests which are run after the
backend deployment.

The other althernative is to keep the config functionality as one but
make the integ tests run after CloudFront stage.

### Relates
- Solves #1353 

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
  • Loading branch information
noah-paige committed Jul 8, 2024
1 parent e9c68e5 commit f51bc35
Show file tree
Hide file tree
Showing 76 changed files with 1,963 additions and 1,395 deletions.
2 changes: 2 additions & 0 deletions .git-blame-ignore-revs
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,5 @@

# Enforce frontend styling and remove dead code
fefc45c2cdc4d3107369c4d70210894d098a775c
# Ignore backend ruff format/lint
a35a4c7c4de78f0b38502882382f07aeac815b3c
2 changes: 1 addition & 1 deletion backend/dataall/base/utils/json_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ def to_json(record):
elif isinstance(record, type({'a': 'dict'})):
return json.loads(json.dumps(record, default=json_decoder))
elif type(record) in [str, 'unicode']:
return record
return json.dumps(record)
elif type(record) in [int, float]:
return json.dumps(record)
elif isinstance(record, bool):
Expand Down
2 changes: 1 addition & 1 deletion backend/dataall/core/organizations/api/resolvers.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ def send_query_chatbot(context, source, queryString):


def list_group_organization_permissions(context, source, organizationUri, groupUri):
return OrganizationService.list_group_organization_permissions(organizationUri, groupUri)
return OrganizationService.list_group_organization_permissions(uri=organizationUri, groupUri=groupUri)


def list_invited_organization_permissions_with_descriptions(context, source):
Expand Down
3 changes: 0 additions & 3 deletions backend/dataall/core/resource_lock/db/resource_lock_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,17 @@ class ResourceLock(Base):

resourceUri = Column(String, nullable=False, primary_key=True)
resourceType = Column(String, nullable=False, primary_key=True)
isLocked = Column(Boolean, default=False)
acquiredByUri = Column(String, nullable=True)
acquiredByType = Column(String, nullable=True)

def __init__(
self,
resourceUri: str,
resourceType: str,
isLocked: bool = False,
acquiredByUri: Optional[str] = None,
acquiredByType: Optional[str] = None,
):
self.resourceUri = resourceUri
self.resourceType = resourceType
self.isLocked = isLocked
self.acquiredByUri = acquiredByUri
self.acquiredByType = acquiredByType
91 changes: 53 additions & 38 deletions backend/dataall/core/resource_lock/db/resource_lock_repositories.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,35 +2,23 @@

from dataall.core.resource_lock.db.resource_lock_models import ResourceLock
from sqlalchemy import and_, or_
from sqlalchemy.orm import Session
from time import sleep
from typing import List, Tuple
from contextlib import contextmanager
from dataall.base.db.exceptions import ResourceLockTimeout

log = logging.getLogger(__name__)

MAX_RETRIES = 10
RETRY_INTERVAL = 60

class ResourceLockRepository:
@staticmethod
def create_resource_lock(
session, resource_uri, resource_type, is_locked=False, acquired_by_uri=None, acquired_by_type=None
):
resource_lock = ResourceLock(
resourceUri=resource_uri,
resourceType=resource_type,
isLocked=is_locked,
acquiredByUri=acquired_by_uri,
acquiredByType=acquired_by_type,
)
session.add(resource_lock)
session.commit()

@staticmethod
def delete_resource_lock(session, resource_uri):
resource_lock = session.query(ResourceLock).filter(ResourceLock.resourceUri == resource_uri).first()
session.delete(resource_lock)
session.commit()

class ResourceLockRepository:
@staticmethod
def acquire_locks(resources, session, acquired_by_uri, acquired_by_type):
def _acquire_locks(resources, session, acquired_by_uri, acquired_by_type):
"""
Attempts to acquire one or more locks on the resources identified by resourceUri and resourceType.
Attempts to acquire/create one or more locks on the resources identified by resourceUri and resourceType.
Args:
resources: List of resource tuples (resourceUri, resourceType) to acquire locks for.
Expand All @@ -47,19 +35,22 @@ def acquire_locks(resources, session, acquired_by_uri, acquired_by_type):
and_(
ResourceLock.resourceUri == resource[0],
ResourceLock.resourceType == resource[1],
~ResourceLock.isLocked,
)
for resource in resources
]
resource_locks = session.query(ResourceLock).filter(or_(*filter_conditions)).with_for_update().all()

# Ensure lock record found for each resource
if len(resource_locks) == len(resources):
# Update the attributes of the ResourceLock object
for resource_lock in resource_locks:
resource_lock.isLocked = True
resource_lock.acquiredByUri = acquired_by_uri
resource_lock.acquiredByType = acquired_by_type
if not session.query(ResourceLock).filter(or_(*filter_conditions)).first():
records = []
for resource in resources:
records.append(
ResourceLock(
resourceUri=resource[0],
resourceType=resource[1],
acquiredByUri=acquired_by_uri,
acquiredByType=acquired_by_type,
)
)
session.add_all(records)
session.commit()
return True
else:
Expand All @@ -74,9 +65,9 @@ def acquire_locks(resources, session, acquired_by_uri, acquired_by_type):
return False

@staticmethod
def release_lock(session, resource_uri, resource_type, share_uri):
def _release_lock(session, resource_uri, resource_type, share_uri):
"""
Releases the lock on the resource identified by resource_uri, resource_type.
Releases/delete the lock on the resource identified by resource_uri, resource_type.
Args:
session (sqlalchemy.orm.Session): The SQLAlchemy session object used for interacting with the database.
Expand All @@ -96,7 +87,6 @@ def release_lock(session, resource_uri, resource_type, share_uri):
and_(
ResourceLock.resourceUri == resource_uri,
ResourceLock.resourceType == resource_type,
ResourceLock.isLocked,
ResourceLock.acquiredByUri == share_uri,
)
)
Expand All @@ -105,10 +95,7 @@ def release_lock(session, resource_uri, resource_type, share_uri):
)

if resource_lock:
resource_lock.isLocked = False
resource_lock.acquiredByUri = ''
resource_lock.acquiredByType = ''

session.delete(resource_lock)
session.commit()
return True
else:
Expand All @@ -120,3 +107,31 @@ def release_lock(session, resource_uri, resource_type, share_uri):
session.rollback()
log.error('Error occurred while releasing lock:', e)
return False

@staticmethod
@contextmanager
def acquire_lock_with_retry(
resources: List[Tuple[str, str]], session: Session, acquired_by_uri: str, acquired_by_type: str
):
retries_remaining = MAX_RETRIES
log.info(f'Attempting to acquire lock for resources {resources} by share {acquired_by_uri}...')
while not (
lock_acquired := ResourceLockRepository._acquire_locks(
resources, session, acquired_by_uri, acquired_by_type
)
):
log.info(
f'Lock for one or more resources {resources} already acquired. Retrying in {RETRY_INTERVAL} seconds...'
)
sleep(RETRY_INTERVAL)
retries_remaining -= 1
if retries_remaining <= 0:
raise ResourceLockTimeout(
'process shares',
f'Failed to acquire lock for one or more of {resources=}',
)
try:
yield lock_acquired
finally:
for resource in resources:
ResourceLockRepository._release_lock(session, resource[0], resource[1], acquired_by_uri)
2 changes: 1 addition & 1 deletion backend/dataall/modules/catalog/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ def __init__(self):


class CatalogAsyncHandlersModuleInterface(ModuleInterface):
"""Implements ModuleInterface for datapipelines async lambda"""
"""Implements ModuleInterface for catalog async lambda"""

@staticmethod
def is_supported(modes: Set[ImportMode]):
Expand Down
1 change: 1 addition & 0 deletions backend/dataall/modules/catalog/api/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
fields=[
gql.Field(name='nodeUri', type=gql.ID),
gql.Field(name='parentUri', type=gql.NonNullableType(gql.String)),
gql.Field(name='status', type=gql.NonNullableType(gql.String)),
gql.Field(name='owner', type=gql.NonNullableType(gql.String)),
gql.Field(name='path', type=gql.NonNullableType(gql.String)),
gql.Field(name='label', type=gql.NonNullableType(gql.String)),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ def run_ecs_reindex_catalog_task(engine, task: Task):
CatalogIndexerTask.index_objects(engine, str(task.payload.get('with_deletes', False)))
else:
ecs_task_arn = Ecs.run_ecs_task(
task_definition_param='ecs/task_def_arn/share_management',
container_name_param='ecs/container/share_management',
task_definition_param='ecs/task_def_arn/catalog_indexer',
container_name_param='ecs/container/catalog_indexer',
context=[
{'name': 'with_deletes', 'value': str(task.payload.get('with_deletes', False))},
],
Expand Down
10 changes: 0 additions & 10 deletions backend/dataall/modules/catalog/services/catalog_service.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,6 @@
import logging

from dataall.base.context import get_context
from dataall.core.permissions.services.tenant_policy_service import TenantPolicyService

from dataall.modules.catalog.db.glossary_repositories import GlossaryRepository
from dataall.modules.catalog.db.glossary_models import GlossaryNode
from dataall.modules.catalog.services.glossaries_permissions import MANAGE_GLOSSARIES
from dataall.modules.catalog.indexers.registry import GlossaryRegistry
from dataall.core.permissions.services.tenant_policy_service import TenantPolicyValidationService
from dataall.core.tasks.db.task_models import Task
from dataall.core.tasks.service_handlers import Worker
Expand All @@ -15,10 +9,6 @@
logger = logging.getLogger(__name__)


def _session():
return get_context().db_engine.scoped_session()


class CatalogService:
@staticmethod
def start_reindex_catalog(with_deletes: bool) -> bool:
Expand Down
18 changes: 18 additions & 0 deletions backend/dataall/modules/dashboards/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,3 +75,21 @@ def __init__(self):

DashboardCatalogIndexer()
log.info('Dashboard catalog indexer task has been loaded')


class DashboardAsyncHandlersModuleInterface(ModuleInterface):
"""Implements ModuleInterface for dashboard async lambda"""

@staticmethod
def is_supported(modes: Set[ImportMode]):
return ImportMode.HANDLERS in modes

@staticmethod
def depends_on() -> List[Type['ModuleInterface']]:
from dataall.modules.catalog import CatalogAsyncHandlersModuleInterface

return [CatalogAsyncHandlersModuleInterface]

def __init__(self):
pass
log.info('S3 Dataset handlers have been imported')
Original file line number Diff line number Diff line change
Expand Up @@ -165,9 +165,6 @@ def create_dataset(uri, admin_group, data: dict):
DatasetService.check_imported_resources(dataset)

dataset = DatasetRepository.create_dataset(session=session, env=environment, dataset=dataset, data=data)
ResourceLockRepository.create_resource_lock(
session=session, resource_uri=dataset.datasetUri, resource_type=dataset.__tablename__
)
DatasetBucketRepository.create_dataset_bucket(session, dataset, data)

ResourcePolicyService.attach_resource_policy(
Expand Down Expand Up @@ -413,7 +410,6 @@ def delete_dataset(uri: str, delete_from_aws: bool = False):
ResourcePolicyService.delete_resource_policy(session=session, resource_uri=uri, group=env.SamlGroupName)
if dataset.stewards:
ResourcePolicyService.delete_resource_policy(session=session, resource_uri=uri, group=dataset.stewards)
ResourceLockRepository.delete_resource_lock(session=session, resource_uri=dataset.datasetUri)
DatasetRepository.delete_dataset(session, dataset)

if delete_from_aws:
Expand Down
42 changes: 32 additions & 10 deletions backend/dataall/modules/s3_datasets_shares/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,37 @@ def depends_on() -> List[Type['ModuleInterface']]:
def __init__(self):
from dataall.core.environment.services.environment_resource_manager import EnvironmentResourceManager
from dataall.modules.s3_datasets_shares import api
from dataall.modules.s3_datasets_shares.services.managed_share_policy_service import SharePolicyService
from dataall.modules.s3_datasets_shares.services.s3_share_managed_policy_service import S3SharePolicyService
from dataall.modules.s3_datasets.services.dataset_service import DatasetService
from dataall.modules.datasets_base.services.dataset_list_service import DatasetListService
from dataall.modules.s3_datasets_shares.services.dataset_sharing_service import DatasetSharingService
from dataall.modules.s3_datasets_shares.db.share_object_repositories import ShareEnvironmentResource
from dataall.modules.s3_datasets_shares.services.s3_share_dataset_service import S3ShareDatasetService
from dataall.modules.s3_datasets_shares.db.s3_share_object_repositories import S3ShareEnvironmentResource
from dataall.modules.shares_base.services.share_processor_manager import (
ShareProcessorManager,
ShareProcessorDefinition,
)
from dataall.modules.shares_base.services.shares_enums import ShareableType
from dataall.modules.s3_datasets.db.dataset_models import DatasetTable, DatasetBucket, DatasetStorageLocation

EnvironmentResourceManager.register(S3ShareEnvironmentResource())
DatasetService.register(S3ShareDatasetService())
DatasetListService.register(S3ShareDatasetService())

ShareProcessorManager.register_processor(
ShareProcessorDefinition(ShareableType.Table, None, DatasetTable, DatasetTable.tableUri)
)
ShareProcessorManager.register_processor(
ShareProcessorDefinition(ShareableType.S3Bucket, None, DatasetBucket, DatasetBucket.bucketUri)
)
ShareProcessorManager.register_processor(
ShareProcessorDefinition(
ShareableType.StorageLocation,
None,
DatasetStorageLocation,
DatasetStorageLocation.locationUri,
)
)

EnvironmentResourceManager.register(ShareEnvironmentResource())
DatasetService.register(DatasetSharingService())
DatasetListService.register(DatasetSharingService())
log.info('API of dataset sharing has been imported')


Expand All @@ -55,7 +77,7 @@ def depends_on() -> List[Type['ModuleInterface']]:
]

def __init__(self):
log.info('S3 Sharing handlers have been imported')
log.info('s3_datasets_shares handlers have been imported')


class S3DatasetsSharesCdkModuleInterface(ModuleInterface):
Expand All @@ -67,9 +89,9 @@ def is_supported(modes):

def __init__(self):
import dataall.modules.s3_datasets_shares.cdk
from dataall.modules.s3_datasets_shares.services.managed_share_policy_service import SharePolicyService
from dataall.modules.s3_datasets_shares.services.s3_share_managed_policy_service import S3SharePolicyService

log.info('CDK module data_sharing has been imported')
log.info('CDK module s3_datasets_shares has been imported')


class S3DatasetsSharesECSShareModuleInterface(ModuleInterface):
Expand Down Expand Up @@ -122,4 +144,4 @@ def __init__(self):
)
)

log.info('ECS Share module s3_data_sharing has been imported')
log.info('ECS Share module s3_datasets_shares has been imported')
8 changes: 8 additions & 0 deletions backend/dataall/modules/s3_datasets_shares/api/mutations.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from dataall.base.api import gql
from dataall.modules.s3_datasets_shares.api.resolvers import (
verify_dataset_share_objects,
reapply_share_items_share_object_for_dataset,
)


Expand All @@ -10,3 +11,10 @@
type=gql.Boolean,
resolver=verify_dataset_share_objects,
)

reApplyShareObjectItemsOnDataset = gql.MutationField(
name='reApplyShareObjectItemsOnDataset',
args=[gql.Argument(name='datasetUri', type=gql.NonNullableType(gql.String))],
type=gql.Boolean,
resolver=reapply_share_items_share_object_for_dataset,
)
Loading

0 comments on commit f51bc35

Please sign in to comment.