v2.3.0

What's Changed

⚠️ ⚠️ Important: After upgrading to v2.3.0 environment stacks need to be updated before executing data sharing requests. If the environment stack is not data sharing will fail. To update the environment stacks there are 3 options:

1. Using cdk.json parameter enable_update_dataall_stacks_in_cicd_pipeline --> automatically updates the environments and dataset stacks in the CICD pipeline
2. Waiting for overnight update stack task --> same as the above, but it runs at a daily schedule.
3. Updating environments in Environment > Stack tab > click on Update button --> manual update

New features 🆕

• Introduce dataset lock for data sharing, increasing robustness of parallel data sharing by @anushka-singh in #1072
• Add verification of data sharing and reapplying if "unhealthy" by @noah-paige in #1062
• Enable Central Catalog Glue databases import by @TejasRGitHub in #1021 and list them in worksheets in #1079
• Replace IAM inline policies by configurable Managed Policies for folder and bucket sharing by @SofiaSazonova and @dlpzx in #1068
• Simplify LakeFormation Glue database shares - single shared_db and single resource link table by @dlpzx in #1016 and add sharing guardrails drop permissions in #1055 and update Worksheet database names in UI in #1063
• Add data sharing auto-approval option for datasets by @SofiaSazonova in #988
• Introduce feature flags for topics and confidentiality and custom confidentiality list by @TejasRGitHub in #1049

Enhancements 🥇

• Enable key rotation for KMS in CodePipeline by @mourya-33 in #923
• Add support for custom environment linking text with sanitization by @zsaltys in #934
• Add KMS encryption for Aurora DB secrets by @mourya-33 in #935
• Implement Docker user directives by @mourya-33 in #895 and by @noah-paige in #968
• Add checkov GitHub actions by @dlpzx in #962
• Add word-wrap in strings in share lists by @dlpzx in #972
• Add logic to serialize bytes and bytearray datatypes to string by @awskaran in #973
• Add network information to listValidEnvironments by @dlpzx in #986
• Introduce data.all version parameter by @SofiaSazonova in #991
• Add WAF ACL to Cognito User Pool by @noah-paige in #976 and in #1097
• Add checkov baseline by @noah-paige in #1019
• Add dataset Description on shares UI page by @TejasRGitHub in #1026
• Allow update consumption role ownership by @petrkalos in #1020
• Add validation of AWS account and region in environment creation by @dlpzx in #1043
• Remove policies-updater ECS task by @dlpzx in #1046
• Remove git_release functionality by @dlpzx in #1042
• Clean-up auto create pivot permissions by @mourya-33 in #1075
• Add email notification metadata by @TejasRGitHub in #1082
• Add guardrails to alembic sync upgrade/downgrade by @noah-paige in #1084

Fixes 🪲

• Fix reAuth re-renders glitch by @noah-paige in #918
• Fix s3 bucket sharing for federated roles by @zsaltys in #920
• Fix Disappearing Env Value Request Access Modal by @noah-paige in #919
• Fix Frontend Config Role Issue while switching from Cognito Idp to Custom Auth by @TejasRGitHub in #938
• Investigate why some shares did not go to failed state (issue 932), but remained stuck or in-progress by @anushka-singh in #933
• Fix when migrating from Manually Created Pivot Role to Auto Create Pivot Role by @TejasRGitHub in #948
• Validate consumer roles by @SofiaSazonova in #951
• Fix local dev environment is broken after recent changes by @TejasRGitHub in #967
• Bugfix 956 by @anushka-singh in #961
• Add lakeformation in trust policy of dataset role by @dlpzx in #970
• Add else if condition to get tables into InSync state by @TejasRGitHub in #980
• Fix consumption role filtering by @TejasRGitHub in #975
• Replace dataall prefix by resourcePrefix in data pipeline creation by @dlpzx in #985
• Remove AWS Manged Lake Formation Service Linked Role from Pivot Role Nested Stack by @TejasRGitHub in #999
• Fix created dataset naming convention by @noah-paige in #1002
• Add CloudFormation permission to PivotRoleNestedStack by @TejasRGitHub in #1040
• Fix userguide dockerfile by @dlpzx in #1089
• Create DatasetLock for new datasets by @noah-paige in #1090
• Fix verify share table items and access point share no bucket policy by @noah-paige in #1095
• Add check and reapply for attaching S3 IAM policy by @dlpzx in #1096
• Fix counter on paged responses by @petrkalos in #1091
• Handle Error on clean up share and not get stuck in IN_PROGRESS status by @noah-paige in #1099
• Fix issue in SageMaker Create permissions by @dlpzx in #1102

Refactoring 💻

• Refactor Core/Organization to follow api/services/db layers by @dbalintx in #989
• Refactor Core/Vpc refactoring to follow api/services/db layers by @dlpzx in #1044
• Refactor Enums by @SofiaSazonova in #978

Documentation 📚

• Update Userguide documentation for v2.3 updates by @noah-paige in #1100
• Add alembic documentation by @SofiaSazonova in #1033

Dependencies 📦

• Upgrade Aurora postgreSQL engine 11 --> 13 by @noah-paige in #963
• Upgrade axios package to resolve follow-redirect vulnerability by @noah-paige in #952
• Remove unused packages: jinja2, deprecated by @dlpzx in #969
• Upgrade npm packages: axios, css-tools by @dlpzx in #1052
• Upgrade postcss and add yarn resolutions by @dlpzx in #1059
• Applyboto3==1.34.35 in DeployFrontend action by @anandsumit2000 in #1054
• Upgrade starlette version and dependecies to avoid ReDoS by @dlpzx in #1038
• Upgrade ip package in frontend for yarn and npm by @dlpzx in #1070

New Contributors 👨‍💻 👩‍💻

• @SofiaSazonova made their first contribution in #951
• @awskaran made their first contribution in #973
• @petrkalos made their first contribution in #1020
• @anandsumit2000 made their first contribution in #1054

Full Changelog: v2.2.0...v2.3.0