feat: include iam pivot role as part of the environment cdk stack

[dlpzx]

Feature or Bugfix

• Feature

Detail

⚠️ ⚠️ ⚠️ IMPORTANT ⚠️ ⚠️ ⚠️

IF YOU ARE MIGRATING FROM MANUAL TO CDK-CREATED ROLE, WHILE THE PIPELINE IS UPDATING THE BACKEND CODE, UPDATING ENVIRONMENT/DATASET STACKS AND UPDATING THE FRONTEND CODE YOU WILL EXPERIENCE DOWN TIMES. PLAN THE MIGRATION (AROUND 1-2H) IN A TIME WINDOW THAT DOES NOT IMPACT YOUR USERS.

Configuration of manual or cdk-created pivotRole

With this PR customers will be able to use data.all and link environments using a manually created pivotRole or a cdk-pivotRole deployed as part of the environment stack. What to use is configurable in the cdk.json for each infra account. For example, customers can have manually created pivotRoles in "dev" and cdk-created ones in "prod". The configuration affects all environments of the deployment. In this example, environments linked to "prod" will always deploy a cdk-PivotRole, it is not possible to have a mix of manual and cdk roles in the same infra deployment.

• Added configuration parameter in cdk.json to enable or disable pivotRole creation as part of environment in each development environment:
  "enable_pivot_role_auto_create": "boolean_ENABLE_PIVOT_ROLE_AUTO_CREATE_IN_ENVIRONMENT|DEFAULT=false",

• Creation of SSM Parameter that stores this flag in each of the development environments. (Screenshot outdated, not the parameter is called enablePivotRoleAutoCreate to match the cdk.json parameter name
  

• Modification of the "dataall-pivotrole-name" Secret in central account. For pivotRole created manually it stays in "dataallPivotRole" for cdk-created ones "dataallPivotRole-cdk" to avoid issues with conflicting IAM roles.

• Added CDK nested stack for PivotRole IAM role and add it to the Environment stack. It is deployed depending on the value of the created SSM parameter. Permissions that reference the pivotRole are adapted to use the pivotRole-cdk when it is enabled.

• Conditional pre-requisites box in CreateEnvironment UI view depending on the value of the SSM Parameter. The value of the SSM parameter is written to the .env file of the React application and is used in the frontend/src/views/Environments/EnvironmentCreateForm.js.
  (With enable_pivot_role_auto_create: false)

(With enable_pivot_role_auto_create: true)

Refactor AWS account checks

In the resolvers and in the environment stack deployment originally we performed some checks in the account with boto3 calls assuming that the pivotRole was created. In this PR some of this checks are now effectuated with the cdk look up role and other tests have been moved to other places in the code. In addition, more tests are added to consider all scenarios (e.g. check that pivotRole exists in manual-pivot role deployments).

• Modified check of CDKToolkit CFN stack to avoid using pivotRole

• Added check of pivotRole creation (for manual created pivotRole) with CDK look-up role.

• Added check_environment methods to the update_environment API call.

• Previous check of existing SageMaker Studio domain when MLStudio is enabled is now executed with cdk look up role

• Previous check of Quicksight subscription when Dashboards is enabled now happens after environment creation, when a Dataset is created or when a Dashboard-Quicksight session is started.

• Previous creation of data.all Quicksight default group when Dashboards is enabled now happens after environment creation, when a Dataset is created or when a Dashboard-Quicksight session is started.

Migrating from manual to cdk-created pivotRole

This PR takes into consideration the scenario in which customers already have a deployment of data.all with environments linked using a manually created pivotRole and want to leverage cdk-created pivotRoles.

To do that, customers just need to add the enable_pivot_role_auto_create parameter in their cdk.json configuration and set it to true. Once the CICD pipeline has completed: new linked environments will contain the nested cdk-pivotRole stack (no actions needed) and existing environments can be updated by:
a) manually, by clicking on "update stack" in the environment>stack tab
b) automatically, wait for the stack-updater ECS task that runs daily overnight
c) automatically, set the added update_dataall_stacks_in_cicd_pipeline parameter to true in the cdk.json config file. The stack-updater ECS task will be triggered from the CICD pipeline

• Added configuration parameter in cdk.json to add a CodeBuild stage in CICD pipeline to trigger update of environments and datasets stacks:
  "enable_update_dataall_stacks_in_cicd_pipeline": "boolean_ENABLE_UPDATE_DATAALL_STACKS_IN_CICD_PIPELINE|DEFAULT=false"
  This parameter can be set back to false whenever customers do not foresee many changes in the dataset and environment stacks and back to true when we want to force the update of stacks. This can be useful to ensure changes in the stacks are applied immediately instead of waiting for the stack-updater to run overnight.

• Modified stacks-updater ECS task to run environment updates first, wait until completion and then update dataset stacks. The reason of keeping this order is that the datasets of an environment use Lambda functions created in the environment stack as Custom resources (glue-db-handler). The original Lambda function needs to be updated to use the new pivotRole before being used in the Dataset stack. Plus, Quicksight group creation which happens in the dataset stack uses the pivotRole because the cdk-look-up role has no Quicksight permissions.

• Replaced the CFN AWS::LakeFormation::Resource by a Custom Resource and added the necessary 'lakeformation:De/registerResource` permissions to Pivot Role. Using the Cfn resource directly caused internal Failure errors. To have more control over the update we now execute it as a custom resource (in the same way as we do with the glue-db-handler).

Documentation

• Userguide changes in environment section
• GitHub pages changes (in a different PR)

Additional enhancements

• For the dataset custom resources, now the provider creation has been moved to the environment stack avoiding Lambda policy size limitations, the SSM parameters for the Lambda arn and name are left for reference if needed.
• Modified Default Lake Formation settings custom resource in environment stack: added validation of IAM roles when adding LakeFormation data lake admins, if the IAM roles do not exist, data.all removes them from data lake admins to avoid errors. It now removes pivotRole (manual or cdk) from LakeFormation data lake admins on environment stack deletion.
• Make team IAM roles policies unique by adding environmentUri to their policy names
• bugfix: Fix local imports of shareItemSM for Docker and quicksight properties
• Isolated SageMaker Studio domain stack in a different nested stack in the Environment stack for readability

Relates

• PivotRole stack as CDK stack #251

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[nikpodsh]

I think it'd be nice to have a test that compares a manual pivot role and the automated one. Just in case when we added a new changes in one of them, but forgot to replicated it into the other. What do you think?

[kukushking]

Looks great!