Include IAM pivot role as part of the environment CDK stack

[dlpzx]

Is your idea related to a problem? Please describe.
I am always frustrated when the pivot role needs a new policy and I have to manually update the pivot role in all my data.all environment accounts.

Describe the solution you'd like
I would like an automatic way of creating and updating the IAM pivot role. As a suggestion, we could add it as part of the environment CDK stack. The downside of this is that we loose the explicit granting to data.all access to the account.

With the manual creation of the pivot role we ensure that an account manager has to create this role before data.all links it as an environment. However, this control is also present in the --trust of the bootstraping command that prevents data.all to deploy stacks in the chosen account. For this reason we could afford to add the pivot role in the environment stack and remove the manual creation of it.

In addition, this feature should also be backwards compatible with accounts that created the pivot role independently from the environment stack.

[dlpzx]

Hi @dosiennik, I am reviewing the code pieces that you added to the branch. Thank you for the contribution! I have rebased the branch onto v1.4.0 to get the latest changes.
There are 2 TODOs that you pointed out where we use the pivotRole to execute SDK calls directly from the environment stack. I am adding those points here for visibility and to discuss them:

1. Creation of Quicksight default group "dataall". Quicksight groups are used:
   a) to add QS users when we register them. In this case we also check the existance of the default QS group and create it if the QS group does not exist --> not affected by removing QS group creation from environment
   b) when a dataset is created we grant LF permissions to this group. As with the case a) we check if the group exists directly in the dataset stack (check cdkproxy/stacks/dataset.py) --> not affected by removing QS group creation from environment
   c) !!!! when tables are shared we grant LF permissions to the requester account QS group. In this case we are directly adding the target QS group as principal in the share request (e.g. /backend/dataall/tasks/data_sharing/share_processors/lf_process_cross_account_share.py V1.4.0) If the QS group does not exist I am not sure how LakeFormation will treat it. --> I think we should do the same as with a) and b) and create the QS group in the target if it does not exist.

As a final note: We use one single QS group per AWS account, which does not follow the ata.all principle of data access at team level. This is legacy from the initial implementation, but in the future QS groups should match data.all teams. In other words, the changes in c) will help us whenever we want to implement this QS change.

2. check_sagemaker_studio is a function that checks existing domains created by dataall or outside of data.all and then creates a domain if needed. The studio domain is used later in the ML Studio sessions. I think we can use this opportunity to move this code from the environment to the ML Studio and create it in a lazy mode. --> So we need some changes

3. I am adding this third point. Use of the pivot role in the LF and Glue custom resources --> I checked that you added the pivot role construct and there are no hardcoded API calls with the pivot role so we should be fine :)

Changes needed for:

• Quicksight group in the sharing tasks
• SageMaker domain in the MLStudio stack

[dlpzx]

@dosiennik, right now it is possible to create the pivotRole as part of the environment stack:

1. In the different places mentioned a)creation of QS users inside a QS group, b) granting permissions to QS group at dataset creation and c) granting permissions to QS group in share request. We first check if the QS default group has been created and we create it if it does not exist. --> We do not need the init_quicksight method anymore
2. SageMaker Studio domain look ups are done using the CDK role for look ups instead of using the pivot role. In addition, the SageMaker Studio domain is extracted as a nested stack for clarity.
3. Instead of reading whether or not the pivot role needs to be created from the ECS environment variables, I created an SSM parameter to decide on whether to create (or not) the pivot role as part of the environment.

So far, this works for newly created environments, now I am going to work on its backwards compatibility. Following your example, I added the "-cdk" suffix to the IAM role to avoid issues with LakeFormation data lake orphan admins and Environment custom resources.

[dlpzx]

For existing deployments of data.all that want to switch to environmentCreatedPivotRole. To update previously created environments we have 4 options:

1. Manually click on "Update stack" on the Environment UI > Stack tab
2. Wait for the daily scheduled task that updated all stacks. "stack_updater"
3. Manually trigger the "stack_updated" task with the following command:
   aws ecs run-task --task-definition "arn:aws:ecs:eu-central-1:111111111111:task-definition/dataall-dev-stacks-updater:340" --cluster "arn:aws:ecs:eu-central-1:111111111111:cluster/dataall-dev-cluster" --launch-type "FARGATE" --network-configuration "awsvpcConfiguration={subnets=[subnet-.....,subnet-....,subnet-....], securityGroups=[sg-....],assignPublicIp=DISABLED}"
4. Develop a new stage in the CICD pipeline that triggers the stack updater task with every run of the CodePipeline pipeline

For the moment I will leave the first 3 options documented and add the fourth as a feature request that can be picked in future releases as an enhancement

[dlpzx]

Completed in #409