Resolve Dataset Profiling Glue Job

[noah-paige]

Feature or Bugfix

• Bugfix

Detail

• Specify SPARK_VERSION as an environment variable for pydeequ before import
• Add IAM Permissions to Dataset IAM Role to Allow for Glue Job logging in CloudWatch
• Add LF Permissions to resolve insufficient permissions error thrown when looking for default database

Relates

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[noah-paige]

To note for this PR - Glue Jobs require the permissions to be able to verify the existence of the default database or create the default database if it does not exist AWS Docs

This is needed for Glue Jobs that Use Data Catalog as the Hive metastore which is needed for our Glue Profiling Job

In order to accommodate this requirement, the PR adds code to grant DESCRIBE LF Permissions to the Dataset Principals (Dataset Role, EnvGroup Role, PivotRole) in the dataset Custom Resource Lambda

If the default database does not exist with the above changes in this PR the Glue Profiling Job will still fail as of now...

[dlpzx]

What happens if the default does not exist?

[dlpzx]

So now every time we create a dataset, the dataset admin role will get permissions to the default database. Permissions to this database are needed to run the profiling job. Do you know why? is it generic for any Glue Job you need permissions to the default database?

[dlpzx]

We also need to fix it in modularization-main, but we need to be careful with all the PRs that are currently open

[dlpzx]

And thank you, this is a long-needed fix :)

[nikpodsh]

@dlpzx, @noah-paige I will gladly replicate these changes to mod-main :)

[noah-paige]

So now every time we create a dataset, the dataset admin role will get permissions to the default database. Permissions to this database are needed to run the profiling job. Do you know why? is it generic for any Glue Job you need permissions to the default database?

I know in documentation it calls out that Glue will also try to verify the default DB only if you enable Use Data Catalog as the Hive metastore

The permissions required are only DESCRIBE on the DB level since all Glue is doing is verifying the default DB exists by calling GetDatabase API