feat: Handle Pre-filtering of tables

[anushka-singh]

Feature or Bugfix

• Feature

Detail

• For a dataset to make sense all the tables within a dataset should have their location pointing to the same place as the dataset S3 bucket. However it is possible that a database can have tables which do not point to the same bucket which is perfectly legal in LakeFormation. Therefore we propose that data.all automatically only lists tables that have the same S3 bucket location as the dataset. This will solve a problem for Yahoo where we want to import a database that contains many tables with different buckets. Additionally Catalog UI should also only list prefiltered tables.

Testing

• Tested this in local env. I was able to create and share datasets even after pre-filtering process takes place.
• Will send separate PR for unit testing.

Relates:

• on import prefilter database tables based on table S3 location #745

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

• Does this PR introduce or modify any input fields or queries - this includes
  fetching data from storage outside the application (e.g. a database, an S3 bucket)?
  • Is the input sanitized?
  • What precautions are you taking before deserializing the data you consume?
  • Is injection prevented by parametrizing queries?
  • Have you ensured no eval or similar functions are used?
• Does this PR introduce any functionality or component that requires authorization?
  • How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  • Are you logging failed auth attempts?
• Are you using or adding any cryptographic features?
  • Do you use a standard proven implementations?
  • Are the used keys controlled by the customer? Where are they stored?
• Are you introducing any new policies/roles/users?
  • Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[noah-paige]

Tested the following in AWS Deployment

Pre-existing dataset with 2 tables:

• Dataset Synchronize successfully and loads 2 tables

Pre-existing dataset with 0 table:

• Dataset Synchronize successfully and loads 0 tables

Import new dataset (2 tables in 2 different buckets)

• Synchronize only loads 1 table
• Share Request for dataset with tables filtered and can only add the 1 table
• Create worksheet and can only query the 1 table

ECS Table Sync Task

• Successfully Runs and finds appropriate number of tables for each dataset

I think the code changes look good to me and UI working as expected! - approving to merge to v2m1m0