[KED-1466] SparkHiveDataSet is slow to initialize in Databricks

[MigQ2]

Description

I'm using SparkHiveDataSet to connect with some Databricks tables and I have seen that the code in SparkHiveDataSet.__init__() loads the dataset to find the columns it has and checks if the table already exists. This triggers a short spark action which takes little time (~5 seconds) for each dataset when the catalog is initialized. Therefore, if I have hundreds of SparkHiveDataSets in my catalog initializing a kedro ipython session takes several minutes even if I just want to do a quick analysis with one dataset.

By reading the source code I have seen that the actions come from the following methods, which are invoked when a SparkHiveDataSet is initialized:

• SparkHiveDataSet._exists()
• SparkHiveDataSet._load()

Steps to Reproduce

1. Create many SparkHiveDataSet in the catalog that point to tables declared in Databricks
2. Initialize the kedro catalog

Expected Result

Kedro context and the catalog get initialized quickly (<30 seconds)

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro 0.15.5
• Python 3.7
• Databricks connect 6.4
• Windows 10

Possible solution

Don't trigger any spark action to inspect the schema when initializing the dataset, just when it needs to be accessed with _load() or _save(). Also caching the result could help speeding things up if the dataset is used many times in a single kedro run.

[andrii-ivaniuk]

Fixed in 8dc70bc