[SPARK-42567][SS][SQL] Track load time for state store provider and log warning if it exceeds threshold

[anishshri-db]

What changes were proposed in this pull request?

Track load time for state store provider and log warning if it exceeds threshold

Why are the changes needed?

We have seen that the initial state store provider load can be blocked by external factors such as filesystem initialization. This log enables us to track cases where this load takes too long and we log a warning in such cases.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Augmented some of the tests to verify the logging is working as expected.
Sample logs:

14:58:51.784 WARN org.apache.spark.sql.execution.streaming.state.StateStore: Loaded state store provider in loadTimeMs=2049 for storeId=StateStoreId[ checkpointRootLocation=file:/Users/anish.shrigondekar/spark/spark/target/tmp/streaming.metadata-1f2ff296-1ece-4a0c-b4b4-48aa0e909b49/
state, operatorId=0, partitionId=2, storeName=default ] and queryRunId=a4063603-3929-4340-9920-eca206ebec36
14:58:53.838 WARN org.apache.spark.sql.execution.streaming.state.StateStore: Loaded state store provider in loadTimeMs=2046 for storeId=StateStoreId[ checkpointRootLocation=file:/Users/anish.shrigondekar/spark/spark/target/tmp/streaming.metadata-1f2ff296-1ece-4a0c-b4b4-48aa0e909b49/
state, operatorId=0, partitionId=3, storeName=default ] and queryRunId=a4063603-3929-4340-9920-eca206ebec36
14:58:55.885 WARN org.apache.spark.sql.execution.streaming.state.StateStore: Loaded state store provider in loadTimeMs=2044 for storeId=StateStoreId[ checkpointRootLocation=file:/Users/anish.shrigondekar/spark/spark/target/tmp/streaming.metadata-1f2ff296-1ece-4a0c-b4b4-48aa0e909b49/
state, operatorId=0, partitionId=4, storeName=default ] and queryRunId=a4063603-3929-4340-9920-eca206ebec36

[anishshri-db]

@HeartSaVioR - please take a look. Thx

[HeartSaVioR]

+1, only a nit.

[HeartSaVioR]

+1

[anishshri-db]

@HeartSaVioR - looks like the tests finished fine. Not sure why the Actions result is not updated here

[HeartSaVioR]

Thanks! Merging to master.