Spark : Spark3Util is not setting the spark session being used as active session when executing sensitive functions

[ahshahid]

Some of the functions of Spark3Util which are internally invoking the catalog being used by the current SparkSession, do not get the right Catalog Instance, as the current session is not being set as the active one , in the ThreadLocal of SparkSession. As a result , the catalog instance being retrieved is the wrong one ( of the default spark session which got created first).

The bug shows up if a new session is created and iceberg's SparkSessionCatalog is set in the new session instead of the original Session.
The fix could be

1. Either we document that some methods of Spark3Util be invoked only after setting the current session as active one
   Or
2. The Spark3Util's code sets the session as active one.

Creating a PR based on #2
The PR has the bug test

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'