[ADAP-405] [Feature] Use server_side_parameters as SparkSession config in Spark "session" mode

[alarocca-apixio]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Configuration for server_side_parameters already exists in the Spark configuration. However, these are currently unused in Spark "session" mode. We can pass these into SessionConnectionWrapper and then, when building the SparkSession, we can pass them as Spark "config":

for k, v in server_side_parameters.items():
            builder = builder.config(k, v)

Describe alternatives you've considered

We've tried Spark via Thrift, but this doesn't support all of our necessary use cases, such as configuring KMS encryption for the jobs.

Who will this benefit?

This will enable users to flexibly control the SparkSession that is created, giving users the ability to inject any parameters desired into the SparkSession.

Are you interested in contributing this feature?

Yes

Anything else?

No response

[Fleid]

Hey @alarocca-apixio, could you please check this discussion, I'm trying to regroup all the threads on that topic in one place.
Please let me know if the plan works for you, before we can decide to move forward with this specific issue.

[Fokko]

@alarocca-apixio @Fleid Any progress on this? I would love to see this added. Happy to help on this one, just commented on the PR.

[Fleid]

I asked @JCZuurmond to weigh in on the topic, but we haven't had time to catch-up.
Any update on your side Cor?

[JCZuurmond]

See PR for update 👆