Expose location, clustered_by to dbt-spark

[NielsZeilemaker]

Implemented Location and Clustered By options which were not exposed as options.
https://docs.databricks.com/spark/latest/spark-sql/language-manual/create-table.html

[NielsZeilemaker]

I have a question though, how can I add some tests to verify if the sql statements which are generated are correct?

[NielsZeilemaker]

@drewbanin I've added some unit-tests for the macros as well

[aaronsteers]

I'm extremely excited about this feature! See my question above regarding a common location prefix for all tables in a project or schema.

[jtcohen6]

Thanks for the tremendous lift on this, @NielsZeilemaker! This will flesh out the dbt-spark plugin with a lot of sensible configuration options that work differently/only on Spark.

I have two asks:

• Could you add an AdapterSpecificConfigs block to imply.py that sets all the Spark-specific config? You can check out the ones for Redshift, Snowflake, BigQuery, and Postgres. Basically, this is what tells dbt to grab certain configuration keywords from dbt_project.yml and combine with the ones in model config blocks. You should include configs added in this PR as well as the ones already supported by the plugin (file_format, partition_by), because we're currently missing those, too :)
• Could you add information to the "Model Configuration" section of the README describing the config options you're adding in this PR?

[NielsZeilemaker]

@jtcohen6 I've implemented the changes, let me know if I can do anything else

[jtcohen6]

This is really close. Could you:

• Update location to location_root in two places (README + AdapterSpecificConfigs)
• Fix pep8 errors

[NielsZeilemaker]

@jtcohen6 done, fixed renamed location and fixed pep8

[jtcohen6]

Thank you for the great work!

[aaronsteers]

Fantastic!!! 🥇

[Dandandan]

Maybe would be good to document the persist_docs feature as well?

[NielsZeilemaker]

Its not specific to spark, bigquery has the same feature. Eg i guess it should be documented at the dbt level?

[Dandandan]

For BQ it is on the main dbt website indeed.

We might want to add the same info here https://docs.getdbt.com/docs/profile-spark

[Dandandan]

I suggested additions of persist_docs to the spark-specific page.

[Dandandan]

Btw, this feature (root_location) is incredible useful for the work at our current client!