Destination Databricks: Replicate data into new catalog instead of metastore

[mcfuhrt]

Tell us about the problem you're trying to solve

Currently, a new schema and the corresponding stream are created as a table in Metastor. I would like to store the replicated data in a dedicated catalogue and schema and thus use the three-level namespace.

In my opinion, this would allow use case or department related data to be better organised within databricks.

Describe the solution you’d like

To ensure backwards compatibility with databricks accounts that do not have unity catalog enabled, I would add a boolean to the configuration to enable the new logic. Once enabled, the streams will be replicated to a new catalog including the specified schema. Which catalogue is used can also be configured.

Are you willing to submit a PR?

Yes, PR is on the way

[evantahler]

Closing, as I believe this was fixed by #24657