Destination Databricks: Implement refreshes

[edgao]

closes https://github.com/airbytehq/airbyte-internal-issues/issues/8534. closes https://github.com/airbytehq/airbyte-internal-issues/issues/8835. closes https://github.com/airbytehq/airbyte-internal-issues/issues/8857#issuecomment-2259169235. Structurally identical to all the other refreshes PRs (e.g. #38713).

❗ this PR also unpins the current version, i.e. we will release the connector publicly here.

As a refresher:

• pass the raw table suffix around to the relevant places
• DestinationHandler needs to fetch an InitialStatus for the temp raw table in addition to the real raw table
  • I did some minor code clarification in this file
• StorageOperation needs to implement some new methods
  • See the new integration test class for how we verify those new methods
  • some code shuffling in the StreamOperation to enable that integration test
• some new expectedrecords.jsonl files (CDK bump pulled in some new test cases)
• fixes a bug in check (wasn't deleting the test table). Added a test case for this.
  • (this only really matters for us - we had an older test table without generation_id, which caused check to fail. Real users wouldn't run into this... but it feels like best practice to clean up ourselves)
• fixes a bug in StreamOperation, where we would write an empty file, which then caused an error in the COPY command

I'll do a prerelease image + set up a sync in the perf test workspace, but in the meantime, this should be ready for review.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 31, 2024 4:29pm

[edgao]

• Databricks: update spec+docs #42578
• Destination Databricks: Implement refreshes #40692 👈
• Destination Databricks: Add generation_id+sync_id #40689
• Destination Redshift: Implement refreshes #40567 : 1 other dependent PR (#40712 )
• Destination Redshift: Add generation_id and sync_id #40201
• Destination redshift: Use new interfaces #39008 : 1 other dependent PR (#39334 )
• Destination redshift: Remove encrypted staging option #39141
• Destination redshift: convert test classes to kotlin #38893
• Destination redshift: remove standard inserts #38886
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @edgao and the rest of your teammates on Graphite

[edgao]

the create schema call now happens via destinationhandler.createNamespaces, so remove it from here

[johnny-schmidt]

LGTM.

• the overridden methods do what they seem like they should
• the statements made about databricks in comments, etc are correct
• the net new behavior makes sense with my understanding of what supporting refreshes means
• the tests look thorough and correct but I didn't have time to go over them all in detail