🎉 Destination Databricks: Support Azure storage (#15140) #15329

abaerptc · 2022-08-04T19:24:30Z

What

Adds Azure storage as an option for Databricks destination.
This PR resolves #15140.

How

Adds Azure storage as an alternative to S3.

🚨 User Impact 🚨

No breaking changes.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Integration

None.

Acceptance

Cannot run S3 acceptance test locally.

CLAassistant · 2022-08-04T19:24:35Z

All committers have signed the CLA.

abaerptc · 2022-08-09T13:29:24Z

The Azure integration test passes, but I don't have access to Databricks on AWS, and it's proving difficult to configure Azure Databricks to write to S3, so I'm not able to fully run the S3 integration tests. I have confirmed that the data gets written to S3 fine, but still need to confirm that the tables can be created from the S3 location in Databricks. Could someone from Airbyte please trigger the integration tests to run from this PR?

marcosmarxm · 2022-08-09T17:10:07Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2827034508
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2827034508
🐛 https://gradle.com/s/77uzxvmvh373e

Build Failed

Test summary info:

Could not find result summary

abaerptc · 2022-08-09T21:52:25Z

I pushed a new commit that should fix the error seen in the previous run of the S3 acceptance test.

The Azure acceptance test will require the addition of a new secrets file by an Airbyte member (see sample_secrets/azure_config.json).

marcosmarxm · 2022-08-09T22:42:31Z

/test connector=connectors/destination-databricks

I added azure_config to Airbyte secret manager.

tuliren · 2022-08-10T03:16:58Z

@abaerptc, thank you for the PR! Could you add me as a collaborator to your fork? In this way I can push changes to this branch if needed as I am working on merging it.

@marcosmarxm, there is another PR pending merging for Databricks. Would you mind me doing the coordination and merging it later?

marcosmarxm · 2022-08-10T11:47:43Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2832361238
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2832361238
🐛 https://gradle.com/s/xtf6vwhphaki2

Build Failed

Test summary info:

Could not find result summary

marcosmarxm · 2022-08-10T11:48:06Z

@marcosmarxm, there is another PR pending merging for Databricks. Would you mind me doing the coordination and merging it later?

Sure @tuliren I'd just to confirm tests are running properly here.

marcosmarxm · 2022-08-10T12:37:43Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2832664101
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2832664101
🐛 https://gradle.com/s/nazigo4vbe2pe

Build Failed

Test summary info:

Could not find result summary

abaerptc · 2022-08-10T12:43:46Z

@tuliren You're added as a collaborator.

marcosmarxm · 2022-08-10T13:00:52Z

@abaerptc now the integration tests are running with your changes!

abaerptc · 2022-08-10T13:08:25Z

@abaerptc now the integration tests are running with your changes!

@marcosmarxm I see a few failures with messages like "Invalid configuration value detected for fs.azure.account.key". I think you need to give your Databricks instance access to your Azure account via the Spark configuration property fs.azure.account.key.

TamGB · 2022-09-05T15:26:30Z

Awesome PR @abaerptc !

Just a side note, the AWS version supports destination pattern matching (see airbyte-integrations/connectors/destination-databricks/src/main/java/io/airbyte/integrations/destination/databricks/DatabricksS3StorageConfig.java), is this something you would consider adding? (of course, I am happy to help!)

abaerptc · 2022-09-06T13:33:58Z

Awesome PR @abaerptc !

Just a side note, the AWS version supports destination pattern matching (see airbyte-integrations/connectors/destination-databricks/src/main/java/io/airbyte/integrations/destination/databricks/DatabricksS3StorageConfig.java), is this something you would consider adding? (of course, I am happy to help!)

@TamGB I don't object to having pattern matching added. However, I don't have the availability to work on it. Please go ahead and implement it if you'd find it valuable.

pkudinov · 2022-09-23T19:06:46Z

Is this abstracted enough to add support for Databricks on Google Cloud (i.e. add GCS as a target bucket along with AWS S3, and Azure Blob)?

abaerptc · 2022-09-23T19:20:43Z

Is this abstracted enough to add support for Databricks on Google Cloud (i.e. add GCS as a target bucket along with AWS S3, and Azure Blob)?

You should be able to add another implementation of the data storage method pretty easily based on what I've done here, yes.

abaerptc · 2022-10-10T14:29:49Z

@marcosmarxm @tuliren Any update on getting this merged in?

natalyjazzviolin · 2022-10-12T14:05:41Z

@abaerptc looks like there are conflicts, could you resolve them please?

@tuliren there is a lot of community interest for this PR, would you be able to move it up in your queue for review/merging? Thank you in advance :)

abaerptc · 2022-10-12T18:54:59Z

@natalyjazzviolin Conflicts are resolved.

marcosmarxm · 2022-10-13T23:18:44Z

So sorry @abaerptc for the long delay to reply. @tuliren is actively working to improve databricks destination for other features and to implement your contribution too. Soon he will return to you!

tuliren · 2022-10-14T19:56:16Z

Sorry about the delay.

To test the Azure integration, we would need a separate Databricks cluster. I was not able to do that so far. Given that it has been delayed for a long time, I will do some local testing, and mute the integration test related to Azure for now.

I target to merge it today.

tuliren · 2022-10-15T08:06:50Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/3254956793
✅ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/3254956793
No Python unittests run

Build Passed

Test summary info:

All Passed

tuliren

🎉

tuliren · 2022-10-15T08:22:12Z

/publish connector=connectors/destination-databricks run-tests=false

🕑 Publishing the following connectors:
connectors/destination-databricks
https://github.com/airbytehq/airbyte/actions/runs/3255000004

Connector	Did it publish?	Were definitions generated?
connectors/destination-databricks	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…airbyte into databricks-azure-15140

tuliren · 2022-10-15T08:39:33Z

@abaerptc, thank you very much for your contribution. Sorry about the long delay.

Please don't forget to remove me from your repo.

…rbytehq#15329) * 🎉 Destination Databricks: Support Azure storage (airbytehq#15140) * Update docs and version. * Revert unintentional change to S3 config parsing. * Revert unnecessary additional table property. * change spec.json * Ignore azure integration test * Add issue link * auto-bump connector version Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Liren Tu <tuliren@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

🎉 Destination Databricks: Support Azure storage (#15140)

962411b

abaerptc requested a review from a team as a code owner August 4, 2022 19:24

github-actions bot added the area/connectors Connector related issues label Aug 4, 2022

octavia-squidington-iii added bounty community connectors/destination/azure-blob-storage connectors/destination/databricks labels Aug 4, 2022

Update docs and version.

65b9c01

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 4, 2022

sajarin added internal and removed bounty labels Aug 5, 2022

abaerptc added 2 commits August 8, 2022 14:27

Revert unintentional change to S3 config parsing.

29c3d3e

Merge branch 'master' into databricks-azure-15140

0914d64

Revert unnecessary additional table property.

ede2d24

marcosmarxm and others added 2 commits August 10, 2022 08:12

Merge branch 'master' into databricks-azure-15140

baffd1c

change spec.json

2d0757e

abaerptc mentioned this pull request Oct 10, 2022

🎉 Destination Databricks: Support Azure storage #15140

Closed

Merge remote-tracking branch 'master' into databricks-azure-15140

04bcca2

Ignore azure integration test

efd63a5

tuliren approved these changes Oct 15, 2022

View reviewed changes

tuliren and others added 3 commits October 15, 2022 01:31

Add issue link

14d186d

auto-bump connector version

9280509

Merge branch 'databricks-azure-15140' of https://github.com/abaerptc/…

6afe9a7

…airbyte into databricks-azure-15140

tuliren merged commit 1b134d8 into airbytehq:master Oct 15, 2022

octavia-squidington-iii mentioned this pull request Oct 20, 2022

Bump Airbyte version from 0.40.15 to 0.40.16 #18268

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Destination Databricks: Support Azure storage (#15140) #15329

🎉 Destination Databricks: Support Azure storage (#15140) #15329

abaerptc commented Aug 4, 2022 •

edited by tuliren

Loading

CLAassistant commented Aug 4, 2022 •

edited

Loading

abaerptc commented Aug 9, 2022

marcosmarxm commented Aug 9, 2022 •

edited by github-actions bot

Loading

abaerptc commented Aug 9, 2022

marcosmarxm commented Aug 9, 2022

tuliren commented Aug 10, 2022

marcosmarxm commented Aug 10, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 10, 2022

marcosmarxm commented Aug 10, 2022 •

edited by github-actions bot

Loading

abaerptc commented Aug 10, 2022

marcosmarxm commented Aug 10, 2022

abaerptc commented Aug 10, 2022

TamGB commented Sep 5, 2022

abaerptc commented Sep 6, 2022

pkudinov commented Sep 23, 2022

abaerptc commented Sep 23, 2022

abaerptc commented Oct 10, 2022

natalyjazzviolin commented Oct 12, 2022

abaerptc commented Oct 12, 2022

marcosmarxm commented Oct 13, 2022

tuliren commented Oct 14, 2022

tuliren commented Oct 15, 2022 •

edited by github-actions bot

Loading

tuliren left a comment

tuliren commented Oct 15, 2022 •

edited by github-actions bot

Loading

tuliren commented Oct 15, 2022

🎉 Destination Databricks: Support Azure storage (#15140) #15329

🎉 Destination Databricks: Support Azure storage (#15140) #15329

Conversation

abaerptc commented Aug 4, 2022 • edited by tuliren Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Aug 4, 2022 • edited Loading

abaerptc commented Aug 9, 2022

marcosmarxm commented Aug 9, 2022 • edited by github-actions bot Loading

Build Failed

abaerptc commented Aug 9, 2022

marcosmarxm commented Aug 9, 2022

tuliren commented Aug 10, 2022

marcosmarxm commented Aug 10, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm commented Aug 10, 2022

marcosmarxm commented Aug 10, 2022 • edited by github-actions bot Loading

Build Failed

abaerptc commented Aug 10, 2022

marcosmarxm commented Aug 10, 2022

abaerptc commented Aug 10, 2022

TamGB commented Sep 5, 2022

abaerptc commented Sep 6, 2022

pkudinov commented Sep 23, 2022

abaerptc commented Sep 23, 2022

abaerptc commented Oct 10, 2022

natalyjazzviolin commented Oct 12, 2022

abaerptc commented Oct 12, 2022

marcosmarxm commented Oct 13, 2022

tuliren commented Oct 14, 2022

tuliren commented Oct 15, 2022 • edited by github-actions bot Loading

Build Passed

tuliren left a comment

Choose a reason for hiding this comment

tuliren commented Oct 15, 2022 • edited by github-actions bot Loading

tuliren commented Oct 15, 2022

abaerptc commented Aug 4, 2022 •

edited by tuliren

Loading

CLAassistant commented Aug 4, 2022 •

edited

Loading

marcosmarxm commented Aug 9, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 10, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 10, 2022 •

edited by github-actions bot

Loading

tuliren commented Oct 15, 2022 •

edited by github-actions bot

Loading

tuliren commented Oct 15, 2022 •

edited by github-actions bot

Loading