🎉 Refactor Normalization docker images and upgrade to use dbt 0.21.0 #6959
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ChristopheDuong
commented
Oct 11, 2021
airbyte-workers/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java
Outdated
Show resolved
Hide resolved
|
I'm also trying to fix Oracle integration tests (they are currently disabled)... |
ChristopheDuong
commented
Oct 11, 2021
airbyte-integrations/bases/base-normalization/integration_tests/test_normalization.py
Outdated
Show resolved
Hide resolved
Phlair
approved these changes
Oct 14, 2021
There was a problem hiding this comment.
Awesome, lgtm! I think this makes a lot of sense given different dependencies between dests and the improved dx will be 👌
What's the reasoning behind the choices of mssql, mysql & oracle to be the ones to separate out? Is intended goal to move all destinations toward this structure within normalization?
| @@ -90,9 +93,11 @@ if(!System.getenv().containsKey("SUB_BUILD") || System.getenv().get("SUB_BUILD") | |||
| include ':airbyte-integrations:connectors:destination-oracle' | |||
| include ':airbyte-integrations:connectors:destination-mssql' | |||
|
|
|||
| //Needed by destination-bugquery | |||
There was a problem hiding this comment.
I like to think this was on purpose 😆
Comment on lines
+26
to
+27
| if DestinationType.POSTGRES.value not in destinations_to_test: | ||
| destinations_to_test.append(DestinationType.POSTGRES.value) |
There was a problem hiding this comment.
what's the reason for enforcing postgres like this?
There was a problem hiding this comment.
some test functions in that class are explicitly requiring postgres destination (but omit tests on other destinations)
normalization for "regular" destinations is based on dbt's core "mono-repo" so we don't actually need to produce multiple docker images there, the code base is the same with the macros making the slight tweaks in syntax. Unfortunately for mssql, mysql and oracle, those are community contributed adapters to dbt, so they each live in a separate git repo and have their own dbt version dependency which may be different from each other, and not always in sync with core dbt...
|
|
/test connector=connectors/destination-postgres
|
|
/test connector=bases/base-normalization
|
This was referenced Oct 15, 2021
schlattk
pushed a commit
to schlattk/airbyte
that referenced
this pull request
Jan 4, 2022
…irbytehq#6959) * Split normalization docker images for some connectors with specifics dependencies * Regenerate (airbytehq#7003)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
How
Describe the solution
Closes #2054
Closes #6872
Recommended reading order
Build related changes:
airbyte-workers/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.javaairbyte-workers/src/main/java/io/airbyte/workers/normalization/DefaultNormalizationRunner.javaairbyte-integrations/bases/base-normalization/build.gradlesettings.gradledocker-compose.build.yamlairbyte-integrations/bases/base-normalization/DockerfileDX with normalization tests:
airbyte-integrations/bases/base-normalization/README.mdairbyte-integrations/bases/base-normalization/integration_tests/dbt_integration_test.pyairbyte-integrations/bases/base-normalization/integration_tests/test_normalization.pyairbyte-integrations/bases/base-normalization/integration_tests/test_ephemeral.pyContext
Docker images for normalization
We currently have additional packages to install inside normalization docker image due to:
Some of these destinations have a hard dependency on a certain dbt version (0.19.0) and can't be upgraded to more recent releases.
However, we do want to support the latest dbt versions as they come.
As the number of supported destinations for normalization grows, we may run into dbt versions conflicts.
So this PR produces a dedicated docker image for each "custom" destination (outside of dbt core support).
Docker image version
Instead of publishing a new docker image at each normalization PR, we can publish normalization images when releasing Airbyte versions. (additionally, we can run normalization in dev mode without change to codebase now)
Integration tests stability & useability
When running integration tests, we noticed flakiness in results. Maube due to concurrent runs by multiple people/CI jobs.
So a random schema where to run tests should better isolate tests.
It is now easier to select which part of normalization tests to run per destination via an env variable