move basic normalization to connectors_base build #7068

cgardens · 2021-10-15T02:32:34Z

Closes #7069

What

The platform depends on a pinned version of normalization so rebuilding every time we build the platform doesn't add any better build guarantees.
The platform should just treat normalization like it would any other pinned artifact.

How

Only build basic normalization as part of the CONNECTORS_BASE build.
Change how normalization publishes multiple images to happen as part of the build for base-normalization.
Update the manage.sh script to handle pushing multiple images for normalization (it is the only "connector", for now, that publishes multiple images)
Update the check_image_exists.sh script to make sure that the normalization images are actually available on dockerhub (we actually weren't doing this at all before 😅 )

Tradeoff

The one tradeoff here is that if you are developing on normalization and for some reason you need to run the platform with the dev version, you need to remember to rebuild it manually because it won't happen as part of composeBuild. This is a pretty low cost be cause 1. the integration tests in normalization should be where we are running tests 2. You already have to manually update the normalization version to dev in order to run against a dev version of normalization, so you're already doing a manual process anyway.

CLAassistant · 2021-10-15T02:32:37Z

All committers have signed the CLA.

cgardens · 2021-10-15T03:06:29Z

docker-compose.build.yaml

-  normalization:
-    image: airbyte/normalization:${VERSION}
-    build:
-      dockerfile: Dockerfile
-      context: airbyte-integrations/bases/base-normalization
-      labels:
-        io.airbyte.git-revision: ${GIT_REVISION}
-  normalization-mssql:
-    image: airbyte/normalization-mssql:${VERSION}
-    build:
-      dockerfile: mssql.Dockerfile
-      context: airbyte-integrations/bases/base-normalization
-      labels:
-        io.airbyte.git-revision: ${GIT_REVISION}
-  normalization-mysql:
-    image: airbyte/normalization-mysql:${VERSION}
-    build:
-      dockerfile: mysql.Dockerfile
-      context: airbyte-integrations/bases/base-normalization
-      labels:
-        io.airbyte.git-revision: ${GIT_REVISION}
-  normalization-oracle:
-    image: airbyte/normalization-oracle:${VERSION}
-    build:
-      dockerfile: oracle.Dockerfile
-      context: airbyte-integrations/bases/base-normalization
-      labels:
-        io.airbyte.git-revision: ${GIT_REVISION}


@ChristopheDuong why were these added here?

Since it is not possible to “edit” and choose what normalization version you want like you can with connectors, I switched normalization version tags to follow airbyte core versions instead as discussed in this thread (I also discussed it with people separately and overall people were supportive of the switch) https://airbytehq.slack.com/archives/C019WEENQRM/p1633710746415900

See also comment making it part of composeBuild: #2054 (comment)

Overall it’d make it easier to publish new normalization images (when releasing airbyte) instead of doing it whenever a PR is merged…. (the docker image is not useable until there is a release of airbyte anyway)

And now that we have multiple docker images for normalization to publish, we'd have to include them in the docker-compose-build.yaml to make them as part of airbyte release?

We could revert back but then we need to adapt the publish script and the DefaultNormalizationRunnerFactory too

Maybe a good way forward is to split normalisation out to use it's own versioning and inject this version by env var.

After this we split the builds so normalisation is a separate build from the platform. In this sense, we have 3 build paths 1) connectors 2) normalisation 3) platform. This will allow the platform team to remove Python from our/their deps.

To simplify things, we can continue to release normalisation with the current OSS release. We can also continue to pin to the Airbyte version. There is some unnecessary work, but it's mostly automated so no sweat off our backs.

We can also start with a single global normalisation version. We can add individual connector normalisation versions as we run into those cases.

For now, I'm reverting back to how we had it that normalization has its own version that is de coupled from platform. I think the next thing to figure out is how we want to package normalization in the future. It seems like either packaging mysql normalization in the mysql destination container or at least having that container specify the name of the container that should be used to normalize it seem like 2 reasonable paths forward but those are just idea, open to others.

ChristopheDuong · 2021-10-15T09:17:08Z