Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove worker dependency from normalization module version #5349

Closed
marcosmarxm opened this issue Aug 11, 2021 · 4 comments
Closed

Remove worker dependency from normalization module version #5349

marcosmarxm opened this issue Aug 11, 2021 · 4 comments
Labels

Comments

@marcosmarxm
Copy link
Member

Tell us about the problem you're trying to solve

public static final String NORMALIZATION_IMAGE_NAME = "airbyte/normalization:0.1.38";

This can lead to a problem when we upgrade normalization. We need to wait for a new Airbyte release to be able to use. Or, if we introduce a bug in normalization people can't downgrade to a older version and keep using Airbyte.

Describe the solution you’d like

Probably we could add a env variable to be more flexible choosing the normalization operation version.

Describe the alternative you’ve considered or used

None

Additional context

Add any other context or screenshots about the feature request here.

Are you willing to submit a PR?

Yes

@marcosmarxm marcosmarxm added the type/enhancement New feature or request label Aug 11, 2021
@jrhizor
Copy link
Contributor

jrhizor commented Aug 11, 2021

I think what we've discussed before is having the normalization version be in sync with the overall Airbyte version. It's very hard to guarantee a range of versions compatible with a normalization version, so it's effectively tied to the same version anyway at the moment (which is why it isn't configurable).

@jrhizor
Copy link
Contributor

jrhizor commented Aug 11, 2021

related: #2054

@ChristopheDuong
Copy link
Contributor

Even if normalization is tied to an airbyte version, it could sometimes be helpful to have the ability to hot-swap/plug another normalization image

But this should probably be an exception rather than the general usage

@evantahler
Copy link
Contributor

Closing this in favor of #16671 which will strongly decouple normalization from the workers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants