Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AllowedHosts for all Database Sources #22372

Open
evantahler opened this issue Feb 3, 2023 · 6 comments
Open

Add AllowedHosts for all Database Sources #22372

evantahler opened this issue Feb 3, 2023 · 6 comments
Assignees
Labels
from/connector-ops frozen Not being actively worked on team/db-dw-sources Backlog for Database and Data Warehouse Sources team

Comments

@evantahler
Copy link
Contributor

evantahler commented Feb 3, 2023

As part of the network isolation epic, all connectors will need to provide which hosts the connector container will be granted access to.

For databases, this information is dynamic and read from the connector's config. It probably looks a lot like:

allowedHosts:
hosts:
- "${host}"
- "${tunnel_method.tunnel_host}"

This information is to be added to the actor_definitions. A re-publish of the connector is not required.

@DevPJ9
Copy link

DevPJ9 commented Feb 3, 2023

Hello @evantahler I would like to work on this.

@evantahler evantahler added the team/db-dw-sources Backlog for Database and Data Warehouse Sources team label Feb 3, 2023
@evantahler evantahler changed the title Add AllowedHosts to all Database Sources Add AllowedHosts for all Database Sources Feb 3, 2023
@bleonard bleonard removed their assignment Feb 22, 2023
@bleonard
Copy link
Contributor

I seem to remember manybe we had an OC issue with allowed hosts a while back. Am I crazy @evantahler ? Just want to make sure not to mess it up.

@evantahler
Copy link
Contributor Author

There was a platform-level OC issue (#21923) which was due to the weirdness that is a reset-connection. Adding these values for a connector, even if they are wrong, shouldn't break anything in production.

The path to ensuring your allowedHosts entries are properly interpolating config information is to look at the logs when the containers boot. Look for a log line that interpolates the allowedHosts properly. This is the failure case:

airbyte-worker | 2023-01-26 19:42:58 ERROR i.a.w.u.ConfigReplacer(getAllowedHosts):60 - The allowedHost value, '${host}', is expecting an interpolation value from the connector's configuration, but none is present

More information @ #21950

@prateekmukhedkar
Copy link
Contributor

Work done: PR reviewed for non-GA source connectors.
Work remaining:

  1. Close out AlloyDB PR Adding allowedHost for Postgres AlloyDB #23596
  2. Publish all new connectors

@prateekmukhedkar
Copy link
Contributor

Plan to publish all database connectors early next week.

@prateekmukhedkar
Copy link
Contributor

Done for all DB Sources except MongoDB, Redshift and Azure Table DB Source because of previously failing integration tests. Moving this to blocked/hold lane until we fix these tests and get them in a publishable state. Issue created for tracking publish https://github.com/airbytehq/airbyte-internal-issues/issues/2714

@bleonard bleonard added the frozen Not being actively worked on label Mar 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
from/connector-ops frozen Not being actively worked on team/db-dw-sources Backlog for Database and Data Warehouse Sources team
Projects
None yet
Development

No branches or pull requests

4 participants