Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Postgres source via TLS/SSL #6008

Closed
5 tasks
Tracked by #5628
alexandr-shegeda opened this issue Sep 13, 2021 · 3 comments
Closed
5 tasks
Tracked by #5628

Add support for Postgres source via TLS/SSL #6008

alexandr-shegeda opened this issue Sep 13, 2021 · 3 comments

Comments

@alexandr-shegeda
Copy link
Contributor

alexandr-shegeda commented Sep 13, 2021

Tell us about the problem you're trying to solve

We want to support TLS encryption when connecting to the Postgres source.

Note that we do NOT need to support certificate verification as part of this issue -- just encryption of data over the wire. In other words, the focus is protecting against eavesdropping, not man-in-the-middle attacks. See the document linked for more details.

Describe the solution you’d like

Go through each source/destination in the must-have list. If the connector doesn't support encryption at all then create a ticket to support TLS/SSL for it.

The acceptance criteria for each ticket is:

  • Implement encryption support in the connector if not already implemented. Where possible, support encryption by default. If encryption-by-default is a bad idea (for example, if most MySQL versions do not support encryption and would require special work from the DB administrator) then expose it as an option in the connector specification, and encrypt when the user requests it.
  • The external documentation of the connector mentions that encryption is supported
  • If encryption is exposed as an option, add in the connector spec and docs a recommendation to use it (for example, MSSQL source mentions that encryption without server certification is used for testing purposes only, which is not true, see the doc above)
  • Encrypted connections are tested as part of either a custom integration test or acceptance tests. Where possible, test it using a test container. If that's impossible and it must be tested on a real DB instance, create a DB instance in AWS ideally using terraform (but if TF is too hard just create it manually and make a ticket to encode it in TF)
  • Create a PR

Implementation hints

There is a difference when implementing this for sources & destinations because destinations might need to change normalization as well.

When implementing this for sources, it's probably as simple as setting a flag e.g: Mysql uses the --ssl=REQUIRED flag.

When implementing for destinations it might be very similar, but there will be two places to edit this: in the destination connector itself and in the normalization module. It might be easiest to ask the Python team to implement the piece around normalization, but it really shouldn't be that complicated e.g: if it's just adding a flag -- it's ideal if you can implement it yourself since you'll learn a bit about normalization, but this is not a primary goal of this ticket. The goal is to support TLS as soon as possible.

@alexandr-shegeda alexandr-shegeda added airbyte-cloud area/connectors Connector related issues lang/java priority/high High priority type/enhancement New feature or request labels Sep 13, 2021
@alexandr-shegeda alexandr-shegeda changed the title Postgres source Add support for Postgres source via TLS/SSL Sep 16, 2021
@etsybaev etsybaev self-assigned this Sep 22, 2021
@etsybaev
Copy link
Contributor

etsybaev commented Sep 22, 2021

SSL had been already added as part of #2757 by @marcosmarxm

@etsybaev
Copy link
Contributor

Seems to work Was able to connect to DB created by PostgresSourceSSLTest test and this docker file (https://github.com/airbytehq/airbyte/tree/master/tools/integrations-test-ss
Selection_055
l
Selection_056
)

@alexandr-shegeda
Copy link
Contributor Author

TLS/SSL functionality was implemented for the Postgres source connector in the scope of the PR 2757

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Archived in project
Development

No branches or pull requests

3 participants