🎉 Redshift Source/Destination SSL Support #6965

VitaliiMaltsev · 2021-10-12T08:44:15Z

What

We want to support TLS encryption when connecting to the Redshift source/destination.

Note that we do NOT need to support certificate verification as part of this issue -- just encryption of data over the wire. In other words, the focus is protecting against eavesdropping, not man-in-the-middle attacks.

How

Added option to connect using SSL.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

CLAassistant · 2021-10-12T08:44:20Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

vmaltsev seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

# Conflicts: # docs/integrations/destinations/redshift.md # docs/integrations/sources/redshift.md

VitaliiMaltsev · 2021-10-13T11:07:49Z

/test connector=connectors/source-redshift

🕑 connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1337176765
❌ connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1337176765
🐛 https://gradle.com/s/uvfwyc7q3auqs
🕑 connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1337176765
✅ connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1337176765
No Python unittests run

VitaliiMaltsev · 2021-10-13T11:57:26Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1337324630
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1337324630
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 360    210    42%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         151     39    74%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1063    405    62%

tuliren

Looks good!

The following PR summary is actually very useful. Can you add it to the README or the connector doc?

Note that we do NOT need to support certificate verification as part of this issue -- just encryption of data over the wire. In other words, the focus is protecting against eavesdropping, not man-in-the-middle attacks.

I blocked this PR because the connector version have not been bumped yet. Please also don't forget to publish the new connectors after bumping the versions.

airbyte-integrations/connectors/destination-redshift/src/main/resources/spec.json

airbyte-integrations/connectors/source-redshift/src/main/resources/spec.json

…s to true

VitaliiMaltsev · 2021-10-13T17:20:14Z

Looks good!

The following PR summary is actually very useful. Can you add it to the README or the connector doc?

Note that we do NOT need to support certificate verification as part of this issue -- just encryption of data over the wire. In other words, the focus is protecting against eavesdropping, not man-in-the-middle attacks.

I blocked this PR because the connector version have not been bumped yet. Please also don't forget to publish the new connectors after bumping the versions.

Redshift Source and Destination versions bumped

tuliren

Thank you.

Please also publish the new connectors.

# Conflicts: # airbyte-config/init/src/main/resources/config/STANDARD_DESTINATION_DEFINITION/f7a7d195-377f-cf5b-70a5-be6b819019dc.json # airbyte-config/init/src/main/resources/seed/destination_definitions.yaml # airbyte-integrations/connectors/destination-redshift/Dockerfile # airbyte-integrations/connectors/destination-redshift/src/main/java/io/airbyte/integrations/destination/redshift/RedshiftCopyS3Destination.java # docs/integrations/destinations/redshift.md

VitaliiMaltsev · 2021-10-14T08:10:17Z

/publish connector=connectors/source-redshift

🕑 connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1340912398
✅ connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1340912398

VitaliiMaltsev · 2021-10-14T08:23:14Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1340957570
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1340957570

* add tls option to spec * Redshift Source add acceptance test * Redshift Destination add ssl field to spec * add RedshiftDestinationAcceptanceTestSSL * fix checkstyle * added changelog * update docs * bump versions of Redshift Source and Destination \ changed default tls to true Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>

vmaltsev added 5 commits October 8, 2021 14:51

add tls option to spec

c2ec467

Redshift Source add acceptance test

584f9e6

Redshift Destination add ssl field to spec

e18f259

add RedshiftDestinationAcceptanceTestSSL

2029ef6

fix checkstyle

9c3823f

github-actions bot added the area/connectors Connector related issues label Oct 12, 2021

VitaliiMaltsev changed the title ~~tada: Redshift Source/Destination SSL Support~~ 🎉 Redshift Source/Destination SSL Support Oct 12, 2021

added changelog

ea9bd32

github-actions bot added the area/documentation Improvements or additions to documentation label Oct 12, 2021

VitaliiMaltsev temporarily deployed to more-secrets October 12, 2021 09:33 Inactive

VitaliiMaltsev requested review from alexandr-shegeda and irynakruk October 12, 2021 13:41

VitaliiMaltsev linked an issue Oct 12, 2021 that may be closed by this pull request

Redshift Src/Destination #6429

Closed

5 tasks

alexandr-shegeda approved these changes Oct 12, 2021

View reviewed changes

update docs

fc764b9

VitaliiMaltsev requested a review from tuliren October 12, 2021 14:05

VitaliiMaltsev marked this pull request as ready for review October 12, 2021 14:05

VitaliiMaltsev temporarily deployed to more-secrets October 12, 2021 14:05 Inactive

irynakruk approved these changes Oct 12, 2021

View reviewed changes

Merge branch 'master' into vmaltsev/6429-redshift-ssl

dc23cd1

# Conflicts: # docs/integrations/destinations/redshift.md # docs/integrations/sources/redshift.md

jrhizor temporarily deployed to more-secrets October 13, 2021 11:12 Inactive

jrhizor temporarily deployed to more-secrets October 13, 2021 11:49 Inactive

jrhizor temporarily deployed to more-secrets October 13, 2021 11:59 Inactive

tuliren suggested changes Oct 13, 2021

View reviewed changes

airbyte-integrations/connectors/destination-redshift/src/main/resources/spec.json Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-redshift/src/main/resources/spec.json Outdated Show resolved Hide resolved

bump versions of Redshift Source and Destination \ changed default tl…

8e00b9d

…s to true

VitaliiMaltsev temporarily deployed to more-secrets October 13, 2021 17:19 Inactive

VitaliiMaltsev requested a review from tuliren October 13, 2021 17:21

tuliren approved these changes Oct 13, 2021

View reviewed changes

VitaliiMaltsev temporarily deployed to more-secrets October 14, 2021 08:09 Inactive

jrhizor temporarily deployed to more-secrets October 14, 2021 08:12 Inactive

jrhizor temporarily deployed to more-secrets October 14, 2021 08:26 Inactive

VitaliiMaltsev merged commit ae9048c into master Oct 14, 2021

VitaliiMaltsev deleted the vmaltsev/6429-redshift-ssl branch October 14, 2021 09:29

jrhizor mentioned this pull request Oct 19, 2021

Bump Airbyte version from 0.30.20-alpha to 0.30.21-alpha #7145

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Redshift Source/Destination SSL Support #6965

🎉 Redshift Source/Destination SSL Support #6965

VitaliiMaltsev commented Oct 12, 2021 •

edited

Loading

CLAassistant commented Oct 12, 2021

VitaliiMaltsev commented Oct 13, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Oct 13, 2021 •

edited by github-actions bot

Loading

tuliren left a comment •

edited

Loading

VitaliiMaltsev commented Oct 13, 2021

tuliren left a comment

VitaliiMaltsev commented Oct 14, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Oct 14, 2021 •

edited by github-actions bot

Loading

🎉 Redshift Source/Destination SSL Support #6965

🎉 Redshift Source/Destination SSL Support #6965

Conversation

VitaliiMaltsev commented Oct 12, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

CLAassistant commented Oct 12, 2021

VitaliiMaltsev commented Oct 13, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Oct 13, 2021 • edited by github-actions bot Loading

tuliren left a comment • edited Loading

Choose a reason for hiding this comment

VitaliiMaltsev commented Oct 13, 2021

tuliren left a comment

Choose a reason for hiding this comment

VitaliiMaltsev commented Oct 14, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Oct 14, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Oct 12, 2021 •

edited

Loading

VitaliiMaltsev commented Oct 13, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Oct 13, 2021 •

edited by github-actions bot

Loading

tuliren left a comment •

edited

Loading

VitaliiMaltsev commented Oct 14, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Oct 14, 2021 •

edited by github-actions bot

Loading