🎉 Destination postgres: Add SSL certificates and update normalization #14743

andriikorotkov · 2022-07-15T08:24:07Z

What

Add SSL certificates to destination postgres and destination postgres strict encrypt.

How

🚨 User Impact 🚨

Added a dropdown for choosing SSL mode. Added fields for SSL modes verify-full and verify-ca to add root key and client keys, and added a password for custom keystore.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…4034_ssl_certificate_postgres

…ncrypted key with .pk8 extension

…tion

…4034_ssl_certificate_postgres

…s for SSL certificates for all postgres connectors

…4034_ssl_certificate_postgres

…4034_ssl_certificate_postgres � Conflicts: � airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile � docs/integrations/sources/postgres.md

…4034_ssl_certificate_postgres

…4034_ssl_postgres_destination_certs_1

andriikorotkov · 2022-08-08T08:23:06Z

/test connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2816461713
✅ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2816461713
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     157      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         186     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1421    624    56%

Build Passed

Test summary info:

All Passed

…4034_ssl_postgres_destination_certs_1

andriikorotkov · 2022-08-09T13:05:02Z

/test connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2825469672
✅ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2825469672
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     157      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         186     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1421    624    56%

Build Passed

Test summary info:

All Passed

andriikorotkov · 2022-08-09T13:36:56Z

/publish connector=connectors/destination-postgres

🕑 Publishing the following connectors:
connectors/destination-postgres
https://github.com/airbytehq/airbyte/actions/runs/2825667775

Connector	Did it publish?	Were definitions generated?
connectors/destination-postgres	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…4034_ssl_postgres_destination_certs_1

andriikorotkov · 2022-08-10T15:21:18Z

/publish connector=connectors/destination-postgres

🕑 Publishing the following connectors:
connectors/destination-postgres
https://github.com/airbytehq/airbyte/actions/runs/2833768889

Connector	Did it publish?	Were definitions generated?
connectors/destination-postgres	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…4034_ssl_postgres_destination_certs_1

andriikorotkov · 2022-08-15T06:59:33Z

/test connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2858997011
✅ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2858997011
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     157      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         186     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1421    624    56%

Build Passed

Test summary info:

All Passed

andriikorotkov · 2022-08-15T08:38:43Z

/publish connector=connectors/destination-postgres

🕑 Publishing the following connectors:
connectors/destination-postgres
https://github.com/airbytehq/airbyte/actions/runs/2859466759

Connector	Did it publish?	Were definitions generated?
connectors/destination-postgres	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

andriikorotkov · 2022-08-15T11:21:38Z

/test connector=connectors/destination-postgres-strict-encrypt

🕑 connectors/destination-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2860288048
✅ connectors/destination-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2860288048
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     157      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         186     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1421    624    56%

Build Passed

Test summary info:

All Passed

andriikorotkov · 2022-08-15T12:41:57Z

/publish connector=connectors/destination-postgres run-tests=false

🕑 Publishing the following connectors:
connectors/destination-postgres
https://github.com/airbytehq/airbyte/actions/runs/2860697617

Connector	Did it publish?	Were definitions generated?
connectors/destination-postgres	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

andriikorotkov · 2022-08-15T13:11:37Z

/publish connector=connectors/destination-postgres-strict-encrypt run-tests=false auto-bump-version=false

🕑 Publishing the following connectors:
connectors/destination-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2860848045

Connector	Did it publish?	Were definitions generated?
connectors/destination-postgres-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…#14743) * added ssl certificates for postgres source * added command for remove client private key after transformation to encrypted key with .pk8 extension * added connection with CA and client certificates for postgres destination * updated code style * moved common methods to the common class * moved common methods to the common class * fixed remarks * updated postgres source tests * added minor changes to spec and added fixes to password mechanism * updated postgres source tests * updated strict-encrypt postgres source and destination and added tests for SSL certificates for all postgres connectors * fixed check style * updated documentation and versions of connectors * updated ordrs in test spec * fixed minor remarks in specs and expected_specs * fixed minor remarks in specs and expected_specs * fixed Dockerfile * fixed remarks * fixed remarks * fixed remarks * fixed remarks * fixed remarks * fixed code style * fixed connectors version in definition file * updated postgres destination normalization * updated postgres destination tests * fixed code style for postgres source and destination * pulled master changes * removed allow mode for destination-postgres-strect-encrypt * updated connectors version * fixed custom DBT transformation and enabled test for it * updated normalization version * updated keystore password generation method * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

andriikorotkov added 30 commits June 15, 2022 13:45

added ssl certificates for postgres source

9fef1bb

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

3f2da64

…4034_ssl_certificate_postgres

added command for remove client private key after transformation to e…

9a48ac3

…ncrypted key with .pk8 extension

added connection with CA and client certificates for postgres destina…

7c5fe79

…tion

updated code style

3c70c36

moved common methods to the common class

cdea269

moved common methods to the common class

c4f623e

fixed remarks

a2266b5

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

a0c49ac

…4034_ssl_certificate_postgres

updated postgres source tests

fa65a81

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

da7c8ae

…4034_ssl_certificate_postgres

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

4189a19

…4034_ssl_certificate_postgres

added minor changes to spec and added fixes to password mechanism

59e467f

updated postgres source tests

5d115da

updated strict-encrypt postgres source and destination and added test…

9ae6e8c

…s for SSL certificates for all postgres connectors

fixed check style

6f8f726

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

4c9cd85

…4034_ssl_certificate_postgres

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

d5a408a

…4034_ssl_certificate_postgres

updated documentation and versions of connectors

1901f35

updated ordrs in test spec

c7044ff

fixed minor remarks in specs and expected_specs

a48774f

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

ccda43b

…4034_ssl_certificate_postgres � Conflicts: � airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile � docs/integrations/sources/postgres.md

fixed minor remarks in specs and expected_specs

0d4c5a5

fixed Dockerfile

9f50f19

fixed remarks

a83a378

fixed remarks

c57ae51

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

38f41f6

…4034_ssl_certificate_postgres

fixed remarks

8137628

fixed remarks

d78ec36

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

db80b1d

…4034_ssl_certificate_postgres

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

c37d273

…4034_ssl_postgres_destination_certs_1

github-actions bot removed area/worker Related to worker normalization area/platform issues related to the platform labels Aug 8, 2022

andriikorotkov temporarily deployed to more-secrets August 8, 2022 08:24 Inactive

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

d7835e0

…4034_ssl_postgres_destination_certs_1

andriikorotkov temporarily deployed to more-secrets August 9, 2022 12:15 Inactive

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

ef615ad

…4034_ssl_postgres_destination_certs_1

andriikorotkov temporarily deployed to more-secrets August 10, 2022 13:37 Inactive

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

0317821

…4034_ssl_postgres_destination_certs_1

andriikorotkov temporarily deployed to more-secrets August 14, 2022 20:42 Inactive

updated keystore password generation method

4b9a905

andriikorotkov temporarily deployed to more-secrets August 15, 2022 06:58 Inactive

auto-bump connector version [ci skip]

be4a3ca

andriikorotkov merged commit d63a24e into master Aug 15, 2022

andriikorotkov deleted the akorotkov/4034_ssl_postgres_destination_certs_1 branch August 15, 2022 13:32

octavia-squidington-iii mentioned this pull request Aug 19, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1-alpha #15775

Closed

evantahler mentioned this pull request Aug 22, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1 #15857

Merged

bffrost mentioned this pull request Feb 9, 2023

Destination Postgres: CloudSQL allow verify-ca with client certificate and client key #22660

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Destination postgres: Add SSL certificates and update normalization #14743

🎉 Destination postgres: Add SSL certificates and update normalization #14743

andriikorotkov commented Jul 15, 2022 •

edited

Loading

andriikorotkov commented Aug 8, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 9, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 9, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 10, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

🎉 Destination postgres: Add SSL certificates and update normalization #14743

🎉 Destination postgres: Add SSL certificates and update normalization #14743

Conversation

andriikorotkov commented Jul 15, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

andriikorotkov commented Aug 8, 2022 • edited by github-actions bot Loading

Build Passed

andriikorotkov commented Aug 9, 2022 • edited by github-actions bot Loading

Build Passed

andriikorotkov commented Aug 9, 2022 • edited by github-actions bot Loading

andriikorotkov commented Aug 10, 2022 • edited by github-actions bot Loading

andriikorotkov commented Aug 15, 2022 • edited by github-actions bot Loading

Build Passed

andriikorotkov commented Aug 15, 2022 • edited by github-actions bot Loading

andriikorotkov commented Aug 15, 2022 • edited by github-actions bot Loading

Build Passed

andriikorotkov commented Aug 15, 2022 • edited by github-actions bot Loading

andriikorotkov commented Aug 15, 2022 • edited by github-actions bot Loading

andriikorotkov commented Jul 15, 2022 •

edited

Loading

andriikorotkov commented Aug 8, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 9, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 9, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 10, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading