Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add SSH tunneling to Elastic Search Destination #17805

Merged
merged 14 commits into from
Oct 14, 2022
Merged

Add SSH tunneling to Elastic Search Destination #17805

merged 14 commits into from
Oct 14, 2022

Conversation

grishick
Copy link
Contributor

@grishick grishick commented Oct 10, 2022 •
edited
Loading

What

Add support for accessing Elastic Search destination via an SSH tunnel

How

  • Add SSH tunneling support for destinations that are configured with a URL instead of a host/port pair
  • Add SSH wrapped integration tests to destination-elasticsearch
  • Modify ElasticsearchDestination class to return an instance of SshWrappedDestination

Recommended reading order

  1. SshTunnel.java
  2. SshTunnelTest.java
  3. SshWrappedDestination.java
  4. ElasticsearchDestination.java
  5. SshElasticsearchDestinationAcceptanceTest.java
  6. SshKeyElasticsearchDestinationAcceptanceTest.java
  7. SshPasswordElasticsearchDestinationAcceptanceTest.java

🚨 User Impact 🚨

Users can now use SSH tunnel with Elastic Search destination

Pre-merge Checklist

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-scylla
  • destination-mssql-strict-encrypt
  • destination-mongodb
  • destination-mysql
  • destination-clickhouse-strict-encrypt
  • destination-oracle
  • destination-gcs
  • source-postgres-strict-encrypt
  • source-jdbc
  • source-mongodb-v2
  • source-oracle-strict-encrypt
  • destination-pulsar
  • source-elasticsearch
  • destination-mssql
  • destination-kafka
  • source-alloydb-strict-encrypt
  • destination-clickhouse
  • destination-dev-null
  • source-postgres
  • destination-csv
  • destination-oracle-strict-encrypt
  • destination-redshift
  • source-scaffold-java-jdbc
  • destination-rockset
  • source-mongodb-strict-encrypt
  • source-alloydb
  • destination-azure-blob-storage
  • destination-local-json
  • source-relational-db
  • source-cockroachdb
  • source-e2e-test
  • destination-pubsub
  • destination-s3
  • source-e2e-test-cloud
  • source-bigquery
  • destination-mqtt
  • destination-e2e-test
  • destination-postgres
  • source-tidb
  • destination-snowflake
  • source-clickhouse
  • destination-dynamodb
  • destination-jdbc
  • source-mssql-strict-encrypt
  • source-cockroachdb-strict-encrypt
  • destination-cassandra
  • source-clickhouse-strict-encrypt
  • source-oracle
  • destination-mysql-strict-encrypt
  • destination-kinesis
  • destination-postgres-strict-encrypt
  • destination-r2
  • source-snowflake
  • source-redshift
  • source-mssql
  • destination-elasticsearch-strict-encrypt
  • source-kafka
  • source-mysql
  • destination-meilisearch
  • destination-bigquery-denormalized
  • source-mysql-strict-encrypt
  • destination-bigquery
  • destination-mongodb-strict-encrypt
  • source-db2-strict-encrypt
  • source-db2
  • destination-tidb
  • destination-elasticsearch
  • destination-keen
  • destination-redis
  • destination-databricks
  • destination-mariadb-columnstore
  • source-sftp

@grishick
Copy link
Contributor Author

grishick commented Oct 10, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-postgres-strict-encrypt

🕑 connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3222394798
✅ connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3222394798
No Python unittests run

Build Passed

Test summary info:

All Passed

@grishick
Copy link
Contributor Author

grishick commented Oct 10, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-elasticsearch-strict-encrypt

🕑 connectors/destination-elasticsearch-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3222441963
✅ connectors/destination-elasticsearch-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3222441963
No Python unittests run

Build Passed

Test summary info:

All Passed

@grishick grishick force-pushed the greg/elastic-ssh-tunnel branch from 5b0ef2c to 5997506 Compare October 10, 2022 21:38
@github-actions github-actions bot added the area/connectors Connector related issues label Oct 10, 2022
@github-actions

This comment was marked as duplicate.

Sign in to view
@grishick grishick force-pushed the greg/elastic-ssh-tunnel branch from 5997506 to 57f2ae7 Compare October 12, 2022 18:38
@grishick
Copy link
Contributor Author

grishick commented Oct 12, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-elasticsearch

🕑 connectors/destination-elasticsearch https://github.com/airbytehq/airbyte/actions/runs/3237068132
✅ connectors/destination-elasticsearch https://github.com/airbytehq/airbyte/actions/runs/3237068132
No Python unittests run

Build Passed

Test summary info:

All Passed

@github-actions

This comment was marked as duplicate.

Sign in to view
@grishick
Copy link
Contributor Author

grishick commented Oct 12, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-elasticsearch-strict-encrypt

🕑 connectors/destination-elasticsearch-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3237333853
✅ connectors/destination-elasticsearch-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3237333853
No Python unittests run

Build Passed

Test summary info:

All Passed

@grishick grishick changed the title Add support for using URL in tunneled config Add SSH tunneling to Elastic Search Destination Oct 12, 2022
@grishick grishick marked this pull request as ready for review October 12, 2022 20:19
@grishick grishick requested a review from a team as a code owner October 12, 2022 20:19
@github-actions

This comment was marked as duplicate.

Sign in to view
@grishick grishick force-pushed the greg/elastic-ssh-tunnel branch from fe55cca to 6bfebfb Compare October 12, 2022 20:25
@github-actions

This comment was marked as duplicate.

Sign in to view
@grishick grishick linked an issue Oct 12, 2022 that may be closed by this pull request
Add SSH Tunnel options to Elastic Search Destination #16252
Closed
@github-actions

This comment was marked as duplicate.

Sign in to view
@github-actions

This comment was marked as duplicate.

Sign in to view
@grishick grishick force-pushed the greg/elastic-ssh-tunnel branch from 6ac4d99 to 40066d8 Compare October 12, 2022 21:23
@github-actions

This comment was marked as duplicate.

Sign in to view
@grishick grishick force-pushed the greg/elastic-ssh-tunnel branch from 40066d8 to 156667e Compare October 12, 2022 22:12
@github-actions

This comment was marked as duplicate.

Sign in to view
@grishick
Copy link
Contributor Author

grishick commented Oct 14, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-elasticsearch-strict-encrypt

🕑 connectors/destination-elasticsearch-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3253216748
✅ connectors/destination-elasticsearch-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3253216748
No Python unittests run

Build Passed

Test summary info:

All Passed

@grishick grishick force-pushed the greg/elastic-ssh-tunnel branch from e9d87c2 to 5a2fed4 Compare October 14, 2022 22:03
@github-actions

This comment was marked as duplicate.

Sign in to view
@grishick
Copy link
Contributor Author

grishick commented Oct 14, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-elasticsearch

🕑 connectors/destination-elasticsearch https://github.com/airbytehq/airbyte/actions/runs/3253262897
✅ connectors/destination-elasticsearch https://github.com/airbytehq/airbyte/actions/runs/3253262897
No Python unittests run

Build Passed

Test summary info:

All Passed

@grishick
Copy link
Contributor Author

grishick commented Oct 14, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-elasticsearch

🕑 Publishing the following connectors:
connectors/destination-elasticsearch
https://github.com/airbytehq/airbyte/actions/runs/3253332183


Connector Did it publish? Were definitions generated?
connectors/destination-elasticsearch ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@grishick
Copy link
Contributor Author

grishick commented Oct 14, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-elasticsearch-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/destination-elasticsearch-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3253368445


Connector Did it publish? Were definitions generated?
connectors/destination-elasticsearch-strict-encrypt ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@github-actions

This comment was marked as duplicate.

Sign in to view
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets October 14, 2022 22:47 Inactive
@grishick
Copy link
Contributor Author

grishick commented Oct 14, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-postgres-strict-encrypt

🕑 connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3253456195
✅ connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3253456195
No Python unittests run

Build Passed

Test summary info:

All Passed

@grishick grishick merged commit a73ae2b into master Oct 14, 2022
@grishick grishick deleted the greg/elastic-ssh-tunnel branch October 14, 2022 23:12
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Oct 20, 2022
Merged
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
@grishick @jhammarstedt
Add SSH tunneling to Elastic Search Destination (airbytehq#17805)
f5816e6 
* Add support for using URL in tunneled config
* Add support for SSH tunneling to destination-elasticsearch and destination-elasticsearch-strict-encrypt
@grishick grishick added the team/destinations Destinations team's backlog label Nov 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ryankfu ryankfu ryankfu approved these changes

@etsybaev etsybaev etsybaev approved these changes

@edgao edgao Awaiting requested review from edgao

@suhomud suhomud Awaiting requested review from suhomud

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/elasticsearch connectors/destination/elasticsearch-strict-encryp team/destinations Destinations team's backlog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add SSH Tunnel options to Elastic Search Destination
7 participants
@grishick @etsybaev @suhomud @ryankfu @supertopher @octavia-squidington-iii @octavia-squidington-iv