🎉 Source Oracle: Added support for connection via SSH tunnels #6356

etsybaev · 2021-09-21T17:43:27Z

What

The goal of this PR is to add SSH only for the Oracle Source. It does not try to deal with how it can be reused for other sources or destinations.
Took the concept of My PR for Oracle source

How

Handles injecting the SSH tunnel in the check, discover, read methods of Oracle source directly. This makes it relatively straightforward to inject without worrying about touching other jdbc dbs.

Tested with AWS EC2 bastion and RDS Oracle + test normal connection.

Got 2 new options to connect via SSH (with key or pass)

Here is what was in the source DB

Here is the result of data migration via SSH

Also checked that direct connection to DB still works:

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…(aka bastion server)

...ration/java/io/airbyte/integrations/source/oracle/AbstractSshOracleSourceAcceptanceTest.java

sherifnada · 2021-09-22T16:16:13Z

/test connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/1262601110
❌ connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/1262601110
🐛 https://gradle.com/s/wo54vxekbols6

sherifnada · 2021-09-22T16:16:47Z

docs/integrations/sources/oracle.md

@@ -100,6 +100,23 @@ Your database user should now be ready for use with Airbyte.

 Case sensitive. Defaults to the upper-cased user if empty. If the user does not have access to the configured schemas, no tables will be discovered.

+## Connection to Oracle via an SSH Tunnel


Suggested change

## Connection to Oracle via an SSH Tunnel

## Connection via an SSH Tunnel

updated, thanks

sherifnada · 2021-09-22T16:17:13Z

docs/integrations/sources/oracle.md

+5. `SSH Login Username` is the username that Airbyte should use when connection to the bastion server. This is NOT the Oracle username.
+6. If you are using `Password Authentication`, then `SSH Login Username` should be set to the password of the User from the previous step. If you are using `SSH Key Authentication` leave this blank. Again, this is not the Oracle password, but the password for the OS-user that Airbyte is using to perform commands on the bastion.
+7. If you are using `SSH Key Authentication`, then `SSH Private Key` should be set to the RSA Private Key that you are using to create the SSH connection. This should be the full contents of the key file starting with `-----BEGIN RSA PRIVATE KEY-----` and ending with `-----END RSA PRIVATE KEY-----`.
+

 ## Changelog


we should update changelog too

Hi Sherif.
Sure, I was going to to update versions, changelog and secrets before publishing

sherifnada · 2021-09-22T16:18:52Z

...ration/java/io/airbyte/integrations/source/oracle/SshPasswordOracleSourceAcceptanceTest.java

+
+  @Override
+  public Path getConfigFilePath() {
+    return Path.of("secrets/ssh-pwd-config.json");


shouldn't this secret be injected in ci_credentials.sh? where is it being inserted in CI?

Hi Sherif.
Sure, I was going to update versions, changelog, and secrets before publishing. Secrets themselves are already added to LastPass and GitHub like ORACLE_SSH_KEY_TEST_CREDS и ORACLE_SSH_PWD_TEST_CREDS.
For json file name, I just follow the pattern of how it was done in other connectors

…on-via-ssh

sherifnada · 2021-09-22T17:03:32Z

btw, great work on the screenshots! makes it very easy to see what's happening

…on-via-ssh

etsybaev · 2021-09-22T18:01:19Z

/test connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/1262916956
✅ connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/1262916956
No Python unittests run

…on-via-ssh

etsybaev · 2021-09-24T17:37:31Z

/publish connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/1270785025
✅ connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/1270785025

[5968] Source Oracle - Added option to connect top Oracle DB via SSH …

8e2f872

…(aka bastion server)

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 21, 2021

etsybaev linked an issue Sep 21, 2021 that may be closed by this pull request

Add support & test cases for Oracle Source via SSH tunnels #5968

Closed

jrhizor temporarily deployed to more-secrets September 21, 2021 18:23 Inactive

etsybaev requested a review from alexandr-shegeda September 22, 2021 06:31

alexandr-shegeda reviewed Sep 22, 2021

View reviewed changes

...ration/java/io/airbyte/integrations/source/oracle/AbstractSshOracleSourceAcceptanceTest.java Outdated Show resolved Hide resolved

removed commented code

b645e73

etsybaev temporarily deployed to more-secrets September 22, 2021 11:58 Inactive

alexandr-shegeda approved these changes Sep 22, 2021

View reviewed changes

etsybaev temporarily deployed to more-secrets September 22, 2021 12:46 Inactive

etsybaev requested a review from sherifnada September 22, 2021 13:15

etsybaev changed the title ~~[DRAFT_PR]🎉 Source Oracle: Added support for connection via SSH tunnels~~ 🎉 Source Oracle: Added support for connection via SSH tunnels Sep 22, 2021

etsybaev marked this pull request as ready for review September 22, 2021 14:05

etsybaev temporarily deployed to more-secrets September 22, 2021 15:02 Inactive

jrhizor temporarily deployed to more-secrets September 22, 2021 16:18 Inactive

sherifnada approved these changes Sep 22, 2021

View reviewed changes

Merge branch 'master' into etsybaev/5968-source-oracle-added-connecti…

bfaa74b

…on-via-ssh

Added secrets and changelog

344737b

etsybaev temporarily deployed to more-secrets September 22, 2021 17:08 Inactive

Fixed checkstyle

a683b45

etsybaev temporarily deployed to more-secrets September 22, 2021 17:27 Inactive

ievgeniit added 2 commits September 22, 2021 20:37

Merge branch 'master' into etsybaev/5968-source-oracle-added-connecti…

6e10d85

…on-via-ssh

Fixed checkstyle foe python files (how did they get into master?)

5899a77

github-actions bot added the CDK Connector Development Kit label Sep 22, 2021

etsybaev temporarily deployed to more-secrets September 22, 2021 17:43 Inactive

jrhizor temporarily deployed to more-secrets September 22, 2021 18:03 Inactive

Merge branch 'master' into etsybaev/5968-source-oracle-added-connecti…

2bf24a6

…on-via-ssh

github-actions bot removed the CDK Connector Development Kit label Sep 23, 2021

etsybaev temporarily deployed to more-secrets September 23, 2021 06:38 Inactive

etsybaev temporarily deployed to more-secrets September 23, 2021 08:35 Inactive

etsybaev temporarily deployed to more-secrets September 23, 2021 09:04 Inactive

etsybaev temporarily deployed to more-secrets September 23, 2021 11:15 Inactive

Merge branch 'master' into etsybaev/5968-source-oracle-added-connecti…

428db3f

…on-via-ssh

etsybaev temporarily deployed to more-secrets September 23, 2021 14:50 Inactive

Merge branch 'master' into etsybaev/5968-source-oracle-added-connecti…

2228d29

…on-via-ssh

etsybaev temporarily deployed to more-secrets September 24, 2021 07:39 Inactive

Increased version

f1f843d

etsybaev temporarily deployed to more-secrets September 24, 2021 17:01 Inactive

jrhizor temporarily deployed to more-secrets September 24, 2021 17:40 Inactive

etsybaev merged commit 6658ecd into master Sep 24, 2021

etsybaev deleted the etsybaev/5968-source-oracle-added-connection-via-ssh branch September 24, 2021 17:52

jrhizor mentioned this pull request Sep 25, 2021

Bump Airbyte version from 0.29.21-alpha to 0.29.22-alpha #6450

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Oracle: Added support for connection via SSH tunnels #6356

🎉 Source Oracle: Added support for connection via SSH tunnels #6356

etsybaev commented Sep 21, 2021 •

edited

Loading

sherifnada commented Sep 22, 2021 •

edited by github-actions bot

Loading

sherifnada Sep 22, 2021

etsybaev Sep 22, 2021

sherifnada Sep 22, 2021

etsybaev Sep 22, 2021

sherifnada Sep 22, 2021

etsybaev Sep 22, 2021

sherifnada commented Sep 22, 2021

etsybaev commented Sep 22, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 24, 2021 •

edited by github-actions bot

Loading

		@@ -100,6 +100,23 @@ Your database user should now be ready for use with Airbyte.

		Case sensitive. Defaults to the upper-cased user if empty. If the user does not have access to the configured schemas, no tables will be discovered.

		## Connection to Oracle via an SSH Tunnel

	## Connection to Oracle via an SSH Tunnel
	## Connection via an SSH Tunnel

🎉 Source Oracle: Added support for connection via SSH tunnels #6356

🎉 Source Oracle: Added support for connection via SSH tunnels #6356

Conversation

etsybaev commented Sep 21, 2021 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

sherifnada commented Sep 22, 2021 • edited by github-actions bot Loading

sherifnada Sep 22, 2021

Choose a reason for hiding this comment

etsybaev Sep 22, 2021

Choose a reason for hiding this comment

sherifnada Sep 22, 2021

Choose a reason for hiding this comment

etsybaev Sep 22, 2021

Choose a reason for hiding this comment

sherifnada Sep 22, 2021

Choose a reason for hiding this comment

etsybaev Sep 22, 2021

Choose a reason for hiding this comment

sherifnada commented Sep 22, 2021

etsybaev commented Sep 22, 2021 • edited by github-actions bot Loading

etsybaev commented Sep 24, 2021 • edited by github-actions bot Loading

etsybaev commented Sep 21, 2021 •

edited

Loading

sherifnada commented Sep 22, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 22, 2021 •

edited by github-actions bot

Loading

etsybaev commented Sep 24, 2021 •

edited by github-actions bot

Loading