Source MongoDb: added support via TLS/SSL #6364

irynakruk · 2021-09-22T04:08:14Z

What

Added support of TLS encryption by default when connecting to the MongoDB source: MongoDb Atlas Cluster and Replica Set.
For Standalone MongoDb instance added possibility to select whether to encrypt, as it may require additional configuration from customer.
Provided small fix for incremental sync.

How

For MongoDb Atlas Cluster and Replica Set added ssl=true to connection param.
For Standalone MongoDb instance it is configurable.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

fixed replica connection; fixed incremental read;

…it by default for other types

irynakruk · 2021-09-23T15:01:41Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1266351264
❌ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1266351264
🐛

irynakruk · 2021-09-23T16:42:43Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1266699799
❌ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1266699799
🐛 https://gradle.com/s/hbdjbvm4v2kvo

sherifnada · 2021-09-23T16:41:22Z

airbyte-db/lib/src/main/java/io/airbyte/db/mongodb/MongoUtils.java

+        case TIMESTAMP -> new BsonTimestamp(Long.parseLong(value));
+        case DATE_TIME -> new BsonDateTime(new DateTime(value).getValue());
+        case OBJECT_ID -> new ObjectId(value);
+        default -> null;


instead of default null why don't we return string?

Also I don't see a mapping from string or object, how come?

I've changed to String, thanks! Also added String and Symbol.
This mapping needed only for cursor field, since it couldn't be object or array, it isn't present here.

sherifnada · 2021-09-23T16:41:47Z

airbyte-integrations/connectors/source-mongodb-v2/README.md

@@ -26,7 +26,29 @@ the Dockerfile.
 We use `JUnit` for Java tests.

 ### Test Configuration
-No specific configuration needed for testing, MongoDb Test Container is used.
+
+In order to test the MongoDb source, you need to have a standalone instance, a replica set or Atlas cluster.


it doesn't work with a docker image anymore?

It works, but I've also added test for Atlas Cluster with ssl, I've corrected doc.

sherifnada · 2021-09-23T16:42:47Z

airbyte-integrations/connectors/source-mongodb-v2/src/main/resources/spec.json

@@ -98,13 +105,6 @@
        "default": "admin",
        "examples": ["admin"],
        "order": 4
-      },


you will need to make additionalProperties=true for this to be backwards compatible, and potentially handle it manually in the connector code

Thanks a lot, I've added.

sherifnada · 2021-09-23T16:47:25Z

docs/integrations/sources/mongodb-v2.md

+### TLS/SSL on a Connection
+
+It is recommended to use encrypted connection.
+Connection with TLS/SSL security protocol for MongoDb Atlas Cluster and Replica Set instances is enabled by default. Encryption is based on the underlying TLS/SSL support in the JDK.


suggest removing since it's an implementation detail

Suggested change

Connection with TLS/SSL security protocol for MongoDb Atlas Cluster and Replica Set instances is enabled by default. Encryption is based on the underlying TLS/SSL support in the JDK.

Connection with TLS/SSL security protocol for MongoDb Atlas Cluster and Replica Set instances is enabled by default.

irynakruk · 2021-09-24T01:34:36Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1268140175
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1268140175
No Python unittests run

irynakruk · 2021-09-24T01:42:47Z

/publish connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1268154279
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1268154279

irynakruk added 3 commits September 17, 2021 20:19

updated ssl configs;

93a1a26

fixed replica connection; fixed incremental read;

added MongoDbSourceAtlasAcceptanceTest with ssl enabled

dd04c26

updated docs, moved TLS option for standalone instance type, enabled …

b3de9aa

…it by default for other types

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 22, 2021

irynakruk linked an issue Sep 22, 2021 that may be closed by this pull request

Add support for MongoDB source via TLS/SSL #6014

Closed

5 tasks

Update mongodb-v2.md

d78e3f5

irynakruk changed the title ~~Source MongoDB - Add support for MongoDB source via TLS/SSL~~ Source MongoDb - added support via TLS/SSL Sep 22, 2021

irynakruk changed the title ~~Source MongoDb - added support via TLS/SSL~~ Source MongoDb: added support via TLS/SSL Sep 22, 2021

irynakruk temporarily deployed to more-secrets September 22, 2021 04:19 Inactive

irynakruk requested review from alexandr-shegeda and DoNotPanicUA September 22, 2021 12:17

updated README.md

7b9093d

irynakruk temporarily deployed to more-secrets September 22, 2021 15:32 Inactive

DoNotPanicUA approved these changes Sep 23, 2021

View reviewed changes

alexandr-shegeda approved these changes Sep 23, 2021

View reviewed changes

updated spec.json

1b08e06

irynakruk temporarily deployed to more-secrets September 23, 2021 14:42 Inactive

irynakruk marked this pull request as ready for review September 23, 2021 14:49

irynakruk requested a review from sherifnada September 23, 2021 14:49

jrhizor temporarily deployed to more-secrets September 23, 2021 15:04 Inactive

merged master

d3e708c

irynakruk temporarily deployed to more-secrets September 23, 2021 16:12 Inactive

jrhizor temporarily deployed to more-secrets September 23, 2021 16:45 Inactive

sherifnada approved these changes Sep 23, 2021

View reviewed changes

Code review changes

d8411a4

irynakruk temporarily deployed to more-secrets September 24, 2021 01:03 Inactive

updated ci_credentials.sh

66f8dd4

irynakruk temporarily deployed to more-secrets September 24, 2021 01:09 Inactive

jrhizor temporarily deployed to more-secrets September 24, 2021 01:37 Inactive

jrhizor temporarily deployed to more-secrets September 24, 2021 01:44 Inactive

irynakruk temporarily deployed to more-secrets September 24, 2021 02:08 Inactive

This was referenced Sep 24, 2021

Redshift Src/Destination #6429

Closed

BigQuery Src/Destination #6431

Closed

Snowflake Src/Destination #6432

Closed

DB2 source #6434

Closed

Clickhouse source TLS #6435

Closed

Kafka source/destination #6436

Closed

irynakruk temporarily deployed to more-secrets September 24, 2021 12:17 Inactive

irynakruk temporarily deployed to more-secrets September 24, 2021 13:15 Inactive

irynakruk merged commit 5551698 into master Sep 24, 2021

irynakruk deleted the irynakruk/6014-mongodb-tsl branch September 24, 2021 13:22

jrhizor mentioned this pull request Sep 25, 2021

Bump Airbyte version from 0.29.21-alpha to 0.29.22-alpha #6450

Merged

tanyaaae12 mentioned this pull request Apr 25, 2024

[connector-builder] Record Selector doesn't return result #37566

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source MongoDb: added support via TLS/SSL #6364

Source MongoDb: added support via TLS/SSL #6364

irynakruk commented Sep 22, 2021 •

edited

Loading

irynakruk commented Sep 23, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 23, 2021 •

edited by github-actions bot

Loading

sherifnada Sep 23, 2021

irynakruk Sep 24, 2021

sherifnada Sep 23, 2021

irynakruk Sep 24, 2021

sherifnada Sep 23, 2021

irynakruk Sep 24, 2021

sherifnada Sep 23, 2021

irynakruk Sep 24, 2021

irynakruk commented Sep 24, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 24, 2021 •

edited by github-actions bot

Loading

	Connection with TLS/SSL security protocol for MongoDb Atlas Cluster and Replica Set instances is enabled by default. Encryption is based on the underlying TLS/SSL support in the JDK.
	Connection with TLS/SSL security protocol for MongoDb Atlas Cluster and Replica Set instances is enabled by default.

Source MongoDb: added support via TLS/SSL #6364

Source MongoDb: added support via TLS/SSL #6364

Conversation

irynakruk commented Sep 22, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

irynakruk commented Sep 23, 2021 • edited by github-actions bot Loading

irynakruk commented Sep 23, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

irynakruk commented Sep 24, 2021 • edited by github-actions bot Loading

irynakruk commented Sep 24, 2021 • edited by github-actions bot Loading

irynakruk commented Sep 22, 2021 •

edited

Loading

irynakruk commented Sep 23, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 23, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 24, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 24, 2021 •

edited by github-actions bot

Loading