Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Source Mongo DB: Failed to fetch schema if user doesn't have permissions for all collections in db #8752

Closed
sashaNeshcheret opened this issue Dec 13, 2021 · 2 comments · Fixed by #9238
Closed

Source Mongo DB: Failed to fetch schema if user doesn't have permissions for all collections in db #8752

sashaNeshcheret opened this issue Dec 13, 2021 · 2 comments · Fixed by #9238
Assignees
@VitaliiMaltsev
Labels
connectors/source/mongodb connectors/sources-database type/bug Something isn't working

Comments

@sashaNeshcheret
Copy link
Contributor

Environment

  • Airbyte version: v0.33.5-alpha
  • OS Version / Instance: Ubuntu 20.04
  • Deployment: Docker
  • Source Connector and version: Source Mongodb 0.1.8
  • Destination Connector and version: Destination PostgreSQL 0.3.12
  • Severity: Medium
  • Step where error happened: Deploy / Sync job / Setup new connection / Update connector / Upgrade Airbyte

Current Behavior

Failed to fetch schema if user doesn't have permissions for all collections in db.
We create user with role that have permission for 1 collection, and added listCollections privilege to this role.

Expected Behavior

Success fetching collections that user have access

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG

2021-12-13 15:14:47 INFO () TemporalAttemptExecution(get):116 - Executing worker wrapper. Airbyte version: 0.33.5-alpha
2021-12-13 15:14:47 INFO () LineGobbler(voidCall):82 - Checking if airbyte/source-mongodb-v2:0.1.8 exists...
2021-12-13 15:14:47 INFO () LineGobbler(voidCall):82 - airbyte/source-mongodb-v2:0.1.8 was found locally.
2021-12-13 15:14:47 INFO () DockerProcessFactory(create):127 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/7f1dc89b-2468-4bb2-89be-ea24aabf862c/0 --network host --log-driver none airbyte/source-mongodb-v2:0.1.8 discover --config source_config.json
2021-12-13 15:14:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:49 �[32mINFO�[m i.a.i.s.m.MongoDbSource(main):64 - {} - starting source: class io.airbyte.integrations.source.mongodb.MongoDbSource
2021-12-13 15:14:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:49 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):76 - {} - Running integration: io.airbyte.integrations.source.mongodb.MongoDbSource
2021-12-13 15:14:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:49 �[32mINFO�[m i.a.i.b.IntegrationCliParser(parseOptions):118 - {} - integration args: {discover=null, config=source_config.json}
2021-12-13 15:14:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:49 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):80 - {} - Command: DISCOVER
2021-12-13 15:14:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:49 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):81 - {} - Integration config: IntegrationConfig{command=DISCOVER, configPath='source_config.json', catalogPath='null', statePath='null'}
2021-12-13 15:14:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:49 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2021-12-13 15:14:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:49 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2021-12-13 15:14:49 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:49 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2021-12-13 15:14:50 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:50 �[32mINFO�[m c.m.d.l.SLF4JLogger(info):71 - {} - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'}
2021-12-13 15:14:50 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:50 �[32mINFO�[m c.m.d.l.SLF4JLogger(info):71 - {} - Cluster description not yet available. Waiting for 30000 ms before timing out
2021-12-13 15:14:50 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:50 �[32mINFO�[m c.m.d.l.SLF4JLogger(info):71 - {} - Opened connection [connectionId{localValue:1, serverValue:104}] to localhost:27017
2021-12-13 15:14:50 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:50 �[32mINFO�[m c.m.d.l.SLF4JLogger(info):71 - {} - Opened connection [connectionId{localValue:2, serverValue:103}] to localhost:27017
2021-12-13 15:14:50 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:50 �[32mINFO�[m c.m.d.l.SLF4JLogger(info):71 - {} - Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=38070184}
2021-12-13 15:14:50 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-12-13 15:14:50 �[32mINFO�[m c.m.d.l.SLF4JLogger(info):71 - {} - Opened connection [connectionId{localValue:3, serverValue:105}] to localhost:27017
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - Exception in thread "main" com.mongodb.MongoCommandException: Command failed with error 13 (Unauthorized): 'not authorized on sales to execute command { aggregate: "orders", readConcern: { level: "majority" }, pipeline: [ { $project: { arrayofkeyvalue: { $objectToArray: "$$ROOT" } } }, { $unwind: "$arrayofkeyvalue" }, { $group: { _id: null, allkeys: { $addToSet: "$arrayofkeyvalue.k" } } } ], cursor: {}, $db: "sales", lsid: { id: UUID("7a7c3225-9960-4bf2-ab8c-55fd7286c7fa") } }' on server localhost:27017. The full response is {"ok": 0.0, "errmsg": "not authorized on sales to execute command { aggregate: "orders", readConcern: { level: "majority" }, pipeline: [ { $project: { arrayofkeyvalue: { $objectToArray: "$$ROOT" } } }, { $unwind: "$arrayofkeyvalue" }, { $group: { _id: null, allkeys: { $addToSet: "$arrayofkeyvalue.k" } } } ], cursor: {}, $db: "sales", lsid: { id: UUID("7a7c3225-9960-4bf2-ab8c-55fd7286c7fa") } }", "code": 13, "codeName": "Unauthorized"}
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:195)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:398)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:319)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:114)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:601)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:81)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:249)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:214)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:113)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:328)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:318)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.operation.CommandOperationHelper.executeCommandWithConnection(CommandOperationHelper.java:201)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.operation.CommandOperationHelper.lambda$executeCommand$4(CommandOperationHelper.java:189)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:583)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:189)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.operation.AggregateOperationImpl.execute(AggregateOperationImpl.java:195)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.operation.AggregateOperation.execute(AggregateOperation.java:306)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.internal.operation.AggregateOperation.execute(AggregateOperation.java:46)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:184)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:135)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:92)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at com.mongodb.client.internal.MongoIterableImpl.cursor(MongoIterableImpl.java:97)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at io.airbyte.db.mongodb.MongoUtils.getFieldsName(MongoUtils.java:200)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at io.airbyte.db.mongodb.MongoUtils.getUniqueFields(MongoUtils.java:187)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at io.airbyte.integrations.source.mongodb.MongoDbSource.discoverInternal(MongoDbSource.java:119)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at io.airbyte.integrations.source.mongodb.MongoDbSource.discoverInternal(MongoDbSource.java:41)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at io.airbyte.integrations.source.relationaldb.AbstractDbSource.discoverWithoutSystemTables(AbstractDbSource.java:131)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at io.airbyte.integrations.source.relationaldb.AbstractDbSource.getTables(AbstractDbSource.java:321)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at io.airbyte.integrations.source.relationaldb.AbstractDbSource.discover(AbstractDbSource.java:82)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:108)
2021-12-13 15:14:50 ERROR () LineGobbler(voidCall):82 - at io.airbyte.integrations.source.mongodb.MongoDbSource.main(MongoDbSource.java:65)
2021-12-13 15:14:50 INFO () TemporalAttemptExecution(lambda$getWorkerThread$2):170 - Completing future exceptionally...
io.airbyte.workers.WorkerException: Discover job subprocess finished with exit code 1
at io.airbyte.workers.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:70) ~[io.airbyte-airbyte-workers-0.33.5-alpha.jar:?]
at io.airbyte.workers.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:24) ~[io.airbyte-airbyte-workers-0.33.5-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:167) ~[io.airbyte-airbyte-workers-0.33.5-alpha.jar:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
2021-12-13 15:14:50 INFO () TemporalAttemptExecution(get):137 - Stopping cancellation check scheduling...
2021-12-13 15:14:50 WARN () POJOActivityTaskHandler$POJOActivityImplementation(execute):243 - Activity failure. ActivityId=7ca4ac3e-a848-3499-b5d5-ced8ba8f15d3, activityType=Run, attempt=1
java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Discover job subprocess finished with exit code 1
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2063) ~[?:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.33.5-alpha.jar:?]
at io.airbyte.workers.temporal.discover.catalog.DiscoverCatalogActivityImpl.run(DiscoverCatalogActivityImpl.java:82) ~[io.airbyte-airbyte-workers-0.33.5-alpha.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]
at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:277) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:216) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:181) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:192) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:154) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.0.4.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: io.airbyte.workers.WorkerException: Discover job subprocess finished with exit code 1
at io.airbyte.workers.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:70) ~[io.airbyte-airbyte-workers-0.33.5-alpha.jar:?]
at io.airbyte.workers.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:24) ~[io.airbyte-airbyte-workers-0.33.5-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:167) ~[io.airbyte-airbyte-workers-0.33.5-alpha.jar:?]
... 1 more

Steps to Reproduce

Are you willing to submit a PR?

Remove this with your answer.
@sashaNeshcheret sashaNeshcheret added type/bug Something isn't working needs-triage and removed needs-triage labels Dec 13, 2021
@sashaNeshcheret sashaNeshcheret mentioned this issue Dec 13, 2021
Closed
@sherifnada sherifnada moved this to Prioritized for Scoping in GL Roadmap Dec 17, 2021
@alexandr-shegeda alexandr-shegeda moved this from Prioritized for scoping to Ready for implementation in GL Roadmap Dec 23, 2021
@VitaliiMaltsev VitaliiMaltsev self-assigned this Dec 28, 2021
@oustynova oustynova moved this from Ready for implementation to Implementation in progress in GL Roadmap Dec 28, 2021
@VitaliiMaltsev VitaliiMaltsev moved this from Implementation in progress to Internal review in GL Roadmap Jan 11, 2022
@VitaliiMaltsev VitaliiMaltsev moved this from Internal review to Airbyte review in GL Roadmap Jan 11, 2022
@VitaliiMaltsev VitaliiMaltsev linked a pull request Jan 11, 2022 that will close this issue
Source MongoDB fetch authorized collections only #9238
Merged
40 tasks
@alexandr-shegeda alexandr-shegeda moved this from Airbyte review to Done in GL Roadmap Jan 14, 2022
@sajid-r
Copy link

sajid-r commented Jan 23, 2022

Thanks for the fix @VitaliiMaltsev , this was a blocker for us.
But, discovery fails eventually, with user(s) that have all permissions to a collection and listCollections permission for the DB. Does it require any additional DB permissions?

Error:

com.mongodb.MongoCommandException: Command failed with error 13 (Unauthorized): 'not authorized on dbName to execute command { aggregate: "converted_originalll", readConcern: { level: "majority" }, pipeline: [ { $limit: 10000 }, { $project: { arrayofkeyvalue: { $objectToArray: "$$ROOT" } } }, { $unwind: "$arrayofkeyvalue" }, { $group: { _id: null, allkeys: { $addToSet: "$arrayofkeyvalue.k" } } } ], cursor: {}, $db: "dbName" }'

@VitaliiMaltsev
Copy link
Contributor

@sajid-r if you want fetch authorized collections only user should not have listCollections permission for the DB. For this user need to have find permissions at the collection level.

For example
db.createRole( { role: "readrole", privileges: [ { resource: { db: "customDB", collection: "test1" }, actions: [ "find" ] }, { resource: { db: "customDB", collection: "test2" }, actions: [ "find" ] } ], roles: [] } )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VitaliiMaltsev VitaliiMaltsev

Labels
connectors/source/mongodb connectors/sources-database type/bug Something isn't working
Projects
No open projects
GL Roadmap
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Source MongoDB fetch authorized collections only airbytehq/airbyte
5 participants
@bleonard @sajid-r @sashaNeshcheret @VitaliiMaltsev @igrankova