-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
No server chosen by com.mongodb.reactivestreams: Failed looking up SRV record #37618
Comments
/cc @evanchooly (mongodb), @loicmathieu (mongodb) |
Hi, |
Hello, good afternoon,
keep as important note -as a tested workaround- that issue does not appear upon combining Quarkus 3.1.0 and connection string containing exact mongo shards formatted as: So after many thorough test seems that even Quarkus 3.1.0 version faces inability to establish connection when a SRV mongo connection string is provided. Exact logs for aforementioned combination: 15:09:53 INFO traceId=, parentId=, spanId=, sampled= [or.mo.dr.cluster] (cluster-6578779b5cf3934a85277080) No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, srvResolutionException=com.mongodb.MongoConfigurationException: Failed looking up SRV record for '_mongodb._tcp.dxl-gr-preprod-nuler.mongodb.net'., connectionMode=MULTIPLE, serverDescriptions=[]}. Waiting for 30000 ms before timing out
15:09:56 INFO traceId=0caf882656bffbdbd9cae0ca6fbfeb3f, parentId=, spanId=c9077c52e865326b, sampled=false [or.mo.dr.cluster] (executor-thread-2) No server chosen by com.mongodb.reactivestreams.client.internal.ClientSessionHelper$$Lambda$790/0x00000000903f70f8@4180c2b from cluster description ClusterDescription{type=REPLICA_SET, srvResolutionException=com.mongodb.MongoConfigurationException: Failed looking up SRV record for '_mongodb._tcp.dxl-gr-preprod-nuler.mongodb.net'., connectionMode=MULTIPLE, serverDescriptions=[]}. Waiting for 30000 ms before timing out
15:09:57 INFO traceId=, parentId=, spanId=, sampled= [or.mo.dr.cluster] (cluster-6578779b5cf3934a85277080) No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, srvResolutionException=com.mongodb.MongoConfigurationException: Failed looking up SRV record for '_mongodb._tcp.dxl-gr-preprod-nuler.mongodb.net'., connectionMode=MULTIPLE, serverDescriptions=[]}. Waiting for 30000 ms before timing out_
_15:10:23 ERROR traceId=ad5511d672b3e0370c637d1fb918f448, parentId=, spanId=12c2d61cc5444467, sampled=true [io.qu.mu.ru.MutinyInfrastructure] (vert.x-eventloop-thread-0) Mutiny had to drop the following exception: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, srvResolutionException=com.mongodb.MongoConfigurationException: Failed looking up SRV record for '_mongodb._tcp.dxl-gr-preprod-nuler.mongodb.net'., servers=[]
at com.mongodb.internal.connection.BaseCluster.createTimeoutException(BaseCluster.java:428)
at com.mongodb.internal.connection.BaseCluster.handleServerSelectionRequest(BaseCluster.java:324)
at com.mongodb.internal.connection.BaseCluster.access$700(BaseCluster.java:69)
at com.mongodb.internal.connection.BaseCluster$WaitQueueHandler.run(BaseCluster.java:503)
at java.base/java.lang.Thread.run(Unknown Source) As upgrading further in 3.6.1 will require extended effort in a series of services, would you please update us if there are implemented enhancements in mongo connection establishment area in Quarkus 3.6.1 that could resolve the above issue and which are they in high level? Thank you all for your assistance |
Hello, we are experiencing similar issues with Quarkus 3.5.3, we have these stacktraces showing every once in a while when connected to a MongoDB Atlas cluster: |
hello all! |
Hello @loicmathieu, we would like to ask if there is an ongoing investigation on the above issue. |
Facing the same problem here |
@va1erian @roiar @charalampos-micholias in JVM mode we use the default MongoDB DNS resolver, nothing specific to Quarkus. Did you deploy to native and use the experimental DNS resolver? |
@loicmathieu we deploy a regular JVM without touching the dns resolver configuration. However note that this issue disappeared at one point when we upgraded Quarkus, I can't recall the exact version but it has been a couple of month already. We are currently running Quarkus 3.10 |
We're facing exactly the same issue running Quarkus 3.10. Relevant Log Messages:
If we remove the srv Record from the config and add all the single database entries, it is working. |
See #31971 (comment) for a possible workaround |
We believe that #43222 fixes this, so I will close it for now, but if the issue persists with |
Hello @geoand , we are still facing the issue with Quarkus 3.14.4 (native), the only workaround we were able to apply with major effects, was ditching mongo+srv:// in favor of simple mongo:// connection string. (ref. we used: https://stackoverflow.com/questions/78713411/com-mongodb-mongoconfigurationexception-failed-looking-up-txt-record-for-host) We inspected the solution in #43222, and concluded that to our case the retry is not effective, since that once the error is emitted by the Mongo connection attempt, nothing will work until the pod / application is restarted and some period (1 - 10min) passes. We contacted Mongo Atlas support to little help, except an upsell that we bought, and the problem persisted. Additional info: We have at least 4 Quarkus microservices running 3.8.6 (LTS) (Native) and 3.14 that are not affected by the issue. |
Can you elaborate on this a little more please? |
Thanks for the reply, I'll try to elaborate a bit more: Application will emit the following exception for a couple o seconds, then health-check will fail and app restarted, then we will randomly face the same problem for a few minutes, until connection is successful, then the cycle repeats, so in this case it seems the retry (part of the solution of #43222 ) doesn't work because the srv resolution will persistently fail for minutes at times.
this shorter stack trace is also logged:
We are also trying find if there are any problems with DNS resolution in our infrastructure, but so far, nothing. Edit: seems to be related to #32313 |
I wonder if |
Great, we will try-it out and report the results, thank you very much! |
🙏🏽 |
Describe the bug
Issue using Quarkus 2.15.0 version
When trying a connection attempt in a mongo shard through a Quarkus application version >= 2.15.0 below error is propagated:
Issue using Quarkus 3.1.0 version
As described on #32991 this issue is resolved in "Quarkus 3.0.2.Final" version
Based on the aforementioned proposal, upon upgrading on Quarkus "3.1.0.Final" a new error (resulting to same mongo connection inability) takes place.
Below you may find exact logs:
Dedicated Script for further logs:
In attempt to have further view on Mongo SRV attributes we have created a dedicated script - triggered each time mongo data retrieval- is attempted that provides below logs:
Expected behavior
Quarkus should connect to Mongo Atlas without this SRV lookup or MongoSocketOpenException error
Actual behavior
Connection failure takes place when reaching the established time thresshold
"Exception opening socket}, caused by {java.net.ConnectException: Connection refused}}]}. Waiting for 30000 ms before timing out"
How to Reproduce?
Using the free tier mongo atlas account,
Output of
uname -a
orver
MINGW64_NT-10.0-22621 E0624331 3.3.3-341.x86_64 2022-01-17 11:45 UTC x86_64 Msys
Output of
java -version
java version "17.0.7" 2023-04-18 LTS
Quarkus version or git rev
2.15.0 and 3.1.0
Build tool (ie. output of
mvnw --version
orgradlew --version
)No response
Additional information
Dedicated script getMongoDebugLogs() :
The text was updated successfully, but these errors were encountered: