Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mlaunch fails launching a sharded cluster on evergreen #691

Closed
p-mongo opened this issue Jun 6, 2019 · 6 comments · Fixed by #790
Closed

mlaunch fails launching a sharded cluster on evergreen #691

p-mongo opened this issue Jun 6, 2019 · 6 comments · Fixed by #790
Labels
bug mlaunch
Milestone
1.6.4

Comments

@p-mongo
Copy link
Contributor

p-mongo commented Jun 6, 2019 •
edited
Loading

I ran this:

[2019/06/06 13:57:13.155] + mlaunch --dir /data/mci/f248d67c080a7afe47e162395af1375f/src/../drivers-tools/.evergreen/orchestration/db --setParameter enableTestCommands=1 --filePermissions 0666 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem --replicaset --sharded 2 --name ruby-driver-rs --auth --username alice --password wland
[2019/06/06 13:57:35.647] launching: "mongod" on port 27018
[2019/06/06 13:57:35.647] launching: "mongod" on port 27019
[2019/06/06 13:57:35.647] launching: "mongod" on port 27020
[2019/06/06 13:57:35.647] launching: "mongod" on port 27021
[2019/06/06 13:57:35.647] launching: "mongod" on port 27022
[2019/06/06 13:57:35.647] launching: "mongod" on port 27023
[2019/06/06 13:57:35.647] Traceback (most recent call last):
[2019/06/06 13:57:35.647] launching: config server on port 27024
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/src/../drivers-tools/.evergreen/orchestration/python/bin/mlaunch", line 10, in 
[2019/06/06 13:57:35.647] replica set 'configRepl' initialized.
[2019/06/06 13:57:35.647]     sys.exit(main())
[2019/06/06 13:57:35.647] replica set 'shard01' initialized.
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/mtools/mlaunch/mlaunch.py", line 2068, in main
[2019/06/06 13:57:35.647] replica set 'shard02' initialized.
[2019/06/06 13:57:35.647] launching: mongos on port 27017
[2019/06/06 13:57:35.647] adding shards. can take up to 30 seconds...
[2019/06/06 13:57:35.647]     tool.run()
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/mtools/mlaunch/mlaunch.py", line 542, in run
[2019/06/06 13:57:35.647]     getattr(self, self.args['command'])()
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/mtools/mlaunch/mlaunch.py", line 773, in init
[2019/06/06 13:57:35.647]     roles=roles)
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/mtools/mlaunch/mlaunch.py", line 1708, in _add_user
[2019/06/06 13:57:35.647]     **opts)
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/database.py", line 1041, in add_user
[2019/06/06 13:57:35.647]     session=session, **kwargs)
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/database.py", line 961, in _create_or_update_user
[2019/06/06 13:57:35.648]     self.command(command_name, name, session=session, **opts)
[2019/06/06 13:57:35.648]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/database.py", line 532, in command
[2019/06/06 13:57:35.648]     codec_options, session=session, **kwargs)
[2019/06/06 13:57:35.648]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/database.py", line 439, in _command
[2019/06/06 13:57:35.648]     client=self.__client)
[2019/06/06 13:57:35.648]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/pool.py", line 522, in command
[2019/06/06 13:57:35.648]     self._raise_connection_failure(error)
[2019/06/06 13:57:35.648]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/pool.py", line 677, in _raise_connection_failure
[2019/06/06 13:57:36.186]     raise error
pymongo.errors.NotMasterError: not master
[2019/06/06 13:57:36.186] Command failed: command encountered problem: exit status 1

Note that mlaunch said it may take up to 30 seconds to add shards, but then failed after 1 second due to a not master error.

Should server selection timeout be increased?

Environment

Software Version
mtools 1.5.3
MongoDB server 4.1.9
Operating system ubuntu1404

[2019/06/06 13:57:13.052] Successfully installed mtools-1.5.3 psutil-5.4.2 pymongo-3.6.1 python-dateutil-2.7.0 six-1.12.0
@p-mongo
Copy link
Contributor Author

p-mongo commented Feb 14, 2020

Any update/thoughts on this? I am moving all of our configurations to mlaunch and this issue is a significant problem because it fails the entire test run.

[2020/02/14 00:44:51.147] + mlaunch --dir /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/db --binarypath /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin --setParameter enableTestCommands=1 --setParameter diagnosticDataCollectionEnabled=false --replicaset --sharded 2 --name ruby-driver-rs --mongos 2 --auth --username bob --password pwd123 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server-second-level-bundle.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27019
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27020
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27021
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27022
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27023
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27024
[2020/02/14 00:45:15.439] launching: config server on port 27025
[2020/02/14 00:45:15.439] replica set 'configRepl' initialized.
[2020/02/14 00:45:15.439] replica set 'shard01' initialized.
[2020/02/14 00:45:15.439] replica set 'shard02' initialized.
[2020/02/14 00:45:15.439] launching: /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27017
[2020/02/14 00:45:15.439] launching: /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27018
[2020/02/14 00:45:15.439] adding shards. can take up to 30 seconds...
[2020/02/14 00:45:15.935] Traceback (most recent call last):
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/bin/mlaunch", line 8, in 
[2020/02/14 00:45:15.935]     sys.exit(main())
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 2068, in main
[2020/02/14 00:45:15.935]     tool.run()
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 542, in run
[2020/02/14 00:45:15.935]     getattr(self, self.args['command'])()
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 773, in init
[2020/02/14 00:45:15.935]     roles=roles)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 1708, in _add_user
[2020/02/14 00:45:15.935]     **opts)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 1339, in add_user
[2020/02/14 00:45:15.935]     session=session, **kwargs)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 1253, in _create_or_update_user
[2020/02/14 00:45:15.935]     self.command(command_name, name, session=session, **opts)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 733, in command
[2020/02/14 00:45:15.935]     codec_options, session=session, **kwargs)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 630, in _command
[2020/02/14 00:45:15.935]     client=self.__client)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/pool.py", line 618, in command
[2020/02/14 00:45:15.935]     self._raise_connection_failure(error)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/pool.py", line 613, in command
[2020/02/14 00:45:15.935]     user_fields=user_fields)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/network.py", line 167, in command
[2020/02/14 00:45:15.935]     parse_write_concern_error=parse_write_concern_error)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/helpers.py", line 136, in _check_command_response
[2020/02/14 00:45:15.935]     raise NotMasterError(errmsg, response)
[2020/02/14 00:45:15.935] pymongo.errors.NotMasterError: not master
[2020/02/14 00:45:15.950] Command failed: command encountered problem: error waiting on process '1b99b8da-e02f-4b16-9f75-f37f58a0df05': exit status 1

The diagnostics here could be better. Is mlaunch requesting a primary server? If yes, the error should happen after server selection timeout. Does mlaunch change server selection timeout from the default? I assume the default value is more than 1 second or so. I am not seeing primary changes in server logs - each replica set elected its primary one time.

What other information can I provide here?

Run log: http://speed:9393/repos/mongodb/mongo-ruby-driver/pulls/1691/evergreen-log/mongo_ruby_driver_jruby_auth__mongodb_version~4.2_topology~sharded_cluster_auth_and_ssl~auth_and_ssl_ruby~jruby_9.2_os~ubuntu1604_patch_feadf5ebbba8774f6deaba0dc9cb05fc18d2caa7_5e463396e3c3311d0f6cd534_20_02_14_05_43_51#first-failure

Server logs: http://speed:9393/eg/mongo-ruby-driver/versions/5e4634db7742ae0eebc9db86/builds/mongo_ruby_driver_jruby_auth__mongodb_version~4.2_topology~sharded_cluster_auth_and_ssl~auth_and_ssl_ruby~jruby_9.2_os~ubuntu1604_patch_feadf5ebbba8774f6deaba0dc9cb05fc18d2caa7_5e463396e3c3311d0f6cd534_20_02_14_05_43_51/artifact-logs

p-mongo pushed a commit to p-mongo/mtools that referenced this issue Feb 14, 2020
@p
Wait for each RS to elect a primary
84f4f70 
In a sharded replica set, wait for each of the shards to elect
a primary before continuing to hopefully fix rueckstiess#691
@p-mongo p-mongo mentioned this issue Feb 14, 2020
Merged
@p-mongo
Copy link
Contributor Author

p-mongo commented Feb 14, 2020

Repro: https://github.com/p-mongo/tests/tree/master/mlaunch-not-master

kevinadi pushed a commit that referenced this issue Feb 17, 2020
@p-mongo @p
Wait for each RS to elect a primary (#790)
b94620c 
In a sharded replica set, wait for each of the shards to elect
a primary before continuing to hopefully fix #691

Co-authored-by: Oleg Pudeyev <p@users.noreply.github.com>
@stennie stennie added this to the 1.6.4 milestone Feb 17, 2020
@p-mongo
Copy link
Contributor Author

p-mongo commented Feb 17, 2020

Actually this issue is still happening, my fix either didn't do anything or was not sufficient.

@ShaneHarvey
Copy link
Contributor

ShaneHarvey commented Feb 18, 2020 •
edited
Loading

I believe the bug is that mlaunch connects directly to a single mongod to add the initial user here:
https://github.com/p-mongo/mtools/blob/80be43b/mtools/mlaunch/mlaunch.py#L1655

    def _add_user(self, port, name, password, database, roles):
        con = self.client('localhost:%i'%port)

It should be connecting to the replica set as a whole, like this:

    def _add_user(self, port, name, password, database, roles):
        con = self.client('localhost:%i'%port, replicaSet=shard_replica_set_name)

or:

    def _add_user(self, port, name, password, database, roles):
        con = self.client('localhost:%i'%port)
        ismaster = con['admin'].command('isMaster')
        set_name = ismaster.get('setName')
        if set_name:
            con = self.client('localhost:%i'%port, replicaSet=set_name)

p-mongo pushed a commit to p-mongo/mtools that referenced this issue Feb 19, 2020
@p
fix rueckstiess#691 for real
6cf8388
p-mongo pushed a commit to p-mongo/mtools that referenced this issue Feb 24, 2020
@p
fix rueckstiess#691 for real
0820675
@p-mongo
Copy link
Contributor Author

p-mongo commented Feb 25, 2020

I applied @ShaneHarvey 's fix in https://github.com/p-mongo/mtools/commits/wait-for-rs, but now I am frequently getting this error:

[2020/02/24 18:21:13.011] + eval mlaunch --dir /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/db --binarypath /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin --setParameter enableTestCommands=1 --setParameter diagnosticDataCollectionEnabled=false --replicaset --sharded 2 --name ruby-driver-rs --mongos 2 --auth --username bob --password pwd123 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server-second-level-bundle.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem
[2020/02/24 18:21:13.011] ++ mlaunch --dir /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/db --binarypath /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin --setParameter enableTestCommands=1 --setParameter diagnosticDataCollectionEnabled=false --replicaset --sharded 2 --name ruby-driver-rs --mongos 2 --auth --username bob --password pwd123 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server-second-level-bundle.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem
[2020/02/24 18:21:34.357] Finished processing dependencies for mtools-legacy==1.5.4
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27019
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27020
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27021
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27022
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27023
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27024
[2020/02/24 18:21:34.357] launching: config server on port 27025
[2020/02/24 18:21:34.357] replica set 'configRepl' initialized.
[2020/02/24 18:21:34.357] replica set 'shard01' initialized.
[2020/02/24 18:21:34.357] replica set 'shard02' initialized.
[2020/02/24 18:21:34.357] launching: /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27017
[2020/02/24 18:21:34.357] launching: /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27018
[2020/02/24 18:21:34.357] adding shards. can take up to 30 seconds...
[2020/02/24 18:21:34.357] Traceback (most recent call last):
[2020/02/24 18:21:34.357]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/bin/mlaunch", line 9, in 
[2020/02/24 18:21:34.357]     load_entry_point('mtools-legacy==1.5.4', 'console_scripts', 'mlaunch')()
[2020/02/24 18:21:34.357]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 2073, in main
[2020/02/24 18:21:34.357]     tool.run()
[2020/02/24 18:21:34.357]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 542, in run
[2020/02/24 18:21:34.357]     getattr(self, self.args['command'])()
[2020/02/24 18:21:34.357]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 773, in init
[2020/02/24 18:21:34.357]     roles=roles)
[2020/02/24 18:21:34.357]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 1701, in _add_user
[2020/02/24 18:21:34.358]     v = con['admin'].command('isMaster').get('maxWireVersion', 0)
[2020/02/24 18:21:34.358]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/database.py", line 730, in command
[2020/02/24 18:21:34.358]     read_preference, session) as (sock_info, slave_ok):
[2020/02/24 18:21:34.358]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
[2020/02/24 18:21:34.358]     return self.gen.next()
[2020/02/24 18:21:34.358]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1298, in _socket_for_reads
[2020/02/24 18:21:34.358]     server = self._select_server(read_preference, session)
[2020/02/24 18:21:34.358]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1253, in _select_server
[2020/02/24 18:21:34.359]     server = topology.select_server(server_selector)
[2020/02/24 18:21:34.359]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/topology.py", line 235, in select_server
[2020/02/24 18:21:34.359]     address))
[2020/02/24 18:21:34.359]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/topology.py", line 193, in select_servers
[2020/02/24 18:21:34.359]     selector, server_timeout, address)
[2020/02/24 18:21:34.359]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/topology.py", line 209, in _select_servers_loop
[2020/02/24 18:21:34.359]     self._error_message(selector))
[2020/02/24 18:21:34.359] pymongo.errors.ServerSelectionTimeoutError: No replica set members match selector "Primary()"
[2020/02/24 18:21:34.869] Command failed: command encountered problem: error waiting on process '95343b3a-3e70-403f-884f-a811b90cecc2': exit status 1

I tried setting the server selection timeout explicitly in p-mongo@b932e76 since the error message does not indicate what the timeout is, and you can see in the above log that there is a 20 second gap between the second and the third line indicating the client did wait for some time (timestamps are inaccurately reported by evergreen, if you are wondering why a bunch of log entries have the same timestamp). The timeout seems to have not had any effect.

@ShaneHarvey
Copy link
Contributor

ShaneHarvey commented Feb 25, 2020 •
edited
Loading

Can you try adding serverSelectionTimeoutMS=10000 to the replica set client too?: https://github.com/p-mongo/mtools/blob/b932e763fb59b9b20972036ea792e0d58b024aba/mtools/mlaunch/mlaunch.py#L1700

p-mongo pushed a commit to p-mongo/mtools that referenced this issue Mar 8, 2020
@p
fix rueckstiess#691 for real
67b8f3e
p-mongo pushed a commit to p-mongo/mtools that referenced this issue Mar 8, 2020
@p
fix rueckstiess#691 for real
83df288
@p-mongo p-mongo mentioned this issue Mar 8, 2020
Closed
p-mongo pushed a commit to p-mongo/mtools that referenced this issue Mar 8, 2020
@p
fix rueckstiess#691 for real
dff3b0b
stennie pushed a commit that referenced this issue Jun 15, 2020
@p @stennie
fix #691 for real
ae7ec0e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug mlaunch
Projects
None yet
Milestone
1.6.4
Development

Successfully merging a pull request may close this issue.

Wait for each RS to elect a primary p-mongo/mtools
3 participants
@stennie @ShaneHarvey @p-mongo