[mongo] dd-agent v5.6.0+ cannot be used with Dockerized mongo replica set

[theartoflogic]

In version 5.6.0 of the dd-agent, the mongo.py check script was updated (PR #1961) to create a new connection when getting replica set data. The new connection is made using the following code:

cli = pymongo.mongo_client.MongoClient(
    server,
    socketTimeoutMS=timeout,
    replicaset=setname,
    read_preference=pymongo.ReadPreference.NEAREST,
    **ssl_params)

The problem is that it sets the replicaset parameter, which according to the pymongo docs means that it will try to connect to each member in the replica set:

The driver will verify that all servers it connects to match this name. Implies that the hosts specified are a seed list and the driver should attempt to find all members of the set.

When the dd-agent is running on the Docker host machine (i.e. not Dockerized), it cannot connect to the other members in the replica set, since they are running inside of Docker containers and are connected by an internal network within the context of Docker. In my case, I'm using Weave to connect the Docker containers together.

When the dd-agent runs the mongo check script it returns the error:

instance #0 [ERROR]: 'mongo-shard-01-01.weave.local:27018: [Errno -2] Name or service not known,mongo-shard-01-arbiter.weave.local:27018: [Errno -2] Name or service not known,mongo-shard-01-02.weave.local:27018: [Errno -2] Name or service not known'

As you can see, it's trying to connect to the other members of the replica set and is failing with the error Name or service not known, since they are not accessible from the host machine where the dd-agent is running.

[remh]

Thanks for the detailed feedback @theartoflogic !

If i understand properly, a reasonable fix for this would be to add a flag to disable the connections to the other replicas ?

[theartoflogic]

@remh Yeah, I think that would solve the issue for now. I'm curious if there's another solution that could still give the same functionality, although I'm unsure of the exact nature of that code and what it's intended purpose is.

I'll look into it a little more to understand it better and see if there's another solution, but I think either way a flag to disable that would probably be good.

[gmmeyer]

Hey there @theartoflogic!

Sorry for taking so long on this! There is another solution, which we use on our integration tests. Unfortunately, it takes some manual intervention in docker.

If you look at the script that sets up our dockerized mongo for our Mongo Check, you can see what we do. We have to manually configure the host and port for mongo

mongo --eval "cfg = rs.conf(); cfg.members[0].host = '$SHARD00_IP:$PORT'; rs.reconfig(cfg); printjson(rs.conf());" localhost:$PORT

I know this is not a great situation. However, it does work.

If you found another solution in the interim, we'd be very happy to hear it! Thank you!

[theartoflogic]

Note: I no longer manage the system that this bug pertains to, but I'll still follow up to help resolve the problem.

@gmmeyer I'm not sure how that would solve the issue.

The issue is with the fact that the DataDog agent is running on the Docker host machine (not in a Docker container and, therefore, not on the Weave network), while the Mongo replicas are all running inside of Docker, and on different servers for the most part (except the one Mongo instance I'm trying to test with the DD Agent that lives on the same machine).

It doesn't matter what I set the host and port to in Mongo for the replicas, since DataDog won't be able to access them, since the DD agent wouldn't be running on the Weave network.

[gmmeyer]

Fair enough, admittedly I don't know very much about weave.

This is a crappy bug. Thank you for the report.

[theartoflogic]

I think @remh's idea of making it configurable as to whether it tries to connect to other replicas would solve the problem. That way if someone has a setup like mine they could disable replica checks. Otherwise, they could keep it enabled if they want.