MongoDB Ruby client driver compatibility

[ricbartm]

Hello folks!!

We are facing some trouble with DataDog dd-trace-rb and out MongoDB integration from a Ruby service we operate.

We are running MongoDB Server 3.4, MongoDB Ruby client version 2.4 and dd-trace-rb version 1.9.0. The situation is as follows:

1. We want to upgrade to a newer MongoDB Server version (e.g. 3.6 and above) because MongoDB Server 3.4 end-of-life-support ends on September 2019
2. We need a newer MongoDB Ruby client driver (>=2.5) to have full compatibility with MongoDB Server 3.6 and above.
3. This gem dd-trace-rb 1.9.0 (but also apparently in latest version 0.21.0) supports MongoDB Ruby client driver up to version 2.4, so upgrading MongoDB Ruby client driver to a newer version would break DataDog APM integration.

We tried the upgrade anyways, and we have seen APM spans are lost on the integration of our service with MongoDB server, loosing a lot of visibility. MongoDB Server 3.4 support end-of-life ends in September 2019 (6 months from now).

This ticket aims to make this issue more visible to other MongoDB-from-ruby users and gather advice/support for other folks suffering this issue. In parallel I've opened a official support ticket with DataDog to see what alternatives we have.

[ricbartm]

We will try dd-trace-rb version 0.20.0 or above to see if PR #704 improves the situation

[ricbartm]

The compatibility issues I pointed out are officially described in the following pages

https://docs.datadoghq.com/tracing/languages/ruby/#compatibility
https://docs.mongodb.com/ecosystem/drivers/driver-compatibility-reference/#ruby-driver-compatibility

[ricbartm]

Confirmed that ddtrace version 0.21.0 suffers this issue too

[delner]

@ricbartm Thanks for the heads up. Just to clarify, this is a request to support mongo >= 2.5?

[ricbartm]

Hello @delner . Thanks for the answer.

The sort answer is yes, please. Nevertheless, I'd like to know if from your experience there is any workaround that we could apply/try. I also understand the answer is "nope, we need to patch every of the public methods of the newer MongoDB client because it changed way too much since 2.4".

I guess this issue I'm describing will get more relevant with time, given that MongoDB server 3.4 end of life end on September 2019, so as soon as people using the combo MongoDB+Ruby+DataDog APM, they will find APM rendering useless on the MongoDB query observability part.

Thanks again!

[ricbartm]

We have been able to reproduce the issue under Unicorn web server, but not under a standalone Rails console.

So, here is what is happening:

• dd-tracer patches the Mongo driver by modifying Mongo::Client class
• This patch consists of pinning each cluster address with the dd tracer object:

            # attach the PIN to all cluster addresses. One of them is used
            # when executing a Command and it is attached to the Monitoring
            # Event instance.
            cluster.addresses.each { |x| pin.onto(x) }

https://github.com/DataDog/dd-trace-rb/blob/0.21-dev/lib/ddtrace/contrib/mongodb/instrumentation.rb#L63

• If the address class is not pinned when the event is received by the Datadog tracer, the event is ignored and the span is not created:

      class MongoCommandSubscriber
        def started(event)
          pin = Datadog::Pin.get_from(event.address)
          return unless pin && pin.enabled?

https://github.com/DataDog/dd-trace-rb/blob/0.19-dev/lib/ddtrace/contrib/mongodb/subscribers.rb#L9

• We identified the problem being MongoCommandSubscriber::started receiving an event where the address was not pinned, meaning Datadog::Pin.get_from(event.address) returned nil

• After further investigation, the root cause is:

  • The DD patching is only done during the initialisation of the unicorn master
  • When a Unicorn worker is created, the MongoDB connections are recreated (https://docs.mongodb.com/ruby-driver/master/tutorials/ruby-driver-create-client/#usage-with-forking-servers)
  • The new cluster contains Address instances that are not pinned and therefore events are not traced

• We did more tests by removing the after_fork code, and traces are working again

We need to find a way to re-patch MongoDB client during the after_fork. We'll start forking the tag 0.21.0 of the gem and implementing a solution that works for us. We have plans have that change reviewed internally with aims to be accepted upstream.

Kudos to @sgirones for the findings!!

[delner]

Excellent write-up; thanks a ton for this!

The explanation does make a lot of sense, and it's a downside of Pin driven tracing, which is something we usually avoid. Ultimately we'd like to get rid of pins in Mongo altogether, if it makes sense and can be managed.

In the interim, if you do end up creating a patch, I'd be very happy to look at it.

[delner]

A fix has been merged to master and should roll with 0.21.2.