Add Sender.queue_size limits

[djmitche]

This adds a queue_size limit to the Sender class, with messages beyond that limit dropped. We support Ruby-2.1.0, which doesn't have a SizedQueue that supports "attempted" sends (it always blocks the sender if the queue is full), so we need to limit the queue size out-of-band. Also, we want to limit the size based on bytes, not message count. Because the @message_queue_bytesize is accessed concurrently in a read-modify-write pattern, it requires a mutex.

Things I'd like feedback on:

• Ideas on other ways to do this that don't involve two mutex syncs on each message
• The tests - is there something similar to Waiter already defined? Or another more rspec-y way to test this kind of concurrency-related behavior?
• Logging on drops - this will log every drop, which is probably exactly the wrong thing to do. Other options:
  • Log only the first drop
  • Log edges (not dropping -> dropping / dropping -> not dropping)
  • Log the first and then every N messages dropped

This isn't quite done yet. Still to do:

• Thread queue_size through Statsd constructor
• More tests, at least covering resuming sending
• Docs
• Changelog

[remeh]

Small feedback @djmitche

• I don't think Ruby is supporting any sort of atomic integer whatsoever so I would probably have done the same...
• Can't help here sorry
• I think we want to use the internal telemetry of the client to report the amount of drops. However about the logs, that's a good question, we should check what's the behavior in our other implementations (i.e. datadog-go).

You have been with the byte size and not the messages count, I would probably have done the same, but it may make sense to validate it has been done the same way in other implementations and to discuss about it with the team otherwise.

[djmitche]

datadog-go doesn't log anything, but does count it in datadog.dogstatsd.client.packets_dropped_queue. I'll do something similar in this client.

[djmitche]

This is based on the Go client, which uses WithSenderQueueSize:

WithSenderQueueSize sets the size of the sender queue in number of buffers.

After data has been serialized in a buffer they're pushed to a queue that the sender will consume and then each one ot the agent.

The default value 0 will set the option to the optimal size for the transport protocol used: 2048 for UDP and named pipe and 512 for UDS.

So that should be easy to emulate with the ruby version.

[djmitche]

Also, somehow I convinced myself that Ruby doesn't have a GIL. It does. So access to an integer is automatically serialized by that GIL.

[djmitche]

I've asked for some advice on how best to test this. I don't think Waiter is a good idea.

[djmitche]

Ivo has suggested using dependency-injection to inject a fake Queue implementation here, mocking out the background thread's behavior. I had tried mocking Thread, without success, so I'll try Queue instead.

[djmitche]

Supporting those ancient Ruby versions is a real drag.

[djmitche]

Hm, it seems webrick might have dropped support for ruby-2.0??

[djmitche]

I think the Ruby-2.0 failures are unrelated to this, so maybe we can handle that with #229 ?

[djmitche]

Merged from main -- looks like I changed all the same places as @abicky :)

[remeh]

Nice!