Consider high water mark for sending messages

[mschubert]

From ropensci/drake#933 (comment):

The problem is that all of these transfers are being started concurrently. If Qsys$private$send() blocked execution until the transfer was complete, then the memory requirements would be limited to the dependencies for one target. However, the current situation is that Qsys$private$send() returns as soon as the transfer is scheduled (because rzmq::send.socket() returns as soon as the transfer is scheduled), and so drake schedules the next target, which means filling the buffer for the next transfer before the first has finished. The result is that dependencies for many targets (whether they are the same data or not) are buffered on the master at the same time.

This could be addressed using ZMQ_HWM.

Alternative is using blocked sending: ropensci/drake#933 (comment)

It looks like pbdZMQ uses blocking connections by default. This is also the behavior of the rzmq-compatibility wrapper function, so blocking will come along for the ride by default if clustermq switches to using pbdZMQ.

[wlandau]

Do you plan to expose the high water mark? Some workflows are not memory intensive and could still benefit from the existing non-blocking behavior.

[mschubert]

I wasn't planning on to, but will have to investigate how ZeroMQ treats this exactly with its IO threads.

Generally, I wouldn't expect this to be an issue because we're blocking on receiving anyway, and the workers will be saturated in either case.

If there's a problem with that approach that needs user/package intervention I will expose it, otherwise not.

[mschubert]

@brendanf This may be already fixed in the v0.9 branch, please test if you have the time