Problem: threadsafe socket types don't support SNDMORE / multi-frame messages

[minrk]

Probably the most valuable feature in libzmq for me is the multi-frame message format, where chunks of a message can be serialized separately, without concatenation, but the message is still delivered as a unit.

The new threadsafe sockets have abandoned support for SNDMORE, presumably in the interest of threadsafety. However, as far as I am concerned, this means all the new sockets may as well not exist since they can't support messages in this format.

Is there any hope of non-contiguous messages in the threadsafe sockets? Ideally via the same SNDMORE mechanism as all the already working sockets?

If there are no plans for multi-frame message support, the docs should probably be updated to remove the note about RADIO-DISH replacing PUB-SUB and SERVER-CLIENT replacing ROUTER-DEALER, since they can't really aspire to do that without multiframe message support.

[bluca]

Unfortunately I don't think it can be done with thread safe sockets, simply because there's no way to guarantee any ordering - if 2 threads send 2 frames each, there's no telling in which order they'll get to the I/O thread with the current implementation :-(

[bluca]

@somdoron what do you think?

[sigiesec]

I can understand that this is not possible to implement. However, I would like to second that then, ROUTER/DEALER cannot be deprecated/removed. For our use, multi-frame messages are essential as well.

[somdoron]

I don't think we can implement SNDMORE with thread safe sockets. some scenario

1. While I write multipart message can I receive in another socket?
2. While I write multipart message can another socket send?
3. What if it take me a long time sending the last time, does all the rest block?

I'm not a big fan of multipart and I think we can live without them, however I understand some people like them. I think the thread safe socket alternative is important and we do see people use them.

We can drop the plan to deprecate or remove not thread safe sockets and fix the docs.

I did have a plan for multipart like support for thread safe socket through a binary message, like czmq bsend and zproto protocol. I even started implement that a year ago, I can try and complete it.

[minrk]

I'd trade threadsafe for SNDMORE on the new sockets any day, but I totally understand that it solves a problem for some. I'm fine if all that's done is removing the deprecation comment in the docs.

I could imagine, for instance, a probably janky solution where send(SNDMORE) acquired a socket-specific message-assembly lock and the final send(0) released it (don't ask me how to keep track of the lock!), preserving the atomicity of a multipart message.

I did have a plan for multipart like support for thread safe socket through a binary message

That would be cool! As long as:

1. it supports a sequence of non-contiguous buffers (and doesn't concat a copy under the covers)
2. the same multipart send API works for all socket types

[guidog]

I'd like to support the point minrk made.
Multiple frames are an essential feature of ZMQ.
I can't see any reason to drop it.

[bluca]

For sending perhaps using thread-local storage for the pipe would help - so each thread transparently has its own sending queue.
But it won't work for receiving.

Anyway I think there's been enough feedback, and I too think it's reasonable to remove the deprecation notice and keep both types. I'll send a PR for that soon.

[bluca]

#2700

[somdoron]

One of the reasons I don't like multipart is that it is making adding transport more complicated.
For example, adding UDP support for radio/dish was very easy because I didn't support multipart.

Also implement ZMTP that only support single part is easier and adding other protocols like websocket will be easy.

With that, it seems multipart is an important part of zeromq, so we should think of away to add it to the thread safe sockets.

@minrk with binary message we do loose the 'zero copy' because we serialize the message before actually enqueue it to the IO thread.

We can have send/recv methods that accept an array of msg. For the recv method, if the size of the array is too small we can return an error message instead of fetching the messages.

Locking the socket until last message is being sent/receiving cannot really work. One of the patterns that we might want to use with thread safe socket is one thread sending and another one receiving, for that if the sending is taking time (for example, serializing something) we will block the receiving thread.

[minrk]

One of the patterns that we might want to use with thread safe socket is one thread sending and another one receiving, for that if the sending is taking time (for example, serializing something) we will block the receiving thread.

In my mental sketch, this would be a message-assembly send-only lock that wouldn't block recv, so it only interacts with any global state when !SNDMORE. I won't push it, since you know a lot more about these internals than I do.

We can have send/recv methods that accept an array of msg.

That would be great. As a binding maintainer and downstream zmq consumer, I would prefer the same SNDMORE APIs to continue to work for all sockets, but if libzmq gets its own send|recv_multipart function, I can make that work.

[chrisjbillington]

Have there been any developments on this front?

I currently have some complexity in my applications in order to shutdown sockets in other threads - sometimes by terminating contexts, sometimes with the 'self-pipe' trick. This all works, but I've been debugging corner-cases in it, and it is a source of occasional bugs that would be absent if I were using the new thread-safe sockets.

Not having multipart messages is something of a show-stopper for using the new sockets for me, but, compatible with the proposed solutions upthread, I never need to spread out send() or recv() calls over time, I would be happy to always send and receive multipart messages all at once. It's solely about not having to concatenate and divide up the data in application code at each end.

So my code works presently, but if the new sockets move to support multipart, I will switch to using them so that I can simplify my cleanup code in light of thread-safety.

[TYoungSL]

I would be happy to always send and receive multipart messages all at once.

According to the documentation,

The intention is to extend the API to allow scatter/gather of multi-part data.

A scatter/gather version of the send/recv functions that took an array of frames would supplement the existing send/recv single-frame functionality and handle all of the threading concerns.

The existing send/recv functions could be extended to support a new msg_t type that had a 'gather' message that was composed of the other messages. ZMQ_SERVER/ZMQ_PEER sockets would still not support 'more' by flags. The data pointer operations, etc. would all need to be revised to support it (or raise an error that it is not supported) and an additional function to 'unpack' it for the user expressed as a pointer array with count.

e.g.

        struct
        {
            metadata_t *metadata;
            msg_t **msgs;
            uint16_t count;
            uint16_t index;
            msg_free_fn *ffn;
            void *hint;
            uint16_t count;
            uint16_t index;
            unsigned char unused[msg_t_size
                                 - (sizeof (metadata_t *) + sizeof (msg_t **)
                                    + sizeof (uint16_t) + sizeof (uint16_t)
                                    + sizeof (msg_free_fn *) + sizeof (void *)
                                    + 2 + sizeof (uint32_t) + sizeof (group_t))];
            unsigned char type;
            unsigned char flags;
            uint32_t routing_id;
            group_t group;
        } gather;

edit: gotta fit index, count, free and hint inside there; index used only by encoders

[TYoungSL]

Made a tentative implementation of this; no changes to ZMTP or anything, still using MORE/SNDMORE to transmit and receive, but fq's recvpipe wasn't going to cut it, so created an fq_gather class that uses mutexes per pipe and on the manipulation of the collection to ensure pipes with MORE are delivered in isolation per receiving thread (so no changes for the receive side other than checking for MORE on received messages).

On the sender side, it required creating msg::is_gather(), msg::init_gather(msg, msgs, count, freefn, hint), some other helpers, and changing the v3_1, v2, v1 encoders to iterate through the gather message's sub-messages, and telling next_step not to release the message by it's last bool param until the gather message's contents were exhausted.