PUB - SUB sockets randomly stop working?

[vreascul]

Hello,

We are using ZMQ for our server (reactphp/zmq) and our client (zeromq/jeromq). We run multiple instances of the server and each server has several clients connecting to it. We use several other sockets but we are facing an issue with the TCP communication between the PUB socket on the server and the SUB ones on the clients.
The issue is very odd as it is not easy to replicate but from time to time the client stops receiving messages from the server. This happens on all of our server instances and their clients once every 2-3 months but we have a new server instance that replicates this issue in around 20 - 30 minutes. Our servers are "identical" and this points us to a network difference.

We made a simple app that just opens several threads with SUB sockets and listen on the same topic. They all fail eventually but we observed that not all of them fail at the same time. Also, restarting a client solves the issue so we think that the issue is most likely with the client.
The server logs indicate that the send is called on the PUB socket and the listening thread is still running on the client calling recvStr on the SUB socket once every several seconds. This points us to a ZMQ issue but we don't know it for sure.

In any case, right now we are looking into the jeromq implementation but we are wondering if anyone else experienced something like this and maybe has some pointers.

Thank you.

[galves]

Are you using TCP and on what OS? Note that socket.setTCPKeepAliveInterval & setTCPKeepAlive don't work on JeroMQ so do not rely on them to reconnect the socket after network loss.

You might need to enable tcp connection polling on your client/server and see if that helps. e.g.

echo 600 > /proc/sys/net/ipv4/tcp_keepalive_time

echo 60 > /proc/sys/net/ipv4/tcp_keepalive_intvl

echo 20 > /proc/sys/net/ipv4/tcp_keepalive_probes

[andreaslarssen]

I work with @vreascul on this project. For now we run it on Windows (7, 8 and 10), as well as a Raspbian. Should run on Ubuntu and OSX in the future as well.

Thanks for the tip, @galves, we'll look into it and come back with the findings.

[kbh4r4th]

@vreascul, @andreaslarssen We also face the same issue in our production, when we migrated to jeromq. Did you guys get any idea/finding to solve the issue?

@galves We are yet to check & enable TCP connection polling.

[andreaslarssen]

Hi @b6arat6. We're experimenting with tcp config settings, but it hasn't solved anything yet. If that indeed solves the issue, it's still kinda troubling for us as we have to config tcp connections in install scripts (we're building an install-able client).

I tried at an earlier stage to heartbeat from server to client and reconnect on missed heartbeats, but that too failed (that off course might have been my implementation that failed as it seems like a straight forward way to achieve a steady connection). After that, @vreascul took over, and he has tried to listen for disconnect events, but with no luck.

Next step might be to re-try the server->client heartbeating or use zmq java bindings instead of jeromq, even though we have to download and install zmq with our client installer.

Did you guys just get aware of the issue, or have you puzzled with this for a while? I'm really interested in what you've tried to solve it so we don't have to go down that same road. And also how this issue manifests itself. For us, one server -> clients connection works (almost) like a charm, while the other (identical setup) fails every X minutes / hours. When I say identical, I'm not counting the router / firewall between the server -> clients, as well as the clients network. While I don't think there are any "jeromq hating" configurations in the firewall or network, an unstable network for example might trigger the issue at one place, not the other.

Atm. @vreascul is working on this and digging into the jeromq source, so he might have some details if you have any specific questions.

[andreaslarssen]

Also, there must be someone out there running the PUB/SUB pattern with success? Someone who can give an example of an implementation that works? How are people handling sub socket disconnects if jeromq is not?

[colini]

We've been running jeromq 0.3.4 and 0.3.5 PUB/SUB for several years on Ubuntu and it has worked fine. SUB sockets connect to the PUB. We do send a heartbeat every second if no data has been published. No TCP kernel tweaks.

Maybe try a packet sniffer to see what's happening on the wire?

[andreaslarssen]

No tcp disconnects on the sub side? How do you handle hearbeats on the sub side?

[colini]

No disconnects that I'm aware of. The SUB clients don't do anything with the heartbeat message. It may not even be necessary.

[andreaslarssen]

Does that mean the heartbeat is necessary to keep the pub socket open? If you're not using it do discover disconnects I mean?

[andreaslarssen]

@vreascul tried setting both the keeptime and the interval for tcp sockets (in windows), but it has no effect on the issue. Next step is trying the pub heartbeat again.

[colini]

TBH, it was the first zeromq application I had written and I don't remember why I put the hb there. Looking at the code, the SUB client does not use it to detect disconnects or reconnect, etc. It just receives the hb and ignores it.

My understanding is that connect/reconnect is one of zeromq's main features -- it's supposed to take care of that for you.

If you're still stuck I recommend sniffing the traffic on the network. This is a good article that explains how to do that with wireshark: http://www.codeproject.com/Articles/863889/ZeroMQ-Diving-Into-The-Wire

[andreaslarssen]

We've used wireshark at an earlier point to confirm that the traffic is actually sent, and it is.

I was also under the impression that connect/reconnect was one of the zmq features, but it seems like that's not the case:

• http://stackoverflow.com/questions/26112992/zeromq-pub-sub-on-unreliable-connection.
• (PUB/SUB) Sub Silent Disconnect on Unreliable Connection libzmq#1199

They also suggest the heartbeating, so that's what we'll try (again) next,

[kbh4r4th]

@andreaslarssen @vreascul Our system has n servers (n-PUB), n clients (n-SUB). n-SUB have subscribed to n-PUB. The system had been working without any issues related to message loss until we migrated to jeromq-0.3.2 (from maven repo) from zmq.jar (directly, not from maven repo) along with native libs, just a week ago.

Of n-SUB, 1-SUB stops receiving messages from 1-PUB after sometime (did not note the interval), whereas (n-1)-SUB continue to receive messages from the 1-PUB.

We already have heart beat mechanism in place - Each PUB sends a heartbeat at every 3 mins to n-SUB. If a SUB fails to receive any heartbeats in last 10 mins from any PUB, [ It closes the socket with org.zeromq.ZMQ.Socket.close(), re-connects (i.e. connects to n-PUB all over again with org.zeromq.ZMQ.Socket.connect(String addr)) ] - Action1.

When the issue happens, neither message nor heartbeat is received at the SUB, though Action1 is taken.

The issue was solved, after 1. restarting the SUB, 2. restarting the PUB. But, the issue was back again, the very next day.

We are yet to enable TCP connection polling.

Our VMs has,
tcp_keepalive_time = 7200
tcp_keepalive_intvl = 75
Tcp_keepalive_probes = 9

@galves Will 0mq at SUB side identify the connection broken state based on tcp config & reconnect itself? Is it what called as tcp config in Linux helping/aiding 0mq to recover in such instances?

Wireshark - We are yet to try.

I will keep posting our progress.

Could you let us know the version of jeromq that you have been using?

[andreaslarssen]

@b6arat6:

We're using 0.3.5 via Maven.
About the connection broken state, I would have to ask @vreascul.

[vreascul]

Hello,

We are monitoring the SUB socket (on all events) and we can't see any event even if we disconnect the network. I think that this might by design as PUB and SUB sockets seem to be made for really fast one way communication and probably they had to drop any extra chattering for the performance. We did not use Wireshark so far and i have a limited understanding of the underlying socket (java.nio.channels.SocketChannel) that jeromq is using for SUB so i can not tell for sure that this is the case but it is my best guess.

For us, manually restarting the client (the SUB socket) solves the issue so we just need to find out when the issue happens and restart the SUB socket. For this we are looking once again into sending pings and , hopefully, finding a way to restart the SUB socket successfully (we tried this approach in the past and it did not work but i'm not sure what happened as i was not involved in this project at that time).

As for using OS settings for the sockets keepalive, we tried this in windows with no success. Not sure how these settings are supposed to interact with the java.nio.channels.SocketChannel but the fact that it did not fix the issue points us in these directions:

1. the issue is not with the socket (since we can not see the socket failing we can not really rule this out but it is very unlikely)
2. The SocketChannel is not really using the "thing" that the windows settings are for.
3. I messed up setting those register values up (https://technet.microsoft.com/en-us/library/cc957549.aspx and https://technet.microsoft.com/en-us/library/cc957548.aspx)

Not sure if this answers you question about the broken state but if you need something more specific please let me know.

Regards!