ros / rosbridge freezes with multiple clients

[ppm-tahi]

Hi,

Maybe we are triing to use ROS + ROSBridge on a wrong way, or it has some problems in the request queues, but our problem is the following:

• we are using rosbridge.js from FlexGui (ROS based flexible, platform independent GUI), it is here in the ros git
• we are using ROS topics, subscribing to them and it is also possible to publish values or advertise topics with the clients, but ros / rosbridge fails with using it with more clients paralell, even if we are only subscribing and waiting for the topic updates
• if we are starting 8-10 FlexGui really fast (~1 sec after each other), the CPU usage of the ROS server (Ubuntu 14.04 LTS + ROS indigo / jade installed from the binary, 4 cores, 2-4gb memory) goes up to 100% per core and stops. We have to wait 5-10 minutes (it depends, sometimes it never ...) to go back to normal. This 8-10 clients are trying to subscribe to 3-4 topics (the topic are published by another client which is connected with rosbridge, but written in cpp), we can see in the roslog that they are trying to subscribe, but they never gets the topic updates (published values).
• when the number of the clients in ros are increasing really fast, it is much more noticable
• it is working, if we start the clients with 4-5sec delay, so when the prev. client subscribed to all of the variables

Sometimes we gets errors in roslog, that a rosapi service (which should be there by default) is not existing (rosapi/topics, rosapi/bublishers)

Could you help us solve this problem? I am not sure which side is the faulty one, so maybe some rosbridge settings could solve the problem.

My personal idea is that we make the request queue absolutely full which uses all of the server's memory / cpu and after a time it collapses, but ofc. I am not sure, this is just an idea to start brainstorming...

Thanks in advance!
Balint Tahi
PPM AS

[ppm-tahi]

Maybe this error message will give some hint:

{"function":"log","name":"/rosbridge_websocket","level":8,"topics":["/FlexGuiPc/Variables/Integer10","/FlexGuiPc/Variables/Integer6","/FlexGuiPc/Variables/Integer4","/FlexGuiPc/Variables/Integer5","/FlexGuiPc/Variables/Integer2","/FlexGuiPc/Variables/Integer3","/FlexGuiPc/Variables/Integer7","/FlexGuiPc/Variables/Integer1","/rosout","/FlexGuiPc/Variables/Integer12","/FlexGuiPc/Variables/Integer8","/FlexGuiPc/Variables/Integer9","/FlexGuiPc/Variables/Integer13","/rosout_agg","/FlexGuiPc/Variables/Output2","/FlexGuiPc/Variables/Integer14","/FlexGuiPc/Variables/Output1","/FlexGuiPc/Variables/Integer11","/FlexGuiPc/Variables/Integer15"],"header":{"stamp":{"secs":1478507508,"nsecs":122277975},"frame_id":"","seq":2009},"file":"protocol.py","msg":"[Client 78] [id: call_service:rosapi/publishers:95] call_service ROSServiceIOException: Unable to communicate with service [/rosapi/publishers], address [rosrpc://ubuntu-Virtual-Machine:39835]","line":363}

Sometimes we get the same issue, just with /rosapi/topics.

[ppm-tahi]

I wrote a small tester, if you need, please let me know!
It publishes 20 topics and updating the values with a selected rate. Then it starts 5-10-20 etc (as many as you want) subscribers, which subscribes all of the topics and calls rosapi/publishers for each topic in every 10 sec. Please see the result above. This is what is happening after a few sec.

[T045T]

Can you be a little more precise in describing the error? For example, it would be super useful to know whether the CPU usage comes from the rosbridge node, the roscore, the rosapi node, or something else entirely.
If your test script can reliably trigger the error, that would be great to have, too :)

[ppm-tahi]

https://www.dropbox.com/s/hswgtlf3uztwyus/rostest.zip?dl=1

With this simple website you can test and I hope you will get the same result as us, just use the magic numbers setted up by default. You should wait a couple of sec before the red error comes.

publisher.html: you can set the parameters of the test (topic number, subscriber number, subscriber open delay, publish value delay)
subscriber.html: this will subscribe to all topics in ROS (rostopic list, then rostopic info, then subscribe) and checks the publishers for each topic in every 10s.

If I order the processes in System Monitor (Ubuntu 14.04) by CPU or Memory descending, I can see rosbridge on top. With this tester I can not achieve 100% CPU useage with 4 cores and 2gb of ram, but the error is still there like on the attached image.

If we want to connect many robots, PLCs, devices to the same ROS server in a factory, it wont be so hard to reach 30 topics and 10 subscribers, that is why I choose this default values.

[T045T]

Sorry, I wasn't able to reproduce your problem using that script yet.
It does sound like there might be a race condition somewhere, so I'll keep trying, but I might not get as lucky as you did in reproducing the bug.

[aishwaryap]

I seem to have the same issue. Has there been any further work on this?

[prad09]

facing the same issue, trying to connect with multiple clients from roslibjs, after 8 or higher number of clients the ros bridge server halts. :/

[flexgui]

I still have the same issue. Now, I am able to kill it with only 1 client. I am getting this error: [ERROR] [1550696982.668135]: [Client 0] [id: call_service:/rosapi/topic_type:1278] call_service ROSServiceIOException: Unable to communicate with service [/rosapi/topic_type], address [rosrpc://asus-ubuntu:39955]

I have only 1 client through rosbridge + more python, with rospy. Sometimes this error just pops up, 3-4 in a row, then it is working again.

[ghost]

Following up on this..
Anyone found a solution for multi machine connectivity issue?

[MoffKalast]

Can confirm this is still the case, rosbridge seems to break down in tears when more than one client connects.

[mvollrath]

Until #464 is merged one client can block messages to all other clients.

[MoffKalast]

It's been merged and I've tested it out. Seems to solve all multiclient issues for my setup. Thanks for the help :D

[nickvaras]

Would the issue in question here explain why I can reproduce at will this following misbehavior?

[github-actions]

This issue has been marked as stale because it has been open for 180 days with no activity. Please remove the stale label or add a comment to keep it open.