Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[xmlrpcpp] close socket when connection can't be accepted. Fixes #914 #960

Merged
merged 1 commit into from
Jan 17, 2017

Conversation

mgrrx
Copy link
Contributor

@mgrrx mgrrx commented Jan 13, 2017

This PR fixes #914. As suggested by @trainman419 and confirmed by @ruffsl, uncommenting line

did the trick.
I also changed the return value of handleEvent in that case.

@jspricke
Copy link
Member

Looks like this has been there forever (at least I didn't find an argument against it):
http://xmlrpcpp.cvs.sourceforge.net/viewvc/xmlrpcpp/xmlrpc%2B%2B/src/XmlRpcServer.cpp?revision=1.1&view=markup
So +1 for fixing :).

@mikepurvis
Copy link
Member

I wish we had a test or something to exercise this execution path. It seems really hinky to have a "fix" that's essentially just uncommenting something previously commented.

@mgrrx
Copy link
Contributor Author

mgrrx commented Jan 14, 2017

I carefully checked when this statement is executed. Basically, it's only called if ::accept returns -1. In my opinion, all the errors described in the manpage suggest to close the file descriptor

@mikepurvis
Copy link
Member

If you've confirmed in #914 that this eliminates a source of leaked sockets, then that's enough motivation for me. +1 to merge for now; we can keep an eye out for any regressions due to this in our testing at CPR.

@dirk-thomas
Copy link
Member

Sounds good to me.

@dirk-thomas dirk-thomas merged commit 10b6fc7 into ros:kinetic-devel Jan 17, 2017
@mikepurvis
Copy link
Member

We're seeing some assertion failure crashes now which may be due to this change. This one is a stack trace for one which occurred in the context of a costmap_2d unit test run:

  File "../nptl/sysdeps/unix/sysv/linux/raise.c", line 56, in __GI_raise
  File "abort.c", line 89, in __GI_abort
  File "../sysdeps/posix/libc_fatal.c", line 175, in __libc_message
  File "fortify_fail.c", line 38, in __GI___fortify_fail
  File "chk_fail.c", line 28, in __GI___chk_fail
  File "fdelt_chk.c", line 25, in __fdelt_chk
  File "../../../../../ros-comm-f6da1ec87d6ed4653e65ac2dbae1325f7bc1238d/utilities/xmlrpcpp/src/XmlRpcDispatch.cpp", line 100, in XmlRpc::XmlRpcDispatch::work
  File "../../../../../ros-comm-f6da1ec87d6ed4653e65ac2dbae1325f7bc1238d/utilities/xmlrpcpp/src/XmlRpcServer.cpp", line 133, in XmlRpc::XmlRpcServer::work
  File "../../../../../ros-comm-f6da1ec87d6ed4653e65ac2dbae1325f7bc1238d/clients/roscpp/src/libros/xmlrpc_manager.cpp", line 272, in ros::XMLRPCManager::serverThreadFunc
  File "/usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0", in ?? ()
  File "pthread_create.c", line 312, in start_thread
  File "../sysdeps/unix/sysv/linux/x86_64/clone.S", line 111, in clone

@jasonimercer @deng02

@mikepurvis
Copy link
Member

@dirk-thomas @mgrrx We haven't been able to bottom this out, but it's impacting our ability to test everything else. IMO it should be reverted until it can be proven safe.

mikepurvis added a commit to clearpathrobotics/ros_comm that referenced this pull request Feb 7, 2017
mikepurvis added a commit that referenced this pull request Feb 7, 2017
mikepurvis added a commit that referenced this pull request Feb 7, 2017
dirk-thomas added a commit that referenced this pull request Feb 7, 2017
@dirk-thomas
Copy link
Member

I completely agree with your conclusion.

rsinnet pushed a commit to MisoRobotics/ros_comm that referenced this pull request Jun 19, 2017
[xmlrpcpp] close socket when connection can't be accepted. Fixes ros#914
rsinnet pushed a commit to MisoRobotics/ros_comm that referenced this pull request Jun 19, 2017
trainman419 added a commit to trainman419/ros_comm that referenced this pull request Nov 22, 2017
Add integration tests for the XmlRpcServer interacting with an
XmlRpcClient in the same process, particularly around the behavior when
the server process runs out of available file handles.

Update the XmlRpcServer with two different mitigations for file handle
exhaustion (Fixes ros#914, replaces ros#960 and ros#977):
 * Measure the number of free file handles and reject incoming
connections if the pool of free file descriptors is too small. This
actively rejects incoming clients instead of waiting for complete file
handle exhaustion, which would leave clients in a pending state until a
file descriptor becomes free.
 * If accept fails due to complete file handle exhaustion, temporarily
stop calling accept on this socket. This prevents a busy-loop where
poll() believes the listening socket is readable, but accept() fails to
allocate a file descriptor and leaves the socket in a readable state.
trainman419 added a commit to trainman419/ros_comm that referenced this pull request Dec 14, 2017
Add integration tests for the XmlRpcServer interacting with an
XmlRpcClient in the same process, particularly around the behavior when
the server process runs out of available file handles.

Update the XmlRpcServer with two different mitigations for file handle
exhaustion (Fixes ros#914, replaces ros#960 and ros#977):
 * Measure the number of free file handles and reject incoming
connections if the pool of free file descriptors is too small. This
actively rejects incoming clients instead of waiting for complete file
handle exhaustion, which would leave clients in a pending state until a
file descriptor becomes free.
 * If accept fails due to complete file handle exhaustion, temporarily
stop calling accept on this socket. This prevents a busy-loop where
poll() believes the listening socket is readable, but accept() fails to
allocate a file descriptor and leaves the socket in a readable state.
dirk-thomas pushed a commit that referenced this pull request Dec 19, 2017
* Tests and bug fixes for XmlRpcServer

Add integration tests for the XmlRpcServer interacting with an
XmlRpcClient in the same process, particularly around the behavior when
the server process runs out of available file handles.

Update the XmlRpcServer with two different mitigations for file handle
exhaustion (Fixes #914, replaces #960 and #977):
 * Measure the number of free file handles and reject incoming
connections if the pool of free file descriptors is too small. This
actively rejects incoming clients instead of waiting for complete file
handle exhaustion, which would leave clients in a pending state until a
file descriptor becomes free.
 * If accept fails due to complete file handle exhaustion, temporarily
stop calling accept on this socket. This prevents a busy-loop where
poll() believes the listening socket is readable, but accept() fails to
allocate a file descriptor and leaves the socket in a readable state.

* Add test depend on boost

* Run astyle
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants