How to set SO_REUSEPORT

[ChristianRobl3D]

I've encountered another issue in our server project: We need to set SO_REUSEPORT.
We already use endpoint:set_reuse_addr(true) with our TCP server (SOCK_STREAM), which partially helps. To solve the last issues we need to set SO_REUSEPORT.

The project uses the latest websocketpp 0.7.0 on Mac OS X.

As suggested I already uses the socket_init_handler ([http://stackoverflow.com/questions/23023317/proper-set-socket-init-handler-syntax-or-modify-source-to-turn-on-tcp-nodelay-wi/23031124#23031124] and [https://groups.google.com/forum/#!msg/websocketpp/rvBcIJ940Bc/zxpZf9AOb0IJ]).
I've also set the tcp_pre_init_handler.

The modified websocketpp-provided example "echo_server.cpp" shows the issue quite nicely; I've added the following lines:
`
void on_socket_init(websocketpp::connection_hdl hdl, boost::asio::ip::tcp::socket & s)
{
std::cerr << "on_socket_init...\n";
int reuse = 1;
int ihdl = s.lowest_layer().native_handle();

if (setsockopt(ihdl, SOL_SOCKET, SO_REUSEPORT, (const char*)&reuse, sizeof(reuse)) < 0)
perror(“setsockopt(SO_REUSEADDR) failed");
}

...
echo_server.set_socket_init_handler(bind(&on_socket_init, ::_1, ::_2));
//echo_server.set_tcp_pre_init_handler(bind(&on_tcp_init, ::_1));
...
std::cerr << "pre listen...\n";
echo_server.listen(9002);
std::cerr << "post listen...\n";
...
`

Starting a second instance of the server, it fails with
[info] asio listen error: system:48 (Address already in use) Underlying Transport Error
Unfortunately the handlers are called when the first client connects and therefore after endpoint.listen(), which actually creates the socket, sets SO_REUSEADDR and bind()s the socket.
Trying to set SO_REUSEPORT in the handler has no effect.

The output before a client has connected:
pre listen... post listen...
As soon as a client connects:
on_socket_init...

When adding tcp_pre_init_handler as well, I can observe it being called even after on_socket_init.

By modifying (as a test) the boost::asio code to also set SO_REUSEPORT for TCP/non-datagram oriented sockets, I can see that starting two (or more) instances of the server then is possible.

(How) is it possible to set SO_REUSEPORT before the bind() takes place?

If this is currently not possible, can you please add a callback / another handler (preferably already providing the socket like the init_socket_handler), which gets called in endpoint.listen() after
m_acceptor->open(ep.protocol(),bec);
and before
m_acceptor->bind(ep,bec);

Many thanks in advance!

[zaphoyd]

I pushed a change to develop branch that moves the socket_init_handler hook from the pre_init phase (which is after bind/listen/accept) to the init_asio phase, which is before. Can you test if this helps?

[ChristianRobl3D]

I just tested it:
The handler is now called early enough, which should do the job, but setting the option still fails.
Even worse, setting the no_delay option as shown in some comments and examples now also fails.

Printing the native handle's value s.lowest_layer().native_handle() in the handler shows "-1". Thus, setting the SO_REUSEADDR in on_socket_init() fails.

For a test I also set an asio option to see the values of the error code:
boost::system::error_code ec;
boost::asio::ip::tcp::no_delay option(true);
s.lowest_layer().set_option(option, ec);
if (ec)
std::cerr << "Error: " << ec.value() << " " << ec.message().c_str() << "\n";

ec result is Error: 9 Bad file descriptor

So IMHO the reason is the socket, that is not initialised at that time.

[ChristianRobl3D]

FYI
I've now modified a local copy of websocket++ 0.7.0, which enables me to set SO_REUSEPORT just at the right time.
I needed to modify transport/asio/endpoint.hpp:
void listen(lib::asio::ip::tcp::endpoint const & ep, lib::error_code & ec):
{
...
m_acceptor->open(ep.protocol(),bec);
if (!bec) {
m_acceptor->set_option(lib::asio::socket_base::reuse_address(m_reuse_addr),bec);
}
// ------------
{
int reuse = 1;
if (::setsockopt(m_acceptor->native_handle(), SOL_SOCKET, SO_REUSEPORT, (const char*)&reuse, sizeof(reuse)) < 0)
std::cerr << "setsockopt(SO_REUSEPORT) failed!!!, handle=" << m_acceptor->native_handle() << "\n";
}
// ------------
if (!bec) {
m_acceptor->bind(ep,bec);
}
...

Its the only location I found that:
a) the socket is present
b) bind(9 has not been called.

Can you change to have a signal handler called in the spot, where I added - admittedly not very nice - the setsockop() call?
This way we could add SO_REUSEPORT, etc. in a much cleaner way.

Thank you in advance!

[vadz]

Please notice that the commit 6f128a5 broke my code using set_socket_init_handler() in the client: I just do socket.lowest_layer().set_option(boost::asio::ip::tcp::no_delay(true)) in it (my socket is a TLS one, hence lowest_layer()) and it now fails because the socket is invalid. I simply reverted it locally for now, but hopefully a better solution can be found for 0.8.0.

[vincedupuis]

@vadz Did you manage to set the no_delay option?

[vadz]

I didn't return to this code since then, so I only "managed" to do it by reverting the commit that broke it, as written above.

[vans163]

Why is this still unfixed? I have same problem in latest master. NO_DELAY is pretty much the point of websockets in the first place, otherwise just use http polling.

MatusKysel patch fixes it for me BTW: SophiaTX@c7783e2

Cleanest solution is probably to have a pre_init and post_init handler and also a way to determine if the passed socket is the actual listen_socket or not, since some options you wanna set on the listen_socket others on the accepted sockets.

[zaphoyd]

This regression introduced in 0.8.0 that broke setting socket options on already accepted connections has been fixed in the develop branch. For clarification, the socket_init handler should present a valid, accepted, socket fd that options like TCP_NODELAY can be set on. To set socket options on the listening socket use the tcp_pre_bind handler, which runs during the call to listen, after the acceptor has initialized the listen socket but before bind and listen have been called. Socket options like SO_REUSEPORT or IPV6_ONLY should be set here.