How to use jsonrpsee as service in dual stack to support both ipv4/ipv6

[niklasad1]

Howdy all, akin to https://github.com/tokio-rs/axum/blob/main/examples/listen-multiple-addrs/src/main.rs, would it be possible to have the jsonrpsee server accept connections on either a V6 socket or a V4 socket? I wasn't sure if there were issues with how Hyper/Tokio gets wrapped? Otherwise, would it be possible to do a v4/v6 fallback?

Solution:

async fn run_server() -> ServerHandle {
	let localhost_v4 = SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 9944);
	let listener_v4 = TcpListener::bind(&localhost_v4).await.unwrap();

	let localhost_v6 = SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 9944);
	let listener_v6 = TcpListener::bind(&localhost_v6).await.unwrap();

	let rpc_api = Methods::new();
	let (stop_handle, server_handle) = stop_channel();

	let svc = jsonrpsee::server::Server::builder()
		.set_http_middleware(tower::ServiceBuilder::new().layer(CorsLayer::permissive()))
		.to_service_builder()
		.build(rpc_api, stop_handle.clone());

	loop {
		let (stream, _remote_addr) = tokio::select! {
			result = listener_v4.accept() => {
				result.unwrap()
			}
			result = listener_v6.accept() => {
				result.unwrap()
			}
			_ = server_handle.clone().shutdown() => {
				break;
			}
		};

		let svc = svc.clone();

		tokio::spawn(async move {
                        // you may want to enable graceful shutdown here
                        // https://docs.rs/hyper/0.14.28/hyper/server/conn/struct.Connection.html#method.graceful_shutdown
			if let Err(e) = hyper::server::conn::Http::new().serve_connection(stream, svc).with_upgrades().await {
				eprintln!("Error while serving HTTP connection: {e}");
			}
		});
	}

	server_handle
}

[zeeshanlakhani]

@niklasad1 to ask one more Q on this. I've implemented:

        let runtime_hdl = Handle::current();
        let listener_v4 = TcpListener::bind(&self.v4_addr).await?;
        let listener_v6 = TcpListener::bind(&self.v6_addr).await?;
        let (stop_hdl, server_hdl) = stop_channel();

        let svc = jsonrpsee::server::Server::builder()
            .custom_tokio_runtime(runtime_hdl.clone())
            .set_http_middleware(middleware)
            .set_id_provider(Box::new(RandomStringIdProvider::new(16)))
            .set_message_buffer_capacity(self.capacity as u32)
            .to_service_builder()
            .build(module.into_inner(), stop_hdl.clone());

        let server_hdl2 = server_hdl.clone();
        runtime_hdl.clone().spawn(async move {
            loop {
                let stream = select! {
                    result = listener_v4.accept() => {
                        if let Ok((stream, _remote_addr)) = result {
                            stream
                        } else {
                            continue
                        }
                    }
                    result = listener_v6.accept() => {
                        if let Ok((stream, _remote_addr)) = result {
                            stream
                        } else {
                            continue
                        }
                    }
                    _ = server_hdl2.clone().stopped() => {
                        break;
                    }
                };

                let svc = svc.clone();
                runtime_hdl.spawn(async move {
                    if let Err(err) = hyper::server::conn::Http::new()
                        .serve_connection(stream, svc)
                        .with_upgrades()
                        .await
                    {
                        error!("error while serving HTTP connection: {err}");
                    }
                });
            }
        });

        Ok(server_hdl)

While connections to both sockets work, when previously using jsonrpsee::server's start method, I was able to take the ServerHandle and use it as part of our own shutdown method (elsewhere in the codebase), calling:

  let _ = ws_hdl.stop();
  ws_hdl.stopped().await;

But, now this won't work. I attempted returning both the server_handle and stop_handle and calling

  let _ = ws_hdl.stop();
  ws_stop_hdl.shutdown().await;

But, this still doesn't seem to control the shutdown correctly. Any ideas here that I'm just missing? There are alternatives like listening on a specified channel for the shutdown, but previously it worked in a built-in way, essentially. Thoughts?

Thanks.

[niklasad1]

Sorry my bad, the server main loop should use StopHandle::shutdown

and then you should be able to do:

server_hdl.stop();
server_handle.stopped().await;

EDIT: updated my example

[zeeshanlakhani]

Thanks @niklasad1. This makes sense. In your updated example, you mean:

_ = stop_handle.clone().shutdown()

vs _ = server_handle.clone().shutdown() right? :)

[niklasad1]

yepp correct but you may want to enable https://docs.rs/hyper/0.14.28/hyper/server/conn/struct.Connection.html#method.graceful_shutdown on your server.

When you are happy and it works it would be nice if you could make PR with this example :)

[zeeshanlakhani]

Will do @niklasad1. As for as using the graceful shutdown on the conn, would it look something like this?:

        runtime_hdl.clone().spawn(async move {
            loop {
                let stream = select! {
                    result = listener_v4.accept() => {
                        if let Ok((stream, _remote_addr)) = result {
                            stream
                        } else {
                            continue
                        }
                    }
                    result = listener_v6.accept() => {
                        if let Ok((stream, _remote_addr)) = result {
                            stream
                        } else {
                            continue
                        }
                    }
                    _ = stop_hdl.clone().shutdown() => break,
                };

                let svc = svc.clone();
                let stop_hdl2 = stop_hdl.clone();
                runtime_hdl.spawn(async move {
                    let conn = hyper::server::conn::Http::new()
                        .serve_connection(stream, svc)
                        .with_upgrades();

                    let stopped = stop_hdl2.shutdown();
                    tokio::pin!(stopped);

                    let res = match future::select(conn, stopped).await {
                        Either::Left((conn, _)) => conn,
                        Either::Right((_, mut conn)) => {
                            Pin::new(&mut conn).graceful_shutdown();
                            conn.await
                        }
                    };

                    if let Err(err) = res {
                        error!(err=?err, "HTTP server connection failed");
                    }
                });
            }
        });

        Ok(server_hdl)

[niklasad1]

looks good

[zeeshanlakhani]

@niklasad1 #1314.