[Query] Netstack close is not closing all the registered endpoints

[AmitKatyal-Sophos]

We are using gvisor netstack with below configuration

User -----> NetStack --------> proxy

1. User creates the TCP connection with the server(with netstack).
2. Server creates a new instance of gvisor netstack for connection.
3. User access the resource over the connection. Server reads the received IP packets and injects them to the gvisor netstack. On receiving TCP sync packet, gvisor netstack invokes the registered TCP handler and creates the new endpoint. On the other end, server dials the proxy and relays the packets from end point to the proxy and vice versa.

Problem statement

If user abruptly closes the connection, then all the TCP connections/endpoints relayed over the connection are not getting closed.

Root cause analysis

Please find below the analysis:

1. TCP connection handlers (i.e. go routines relaying the packets from endpoint to proxy and vice versa).
   Reader go routine is blocked on the Read() from the endpoint. As there is no data to read, Read go routine is registering for the waiter.ReadableEvents. Please find below the code snippet.

https://github.com/google/gvisor/blob/b250ee717d8797a27ca31cc4ad43e10ba59d88f1/pkg/tcpip/adapters/gonet/gonet.go#L302-L329

2. On abrupt closure of connection, go routines handling read and writes from the connection exits and server calls NetStack close.

NetStack close performs the below actions,

Close internally calls endpoint abort but abort endpoint just sends the RST to the client, and marks the endpoint state to error, but doesn't notify the reader go routine to come out of the blocking read. Since reader go routine is not notified, reader go routing is stuck and keep on waiting for the read ready event. As a result, TCP connections/endpoints are not getting terminated. Please find below the code snippets of netstack close handling.

Netstack close
https://github.com/google/gvisor/blob/b250ee717d8797a27ca31cc4ad43e10ba59d88f1/pkg/tcpip/stack/stack.go#L1725-L1735

Endpoint Abort
As the state is connected, abort will send the RST and return
https://github.com/google/gvisor/blob/b250ee717d8797a27ca31cc4ad43e10ba59d88f1/pkg/tcpip/transport/tcp/endpoint.go#L1006-L1018

IMO nestack close should have wake up the endpoint reader go routine so that connection gets termintated. Is this a known issue or am I missing something here ?

[xjasonlyu]

Did you create a netstack for each websocket?

[AmitKatyal-Sophos]

Thanks for the quick reply, we have speical use-case, hence we can't have the common stack.
The issue we are facing is a very basic use-case. Just abruptly close the connection from the left side and endpoint is not getting closed.
Any idea how in this case the endpoint will get terminated ? Any pointers to debug the issue further. Thanks.

[xjasonlyu]

In my opinion, if I get it correctly, maybe you should close the gonet connections before closing the stack.

[AmitKatyal-Sophos]

yes, then I'll have to track all the connections. Shouldn't this be handled by the net stack close ?

[xjasonlyu]

From my perspective, stack.Close is meant to close the stack itself, not the connections.

Also, the gVisor stack has a stats mechanism to track all the statistics like what netstats does, maybe you'll find it helpful.

[AmitKatyal-Sophos]

As part of stack close, all the registered endpoints are being aborted, but not closed.

https://github.com/google/gvisor/blob/b250ee717d8797a27ca31cc4ad43e10ba59d88f1/pkg/tcpip/transport/tcp/endpoint.go#L1006-L1018

I'll look into statistics. Thanks.