fix: use independent deadline for sending error

[aruiz14]

Issue: 44576

Complementary (but not mutually blocking) to #74

Problem

remotedialer connections (those multiplexed over a single websocket) rely on receiving an Error message (io.EOF in the case of success), in order to consider a connection closed. In some cases, this message is never received, causing associated resources to be kept indefinitely.

Solution

#74 mitigates this problem by adding a synchronization mechanism to detect those stale connections by sharing ids of the active connections. That PR is complementary in the sense that it will make the protocol more robust, while this one should provide a fix for the original problem that caused the Error message not to be sent. I was able to troubleshoot this thanks to the errors printed by the changes from #67.

connection implements the net.Conn interface (including SetWriteDeadline), and is created from a Dialer.
In the case of Rancher, one of the uses is to create a Kubernetes client to access the API of downstream clusters. This client is configured to use remotedialer as part of is Transport, hence using HTTPS over the remotedialer connection.

The HTTP client used seems to be configured by default with a IdleConnTimeout of 90 seconds, after which the connection is automatically closed. However, the intermediate TLS connection layer will send an alert, then proceed to call SetWriteDeadline on the underlying connection with an invalid value in order to prevent any further writes, which makes writeErr to fail, because it uses the value of writeDeadline for sending the Error message.

Given that this is a control message (at the Session level), it should use its own deadline and not rely on the connection anymore.

CheckList

• Test
  • Adapted unit test to cover this unit case. Without the deadline change, it will fail with:

time="2024-06-13T09:29:10+02:00" level=warning msg="[2] encountered error \"deadline exceeded\" while writing error \"connection closed\" to close remotedialer"
    session_serve_test.go:134: message not sent on closed connection
--- FAIL: TestSession_closeConnection (0.00s)

[moio]

Thanks for the review, can this be merged @maxsokolovsky ?