client: always enable TCP keepalives with OS defaults #6834

easwars · 2023-12-06T02:32:39Z

In #6672, we attempted to configure OS defaults for TCP keepalive parameters (keepalive time and interval). But this had the unintended consequence of disabling TCP keepalives completely because of the way things are implemented in the Go stdlib. In this PR, we attempt to do the following:

Summary of changes:

Always enable TCP keepalives on client connections
- This has been the behavior for a while because the Go stdlib enables TCP keepalives as long as the net.Dialer.KeepAlive field is not set to a negative value.
- Explicitly setting the socket option ensures the old behavior persists.
Use OS defaults for TCP keepalive interval and time
- We do this by setting the net.Dialer.KeepAlive field to a negative value, thereby disabling the Go stdlib's override of these values to 15s.

There will be a follow-up PR for the server side.

Addresses #6250

RELEASE NOTES:

client: always enable TCP keepalives with OS defaults

easwars · 2023-12-06T02:33:23Z

@atollena : If you are interested in this change, please let me know and I'll mark you as a reviewer. Thanks.

codecov · 2023-12-06T02:34:59Z

Codecov Report

Merging #6834 (8675d7d) into master (0866ce0) will increase coverage by 0.08%.
Report is 1 commits behind head on master.
The diff coverage is 95.83%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6834      +/-   ##
==========================================
+ Coverage   83.53%   83.61%   +0.08%     
==========================================
  Files         285      286       +1     
  Lines       30754    30770      +16     
==========================================
+ Hits        25690    25729      +39     
+ Misses       3998     3981      -17     
+ Partials     1066     1060       -6

Files	Coverage Δ
dialoptions.go	`88.84% <ø> (ø)`
internal/tcp_keepalive.go	`100.00% <100.00%> (ø)`
internal/transport/proxy.go	`63.88% <100.00%> (ø)`
server.go	`79.00% <ø> (+0.23%)`	⬆️
internal/transport/http2_client.go	`90.23% <50.00%> (+0.15%)`	⬆️

... and 13 files with indirect coverage changes

atollena

That's what I had in mind, thanks!

atollena · 2023-12-06T11:26:11Z

internal/tcp_keepalive.go

+		// disabling the overriding of TCP keepalive parameters by setting the
+		// KeepAlive field to a negative value above, results in OS defaults for
+		// the TCP keealive interval and time parameters.
+		ControlContext: func(_ context.Context, _, _ string, c syscall.RawConn) error {


feel free to ignore: you can use Control instead of ControlContext for one less unused parameter.

Sounds like a minor improvement to me, but I don't feel strongly.

I went with ControlContext over Control (even though the context parameter is unused) since the former is preferred if both are set.

Looks like the ControlContext field was only added in Go1.20, and currently the least supported Go version is still Go1.19. So I'm changing this back to Control.

dfawley · 2023-12-06T17:28:49Z

dialoptions.go

-// TCP keepalive time and interval to 15s.
-// To retain OS defaults, use a net.Dialer with the KeepAlive field set to a
-// negative value.
+// Note: As of Go 1.21, the standard library overrides the OS defaults for TCP


I agree this wording is confusing, and have been trying to think of something better... Maybe:

Note: All supported releases of Go (as of December 2023) override the OS defaults for...

Or "as of the Go 1.21 release" or "up to and including at least Go 1.21" for the parenthetical?

Done. Thanks.

dfawley · 2023-12-06T17:30:08Z

server.go

@@ -813,15 +813,6 @@ func (l *listenSocket) Close() error {
 // Serve returns when lis.Accept fails with fatal errors.  lis will be closed when
 // this method returns.
 // Serve will return a non-nil error unless Stop or GracefulStop is called.
-//
-// Note: As of Go 1.21, the standard library overrides the OS defaults for


Why is this deleted?

Because I will be sending a follow-up PR to fix things on the server-side.

Updated this with the latest commit, based on our discussion on the issue.

dfawley · 2023-12-06T17:32:26Z

dialoptions.go

+// for keepalive time and interval, use a net.Dialer that sets the KeepAlive
+// field set to a negative value, and sets the SO_KEEPALIVE socket option to


"sets (the KeepAlive field) set to a negative". Remove the 2nd "set".

dfawley · 2023-12-06T17:32:49Z

internal/tcp_keepalive.go

+		// disabling the overriding of TCP keepalive parameters by setting the
+		// KeepAlive field to a negative value above, results in OS defaults for
+		// the TCP keealive interval and time parameters.
+		ControlContext: func(_ context.Context, _, _ string, c syscall.RawConn) error {


Sounds like a minor improvement to me, but I don't feel strongly.

dfawley

LGTM, but one question about something else we might want to fix.

dfawley · 2023-12-07T00:16:31Z

internal/transport/http2_client.go

@@ -264,7 +263,7 @@ func newHTTP2Client(connectCtx, ctx context.Context, addr resolver.Address, opts
 	}
 	keepaliveEnabled := false
 	if kp.Time != infinity {
-		if err = syscall.SetTCPUserTimeout(conn, kp.Timeout); err != nil {
+		if err = isyscall.SetTCPUserTimeout(conn, kp.Timeout); err != nil {


It looks like this function sets the TCP user timeout but does not enable keepalives.. should(n't) we update it to do that, too, in case they are disabled?

TCP_USER_TIMEOUT specifies the maximum amount of time that transmitted data may remain unacknowledged before TCP will forcibly close the corresponding connection and return ETIMEDOUT to the application. If the option value is specified as 0, TCP will to use the system default.

I think it is totally a reasonable thing to use it without TCP keepalives.

As per A18, the reason we set TCP_USER_TIMEOUT when gRPC keepalives are configured is to ensure that if things are stuck at the TCP layer, setting the TCP_USER_TIMEOUT will ensure that the connection is closed anyways (even if gRPC keepalives are not able to do the same).

Are you concerned about the case where a user specifies a custom net.Dialer to disable TCP keepalives, and configures gRPC keepalives, and therefore, we end up setting TCP_USER_TIMEOUT, but TCP keepalives are disabled? Even in this case, I feel it should be fine, since TCP_USER_TIMEOUT can work irrespective of whether TCP keepalives are configured on the connection.

Ah no, sorry that makes sense, I'm not sure what I was thinking yesterday.. :)

client: enable TCP keepalives with OS defaults

2d8170b

easwars added the Type: Bug label Dec 6, 2023

easwars added this to the 1.60 Release milestone Dec 6, 2023

easwars requested a review from dfawley December 6, 2023 02:33

easwars assigned dfawley Dec 6, 2023

atollena approved these changes Dec 6, 2023

View reviewed changes

dfawley reviewed Dec 6, 2023

View reviewed changes

dfawley assigned easwars and unassigned dfawley Dec 6, 2023

easwars added 2 commits December 6, 2023 19:54

minor review comments

4a78dcb

use Control instead of ControlContext

2510f50

easwars assigned dfawley and unassigned easwars Dec 6, 2023

update comment on Serve()

8675d7d

dfawley reviewed Dec 7, 2023

View reviewed changes

dfawley assigned easwars and unassigned dfawley Dec 7, 2023

easwars assigned dfawley and unassigned easwars Dec 7, 2023

dfawley approved these changes Dec 7, 2023

View reviewed changes

dfawley assigned easwars and unassigned dfawley Dec 7, 2023

easwars merged commit a03c7f1 into grpc:master Dec 7, 2023
14 checks passed

easwars deleted the tcp_keepalive_fix branch December 7, 2023 22:04

easwars mentioned this pull request Dec 7, 2023

TCP Connection reset after a single lost keepalive packet #6250

Closed

easwars added a commit to easwars/grpc-go that referenced this pull request Dec 7, 2023

client: always enable TCP keepalives with OS defaults (grpc#6834)

e3be3a7

easwars added a commit that referenced this pull request Dec 8, 2023

Cherry-pick #6834 to v1.60.x release branch (#6839)

71e67a9

easwars mentioned this pull request Dec 8, 2023

internal: use OS defaults for TCP keepalive params only on unix #6841

Merged

mmatczuk mentioned this pull request Dec 18, 2023

internal: use OS defaults for TCP keepalive params in Windows #6863

Merged

smira mentioned this pull request Jan 16, 2024

Discovery Service connectivity issues siderolabs/discovery-service#49

Closed

karimra mentioned this pull request Feb 28, 2024

Session reestablishment issues openconfig/gnmic#383

Closed

github-actions bot locked as resolved and limited conversation to collaborators Jul 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client: always enable TCP keepalives with OS defaults #6834

client: always enable TCP keepalives with OS defaults #6834

easwars commented Dec 6, 2023

easwars commented Dec 6, 2023

codecov bot commented Dec 6, 2023 •

edited

Loading

atollena left a comment

atollena Dec 6, 2023

dfawley Dec 6, 2023

easwars Dec 6, 2023

easwars Dec 6, 2023

dfawley Dec 6, 2023

easwars Dec 6, 2023

dfawley Dec 6, 2023

easwars Dec 6, 2023

easwars Dec 6, 2023

dfawley Dec 6, 2023

easwars Dec 6, 2023

dfawley Dec 6, 2023

dfawley left a comment

dfawley Dec 7, 2023

easwars Dec 7, 2023

dfawley Dec 7, 2023

		// for keepalive time and interval, use a net.Dialer that sets the KeepAlive
		// field set to a negative value, and sets the SO_KEEPALIVE socket option to

client: always enable TCP keepalives with OS defaults #6834

client: always enable TCP keepalives with OS defaults #6834

Conversation

easwars commented Dec 6, 2023

easwars commented Dec 6, 2023

codecov bot commented Dec 6, 2023 • edited Loading

Codecov Report

atollena left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfawley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 6, 2023 •

edited

Loading