Retry not always working

[lzambarda]

Hello, I am working with a gRPC client. The server it is communicating with runs in kubernetes and sometimes it may be restarted due to resources and other k8s-related lifecycle issues.

The way I have set up my client is as follow:

retryOptions := []grpc_retry.CallOption{
	grpc_retry.WithBackoff(grpc_retry.BackoffLinear(1000 * time.Millisecond)),
}
c, err := grpc.Dial(address,
	grpc.WithInsecure(),
	grpc.WithStreamInterceptor(grpc_retry.StreamClientInterceptor(retryOptions...)),
	grpc.WithUnaryInterceptor(grpc_retry.UnaryClientInterceptor(retryOptions...)),
)
client := xxx.NewXXXClient(c)
...
for ... {
	data, err := client.GetXXXData(context.Background(),request)
	...
}

Since when I integrated the retry strategy it is much more stable during its long-lasting operation, but sometimes it fails with any of the following errors:

rpc error: code = Unavailable desc = transport is closing

rpc error: code = Unavailable desc = timed out waiting for server handshake

Is there anything wrong with my setup? Am I missing something? I used to have retry options on codes but I removed all of them.
Thanks in advance for your help!

I am using github.com/grpc-ecosystem/go-grpc-middleware v1.2.0

[johanbrandhorst]

This may mean that it's retrying against the same backend and that you need to more aggressively close connections to trigger DNS refreshes. See grpc/grpc-go#3170 (comment). I don't think there's anything we can do here.