crypto/tls: error communicating with OpenSSL 1.0.2k client with high parallel connections

[hkishn]

Go version

go version 1.21

Output of go env in your module/workspace:

GO111MODULE='off'
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/test/.cache/go-build'
GOENV='/home/test/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS='-mod=vendor'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/test/go/pkg/mod'
GONOPROXY='gitlab.protectv.local'
GONOSUMDB='gitlab.protectv.local'
GOOS='linux'
GOPATH='/home/test/go'
GOPRIVATE='gitlab.protectv.local'
GOPROXY='
https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD=''
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build4217941611=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I have a Golang server that is accepting TLS connections from an OpenSSL client. The OpenSSL client is on version OpenSSL 1.0.2k.
The client tries to connect with the server in TLS mode in parallel threads.

What did you see happen?

If I increase the thread count to 100 and try to connect with the server simultaneously. The TLS connection breaks with the error from the client side with the error "Fatal, Description: Unexpected Message"

This use case works work with Golang 1.18. Moreover, if I reduce the thread count to 50, then no error comes.

I think something has changed in the new releases of the Golang which is causing errors if the client tries to connect with the Golang server simultaneously in a large number of threads.

What did you expect to see?

The Golang server should accept connections in parallel as it was working with Golang 1.18.

[gabyhelp]

Related Issues

• crypto/tls: TLS1.3 occasionally fails with "tls: error decoding message" when fetching from https://tls13.1d.pw #41983 (closed)
• crypto/tls: client gets nondescriptive EOF error if server stops connection after client hello #44663 (closed)
• crypto/tls: TLS1.2 connections are getting broken with encryption alert with code 21 but based on RFC 5246 it shouldn't be. #66268 (closed)
• crypto/tls: client connection is staying open after server has closed it #34796 (closed)
• crypto/tls: tls deadlock at handshake stage during my concurrent tests #27136 (closed)
• crypto/tls: error handshake if there are duplicate TLS extensions #51088 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[hkishn]

Related Issues

• crypto/tls: TLS1.3 occasionally fails with "tls: error decoding message" when fetching from https://tls13.1d.pw #41983 (closed)
• crypto/tls: client gets nondescriptive EOF error if server stops connection after client hello #44663 (closed)
• crypto/tls: TLS1.2 connections are getting broken with encryption alert with code 21 but based on RFC 5246 it shouldn't be. #66268 (closed)
• crypto/tls: client connection is staying open after server has closed it #34796 (closed)
• crypto/tls: tls deadlock at handshake stage during my concurrent tests #27136 (closed)
• crypto/tls: error handshake if there are duplicate TLS extensions #51088 (closed)

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

no these issues are not related to issue I reported.

[seankhliao]

I think you'll need to provide a reproducer for the issue, right now it doesn't look very actionable.

[hkishn]

@seankhliao I am trying to build the enviornment to reproduce the issue.
Is this issue similar to crypto/tls: Large session tickets in Go 1.21 can cause Windows Schannel clients to be unable to connect · Issue #63763 · golang/go · GitHub
?

In my setup when I downgrade to golang 1.20, then I won't see the error

[FiloSottile]

Go servers are regularly used with more than 100 parallel connections, so we'll need more information to understand if there's an issue. Either a reproducer, or a PCAP with SSLKEYLOG, as well as the server code.

If you think it might be the same as #63763, you can try adding a WrapTicket that returns differently sized fake tickets and check if that fixes it, but that doesn't explain why it would break based on the number of parallel connections.

[hkishn]

@FiloSottile
I try to create a setup where the issue is reproducing. I also have a tcpdump with SSLKEYLOG file. That I will share.

And I have tried the fix mentioned in #63763. I just called return []byte{0}, nil in the WrapTicket function. This fixed the issue.

Is there any timeline #63763 will be fixed ?

[hkishn]

@seankhliao @FiloSottile

Go servers are regularly used with more than 100 parallel connections, so we'll need more information to understand if there's an issue. Either a reproducer, or a PCAP with SSLKEYLOG, as well as the server code.

If you think it might be the same as #63763, you can try adding a WrapTicket that returns differently sized fake tickets and check if that fixes it, but that doesn't explain why it would break based on the number of parallel connections.

When we configure TLS server with the following callback functions, then we can resolve the issue

 tlsConfig.WrapSession = (&tls.Config{}).EncryptTicket
   tlsConfig.UnwrapSession = (&tls.Config{}).DecryptTicket

Will these changes disable the new implementation of Wrapsession/Unwrapsession done in Golang 1.21?
Is there any other way by which we can resolve or disable this new functionality in golang1.21 ?

[seankhliao]

Duplicate of #63763