Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Go 1.8 unit test fixes #14847

Merged
merged 5 commits into from
Jun 28, 2017
Merged

Go 1.8 unit test fixes #14847

merged 5 commits into from
Jun 28, 2017

Conversation

detiber
Copy link

@detiber detiber commented Jun 22, 2017

Backport upstream golang-1.8 fixes for unit tests.

@detiber
Copy link
Author

detiber commented Jun 22, 2017

[test]

@detiber
Copy link
Author

detiber commented Jun 23, 2017

@liggitt upstream cherry-picks of golang-1.8 fixes

The 'UPSTREAM: 46440' solves a missing import for the related upstream cherry-pick already in the repo, since the test is only run when race detection is disabled for tests it was not causing failures in jenkins, I only noticed it because race detection can only be enabled on x86_64 platforms.

The test failure in jenkins appears to be an unrelated flake.

@liggitt
Copy link
Contributor

liggitt commented Jun 23, 2017

since the test is only run when race detection is disabled for tests it was not causing failures in jenkins

we should absolutely run with race detection on in jenkins

LGTM [test]

@detiber
Copy link
Author

detiber commented Jun 23, 2017

we should absolutely run with race detection on in jenkins

@liggitt completely agree, I'm just wondering if it would make sense to have a job that runs tests with race detection disabled, since it looks like there are multiple tests that are disabled when race detection is enabled:

> git grep '!race'
cmd/cluster-capacity/go/src/github.com/kubernetes-incubator/cluster-capacity/vendor/golang.org/x/sys/plan9/race0.go:// +build plan9,!race
cmd/cluster-capacity/go/src/github.com/kubernetes-incubator/cluster-capacity/vendor/golang.org/x/sys/unix/race0.go:// +build darwin,!race linux,!race freebsd,!race netbsd openbsd solaris dragonfly
cmd/cluster-capacity/go/src/github.com/kubernetes-incubator/cluster-capacity/vendor/golang.org/x/sys/windows/race0.go:// +build windows,!race
cmd/cluster-capacity/go/src/github.com/kubernetes-incubator/cluster-capacity/vendor/k8s.io/client-go/tools/cache/mutation_detector_test.go:// +build !race
cmd/cluster-capacity/go/src/github.com/kubernetes-incubator/cluster-capacity/vendor/k8s.io/kubernetes/pkg/master/master_openapi_test.go:// +build !race
cmd/cluster-capacity/go/src/github.com/kubernetes-incubator/cluster-capacity/vendor/k8s.io/kubernetes/staging/src/k8s.io/client-go/tools/cache/mutation_detector_test.go:// +build !race
cmd/service-catalog/go/src/github.com/kubernetes-incubator/service-catalog/vendor/golang.org/x/sys/plan9/race0.go:// +build plan9,!race
cmd/service-catalog/go/src/github.com/kubernetes-incubator/service-catalog/vendor/golang.org/x/sys/unix/race0.go:// +build darwin,!race linux,!race freebsd,!race netbsd openbsd solaris dragonfly
cmd/service-catalog/go/src/github.com/kubernetes-incubator/service-catalog/vendor/golang.org/x/sys/windows/race0.go:// +build windows,!race
cmd/service-catalog/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/client-go/tools/cache/mutation_detector_test.go:// +build !race
cmd/service-catalog/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/kubernetes/pkg/master/master_openapi_test.go:// +build !race
cmd/service-catalog/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/kubernetes/staging/src/k8s.io/client-go/tools/cache/mutation_detector_test.go:// +build !race
vendor/github.com/fsouza/go-dockerclient/external/golang.org/x/sys/unix/race0.go:// +build darwin,!race linux,!race freebsd,!race netbsd openbsd solaris dragonfly
vendor/golang.org/x/sys/unix/race0.go:// +build darwin,!race linux,!race freebsd,!race netbsd openbsd solaris dragonfly
vendor/golang.org/x/sys/windows/race0.go:// +build windows,!race
vendor/k8s.io/kubernetes/pkg/master/master_openapi_test.go:// +build !race
vendor/k8s.io/kubernetes/staging/src/k8s.io/client-go/tools/cache/mutation_detector_test.go:// +build !race

If it wasn't for building/testing on ppc64le, where race detection cannot be enabled I would have never stumbled across the missing import issue in vendor/k8s.io/kubernetes/pkg/master/master_openapi_test.go.

cc @stevekuznetsov

@liggitt
Copy link
Contributor

liggitt commented Jun 23, 2017

in my mind, a test that disables itself when race detection is on is self-declaring it is buggy. those should be fixed or removed.

@detiber
Copy link
Author

detiber commented Jun 23, 2017

in my mind, a test that disables itself when race detection is on is self-declaring it is buggy. those should be fixed or removed.

@liggitt no objections there, but I can't speak to the reasons those tests were disabled for race detection originally. The one that seems the oddest to me are the ones called race0.go

@stevekuznetsov
Copy link
Contributor

Lots of cluster-capacity and service-catalog ones in there -- /cc @derekwaynecarr @aveshagarwal @jpeeler

@detiber
Copy link
Author

detiber commented Jun 23, 2017

@stevekuznetsov looks like they are all vendored from upstream, though.

@detiber
Copy link
Author

detiber commented Jun 23, 2017

@liggitt test failure looks like a flake.

@stevekuznetsov
Copy link
Contributor

@detiber I've tagged the people who vendored it

@aveshagarwal
Copy link
Contributor

@detiber @stevekuznetsov All those tests are part of vendored stuff in vendored cluster-capacity, so do we want to remove them?

@stevekuznetsov
Copy link
Contributor

@aveshagarwal I can't remember -- are we trying to or are we not trying to run them as part of Origin's unit test super-suite?

@stevekuznetsov
Copy link
Contributor

@aveshagarwal I guess @liggitt's general comment re: turning off race detection still applies

@aveshagarwal
Copy link
Contributor

@stevekuznetsov my understanding was that cluster-capacity and service catalog like vendored stuff is being ignored for any testing.

@stevekuznetsov
Copy link
Contributor

OK, non-issue then for us. @detiber you're probably better off not running those tests either.

@detiber
Copy link
Author

detiber commented Jun 23, 2017

@stevekuznetsov I was not running the tests for vendored dependencies of service catalog or cluster-capacity. The issue that I hit was related to vendor/k8s.io/kubernetes/pkg/master/master_openapi_test.go, which is part of the unit test suite. I would expect that vendor/k8s.io/kubernetes/staging/src/k8s.io/client-go/tools/cache/mutation_detector_test.go might potentially be one that we want to run as well, but is currently disabled due to it being disabled for race detection.

@detiber
Copy link
Author

detiber commented Jun 23, 2017

[test] again

@stevekuznetsov
Copy link
Contributor

I see. We could change make check to run with and without race, but I would be interested to know why those tests disable it first

@detiber
Copy link
Author

detiber commented Jun 23, 2017

@liggitt finally made it through a test run without hitting a flake. Can we get this lined up for merge? These test failures are currently blocking testing on multi-arch builds.

@@ -96,7 +96,7 @@ func TestDialURL(t *testing.T) {
ts.TLS = &tls.Config{Certificates: []tls.Certificate{cert}}
ts.StartTLS()

tlsConfigCopy := utilnet.CloneTLSConfig(tc.TLSConfig)
tlsConfigCopy := tc.TLSConfig
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is actually wrong (it was a bug in the upstream PR, as noted in kubernetes/kubernetes#41758 (comment))

once we're on go1.8 alone, the suggested solution there could work. until then, we need to continue cloning with utilnet.CloneTLSConfig

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liggitt cloning using utilnet.CloneTLSConfig, causes the following error when built/tested with golang-1.8:


2017/06/26 17:30:58 http: TLS handshake error from 127.0.0.1:52074: remote error: tls: bad certificate
2017/06/26 17:30:59 http: TLS handshake error from 127.0.0.1:53096: remote error: tls: bad certificate
2017/06/26 17:30:59 http: TLS handshake error from 127.0.0.1:40544: remote error: tls: bad certificate
2017/06/26 17:30:59 http: TLS handshake error from 127.0.0.1:42000: remote error: tls: bad certificate
--- FAIL: TestDialURL (0.24s)
	dial_test.go:128: insecure: transport's copy of TLSConfig was mutated
		&tls.Config{Rand:io.Reader(nil), Time:(func() time.Time)(nil), Certificates:[]tls.Certificate(nil), NameToCertificate:map[string]*tls.Certificate(nil), GetCertificate:(func(*tls.ClientHelloInfo) (*tls.Certificate, error))(nil), GetClientCertificate:(func(*tls.CertificateRequestInfo) (*tls.Certificate, error))(nil), GetConfigForClient:(func(*tls.ClientHelloInfo) (*tls.Config, error))(nil), VerifyPeerCertificate:(func([][]uint8, [][]*x509.Certificate) error)(nil), RootCAs:(*x509.CertPool)(nil), NextProtos:[]string(nil), ServerName:"", ClientAuth:0, ClientCAs:(*x509.CertPool)(nil), InsecureSkipVerify:true, CipherSuites:[]uint16(nil), PreferServerCipherSuites:false, SessionTicketsDisabled:false, SessionTicketKey:[32]uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, ClientSessionCache:tls.ClientSessionCache(nil), MinVersion:0x0, MaxVersion:0x0, CurvePreferences:[]tls.CurveID(nil), DynamicRecordSizingDisabled:false, Renegotiation:0, KeyLogWriter:io.Writer(nil), serverInitOnce:sync.Once{m:sync.Mutex{state:0, sema:0x0}, done:0x0}, mutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, sessionTicketKeys:[]tls.ticketKey(nil), originalConfig:(*tls.Config)(nil)}
		
		&tls.Config{Rand:io.Reader(nil), Time:(func() time.Time)(nil), Certificates:[]tls.Certificate(nil), NameToCertificate:map[string]*tls.Certificate(nil), GetCertificate:(func(*tls.ClientHelloInfo) (*tls.Certificate, error))(nil), GetClientCertificate:(func(*tls.CertificateRequestInfo) (*tls.Certificate, error))(nil), GetConfigForClient:(func(*tls.ClientHelloInfo) (*tls.Config, error))(nil), VerifyPeerCertificate:(func([][]uint8, [][]*x509.Certificate) error)(nil), RootCAs:(*x509.CertPool)(nil), NextProtos:[]string(nil), ServerName:"", ClientAuth:0, ClientCAs:(*x509.CertPool)(nil), InsecureSkipVerify:true, CipherSuites:[]uint16(nil), PreferServerCipherSuites:false, SessionTicketsDisabled:false, SessionTicketKey:[32]uint8{0x62, 0x6, 0x30, 0x68, 0x7a, 0x42, 0xb9, 0xf5, 0x54, 0x52, 0x0, 0xfa, 0x8b, 0xb7, 0x84, 0x53, 0x79, 0x31, 0xf4, 0x7a, 0xcf, 0xe5, 0x14, 0xf9, 0x18, 0x18, 0xd9, 0xb7, 0x82, 0xd6, 0x1f, 0x7f}, ClientSessionCache:tls.ClientSessionCache(nil), MinVersion:0x0, MaxVersion:0x0, CurvePreferences:[]tls.CurveID(nil), DynamicRecordSizingDisabled:false, Renegotiation:0, KeyLogWriter:io.Writer(nil), serverInitOnce:sync.Once{m:sync.Mutex{state:0, sema:0x0}, done:0x1}, mutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, sessionTicketKeys:[]tls.ticketKey{tls.ticketKey{keyName:[16]uint8{0x19, 0x64, 0x20, 0x31, 0xb5, 0xff, 0xba, 0x6c, 0xde, 0xe2, 0x5d, 0xef, 0x4f, 0x8d, 0xed, 0x22}, aesKey:[16]uint8{0x83, 0x98, 0x50, 0xef, 0x33, 0xfc, 0xc1, 0x96, 0x46, 0xc1, 0xda, 0x8d, 0xae, 0xbc, 0xcc, 0x31}, hmacKey:[16]uint8{0x41, 0x44, 0x5a, 0xe5, 0xb3, 0xc1, 0x36, 0xa5, 0x7a, 0xa0, 0xc2, 0x43, 0xb0, 0xf4, 0x86, 0xf7}}}, originalConfig:(*tls.Config)(nil)}
	dial_test.go:128: secure, no roots: transport's copy of TLSConfig was mutated
		&tls.Config{Rand:io.Reader(nil), Time:(func() time.Time)(nil), Certificates:[]tls.Certificate(nil), NameToCertificate:map[string]*tls.Certificate(nil), GetCertificate:(func(*tls.ClientHelloInfo) (*tls.Certificate, error))(nil), GetClientCertificate:(func(*tls.CertificateRequestInfo) (*tls.Certificate, error))(nil), GetConfigForClient:(func(*tls.ClientHelloInfo) (*tls.Config, error))(nil), VerifyPeerCertificate:(func([][]uint8, [][]*x509.Certificate) error)(nil), RootCAs:(*x509.CertPool)(nil), NextProtos:[]string(nil), ServerName:"", ClientAuth:0, ClientCAs:(*x509.CertPool)(nil), InsecureSkipVerify:false, CipherSuites:[]uint16(nil), PreferServerCipherSuites:false, SessionTicketsDisabled:false, SessionTicketKey:[32]uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, ClientSessionCache:tls.ClientSessionCache(nil), MinVersion:0x0, MaxVersion:0x0, CurvePreferences:[]tls.CurveID(nil), DynamicRecordSizingDisabled:false, Renegotiation:0, KeyLogWriter:io.Writer(nil), serverInitOnce:sync.Once{m:sync.Mutex{state:0, sema:0x0}, done:0x0}, mutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, sessionTicketKeys:[]tls.ticketKey(nil), originalConfig:(*tls.Config)(nil)}
		
		&tls.Config{Rand:io.Reader(nil), Time:(func() time.Time)(nil), Certificates:[]tls.Certificate(nil), NameToCertificate:map[string]*tls.Certificate(nil), GetCertificate:(func(*tls.ClientHelloInfo) (*tls.Certificate, error))(nil), GetClientCertificate:(func(*tls.CertificateRequestInfo) (*tls.Certificate, error))(nil), GetConfigForClient:(func(*tls.ClientHelloInfo) (*tls.Config, error))(nil), VerifyPeerCertificate:(func([][]uint8, [][]*x509.Certificate) error)(nil), RootCAs:(*x509.CertPool)(nil), NextProtos:[]string(nil), ServerName:"", ClientAuth:0, ClientCAs:(*x509.CertPool)(nil), InsecureSkipVerify:false, CipherSuites:[]uint16(nil), PreferServerCipherSuites:false, SessionTicketsDisabled:false, SessionTicketKey:[32]uint8{0x17, 0xbf, 0xb4, 0xda, 0xa2, 0x22, 0x68, 0x9f, 0x5f, 0x1e, 0x34, 0x7b, 0xee, 0xe1, 0x34, 0xad, 0xc2, 0xf6, 0xc6, 0x19, 0xf6, 0x9a, 0xb4, 0x6, 0x9b, 0xb1, 0xe9, 0xf3, 0xa5, 0x24, 0xb6, 0xa9}, ClientSessionCache:tls.ClientSessionCache(nil), MinVersion:0x0, MaxVersion:0x0, CurvePreferences:[]tls.CurveID(nil), DynamicRecordSizingDisabled:false, Renegotiation:0, KeyLogWriter:io.Writer(nil), serverInitOnce:sync.Once{m:sync.Mutex{state:0, sema:0x0}, done:0x1}, mutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, sessionTicketKeys:[]tls.ticketKey{tls.ticketKey{keyName:[16]uint8{0xdc, 0x4c, 0x8, 0xc0, 0xac, 0x3d, 0xe7, 0x2f, 0xa7, 0x62, 0x8b, 0xec, 0xcb, 0xe8, 0x1c, 0xcf}, aesKey:[16]uint8{0x12, 0x4, 0x8e, 0x35, 0x7f, 0x8c, 0x31, 0xb7, 0x78, 0x50, 0x48, 0xe4, 0x38, 0xaa, 0x32, 0xac}, hmacKey:[16]uint8{0xbb, 0xe4, 0xb1, 0x60, 0x84, 0xe9, 0x1e, 0xaa, 0x70, 0x90, 0xd8, 0x0, 0xdf, 0xc7, 0x66, 0x85}}}, originalConfig:(*tls.Config)(nil)}
	dial_test.go:128: secure with roots: transport's copy of TLSConfig was mutated
		&tls.Config{Rand:io.Reader(nil), Time:(func() time.Time)(nil), Certificates:[]tls.Certificate(nil), NameToCertificate:map[string]*tls.Certificate(nil), GetCertificate:(func(*tls.ClientHelloInfo) (*tls.Certificate, error))(nil), GetClientCertificate:(func(*tls.CertificateRequestInfo) (*tls.Certificate, error))(nil), GetConfigForClient:(func(*tls.ClientHelloInfo) (*tls.Config, error))(nil), VerifyPeerCertificate:(func([][]uint8, [][]*x509.Certificate) error)(nil), RootCAs:(*x509.CertPool)(0xc420083ec0), NextProtos:[]string(nil), ServerName:"", ClientAuth:0, ClientCAs:(*x509.CertPool)(nil), InsecureSkipVerify:false, CipherSuites:[]uint16(nil), PreferServerCipherSuites:false, SessionTicketsDisabled:false, SessionTicketKey:[32]uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, ClientSessionCache:tls.ClientSessionCache(nil), MinVersion:0x0, MaxVersion:0x0, CurvePreferences:[]tls.CurveID(nil), DynamicRecordSizingDisabled:false, Renegotiation:0, KeyLogWriter:io.Writer(nil), serverInitOnce:sync.Once{m:sync.Mutex{state:0, sema:0x0}, done:0x0}, mutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, sessionTicketKeys:[]tls.ticketKey(nil), originalConfig:(*tls.Config)(nil)}
		
		&tls.Config{Rand:io.Reader(nil), Time:(func() time.Time)(nil), Certificates:[]tls.Certificate(nil), NameToCertificate:map[string]*tls.Certificate(nil), GetCertificate:(func(*tls.ClientHelloInfo) (*tls.Certificate, error))(nil), GetClientCertificate:(func(*tls.CertificateRequestInfo) (*tls.Certificate, error))(nil), GetConfigForClient:(func(*tls.ClientHelloInfo) (*tls.Config, error))(nil), VerifyPeerCertificate:(func([][]uint8, [][]*x509.Certificate) error)(nil), RootCAs:(*x509.CertPool)(0xc420083ec0), NextProtos:[]string(nil), ServerName:"", ClientAuth:0, ClientCAs:(*x509.CertPool)(nil), InsecureSkipVerify:false, CipherSuites:[]uint16(nil), PreferServerCipherSuites:false, SessionTicketsDisabled:false, SessionTicketKey:[32]uint8{0x47, 0x8, 0x4b, 0x8, 0xf2, 0x22, 0xbd, 0x5a, 0x7c, 0xbe, 0xfd, 0x69, 0xa2, 0xc4, 0xb, 0x2, 0x3, 0xf8, 0x4, 0xd8, 0xcd, 0x70, 0x60, 0x6a, 0x67, 0xe1, 0x82, 0x1c, 0x19, 0x19, 0x2f, 0x1e}, ClientSessionCache:tls.ClientSessionCache(nil), MinVersion:0x0, MaxVersion:0x0, CurvePreferences:[]tls.CurveID(nil), DynamicRecordSizingDisabled:false, Renegotiation:0, KeyLogWriter:io.Writer(nil), serverInitOnce:sync.Once{m:sync.Mutex{state:0, sema:0x0}, done:0x1}, mutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, sessionTicketKeys:[]tls.ticketKey{tls.ticketKey{keyName:[16]uint8{0xdd, 0x8, 0xb0, 0x4e, 0x4, 0xc2, 0xd5, 0x78, 0x8f, 0x56, 0x3c, 0xe, 0xcc, 0x8, 0x9f, 0x2f}, aesKey:[16]uint8{0xcc, 0x50, 0x9f, 0x23, 0x78, 0x9c, 0x5d, 0x3c, 0xf5, 0x85, 0xb0, 0x13, 0xe, 0x1d, 0xac, 0xaa}, hmacKey:[16]uint8{0xb3, 0x93, 0xf5, 0x52, 0x10, 0xe6, 0x6d, 0x59, 0xe8, 0x31, 0xe, 0x8d, 0x2b, 0x7a, 0x1f, 0x0}}}, originalConfig:(*tls.Config)(nil)}
FAIL
coverage: 78.7% of statements
FAIL	github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/util/proxy	0.291s

The error becomes a bit more clear by using diff.ObjectGoPrintSideBySide from k8s.io/apimachinery/pkg/util:

--- FAIL: TestDialURL (0.16s)
	dial_test.go:129: secure with roots: transport's copy of TLSConfig was mutated
		(*tls.Config)(0xc420160480)({                                                               (*tls.Config)(0xc420161980)({
		 Rand: (io.Reader) <nil>,                                                                    Rand: (io.Reader) <nil>,
		 Time: (func() time.Time) <nil>,                                                             Time: (func() time.Time) <nil>,
		 Certificates: ([]tls.Certificate) <nil>,                                                    Certificates: ([]tls.Certificate) <nil>,
		 NameToCertificate: (map[string]*tls.Certificate) <nil>,                                     NameToCertificate: (map[string]*tls.Certificate) <nil>,
		 GetCertificate: (func(*tls.ClientHelloInfo) (*tls.Certificate, error)) <nil>,               GetCertificate: (func(*tls.ClientHelloInfo) (*tls.Certificate, error)) <nil>,
		 GetClientCertificate: (func(*tls.CertificateRequestInfo) (*tls.Certificate, error)) <nil>,  GetClientCertificate: (func(*tls.CertificateRequestInfo) (*tls.Certificate, error)) <nil>,
		 GetConfigForClient: (func(*tls.ClientHelloInfo) (*tls.Config, error)) <nil>,                GetConfigForClient: (func(*tls.ClientHelloInfo) (*tls.Config, error)) <nil>,
		 VerifyPeerCertificate: (func([][]uint8, [][]*x509.Certificate) error) <nil>,                VerifyPeerCertificate: (func([][]uint8, [][]*x509.Certificate) error) <nil>,
		 RootCAs: (*x509.CertPool)(0xc42015e1e0)({                                                   RootCAs: (*x509.CertPool)(0xc42015e1e0)({
		  bySubjectKeyId: (map[string][]int) {                                                        bySubjectKeyId: (map[string][]int) {
		  },                                                                                          },
		  byName: (map[string][]int) (len=1) {                                                        byName: (map[string][]int) (len=1) {
		   (string) (len=20) "0\x121\x100\x0e\x06\x03U\x04\n\x13\aAcme Co": ([]int) (len=1 cap=1) {    (string) (len=20) "0\x121\x100\x0e\x06\x03U\x04\n\x13\aAcme Co": ([]int) (len=1 cap=1) {
		    (int) 0                                                                                     (int) 0
		   }                                                                                           }
		  },                                                                                          },
		  certs: ([]*x509.Certificate) (len=1 cap=1) {                                                certs: ([]*x509.Certificate) (len=1 cap=1) {
		   (*x509.Certificate)(0xc420162000)({                                                         (*x509.Certificate)(0xc420162000)({
		    Raw: ([]uint8) (len=403 cap=405) {                                                          Raw: ([]uint8) (len=403 cap=405) {
		     00000000  30 82 01 8f 30 82 01 39  a0 03 02 01 02 02 11 00  |0...0..9........|              00000000  30 82 01 8f 30 82 01 39  a0 03 02 01 02 02 11 00  |0...0..9........|
		     00000010  aa 62 d9 69 93 70 5a d5  8f 1a e5 e6 7e 58 0d 41  |.b.i.pZ.....~X.A|              00000010  aa 62 d9 69 93 70 5a d5  8f 1a e5 e6 7e 58 0d 41  |.b.i.pZ.....~X.A|
		     00000020  30 0d 06 09 2a 86 48 86  f7 0d 01 01 0b 05 00 30  |0...*.H........0|              00000020  30 0d 06 09 2a 86 48 86  f7 0d 01 01 0b 05 00 30  |0...*.H........0|
		     00000030  12 31 10 30 0e 06 03 55  04 0a 13 07 41 63 6d 65  |.1.0...U....Acme|              00000030  12 31 10 30 0e 06 03 55  04 0a 13 07 41 63 6d 65  |.1.0...U....Acme|
		     00000040  20 43 6f 30 20 17 0d 37  30 30 31 30 31 30 30 30  | Co0 ..700101000|              00000040  20 43 6f 30 20 17 0d 37  30 30 31 30 31 30 30 30  | Co0 ..700101000|
		     00000050  30 30 30 5a 18 0f 32 30  38 34 30 31 32 39 31 36  |000Z..2084012916|              00000050  30 30 30 5a 18 0f 32 30  38 34 30 31 32 39 31 36  |000Z..2084012916|
		     00000060  30 30 30 30 5a 30 12 31  10 30 0e 06 03 55 04 0a  |0000Z0.1.0...U..|              00000060  30 30 30 30 5a 30 12 31  10 30 0e 06 03 55 04 0a  |0000Z0.1.0...U..|
		     00000070  13 07 41 63 6d 65 20 43  6f 30 5c 30 0d 06 09 2a  |..Acme Co0\0...*|              00000070  13 07 41 63 6d 65 20 43  6f 30 5c 30 0d 06 09 2a  |..Acme Co0\0...*|
		     00000080  86 48 86 f7 0d 01 01 01  05 00 03 4b 00 30 48 02  |.H.........K.0H.|              00000080  86 48 86 f7 0d 01 01 01  05 00 03 4b 00 30 48 02  |.H.........K.0H.|
		     00000090  41 00 bd 7c 46 d1 b3 33  f7 b7 5e 06 af 8a 1a 85  |A..|F..3..^.....|              00000090  41 00 bd 7c 46 d1 b3 33  f7 b7 5e 06 af 8a 1a 85  |A..|F..3..^.....|
		     000000a0  5e f3 14 e6 c1 23 84 df  03 98 bb 62 79 4c 5d e2  |^....#.....byL].|              000000a0  5e f3 14 e6 c1 23 84 df  03 98 bb 62 79 4c 5d e2  |^....#.....byL].|
		     000000b0  e2 9d be cc 9e f4 f8 6e  47 ce 27 31 3a ce 4b 67  |.......nG.'1:.Kg|              000000b0  e2 9d be cc 9e f4 f8 6e  47 ce 27 31 3a ce 4b 67  |.......nG.'1:.Kg|
		     000000c0  e7 dc cf ed 48 87 b3 79  78 60 eb aa ae 38 b0 33  |....H..yx`...8.3|              000000c0  e7 dc cf ed 48 87 b3 79  78 60 eb aa ae 38 b0 33  |....H..yx`...8.3|
		     000000d0  8d 77 02 03 01 00 01 a3  68 30 66 30 0e 06 03 55  |.w......h0f0...U|              000000d0  8d 77 02 03 01 00 01 a3  68 30 66 30 0e 06 03 55  |.w......h0f0...U|
		     000000e0  1d 0f 01 01 ff 04 04 03  02 02 a4 30 13 06 03 55  |...........0...U|              000000e0  1d 0f 01 01 ff 04 04 03  02 02 a4 30 13 06 03 55  |...........0...U|
		     000000f0  1d 25 04 0c 30 0a 06 08  2b 06 01 05 05 07 03 01  |.%..0...+.......|              000000f0  1d 25 04 0c 30 0a 06 08  2b 06 01 05 05 07 03 01  |.%..0...+.......|
		     00000100  30 0f 06 03 55 1d 13 01  01 ff 04 05 30 03 01 01  |0...U.......0...|              00000100  30 0f 06 03 55 1d 13 01  01 ff 04 05 30 03 01 01  |0...U.......0...|
		     00000110  ff 30 2e 06 03 55 1d 11  04 27 30 25 82 0b 65 78  |.0...U...'0%..ex|              00000110  ff 30 2e 06 03 55 1d 11  04 27 30 25 82 0b 65 78  |.0...U...'0%..ex|
		     00000120  61 6d 70 6c 65 2e 63 6f  6d 87 04 7f 00 00 01 87  |ample.com.......|              00000120  61 6d 70 6c 65 2e 63 6f  6d 87 04 7f 00 00 01 87  |ample.com.......|
		     00000130  10 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|              00000130  10 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
		     00000140  01 30 0d 06 09 2a 86 48  86 f7 0d 01 01 0b 05 00  |.0...*.H........|              00000140  01 30 0d 06 09 2a 86 48  86 f7 0d 01 01 0b 05 00  |.0...*.H........|
		     00000150  03 41 00 52 c5 44 e8 a3  27 cd af d9 6e 87 4b 97  |.A.R.D..'...n.K.|              00000150  03 41 00 52 c5 44 e8 a3  27 cd af d9 6e 87 4b 97  |.A.R.D..'...n.K.|
		     00000160  27 8c cd da 3b 10 cf f9  9d be 72 c3 23 b1 80 83  |'...;.....r.#...|              00000160  27 8c cd da 3b 10 cf f9  9d be 72 c3 23 b1 80 83  |'...;.....r.#...|
		     00000170  82 7f 43 8e 2c 7b 0f 4b  db 9e 18 b4 17 f4 71 06  |..C.,{.K......q.|              00000170  82 7f 43 8e 2c 7b 0f 4b  db 9e 18 b4 17 f4 71 06  |..C.,{.K......q.|
		     00000180  ff fc a3 4d 7b a1 35 c5  2e 80 4c 48 8c cf c0 23  |...M{.5...LH...#|              00000180  ff fc a3 4d 7b a1 35 c5  2e 80 4c 48 8c cf c0 23  |...M{.5...LH...#|
		     00000190  00 19 3d                                          |..=|                           00000190  00 19 3d                                          |..=|
		    },                                                                                          },
		    RawTBSCertificate: ([]uint8) (len=317 cap=401) {                                            RawTBSCertificate: ([]uint8) (len=317 cap=401) {
		     00000000  30 82 01 39 a0 03 02 01  02 02 11 00 aa 62 d9 69  |0..9.........b.i|              00000000  30 82 01 39 a0 03 02 01  02 02 11 00 aa 62 d9 69  |0..9.........b.i|
		     00000010  93 70 5a d5 8f 1a e5 e6  7e 58 0d 41 30 0d 06 09  |.pZ.....~X.A0...|              00000010  93 70 5a d5 8f 1a e5 e6  7e 58 0d 41 30 0d 06 09  |.pZ.....~X.A0...|
		     00000020  2a 86 48 86 f7 0d 01 01  0b 05 00 30 12 31 10 30  |*.H........0.1.0|              00000020  2a 86 48 86 f7 0d 01 01  0b 05 00 30 12 31 10 30  |*.H........0.1.0|
		     00000030  0e 06 03 55 04 0a 13 07  41 63 6d 65 20 43 6f 30  |...U....Acme Co0|              00000030  0e 06 03 55 04 0a 13 07  41 63 6d 65 20 43 6f 30  |...U....Acme Co0|
		     00000040  20 17 0d 37 30 30 31 30  31 30 30 30 30 30 30 5a  | ..700101000000Z|              00000040  20 17 0d 37 30 30 31 30  31 30 30 30 30 30 30 5a  | ..700101000000Z|
		     00000050  18 0f 32 30 38 34 30 31  32 39 31 36 30 30 30 30  |..20840129160000|              00000050  18 0f 32 30 38 34 30 31  32 39 31 36 30 30 30 30  |..20840129160000|
		     00000060  5a 30 12 31 10 30 0e 06  03 55 04 0a 13 07 41 63  |Z0.1.0...U....Ac|              00000060  5a 30 12 31 10 30 0e 06  03 55 04 0a 13 07 41 63  |Z0.1.0...U....Ac|
		     00000070  6d 65 20 43 6f 30 5c 30  0d 06 09 2a 86 48 86 f7  |me Co0\0...*.H..|              00000070  6d 65 20 43 6f 30 5c 30  0d 06 09 2a 86 48 86 f7  |me Co0\0...*.H..|
		     00000080  0d 01 01 01 05 00 03 4b  00 30 48 02 41 00 bd 7c  |.......K.0H.A..||              00000080  0d 01 01 01 05 00 03 4b  00 30 48 02 41 00 bd 7c  |.......K.0H.A..||
		     00000090  46 d1 b3 33 f7 b7 5e 06  af 8a 1a 85 5e f3 14 e6  |F..3..^.....^...|              00000090  46 d1 b3 33 f7 b7 5e 06  af 8a 1a 85 5e f3 14 e6  |F..3..^.....^...|
		     000000a0  c1 23 84 df 03 98 bb 62  79 4c 5d e2 e2 9d be cc  |.#.....byL].....|              000000a0  c1 23 84 df 03 98 bb 62  79 4c 5d e2 e2 9d be cc  |.#.....byL].....|
		     000000b0  9e f4 f8 6e 47 ce 27 31  3a ce 4b 67 e7 dc cf ed  |...nG.'1:.Kg....|              000000b0  9e f4 f8 6e 47 ce 27 31  3a ce 4b 67 e7 dc cf ed  |...nG.'1:.Kg....|
		     000000c0  48 87 b3 79 78 60 eb aa  ae 38 b0 33 8d 77 02 03  |H..yx`...8.3.w..|              000000c0  48 87 b3 79 78 60 eb aa  ae 38 b0 33 8d 77 02 03  |H..yx`...8.3.w..|
		     000000d0  01 00 01 a3 68 30 66 30  0e 06 03 55 1d 0f 01 01  |....h0f0...U....|              000000d0  01 00 01 a3 68 30 66 30  0e 06 03 55 1d 0f 01 01  |....h0f0...U....|
		     000000e0  ff 04 04 03 02 02 a4 30  13 06 03 55 1d 25 04 0c  |.......0...U.%..|              000000e0  ff 04 04 03 02 02 a4 30  13 06 03 55 1d 25 04 0c  |.......0...U.%..|
		     000000f0  30 0a 06 08 2b 06 01 05  05 07 03 01 30 0f 06 03  |0...+.......0...|              000000f0  30 0a 06 08 2b 06 01 05  05 07 03 01 30 0f 06 03  |0...+.......0...|
		     00000100  55 1d 13 01 01 ff 04 05  30 03 01 01 ff 30 2e 06  |U.......0....0..|              00000100  55 1d 13 01 01 ff 04 05  30 03 01 01 ff 30 2e 06  |U.......0....0..|
		     00000110  03 55 1d 11 04 27 30 25  82 0b 65 78 61 6d 70 6c  |.U...'0%..exampl|              00000110  03 55 1d 11 04 27 30 25  82 0b 65 78 61 6d 70 6c  |.U...'0%..exampl|
		     00000120  65 2e 63 6f 6d 87 04 7f  00 00 01 87 10 00 00 00  |e.com...........|              00000120  65 2e 63 6f 6d 87 04 7f  00 00 01 87 10 00 00 00  |e.com...........|
		     00000130  00 00 00 00 00 00 00 00  00 00 00 00 01           |.............|                 00000130  00 00 00 00 00 00 00 00  00 00 00 00 01           |.............|
		    },                                                                                          },
		    RawSubjectPublicKeyInfo: ([]uint8) (len=94 cap=284) {                                       RawSubjectPublicKeyInfo: ([]uint8) (len=94 cap=284) {
		     00000000  30 5c 30 0d 06 09 2a 86  48 86 f7 0d 01 01 01 05  |0\0...*.H.......|              00000000  30 5c 30 0d 06 09 2a 86  48 86 f7 0d 01 01 01 05  |0\0...*.H.......|
		     00000010  00 03 4b 00 30 48 02 41  00 bd 7c 46 d1 b3 33 f7  |..K.0H.A..|F..3.|              00000010  00 03 4b 00 30 48 02 41  00 bd 7c 46 d1 b3 33 f7  |..K.0H.A..|F..3.|
		     00000020  b7 5e 06 af 8a 1a 85 5e  f3 14 e6 c1 23 84 df 03  |.^.....^....#...|              00000020  b7 5e 06 af 8a 1a 85 5e  f3 14 e6 c1 23 84 df 03  |.^.....^....#...|
		     00000030  98 bb 62 79 4c 5d e2 e2  9d be cc 9e f4 f8 6e 47  |..byL]........nG|              00000030  98 bb 62 79 4c 5d e2 e2  9d be cc 9e f4 f8 6e 47  |..byL]........nG|
		     00000040  ce 27 31 3a ce 4b 67 e7  dc cf ed 48 87 b3 79 78  |.'1:.Kg....H..yx|              00000040  ce 27 31 3a ce 4b 67 e7  dc cf ed 48 87 b3 79 78  |.'1:.Kg....H..yx|
		     00000050  60 eb aa ae 38 b0 33 8d  77 02 03 01 00 01        |`...8.3.w.....|                00000050  60 eb aa ae 38 b0 33 8d  77 02 03 01 00 01        |`...8.3.w.....|
		    },                                                                                          },
		    RawSubject: ([]uint8) (len=20 cap=304) {                                                    RawSubject: ([]uint8) (len=20 cap=304) {
		     00000000  30 12 31 10 30 0e 06 03  55 04 0a 13 07 41 63 6d  |0.1.0...U....Acm|              00000000  30 12 31 10 30 0e 06 03  55 04 0a 13 07 41 63 6d  |0.1.0...U....Acm|
		     00000010  65 20 43 6f                                       |e Co|                          00000010  65 20 43 6f                                       |e Co|
		    },                                                                                          },
		    RawIssuer: ([]uint8) (len=20 cap=358) {                                                     RawIssuer: ([]uint8) (len=20 cap=358) {
		     00000000  30 12 31 10 30 0e 06 03  55 04 0a 13 07 41 63 6d  |0.1.0...U....Acm|              00000000  30 12 31 10 30 0e 06 03  55 04 0a 13 07 41 63 6d  |0.1.0...U....Acm|
		     00000010  65 20 43 6f                                       |e Co|                          00000010  65 20 43 6f                                       |e Co|
		    },                                                                                          },
		    Signature: ([]uint8) (len=64 cap=66) {                                                      Signature: ([]uint8) (len=64 cap=66) {
		     00000000  52 c5 44 e8 a3 27 cd af  d9 6e 87 4b 97 27 8c cd  |R.D..'...n.K.'..|              00000000  52 c5 44 e8 a3 27 cd af  d9 6e 87 4b 97 27 8c cd  |R.D..'...n.K.'..|
		     00000010  da 3b 10 cf f9 9d be 72  c3 23 b1 80 83 82 7f 43  |.;.....r.#.....C|              00000010  da 3b 10 cf f9 9d be 72  c3 23 b1 80 83 82 7f 43  |.;.....r.#.....C|
		     00000020  8e 2c 7b 0f 4b db 9e 18  b4 17 f4 71 06 ff fc a3  |.,{.K......q....|              00000020  8e 2c 7b 0f 4b db 9e 18  b4 17 f4 71 06 ff fc a3  |.,{.K......q....|
		     00000030  4d 7b a1 35 c5 2e 80 4c  48 8c cf c0 23 00 19 3d  |M{.5...LH...#..=|              00000030  4d 7b a1 35 c5 2e 80 4c  48 8c cf c0 23 00 19 3d  |M{.5...LH...#..=|
		    },                                                                                          },
		    SignatureAlgorithm: (x509.SignatureAlgorithm) 4,                                            SignatureAlgorithm: (x509.SignatureAlgorithm) 4,
		    PublicKeyAlgorithm: (x509.PublicKeyAlgorithm) 1,                                            PublicKeyAlgorithm: (x509.PublicKeyAlgorithm) 1,
		    PublicKey: (*rsa.PublicKey)(0xc42015c800)({                                                 PublicKey: (*rsa.PublicKey)(0xc42015c800)({
		     N: (*big.Int)(0xc42013d720)({                                                               N: (*big.Int)(0xc42013d720)({
		      neg: (bool) false,                                                                          neg: (bool) false,
		      abs: (big.nat) (len=8 cap=13) {                                                             abs: (big.nat) (len=8 cap=13) {
		       (big.Word) 0xebaaae38b0338d77,                                                              (big.Word) 0xebaaae38b0338d77,
		       (big.Word) 0xcfed4887b3797860,                                                              (big.Word) 0xcfed4887b3797860,
		       (big.Word) 0x27313ace4b67e7dc,                                                              (big.Word) 0x27313ace4b67e7dc,
		       (big.Word) 0xbecc9ef4f86e47ce,                                                              (big.Word) 0xbecc9ef4f86e47ce,
		       (big.Word) 0xbb62794c5de2e29d,                                                              (big.Word) 0xbb62794c5de2e29d,
		       (big.Word) 0x14e6c12384df0398,                                                              (big.Word) 0x14e6c12384df0398,
		       (big.Word) 0x5e06af8a1a855ef3,                                                              (big.Word) 0x5e06af8a1a855ef3,
		       (big.Word) 0xbd7c46d1b333f7b7                                                               (big.Word) 0xbd7c46d1b333f7b7
		      }                                                                                           }
		     }),                                                                                         }),
		     E: (int) 65537                                                                              E: (int) 65537
		    }),                                                                                         }),
		    Version: (int) 3,                                                                           Version: (int) 3,
		    SerialNumber: (*big.Int)(0xc42013d240)({                                                    SerialNumber: (*big.Int)(0xc42013d240)({
		     neg: (bool) false,                                                                          neg: (bool) false,
		     abs: (big.nat) (len=2 cap=7) {                                                              abs: (big.nat) (len=2 cap=7) {
		      (big.Word) 0x8f1ae5e67e580d41,                                                              (big.Word) 0x8f1ae5e67e580d41,
		      (big.Word) 0xaa62d96993705ad5                                                               (big.Word) 0xaa62d96993705ad5
		     }                                                                                           }
		    }),                                                                                         }),
		    Issuer: (pkix.Name) {                                                                       Issuer: (pkix.Name) {
		     Country: ([]string) <nil>,                                                                  Country: ([]string) <nil>,
		     Organization: ([]string) (len=1 cap=1) {                                                    Organization: ([]string) (len=1 cap=1) {
		      (string) (len=7) "Acme Co"                                                                  (string) (len=7) "Acme Co"
		     },                                                                                          },
		     OrganizationalUnit: ([]string) <nil>,                                                       OrganizationalUnit: ([]string) <nil>,
		     Locality: ([]string) <nil>,                                                                 Locality: ([]string) <nil>,
		     Province: ([]string) <nil>,                                                                 Province: ([]string) <nil>,
		     StreetAddress: ([]string) <nil>,                                                            StreetAddress: ([]string) <nil>,
		     PostalCode: ([]string) <nil>,                                                               PostalCode: ([]string) <nil>,
		     SerialNumber: (string) "",                                                                  SerialNumber: (string) "",
		     CommonName: (string) "",                                                                    CommonName: (string) "",
		     Names: ([]pkix.AttributeTypeAndValue) (len=1 cap=1) {                                       Names: ([]pkix.AttributeTypeAndValue) (len=1 cap=1) {
		      (pkix.AttributeTypeAndValue) {                                                              (pkix.AttributeTypeAndValue) {
		       Type: (asn1.ObjectIdentifier) (len=4 cap=4) {                                               Type: (asn1.ObjectIdentifier) (len=4 cap=4) {
		        (int) 2,                                                                                    (int) 2,
		        (int) 5,                                                                                    (int) 5,
		        (int) 4,                                                                                    (int) 4,
		        (int) 10                                                                                    (int) 10
		       },                                                                                          },
		       Value: (string) (len=7) "Acme Co"                                                           Value: (string) (len=7) "Acme Co"
		      }                                                                                           }
		     },                                                                                          },
		     ExtraNames: ([]pkix.AttributeTypeAndValue) <nil>                                            ExtraNames: ([]pkix.AttributeTypeAndValue) <nil>
		    },                                                                                          },
		    Subject: (pkix.Name) {                                                                      Subject: (pkix.Name) {
		     Country: ([]string) <nil>,                                                                  Country: ([]string) <nil>,
		     Organization: ([]string) (len=1 cap=1) {                                                    Organization: ([]string) (len=1 cap=1) {
		      (string) (len=7) "Acme Co"                                                                  (string) (len=7) "Acme Co"
		     },                                                                                          },
		     OrganizationalUnit: ([]string) <nil>,                                                       OrganizationalUnit: ([]string) <nil>,
		     Locality: ([]string) <nil>,                                                                 Locality: ([]string) <nil>,
		     Province: ([]string) <nil>,                                                                 Province: ([]string) <nil>,
		     StreetAddress: ([]string) <nil>,                                                            StreetAddress: ([]string) <nil>,
		     PostalCode: ([]string) <nil>,                                                               PostalCode: ([]string) <nil>,
		     SerialNumber: (string) "",                                                                  SerialNumber: (string) "",
		     CommonName: (string) "",                                                                    CommonName: (string) "",
		     Names: ([]pkix.AttributeTypeAndValue) (len=1 cap=1) {                                       Names: ([]pkix.AttributeTypeAndValue) (len=1 cap=1) {
		      (pkix.AttributeTypeAndValue) {                                                              (pkix.AttributeTypeAndValue) {
		       Type: (asn1.ObjectIdentifier) (len=4 cap=4) {                                               Type: (asn1.ObjectIdentifier) (len=4 cap=4) {
		        (int) 2,                                                                                    (int) 2,
		        (int) 5,                                                                                    (int) 5,
		        (int) 4,                                                                                    (int) 4,
		        (int) 10                                                                                    (int) 10
		       },                                                                                          },
		       Value: (string) (len=7) "Acme Co"                                                           Value: (string) (len=7) "Acme Co"
		      }                                                                                           }
		     },                                                                                          },
		     ExtraNames: ([]pkix.AttributeTypeAndValue) <nil>                                            ExtraNames: ([]pkix.AttributeTypeAndValue) <nil>
		    },                                                                                          },
		    NotBefore: (time.Time) {                                                                    NotBefore: (time.Time) {
		     sec: (int64) 62135596800,                                                                   sec: (int64) 62135596800,
		     nsec: (int32) 0,                                                                            nsec: (int32) 0,
		     loc: (*time.Location)(<nil>)                                                                loc: (*time.Location)(<nil>)
		    },                                                                                          },
		    NotAfter: (time.Time) {                                                                     NotAfter: (time.Time) {
		     sec: (int64) 65735596800,                                                                   sec: (int64) 65735596800,
		     nsec: (int32) 0,                                                                            nsec: (int32) 0,
		     loc: (*time.Location)(<nil>)                                                                loc: (*time.Location)(<nil>)
		    },                                                                                          },
		    KeyUsage: (x509.KeyUsage) 37,                                                               KeyUsage: (x509.KeyUsage) 37,
		    Extensions: ([]pkix.Extension) (len=4 cap=4) {                                              Extensions: ([]pkix.Extension) (len=4 cap=4) {
		     (pkix.Extension) {                                                                          (pkix.Extension) {
		      Id: (asn1.ObjectIdentifier) (len=4 cap=4) {                                                 Id: (asn1.ObjectIdentifier) (len=4 cap=4) {
		       (int) 2,                                                                                    (int) 2,
		       (int) 5,                                                                                    (int) 5,
		       (int) 29,                                                                                   (int) 29,
		       (int) 15                                                                                    (int) 15
		      },                                                                                          },
		      Critical: (bool) true,                                                                      Critical: (bool) true,
		      Value: ([]uint8) (len=4 cap=4) {                                                            Value: ([]uint8) (len=4 cap=4) {
		       00000000  03 02 02 a4                                       |....|                          00000000  03 02 02 a4                                       |....|
		      }                                                                                           }
		     },                                                                                          },
		     (pkix.Extension) {                                                                          (pkix.Extension) {
		      Id: (asn1.ObjectIdentifier) (len=4 cap=4) {                                                 Id: (asn1.ObjectIdentifier) (len=4 cap=4) {
		       (int) 2,                                                                                    (int) 2,
		       (int) 5,                                                                                    (int) 5,
		       (int) 29,                                                                                   (int) 29,
		       (int) 37                                                                                    (int) 37
		      },                                                                                          },
		      Critical: (bool) false,                                                                     Critical: (bool) false,
		      Value: ([]uint8) (len=12 cap=12) {                                                          Value: ([]uint8) (len=12 cap=12) {
		       00000000  30 0a 06 08 2b 06 01 05  05 07 03 01              |0...+.......|                  00000000  30 0a 06 08 2b 06 01 05  05 07 03 01              |0...+.......|
		      }                                                                                           }
		     },                                                                                          },
		     (pkix.Extension) {                                                                          (pkix.Extension) {
		      Id: (asn1.ObjectIdentifier) (len=4 cap=4) {                                                 Id: (asn1.ObjectIdentifier) (len=4 cap=4) {
		       (int) 2,                                                                                    (int) 2,
		       (int) 5,                                                                                    (int) 5,
		       (int) 29,                                                                                   (int) 29,
		       (int) 19                                                                                    (int) 19
		      },                                                                                          },
		      Critical: (bool) true,                                                                      Critical: (bool) true,
		      Value: ([]uint8) (len=5 cap=5) {                                                            Value: ([]uint8) (len=5 cap=5) {
		       00000000  30 03 01 01 ff                                    |0....|                         00000000  30 03 01 01 ff                                    |0....|
		      }                                                                                           }
		     },                                                                                          },
		     (pkix.Extension) {                                                                          (pkix.Extension) {
		      Id: (asn1.ObjectIdentifier) (len=4 cap=4) {                                                 Id: (asn1.ObjectIdentifier) (len=4 cap=4) {
		       (int) 2,                                                                                    (int) 2,
		       (int) 5,                                                                                    (int) 5,
		       (int) 29,                                                                                   (int) 29,
		       (int) 17                                                                                    (int) 17
		      },                                                                                          },
		      Critical: (bool) false,                                                                     Critical: (bool) false,
		      Value: ([]uint8) (len=39 cap=39) {                                                          Value: ([]uint8) (len=39 cap=39) {
		       00000000  30 25 82 0b 65 78 61 6d  70 6c 65 2e 63 6f 6d 87  |0%..example.com.|              00000000  30 25 82 0b 65 78 61 6d  70 6c 65 2e 63 6f 6d 87  |0%..example.com.|
		       00000010  04 7f 00 00 01 87 10 00  00 00 00 00 00 00 00 00  |................|              00000010  04 7f 00 00 01 87 10 00  00 00 00 00 00 00 00 00  |................|
		       00000020  00 00 00 00 00 00 01                              |.......|                       00000020  00 00 00 00 00 00 01                              |.......|
		      }                                                                                           }
		     }                                                                                           }
		    },                                                                                          },
		    ExtraExtensions: ([]pkix.Extension) <nil>,                                                  ExtraExtensions: ([]pkix.Extension) <nil>,
		    UnhandledCriticalExtensions: ([]asn1.ObjectIdentifier) <nil>,                               UnhandledCriticalExtensions: ([]asn1.ObjectIdentifier) <nil>,
		    ExtKeyUsage: ([]x509.ExtKeyUsage) (len=1 cap=1) {                                           ExtKeyUsage: ([]x509.ExtKeyUsage) (len=1 cap=1) {
		     (x509.ExtKeyUsage) 1                                                                        (x509.ExtKeyUsage) 1
		    },                                                                                          },
		    UnknownExtKeyUsage: ([]asn1.ObjectIdentifier) <nil>,                                        UnknownExtKeyUsage: ([]asn1.ObjectIdentifier) <nil>,
		    BasicConstraintsValid: (bool) true,                                                         BasicConstraintsValid: (bool) true,
		    IsCA: (bool) true,                                                                          IsCA: (bool) true,
		    MaxPathLen: (int) -1,                                                                       MaxPathLen: (int) -1,
		    MaxPathLenZero: (bool) false,                                                               MaxPathLenZero: (bool) false,
		    SubjectKeyId: ([]uint8) <nil>,                                                              SubjectKeyId: ([]uint8) <nil>,
		    AuthorityKeyId: ([]uint8) <nil>,                                                            AuthorityKeyId: ([]uint8) <nil>,
		    OCSPServer: ([]string) <nil>,                                                               OCSPServer: ([]string) <nil>,
		    IssuingCertificateURL: ([]string) <nil>,                                                    IssuingCertificateURL: ([]string) <nil>,
		    DNSNames: ([]string) (len=1 cap=1) {                                                        DNSNames: ([]string) (len=1 cap=1) {
		     (string) (len=11) "example.com"                                                             (string) (len=11) "example.com"
		    },                                                                                          },
		    EmailAddresses: ([]string) <nil>,                                                           EmailAddresses: ([]string) <nil>,
		    IPAddresses: ([]net.IP) (len=2 cap=2) {                                                     IPAddresses: ([]net.IP) (len=2 cap=2) {
		     (net.IP) (len=4 cap=22) {                                                                   (net.IP) (len=4 cap=22) {
		      00000000  7f 00 00 01                                       |....|                          00000000  7f 00 00 01                                       |....|
		     },                                                                                          },
		     (net.IP) (len=16 cap=16) {                                                                  (net.IP) (len=16 cap=16) {
		      00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 01  |................|              00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 01  |................|
		     }                                                                                           }
		    },                                                                                          },
		    PermittedDNSDomainsCritical: (bool) false,                                                  PermittedDNSDomainsCritical: (bool) false,
		    PermittedDNSDomains: ([]string) <nil>,                                                      PermittedDNSDomains: ([]string) <nil>,
		    CRLDistributionPoints: ([]string) <nil>,                                                    CRLDistributionPoints: ([]string) <nil>,
		    PolicyIdentifiers: ([]asn1.ObjectIdentifier) <nil>                                          PolicyIdentifiers: ([]asn1.ObjectIdentifier) <nil>
		   })                                                                                          })
		  }                                                                                           }
		 }),                                                                                         }),
		 NextProtos: ([]string) <nil>,                                                               NextProtos: ([]string) <nil>,
		 ServerName: (string) "",                                                                    ServerName: (string) "",
		 ClientAuth: (tls.ClientAuthType) 0,                                                         ClientAuth: (tls.ClientAuthType) 0,
		 ClientCAs: (*x509.CertPool)(<nil>),                                                         ClientCAs: (*x509.CertPool)(<nil>),
		 InsecureSkipVerify: (bool) false,                                                           InsecureSkipVerify: (bool) false,
		 CipherSuites: ([]uint16) <nil>,                                                             CipherSuites: ([]uint16) <nil>,
		 PreferServerCipherSuites: (bool) false,                                                     PreferServerCipherSuites: (bool) false,
		 SessionTicketsDisabled: (bool) false,                                                       SessionTicketsDisabled: (bool) false,
		 SessionTicketKey: ([32]uint8) (len=32 cap=32) {                                             SessionTicketKey: ([32]uint8) (len=32 cap=32) {
		  00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|              00000000  00 2e 33 15 c1 52 97 e3  0a 7f 82 57 42 51 0f 69  |..3..R.....WBQ.i|
		  00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|              00000010  6d 34 09 24 13 a0 7d c9  2a 0d 28 bb 0a 53 f6 33  |m4.$..}.*.(..S.3|
		 },                                                                                          },
		 ClientSessionCache: (tls.ClientSessionCache) <nil>,                                         ClientSessionCache: (tls.ClientSessionCache) <nil>,
		 MinVersion: (uint16) 0,                                                                     MinVersion: (uint16) 0,
		 MaxVersion: (uint16) 0,                                                                     MaxVersion: (uint16) 0,
		 CurvePreferences: ([]tls.CurveID) <nil>,                                                    CurvePreferences: ([]tls.CurveID) <nil>,
		 DynamicRecordSizingDisabled: (bool) false,                                                  DynamicRecordSizingDisabled: (bool) false,
		 Renegotiation: (tls.RenegotiationSupport) 0,                                                Renegotiation: (tls.RenegotiationSupport) 0,
		 KeyLogWriter: (io.Writer) <nil>,                                                            KeyLogWriter: (io.Writer) <nil>,
		 serverInitOnce: (sync.Once) {                                                               serverInitOnce: (sync.Once) {
		  m: (sync.Mutex) {                                                                           m: (sync.Mutex) {
		   state: (int32) 0,                                                                           state: (int32) 0,
		   sema: (uint32) 0                                                                            sema: (uint32) 0
		  },                                                                                          },
		  done: (uint32) 0                                                                            done: (uint32) 1
		 },                                                                                          },
		 mutex: (sync.RWMutex) {                                                                     mutex: (sync.RWMutex) {
		  w: (sync.Mutex) {                                                                           w: (sync.Mutex) {
		   state: (int32) 0,                                                                           state: (int32) 0,
		   sema: (uint32) 0                                                                            sema: (uint32) 0
		  },                                                                                          },
		  writerSem: (uint32) 0,                                                                      writerSem: (uint32) 0,
		  readerSem: (uint32) 0,                                                                      readerSem: (uint32) 0,
		  readerCount: (int32) 0,                                                                     readerCount: (int32) 0,
		  readerWait: (int32) 0                                                                       readerWait: (int32) 0
		 },                                                                                          },
		 sessionTicketKeys: ([]tls.ticketKey) <nil>,                                                 sessionTicketKeys: ([]tls.ticketKey) (len=1 cap=1) {
		 originalConfig: (*tls.Config)(<nil>)                                                         (tls.ticketKey) {
		})                                                                                             keyName: ([16]uint8) (len=16 cap=16) {
		                                                                                                00000000  7e 98 a0 9f 9e bb a5 b1  c6 23 55 05 dd 47 fe 7a  |~........#U..G.z|
		                                                                                               },
		                                                                                               aesKey: ([16]uint8) (len=16 cap=16) {
		                                                                                                00000000  64 e4 92 27 59 d8 6a ab  e8 c2 c6 c9 eb 7b 2f 00  |d..'Y.j......{/.|
		                                                                                               },
		                                                                                               hmacKey: ([16]uint8) (len=16 cap=16) {
		                                                                                                00000000  ff f7 35 3b 5c 93 5b 6f  10 c3 b0 7b 9e 18 58 41  |..5;\.[o...{..XA|
		                                                                                               }
		                                                                                              }
[ERROR] PID 7: hack/test-go.sh:237: `go test ${gotest_flags} ${test_packages}` exited with status 1.

It looks like some behavior changed between golang-1.7 and golang-1.8 that is now causing the SessionTicketKey to be mutated.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liggitt thanks, what is the proper way to backport that? cherry-pick and replace the Clone calls with utilnet.CloneTLSConfig? Or manually change with a drop commit?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrgh, that wouldn't work, since utilnet.CloneTLSConfig wouldn't trigger the call to serverInit in tlsConfigCopy.

@liggitt I'm not sure how to make this test work for both 1.7 and 1.8, unless we selectively use Clone for golang 1.8 and utilnet.CloneTLSConfig for 1.7, but I'm not sure if that leaves us open for unexpected behavior under golang 1.8.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you'd need to implement CloneTLSConfig using Clone for go1.8

@detiber
Copy link
Author

detiber commented Jun 27, 2017

@liggitt updated to use Clone() in CloneTLSConfig() when golang version is 1.8.

I'm not thrilled with using a partial text comparison against runtime.Version(), I suspect there is a better way to handle this through an existing library or through the reflection api. I'm happy to update based on some guidance on what is considered the appropriate way of handling these types of issues.


if strings.HasPrefix(runtime.Version(), "go1.8") {
// make a copy of the config
tlsConfigCopy := cfg.Clone()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, this won't compile in go1.7... you need to do conditional compilation with two new files:

clone.go:

// +build go1.8
func CloneTLSConfig(cfg *tls.Config) *tls.Config {
  cfgCopy := cfg.Clone()
  cfgCopy.Clone()
  return cfgCopy
}

clone_17.go:

// +build !go1.8

func CloneTLSConfig(cfg *tls.Config) *tls.Config {
   ... existing impl ...
}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liggitt thanks, makes sense. I'll refactor in the morning.

Use Clone() for the implementation of CloneTLSConfig when golang version
is 1.8, otherwise continue to use existing implementation. This avoids
issues related Clone() calling serverInit on the source object if it was
not already previously run.

This ensures that CloneTLSConfig is able to be used consistently for
both golang 1.7 and golang 1.8.

This commit can be dropped when rebased on k8s 1.7, which will require
golang 1.8. Alternatively, this commit can be dropped if we move to
golang 1.8 exclusively and the related upstream Clone() changes are
backported.
@stevekuznetsov
Copy link
Contributor

/cc @deads2k how close are we to moving to 1.8 ?

@detiber
Copy link
Author

detiber commented Jun 27, 2017

@stevekuznetsov upstream is requiring 1.8 only for k8s 1.7. With this PR, I am down to only 2 unit test failures with golang 1.8, but I still have to try and get through integration and e2e tests.

@deads2k
Copy link
Contributor

deads2k commented Jun 27, 2017

/cc @deads2k how close are we to moving to 1.8 ?

A few weeks. Whenever we branch 3.6, we should switch.

@detiber
Copy link
Author

detiber commented Jun 27, 2017

[test] again

@detiber
Copy link
Author

detiber commented Jun 27, 2017

@liggitt test failure appears to be a flake, it failed to build the RPMs, while the other jobs were able to proceed past the point without issue. I'm not sure there is enough debug output to tell why the RPM build even failed.

@detiber
Copy link
Author

detiber commented Jun 27, 2017

@deads2k @stevekuznetsov: outside the 1.8 test fixes in this branch, I'm seeing these two unit test failures:

--- FAIL: TestParseRepository (0.04s)
	git_test.go:101: ParseRepository returned err: parse mbalazs@build.ulx.hu:/var/git/eap-ulx.git: first path segment in URL cannot contain colon
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x229c48]
 
goroutine 7 [running]:
testing.tRunner.func1(0xc42008b2b0)
	/usr/lib/golang/src/testing/testing.go:622 +0x25c
panic(0x274e40, 0x429700)
	/usr/lib/golang/src/runtime/panic.go:489 +0x284
github.com/openshift/origin/pkg/generate/git.TestParseRepository(0xc42008b2b0)
 
--- FAIL: TestConstantMaps (0.08s)
	crypto_test.go:35: discovered cipher tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 not in ciphers map
	crypto_test.go:35: discovered cipher tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 not in ciphers map
	crypto_test.go:35: discovered cipher tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 not in ciphers map
	crypto_test.go:35: discovered cipher tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 not in ciphers map
	crypto_test.go:35: discovered cipher tls.TLS_RSA_WITH_AES_128_CBC_SHA256 not in ciphers map
Building CA...
Building intermediate 1...
Building intermediate 2...
Building server...
Building client...
FAIL
coverage: 12.7% of statements
FAIL	github.com/openshift/origin/pkg/cmd/server/crypto	13.380s

Resolving the crypto one introduces a catch 22, unless we used build tags there, but I didn't see a point, and have been skipping it until we move to golang-1.8.

I created a tracking issue for the other failure here: #14831
That one will require more invasive changes, which would be better handled post branch..

@detiber
Copy link
Author

detiber commented Jun 28, 2017

@liggitt @smarterclayton bump, I'd like to see about getting this into 3.6, since the changes are either test-related or only affect builds on golang-1.8. With these changes, we are down to only 2 unit test failures on golang-1.8.

@deads2k
Copy link
Contributor

deads2k commented Jun 28, 2017

lgtm [merge]

@openshift-bot
Copy link
Contributor

openshift-bot commented Jun 28, 2017

continuous-integration/openshift-jenkins/merge Waiting: You are in the build queue at position: 5

@openshift-bot
Copy link
Contributor

Evaluated for origin merge up to 868a91a

@deads2k
Copy link
Contributor

deads2k commented Jun 28, 2017

re[test]

@openshift-bot
Copy link
Contributor

Evaluated for origin test up to 868a91a

@openshift-bot
Copy link
Contributor

continuous-integration/openshift-jenkins/test FAILURE (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/2769/) (Base Commit: 478cfe2) (PR Branch Commit: 868a91a)

@smarterclayton smarterclayton merged commit 3aeaab3 into openshift:master Jun 28, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants