(stress) server: TestInitHandshakeWrongToken failed #61538

cockroach-teamcity · 2021-03-05T08:35:00Z

(server).TestInitHandshakeWrongToken failed on master@b703e663da8ededaee2e28fc39a24e3880ae54cf:

  github.com/cockroachdb/cockroach/pkg/server.initHandshakeHelper()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:516 +0x93b
  github.com/cockroachdb/cockroach/pkg/server.InitHandshake.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:655 +0x1a6
  github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:140 +0xe2
  github.com/cockroachdb/cockroach/pkg/server.InitHandshake()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:653 +0x17b
  github.com/cockroachdb/cockroach/pkg/server.TestInitHandshakeWrongToken.func7()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake_test.go:214 +0x17d

Goroutine 813 (running) created at:
  net/http.http2bodyWriterState.scheduleBodyWrite()
      /usr/local/go/src/net/http/h2_bundle.go:9085 +0x12d
  net/http.(*http2ClientConn).roundTrip()
      /usr/local/go/src/net/http/h2_bundle.go:7567 +0x953
  net/http.(*http2Transport).RoundTripOpt()
      /usr/local/go/src/net/http/h2_bundle.go:6948 +0x276
  net/http.(*http2Transport).RoundTrip()
      /usr/local/go/src/net/http/h2_bundle.go:6909 +0x4b
  net/http.(*Transport).roundTrip()
      /usr/local/go/src/net/http/transport.go:580 +0xb43
  net/http.(*Transport).RoundTrip()
      /usr/local/go/src/net/http/roundtrip.go:17 +0x46
  net/http.send()
      /usr/local/go/src/net/http/client.go:252 +0x6da
  net/http.(*Client).send()
      /usr/local/go/src/net/http/client.go:176 +0x1d5
  net/http.(*Client).do()
      /usr/local/go/src/net/http/client.go:718 +0x2d7
  net/http.(*Client).Do()
      /usr/local/go/src/net/http/client.go:586 +0x20c
  net/http.(*Client).Post()
      /usr/local/go/src/net/http/client.go:846 +0x231
  github.com/cockroachdb/cockroach/pkg/server.(*tlsInitHandshaker).getPeerCACert()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:346 +0x404
  github.com/cockroachdb/cockroach/pkg/server.(*tlsInitHandshaker).runClient()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:387 +0x290
  github.com/cockroachdb/cockroach/pkg/server.initHandshakeHelper.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:521 +0x1c7
==================
    init_handshake_test.go:203: n1: trusted peer: 127.0.0.1:46001
    init_handshake_test.go:203: n1: trusted peer: 127.0.0.1:46001
    init_handshake_test.go:203: n3: trusted peer: 127.0.0.1:46705
E210305 08:30:40.808831 98982 1@server/init_handshake.go:543  [n1,init-tls-handshake] 2  HMAC error from client when connecting to peer: invalid HMAC signature
E210305 08:30:40.808957 98984 1@server/init_handshake.go:543  [n3,init-tls-handshake] 3  HMAC error from client when connecting to peer: invalid HMAC signature
    init_handshake_test.go:244: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestInitHandshakeWrongToken525656275
    testing.go:1038: race detected during execution of test
--- FAIL: TestInitHandshakeWrongToken (20.09s)

More

Parameters:

TAGS=
GOFLAGS=-race -parallel=4

make stressrace TESTS=TestInitHandshakeWrongToken PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

Epic: CRDB-6663

Jira issue: CRDB-3000

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2021-03-06T07:30:10Z

(server).TestInitHandshakeWrongToken failed on master@15a185606d5e80b47d9fdd0ed4f54cfe29c527c6:

      /usr/local/go/src/net/http/h2_bundle.go:7771 +0x761
  net/http.(*http2Transport).getBodyWriterState.func1()
      /usr/local/go/src/net/http/h2_bundle.go:9038 +0x13d

Goroutine 813 (running) created at:
  github.com/cockroachdb/cockroach/pkg/server.initHandshakeHelper()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:516 +0x93b
  github.com/cockroachdb/cockroach/pkg/server.InitHandshake.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:655 +0x1a6
  github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:140 +0xe2
  github.com/cockroachdb/cockroach/pkg/server.InitHandshake()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:653 +0x17b
  github.com/cockroachdb/cockroach/pkg/server.TestInitHandshakeWrongToken.func7()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake_test.go:214 +0x17d

Goroutine 233 (finished) created at:
  net/http.http2bodyWriterState.scheduleBodyWrite()
      /usr/local/go/src/net/http/h2_bundle.go:9085 +0x12d
  net/http.(*http2ClientConn).roundTrip()
      /usr/local/go/src/net/http/h2_bundle.go:7567 +0x953
  net/http.(*http2Transport).RoundTripOpt()
      /usr/local/go/src/net/http/h2_bundle.go:6948 +0x276
  net/http.(*http2Transport).RoundTrip()
      /usr/local/go/src/net/http/h2_bundle.go:6909 +0x4b
  net/http.(*Transport).roundTrip()
      /usr/local/go/src/net/http/transport.go:580 +0xb43
  net/http.(*Transport).RoundTrip()
      /usr/local/go/src/net/http/roundtrip.go:17 +0x46
  net/http.send()
      /usr/local/go/src/net/http/client.go:252 +0x6da
  net/http.(*Client).send()
      /usr/local/go/src/net/http/client.go:176 +0x1d5
  net/http.(*Client).do()
      /usr/local/go/src/net/http/client.go:718 +0x2d7
  net/http.(*Client).Do()
      /usr/local/go/src/net/http/client.go:586 +0x20c
  net/http.(*Client).Post()
      /usr/local/go/src/net/http/client.go:846 +0x231
  github.com/cockroachdb/cockroach/pkg/server.(*tlsInitHandshaker).getPeerCACert()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:346 +0x404
  github.com/cockroachdb/cockroach/pkg/server.(*tlsInitHandshaker).runClient()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:387 +0x290
  github.com/cockroachdb/cockroach/pkg/server.initHandshakeHelper.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:521 +0x1c7
==================
    init_handshake_test.go:244: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestInitHandshakeWrongToken832884491
    testing.go:1038: race detected during execution of test
--- FAIL: TestInitHandshakeWrongToken (20.10s)

More

Parameters:

TAGS=
GOFLAGS=-race -parallel=4

make stressrace TESTS=TestInitHandshakeWrongToken PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

knz · 2021-03-15T17:39:42Z

This seems to be a data race.

cockroach-teamcity · 2021-03-31T07:43:19Z

(server).TestInitHandshakeWrongToken failed on master@fe14895b103aae7f969ebf90763802629ad209c5:

=== RUN   TestInitHandshakeWrongToken
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestInitHandshakeWrongToken578819282
    test_log_scope.go:74: use -show-logs to present logs inline
    init_handshake_test.go:203: n3: waiting for handshake for 2 peers
    init_handshake_test.go:203: n3: trusted peer: 127.0.0.1:37483
    init_handshake_test.go:203: n3: trusted peer: 127.0.0.1:42849
    init_handshake_test.go:203: n3: generating cert bundle for cluster
    init_handshake_test.go:203: n1: waiting for handshake for 2 peers
    init_handshake_test.go:203: n2: waiting for handshake for 2 peers
    init_handshake_test.go:203: n1: trusted peer: 127.0.0.1:43293
E210331 07:41:41.274751 104604 1@server/init_handshake.go:541  [n2,init-tls-handshake] 1  HMAC error from client when connecting to peer: invalid HMAC signature
    init_handshake_test.go:203: n1: trusted peer: 127.0.0.1:43293
E210331 07:41:41.463056 104603 1@server/init_handshake.go:541  [n1,init-tls-handshake] 2  HMAC error from client when connecting to peer: invalid HMAC signature
    init_handshake_test.go:203: n3: sending cert bundle to peers
    init_handshake_test.go:243: 
        	Error Trace:	init_handshake_test.go:243
        	Error:      	Should be false
        	Test:       	TestInitHandshakeWrongToken
    panic.go:617: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestInitHandshakeWrongToken578819282
--- FAIL: TestInitHandshakeWrongToken (20.52s)

More

Parameters:

TAGS=
GOFLAGS=-parallel=4

make stressrace TESTS=TestInitHandshakeWrongToken PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

knz · 2021-03-31T11:14:30Z

@itsbilal if you want to help with this work in April, this issue could be a candidate.

knz · 2021-04-19T14:34:40Z

I tried to stress this locally with stress, and was unable to find a reproducible case after ~300 runs.

With stressrace also no repro after ~200 runs.

I am tempted to chuck it on #63836
and call it a day?

cockroach-teamcity · 2021-04-22T07:10:12Z

server.TestInitHandshakeWrongToken failed with artifacts on master @ a2b80cb4474f93ac66e59d567c3197dfd482a3be:

=== RUN   TestInitHandshakeWrongToken
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestInitHandshakeWrongToken468991216
    test_log_scope.go:74: use -show-logs to present logs inline
    init_handshake_test.go:203: n1: waiting for handshake for 2 peers
    init_handshake_test.go:203: n3: waiting for handshake for 2 peers
    init_handshake_test.go:203: n3: trusted peer: 127.0.0.1:35029
    init_handshake_test.go:203: n3: trusted peer: 127.0.0.1:46011
    init_handshake_test.go:203: n3: generating cert bundle for cluster
    init_handshake_test.go:203: n2: waiting for handshake for 2 peers
    init_handshake_test.go:203: n1: trusted peer: 127.0.0.1:35649
E210422 07:08:43.416257 110834 1@server/init_handshake.go:541  [n2,init-tls-handshake] 1  HMAC error from client when connecting to peer: invalid HMAC signature
    init_handshake_test.go:203: n1: trusted peer: 127.0.0.1:35649
E210422 07:08:43.475182 110769 1@server/init_handshake.go:541  [n1,init-tls-handshake] 2  HMAC error from client when connecting to peer: invalid HMAC signature
    init_handshake_test.go:203: n3: sending cert bundle to peers
    init_handshake_test.go:243: 
        	Error Trace:	init_handshake_test.go:243
        	Error:      	Should be false
        	Test:       	TestInitHandshakeWrongToken
    panic.go:617: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestInitHandshakeWrongToken468991216
--- FAIL: TestInitHandshakeWrongToken (20.52s)

Reproduce

To reproduce, try:

make stressrace TESTS=TestInitHandshakeWrongToken PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

GOFLAGS=-parallel=4

/cc @itsbilal _{This test on roachdash | Improve this report!}

itsbilal · 2021-04-22T16:36:39Z

Looks like it failed again, so it can't be #63836 right? I'll take a look as this seems pretty concerning.

knz · 2021-04-23T12:40:25Z

yep, thanks for looking

cockroach-teamcity · 2022-12-23T09:22:07Z

server.TestInitHandshakeWrongToken failed with artifacts on master @ 4232883add85a151c423c45904ac4096d04656c5:

=== RUN   TestInitHandshakeWrongToken
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestInitHandshakeWrongToken2354720111
    test_log_scope.go:79: use -show-logs to present logs inline
    init_handshake_test.go:203: n1: waiting for handshake for 2 peers
    init_handshake_test.go:203: n2: waiting for handshake for 2 peers
    init_handshake_test.go:203: n3: waiting for handshake for 2 peers
    init_handshake_test.go:203: n3: trusted peer: 127.0.0.1:34335
    init_handshake_test.go:203: n3: trusted peer: 127.0.0.1:38399
    init_handshake_test.go:203: n3: generating cert bundle for cluster
    init_handshake_test.go:203: n1: trusted peer: 127.0.0.1:37569
    init_handshake_test.go:203: n3: sending cert bundle to peers
    init_handshake_test.go:243: 
        	Error Trace:	/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2572/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/server/server_test_/server_test.runfiles/com_github_cockroachdb_cockroach/pkg/server/init_handshake_test.go:243
        	Error:      	Should be false
        	Test:       	TestInitHandshakeWrongToken
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestInitHandshakeWrongToken2354720111
--- FAIL: TestInitHandshakeWrongToken (20.01s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

server: TestInitHandshakeWrongToken failed #79734 server: TestInitHandshakeWrongToken failed [C-test-failure O-robot T-server-and-security branch-release-22.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 5, 2021

cockroach-teamcity assigned itsbilal Mar 5, 2021

knz changed the title ~~server: TestInitHandshakeWrongToken failed~~ (stress) server: TestInitHandshakeWrongToken failed Mar 15, 2021

knz unassigned itsbilal May 10, 2021

knz mentioned this issue May 10, 2021

server: integrate the TLS auto-negotiation inside the crdb code #60632

Closed

25 tasks

jlinder added the T-server-and-security DB Server & Security label Jun 16, 2021

cockroach-teamcity mentioned this issue Apr 10, 2022

server: TestInitHandshakeWrongToken failed #79734

Open

cockroach-teamcity mentioned this issue Jan 15, 2023

server: TestInitHandshakeWrongToken failed #95263

Closed

cockroach-teamcity mentioned this issue Feb 26, 2023

server: TestInitHandshakeWrongToken failed #97692

Closed

williamkulju added the exception-requested Issue is a release-blocker you want to defer by exception label Jul 12, 2023

exalate-issue-sync bot assigned knz Aug 18, 2023

knz added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. and removed C-test-failure Broken test (automatically or manually discovered). labels Sep 13, 2023

celiala unassigned knz Nov 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(stress) server: TestInitHandshakeWrongToken failed #61538

(stress) server: TestInitHandshakeWrongToken failed #61538

cockroach-teamcity commented Mar 5, 2021 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Mar 6, 2021

knz commented Mar 15, 2021

cockroach-teamcity commented Mar 31, 2021

knz commented Mar 31, 2021

knz commented Apr 19, 2021

cockroach-teamcity commented Apr 22, 2021

itsbilal commented Apr 22, 2021

knz commented Apr 23, 2021

cockroach-teamcity commented Dec 23, 2022

(stress) server: TestInitHandshakeWrongToken failed #61538

(stress) server: TestInitHandshakeWrongToken failed #61538

Comments

cockroach-teamcity commented Mar 5, 2021 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Mar 6, 2021

knz commented Mar 15, 2021

cockroach-teamcity commented Mar 31, 2021

knz commented Mar 31, 2021

knz commented Apr 19, 2021

cockroach-teamcity commented Apr 22, 2021

itsbilal commented Apr 22, 2021

knz commented Apr 23, 2021

cockroach-teamcity commented Dec 23, 2022

cockroach-teamcity commented Mar 5, 2021 •

edited by cockroach-jira-scripts

Loading