Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

server: TestTelemetrySQLStatsIndependence failed #63844

Closed
cockroach-teamcity opened this issue Apr 19, 2021 · 10 comments
Closed

server: TestTelemetrySQLStatsIndependence failed #63844

cockroach-teamcity opened this issue Apr 19, 2021 · 10 comments
Assignees
@solongordon
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

(server).TestTelemetrySQLStatsIndependence failed on release-21.1@c07fe34fc84626d1fea17c3ccb13ed80bf2e7d2e:

        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x2a7
        crypto/tls.(*Conn).Handshake(0xc004e5a000, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9
        net/http.(*persistConn).addTLS.func2(0x0, 0xc004e5a000, 0x0, 0xc0019530e0)
        	/usr/local/go/src/net/http/transport.go:1512 +0x45
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x177
        Leaked goroutine: goroutine 157218 [IO wait]:
        internal/poll.runtime_pollWait(0x7f2aaa8d5f90, 0x72, 0x50c5de0)
        	/usr/local/go/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc0062ace18, 0x72, 0x50c5d00, 0x6eaa6e8, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc0062ace00, 0xc002d53440, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
        net.(*netFD).Read(0xc0062ace00, 0xc002d53440, 0x205, 0x205, 0x203001, 0x58, 0xc0004d9c00)
        	/usr/local/go/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc00061a878, 0xc002d53440, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/net/net.go:182 +0x8e
        crypto/tls.(*atLeastReader).Read(0xc004a06420, 0xc002d53440, 0x205, 0x205, 0xc002d53440, 0x0, 0xc0021b8688)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0x62
        bytes.(*Buffer).ReadFrom(0xc002d5bb00, 0x50ba060, 0xc004a06420, 0x426965, 0x3eb2d00, 0x43afc60)
        	/usr/local/go/src/bytes/buffer.go:204 +0xb1
        crypto/tls.(*Conn).readFromUntil(0xc002d5b880, 0x50c2720, 0xc00061a878, 0x5, 0xc00061a878, 0x10000042f6ec0)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
        crypto/tls.(*Conn).readRecordOrCCS(0xc002d5b880, 0xc0004fe200, 0x114, 0x120)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x115
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).readHandshake(0xc002d5b880, 0xc002b87616, 0xc002b87680, 0x10f, 0x180)
        	/usr/local/go/src/crypto/tls/conn.go:992 +0x6d
        crypto/tls.(*Conn).clientHandshake(0xc002d5b880, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x2a7
        crypto/tls.(*Conn).Handshake(0xc002d5b880, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9
        net/http.(*persistConn).addTLS.func2(0x0, 0xc002d5b880, 0x0, 0xc0014755c0)
        	/usr/local/go/src/net/http/transport.go:1512 +0x45
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x177
        Leaked goroutine: goroutine 157320 [chan receive]:
        net/http.(*persistConn).addTLS(0xc004d09320, 0xc001317aa0, 0x18, 0x0, 0xc001317ab9, 0x3)
        	/usr/local/go/src/net/http/transport.go:1518 +0x1a5
        net/http.(*Transport).dialConn(0xc00019ea00, 0x514de40, 0xc0093990e0, 0x0, 0xc007031440, 0x5, 0xc001317aa0, 0x1c, 0x0, 0xc004d09320, ...)
        	/usr/local/go/src/net/http/transport.go:1588 +0x1d25
        net/http.(*Transport).dialConnFor(0xc00019ea00, 0xc0030b3b80)
        	/usr/local/go/src/net/http/transport.go:1424 +0xc6
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x40f
--- FAIL: TestTelemetrySQLStatsIndependence (11.68s)
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestTelemetrySQLStatsIndependence PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-release-21.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 19, 2021
@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 19, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

(server).TestTelemetrySQLStatsIndependence failed on release-21.1@c139a85ff283ef7da3cb687723d53adf5890f834:

        	/usr/local/go/src/net/http/transport.go:1518 +0x1a5
        net/http.(*Transport).dialConn(0xc00026ac80, 0x514ef00, 0xc001a65bc0, 0x0, 0xc002f94360, 0x5, 0xc007675bc0, 0x1c, 0x0, 0xc00888ac60, ...)
        	/usr/local/go/src/net/http/transport.go:1588 +0x1d25
        net/http.(*Transport).dialConnFor(0xc00026ac80, 0xc001681760)
        	/usr/local/go/src/net/http/transport.go:1424 +0xc6
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x40f
        Leaked goroutine: goroutine 156626 [chan receive]:
        net/http.(*persistConn).addTLS(0xc0079b4b40, 0xc00ae2ace0, 0x18, 0x0, 0xc00ae2acf9, 0x3)
        	/usr/local/go/src/net/http/transport.go:1518 +0x1a5
        net/http.(*Transport).dialConn(0xc00026ac80, 0x514ef00, 0xc001915da0, 0x0, 0xc00b636000, 0x5, 0xc00ae2ace0, 0x1c, 0x0, 0xc0079b4b40, ...)
        	/usr/local/go/src/net/http/transport.go:1588 +0x1d25
        net/http.(*Transport).dialConnFor(0xc00026ac80, 0xc008fde840)
        	/usr/local/go/src/net/http/transport.go:1424 +0xc6
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x40f
        Leaked goroutine: goroutine 156630 [IO wait]:
        internal/poll.runtime_pollWait(0x7f8584a95f90, 0x72, 0x50c6ea0)
        	/usr/local/go/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc001401118, 0x72, 0x50c6e00, 0x6eab6e8, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc001401100, 0xc0048dc900, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
        net.(*netFD).Read(0xc001401100, 0xc0048dc900, 0x205, 0x205, 0x203002, 0x58, 0xc000480400)
        	/usr/local/go/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc002a7a478, 0xc0048dc900, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/net/net.go:182 +0x8e
        crypto/tls.(*atLeastReader).Read(0xc00961d520, 0xc0048dc900, 0x205, 0x205, 0xc0048dc900, 0x0, 0xc002e27688)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0x62
        bytes.(*Buffer).ReadFrom(0xc0072ec980, 0x50bb120, 0xc00961d520, 0x426965, 0x3eb3d40, 0x43b0d20)
        	/usr/local/go/src/bytes/buffer.go:204 +0xb1
        crypto/tls.(*Conn).readFromUntil(0xc0072ec700, 0x50c37e0, 0xc002a7a478, 0x5, 0xc002a7a478, 0x10000042f7f80)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
        crypto/tls.(*Conn).readRecordOrCCS(0xc0072ec700, 0xc00b636100, 0x114, 0x120)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x115
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).readHandshake(0xc0072ec700, 0xc0082ce016, 0xc0082ce000, 0x10f, 0x180)
        	/usr/local/go/src/crypto/tls/conn.go:992 +0x6d
        crypto/tls.(*Conn).clientHandshake(0xc0072ec700, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x2a7
        crypto/tls.(*Conn).Handshake(0xc0072ec700, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9
        net/http.(*persistConn).addTLS.func2(0x0, 0xc0072ec700, 0x0, 0xc00c1fc120)
        	/usr/local/go/src/net/http/transport.go:1512 +0x45
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x177
--- FAIL: TestTelemetrySQLStatsIndependence (11.69s)
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestTelemetrySQLStatsIndependence PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@knz
Copy link
Contributor

knz commented Apr 19, 2021

This is blocking multiple PR merges:

#63837

#63589

Might be related to #63836

I'm going to skip this until it gets fixed.

This was referenced Apr 19, 2021
Merged
Closed
@cockroach-teamcity
Copy link
Member Author

(server).TestTelemetrySQLStatsIndependence failed on release-21.1@7540d056d265a38e3d5ecc129e469fcc0010f918:

=== RUN   TestTelemetrySQLStatsIndependence
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTelemetrySQLStatsIndependence084161242
    test_log_scope.go:74: use -show-logs to present logs inline
    stats_test.go:79: -- test log scope end --
    stats_test.go:79: Leaked goroutine: goroutine 155868 [chan receive]:
        net/http.(*persistConn).addTLS(0xc0038c8120, 0xc0003a11a0, 0x18, 0x0, 0xc0003a11b9, 0x3)
        	/usr/local/go/src/net/http/transport.go:1518 +0x1a5
        net/http.(*Transport).dialConn(0xc000865a40, 0x514ef40, 0xc0093acc60, 0x0, 0xc00c94e480, 0x5, 0xc0003a11a0, 0x1c, 0x0, 0xc0038c8120, ...)
        	/usr/local/go/src/net/http/transport.go:1588 +0x1d25
        net/http.(*Transport).dialConnFor(0xc000865a40, 0xc00735f970)
        	/usr/local/go/src/net/http/transport.go:1424 +0xc6
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x40f
        Leaked goroutine: goroutine 155872 [IO wait]:
        internal/poll.runtime_pollWait(0x7ff2e2d55990, 0x72, 0x50c6ee0)
        	/usr/local/go/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc000ca0018, 0x72, 0x50c6e00, 0x6eab6e8, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc000ca0000, 0xc0091f5d40, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
        net.(*netFD).Read(0xc000ca0000, 0xc0091f5d40, 0x205, 0x205, 0x203000, 0x58, 0xc0021aa000)
        	/usr/local/go/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc009090948, 0xc0091f5d40, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/net/net.go:182 +0x8e
        crypto/tls.(*atLeastReader).Read(0xc000caa480, 0xc0091f5d40, 0x205, 0x205, 0xc0091f5d40, 0x0, 0xc001bfd688)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0x62
        bytes.(*Buffer).ReadFrom(0xc0078d4980, 0x50bb160, 0xc000caa480, 0x426965, 0x3eb3d40, 0x43b0d20)
        	/usr/local/go/src/bytes/buffer.go:204 +0xb1
        crypto/tls.(*Conn).readFromUntil(0xc0078d4700, 0x50c3820, 0xc009090948, 0x5, 0xc009090948, 0x10000042f7f80)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
        crypto/tls.(*Conn).readRecordOrCCS(0xc0078d4700, 0xc00c94e500, 0x114, 0x120)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x115
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).readHandshake(0xc0078d4700, 0xc00bd7c616, 0xc00bd7c600, 0x10f, 0x180)
        	/usr/local/go/src/crypto/tls/conn.go:992 +0x6d
        crypto/tls.(*Conn).clientHandshake(0xc0078d4700, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x2a7
        crypto/tls.(*Conn).Handshake(0xc0078d4700, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9
        net/http.(*persistConn).addTLS.func2(0x0, 0xc0078d4700, 0x0, 0xc0093ad6e0)
        	/usr/local/go/src/net/http/transport.go:1512 +0x45
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x177
--- FAIL: TestTelemetrySQLStatsIndependence (10.19s)
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestTelemetrySQLStatsIndependence PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@knz knz mentioned this issue Apr 19, 2021
Merged
@knz knz added the GA-blocker label Apr 19, 2021
@knz
Copy link
Contributor

knz commented Apr 19, 2021

see discussion in #63851.

@cockroach-teamcity
Copy link
Member Author

(server).TestTelemetrySQLStatsIndependence failed on release-21.1@c07fe34fc84626d1fea17c3ccb13ed80bf2e7d2e:

=== RUN   TestTelemetrySQLStatsIndependence
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTelemetrySQLStatsIndependence443914876
    test_log_scope.go:74: use -show-logs to present logs inline
    stats_test.go:79: -- test log scope end --
    stats_test.go:79: Leaked goroutine: goroutine 159659 [IO wait]:
        internal/poll.runtime_pollWait(0x7f57215749b8, 0x72, 0x7c12aa0)
        	/usr/local/go/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc00409dc18, 0x72, 0x7c12a00, 0x9be88e8, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xe6
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc00409dc00, 0xc0020d0000, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x23f
        net.(*netFD).Read(0xc00409dc00, 0xc0020d0000, 0x205, 0x205, 0x10100c0037b1208, 0x7f56e35c2e90, 0x0)
        	/usr/local/go/src/net/fd_posix.go:55 +0x66
        net.(*conn).Read(0xc000a5e168, 0xc0020d0000, 0x205, 0x205, 0x203001, 0xc0037b10f8, 0x4a2282)
        	/usr/local/go/src/net/net.go:182 +0xec
        crypto/tls.(*atLeastReader).Read(0xc004543680, 0xc0020d0000, 0x205, 0x205, 0xc004543680, 0x205, 0x203000)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0xa7
        bytes.(*Buffer).ReadFrom(0xc0017f3780, 0x7c06d20, 0xc004543680, 0x61fe460, 0x7c0f3e0, 0x73a22f)
        	/usr/local/go/src/bytes/buffer.go:204 +0x159
        crypto/tls.(*Conn).readFromUntil(0xc0017f3500, 0x7c0f3e0, 0xc000a5e168, 0x5, 0xc000a5e168, 0xc0052dc3c3)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0x215
        crypto/tls.(*Conn).readRecordOrCCS(0xc0017f3500, 0xc0017f3500, 0xc001bf2120, 0x114)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x42b
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).readHandshake(0xc0017f3500, 0xc000e98016, 0xc000e98000, 0x10f, 0x180)
        	/usr/local/go/src/crypto/tls/conn.go:992 +0xc5
        crypto/tls.(*Conn).clientHandshake(0xc0017f3500, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x27d
        crypto/tls.(*Conn).Handshake(0xc0017f3500, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0x18e
        net/http.(*persistConn).addTLS.func2(0x0, 0xc0017f3500, 0x0, 0xc00529d5c0)
        	/usr/local/go/src/net/http/transport.go:1512 +0x4b
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x22f
        Leaked goroutine: goroutine 159697 [chan receive]:
        net/http.(*persistConn).addTLS(0xc002362d80, 0xc0054f78c0, 0x18, 0x0, 0xc0054f78d9, 0x3)
        	/usr/local/go/src/net/http/transport.go:1518 +0x258
        net/http.(*Transport).dialConn(0xc0001e0780, 0x7c9b580, 0xc00651f7a0, 0x0, 0xc000c78240, 0x5, 0xc0054f78c0, 0x1c, 0x0, 0xc002362d80, ...)
        	/usr/local/go/src/net/http/transport.go:1588 +0x3054
        net/http.(*Transport).dialConnFor(0xc0001e0780, 0xc0026ea370)
        	/usr/local/go/src/net/http/transport.go:1424 +0x152
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x6b0
--- FAIL: TestTelemetrySQLStatsIndependence (12.75s)
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestTelemetrySQLStatsIndependence PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(server).TestTelemetrySQLStatsIndependence failed on release-21.1@c139a85ff283ef7da3cb687723d53adf5890f834:

        	/usr/local/go/src/net/http/transport.go:1518 +0x258
        net/http.(*Transport).dialConn(0xc0004fda40, 0x7c9b640, 0xc0076b11a0, 0x0, 0xc0005eac60, 0x5, 0xc00796efc0, 0x1c, 0x0, 0xc002268360, ...)
        	/usr/local/go/src/net/http/transport.go:1588 +0x3054
        net/http.(*Transport).dialConnFor(0xc0004fda40, 0xc002c81550)
        	/usr/local/go/src/net/http/transport.go:1424 +0x152
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x6b0
        Leaked goroutine: goroutine 160028 [chan receive]:
        net/http.(*persistConn).addTLS(0xc00467a360, 0xc0026f6640, 0x18, 0x0, 0xc0026f6659, 0x3)
        	/usr/local/go/src/net/http/transport.go:1518 +0x258
        net/http.(*Transport).dialConn(0xc0004fda40, 0x7c9b640, 0xc002514f60, 0x0, 0xc003db2480, 0x5, 0xc0026f6640, 0x1c, 0x0, 0xc00467a360, ...)
        	/usr/local/go/src/net/http/transport.go:1588 +0x3054
        net/http.(*Transport).dialConnFor(0xc0004fda40, 0xc000a54630)
        	/usr/local/go/src/net/http/transport.go:1424 +0x152
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x6b0
        Leaked goroutine: goroutine 160031 [IO wait]:
        internal/poll.runtime_pollWait(0x7effe2317bd0, 0x72, 0x7c12b60)
        	/usr/local/go/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc004689898, 0x72, 0x7c12b00, 0x9be88e8, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xe6
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc004689880, 0xc001f29440, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x23f
        net.(*netFD).Read(0xc004689880, 0xc001f29440, 0x205, 0x205, 0x10000c00227f208, 0x7f000ed7a5e0, 0x0)
        	/usr/local/go/src/net/fd_posix.go:55 +0x66
        net.(*conn).Read(0xc005d86128, 0xc001f29440, 0x205, 0x205, 0x203000, 0xc00227f0f8, 0x4a2282)
        	/usr/local/go/src/net/net.go:182 +0xec
        crypto/tls.(*atLeastReader).Read(0xc000fec920, 0xc001f29440, 0x205, 0x205, 0xc000fec920, 0x205, 0x203000)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0xa7
        bytes.(*Buffer).ReadFrom(0xc002bcc980, 0x7c06de0, 0xc000fec920, 0x61fe500, 0x7c0f4a0, 0x73a22f)
        	/usr/local/go/src/bytes/buffer.go:204 +0x159
        crypto/tls.(*Conn).readFromUntil(0xc002bcc700, 0x7c0f4a0, 0xc005d86128, 0x5, 0xc005d86128, 0xc000bdef83)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0x215
        crypto/tls.(*Conn).readRecordOrCCS(0xc002bcc700, 0xc002bcc700, 0xc003db25a0, 0x114)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x42b
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).readHandshake(0xc002bcc700, 0xc0034be116, 0xc0034be180, 0x10f, 0x180)
        	/usr/local/go/src/crypto/tls/conn.go:992 +0xc5
        crypto/tls.(*Conn).clientHandshake(0xc002bcc700, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x27d
        crypto/tls.(*Conn).Handshake(0xc002bcc700, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0x18e
        net/http.(*persistConn).addTLS.func2(0x0, 0xc002bcc700, 0x0, 0xc0025154a0)
        	/usr/local/go/src/net/http/transport.go:1512 +0x4b
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x22f
--- FAIL: TestTelemetrySQLStatsIndependence (12.84s)
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestTelemetrySQLStatsIndependence PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

craig bot pushed a commit that referenced this issue Apr 19, 2021
@knz @yuzefovich
Merge #63852 #63854
cc35207 
63852: server: skip TestTelemetrySQLStatsIndependence r=tbg a=knz

This test flakes if the reg server is temporarily unavailable.

Informs  #63844, #63851,  #63848

63854: sqlsmith: don't use crdb_internal.create_join_token r=yuzefovich a=yuzefovich

This is an internal builtin, and we probably don't get much when using
it in the sqlsmith. Currently this builtin occasionally leads to an
internal error.

Addresses: #63819.

Release note: None

Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 19, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

(server).TestTelemetrySQLStatsIndependence failed on release-21.1@7540d056d265a38e3d5ecc129e469fcc0010f918:

        	/usr/local/go/src/net/http/transport.go:1518 +0x258
        net/http.(*Transport).dialConn(0xc0003fe8c0, 0x7c9b6a0, 0xc007a82ea0, 0x0, 0xc003321320, 0x5, 0xc0009c8040, 0x1c, 0x0, 0xc00277fd40, ...)
        	/usr/local/go/src/net/http/transport.go:1588 +0x3054
        net/http.(*Transport).dialConnFor(0xc0003fe8c0, 0xc0061a4160)
        	/usr/local/go/src/net/http/transport.go:1424 +0x152
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x6b0
        Leaked goroutine: goroutine 160277 [IO wait]:
        internal/poll.runtime_pollWait(0x7f73a5f41cd8, 0x72, 0x7c12bc0)
        	/usr/local/go/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc003b3ad98, 0x72, 0x7c12b00, 0x9be88e8, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xe6
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc003b3ad80, 0xc000f02d80, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x23f
        net.(*netFD).Read(0xc003b3ad80, 0xc000f02d80, 0x205, 0x205, 0x10000c001ef1208, 0x7f73d19cff68, 0x0)
        	/usr/local/go/src/net/fd_posix.go:55 +0x66
        net.(*conn).Read(0xc0016c2190, 0xc000f02d80, 0x205, 0x205, 0x203000, 0xc001ef10f8, 0x4a2282)
        	/usr/local/go/src/net/net.go:182 +0xec
        crypto/tls.(*atLeastReader).Read(0xc001da6220, 0xc000f02d80, 0x205, 0x205, 0xc001da6220, 0x205, 0x203001)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0xa7
        bytes.(*Buffer).ReadFrom(0xc00513c600, 0x7c06e40, 0xc001da6220, 0x61fe500, 0x7c0f500, 0x73a22f)
        	/usr/local/go/src/bytes/buffer.go:204 +0x159
        crypto/tls.(*Conn).readFromUntil(0xc00513c380, 0x7c0f500, 0xc0016c2190, 0x5, 0xc0016c2190, 0xc006502da3)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0x215
        crypto/tls.(*Conn).readRecordOrCCS(0xc00513c380, 0xc00513c300, 0xc003320000, 0x114)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x42b
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).readHandshake(0xc00513c380, 0xc00028e016, 0xc00028e000, 0x10f, 0x180)
        	/usr/local/go/src/crypto/tls/conn.go:992 +0xc5
        crypto/tls.(*Conn).clientHandshake(0xc00513c380, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x27d
        crypto/tls.(*Conn).Handshake(0xc00513c380, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0x18e
        net/http.(*persistConn).addTLS.func2(0x0, 0xc00513c380, 0x0, 0xc005fd6120)
        	/usr/local/go/src/net/http/transport.go:1512 +0x4b
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x22f
        Leaked goroutine: goroutine 160290 [chan receive]:
        net/http.(*persistConn).addTLS(0xc00276a240, 0xc0009c84c0, 0x18, 0x0, 0xc0009c84d9, 0x3)
        	/usr/local/go/src/net/http/transport.go:1518 +0x258
        net/http.(*Transport).dialConn(0xc0003fe8c0, 0x7c9b6a0, 0xc006efc780, 0x0, 0xc001fdc360, 0x5, 0xc0009c84c0, 0x1c, 0x0, 0xc00276a240, ...)
        	/usr/local/go/src/net/http/transport.go:1588 +0x3054
        net/http.(*Transport).dialConnFor(0xc0003fe8c0, 0xc001060160)
        	/usr/local/go/src/net/http/transport.go:1424 +0x152
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x6b0
--- FAIL: TestTelemetrySQLStatsIndependence (12.82s)
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestTelemetrySQLStatsIndependence PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(server).TestTelemetrySQLStatsIndependence failed on release-21.1@94c64b0bd31c24bb707b32eec714c2af8667135b:

        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x2a7
        crypto/tls.(*Conn).Handshake(0xc0048d7c00, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9
        net/http.(*persistConn).addTLS.func2(0x0, 0xc0048d7c00, 0x0, 0xc009f9b2c0)
        	/usr/local/go/src/net/http/transport.go:1512 +0x45
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x177
        Leaked goroutine: goroutine 156474 [chan receive]:
        net/http.(*persistConn).addTLS(0xc0092c4a20, 0xc0098fb2a0, 0x18, 0x0, 0xc0098fb2b9, 0x3)
        	/usr/local/go/src/net/http/transport.go:1518 +0x1a5
        net/http.(*Transport).dialConn(0xc00019ec80, 0x514f020, 0xc00224ac00, 0x0, 0xc0052ff200, 0x5, 0xc0098fb2a0, 0x1c, 0x0, 0xc0092c4a20, ...)
        	/usr/local/go/src/net/http/transport.go:1592 +0x1d45
        net/http.(*Transport).dialConnFor(0xc00019ec80, 0xc0072ddc30)
        	/usr/local/go/src/net/http/transport.go:1424 +0xc6
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x40f
        Leaked goroutine: goroutine 156531 [IO wait]:
        internal/poll.runtime_pollWait(0x7f00b0570150, 0x72, 0x50c6fc0)
        	/usr/local/go/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc005d50218, 0x72, 0x50c6f00, 0x6eab6e8, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc005d50200, 0xc00317f680, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
        net.(*netFD).Read(0xc005d50200, 0xc00317f680, 0x205, 0x205, 0x203000, 0x58, 0xc0022ca000)
        	/usr/local/go/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc0015d2468, 0xc00317f680, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/net/net.go:182 +0x8e
        crypto/tls.(*atLeastReader).Read(0xc00142a5c0, 0xc00317f680, 0x205, 0x205, 0xc00317f680, 0x0, 0xc002a11688)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0x62
        bytes.(*Buffer).ReadFrom(0xc00636d080, 0x50bb240, 0xc00142a5c0, 0x426965, 0x3eb3dc0, 0x43b0e20)
        	/usr/local/go/src/bytes/buffer.go:204 +0xb1
        crypto/tls.(*Conn).readFromUntil(0xc00636ce00, 0x50c3900, 0xc0015d2468, 0x5, 0xc0015d2468, 0x10000042f8080)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
        crypto/tls.(*Conn).readRecordOrCCS(0xc00636ce00, 0xc001598a00, 0x114, 0x120)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x115
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).readHandshake(0xc00636ce00, 0xc009a96c16, 0xc009a96c00, 0x10f, 0x180)
        	/usr/local/go/src/crypto/tls/conn.go:992 +0x6d
        crypto/tls.(*Conn).clientHandshake(0xc00636ce00, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x2a7
        crypto/tls.(*Conn).Handshake(0xc00636ce00, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9
        net/http.(*persistConn).addTLS.func2(0x0, 0xc00636ce00, 0x0, 0xc001475a40)
        	/usr/local/go/src/net/http/transport.go:1512 +0x45
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x177
--- FAIL: TestTelemetrySQLStatsIndependence (11.62s)
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestTelemetrySQLStatsIndependence PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

adityamaru added a commit to adityamaru/cockroach that referenced this issue Apr 19, 2021
@adityamaru
server: skip TestTelemetrySQLStatsIndependence
f36db2e 
Refs: cockroachdb#63844

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None
@adityamaru adityamaru mentioned this issue Apr 19, 2021
Closed
@adityamaru adityamaru mentioned this issue Apr 19, 2021
Merged
adityamaru added a commit to adityamaru/cockroach that referenced this issue Apr 19, 2021
@adityamaru
server: skip TestTelemetrySQLStatsIndependence
53c5682 
Refs: cockroachdb#63844

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None
@irfansharif irfansharif mentioned this issue Apr 20, 2021
Merged
@cockroach-teamcity
Copy link
Member Author

(server).TestTelemetrySQLStatsIndependence failed on release-21.1@4e822751a488380f3f39a5fe25ebd529920fefbc:

        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x2a7
        crypto/tls.(*Conn).Handshake(0xc006534e00, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9
        net/http.(*persistConn).addTLS.func2(0x0, 0xc006534e00, 0x0, 0xc003e61500)
        	/usr/local/go/src/net/http/transport.go:1512 +0x45
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x177
        Leaked goroutine: goroutine 155626 [chan receive]:
        net/http.(*persistConn).addTLS(0xc0086de000, 0xc001fc9580, 0x18, 0x0, 0xc001fc9599, 0x3)
        	/usr/local/go/src/net/http/transport.go:1518 +0x1a5
        net/http.(*Transport).dialConn(0xc0004f6140, 0x5151000, 0xc0006c2000, 0x0, 0xc004bc6000, 0x5, 0xc001fc9580, 0x1c, 0x0, 0xc0086de000, ...)
        	/usr/local/go/src/net/http/transport.go:1592 +0x1d45
        net/http.(*Transport).dialConnFor(0xc0004f6140, 0xc00476b290)
        	/usr/local/go/src/net/http/transport.go:1424 +0xc6
        created by net/http.(*Transport).queueForDial
        	/usr/local/go/src/net/http/transport.go:1393 +0x40f
        Leaked goroutine: goroutine 155635 [IO wait]:
        internal/poll.runtime_pollWait(0x7f21fe87a0c0, 0x72, 0x50c8e20)
        	/usr/local/go/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc008ccdd18, 0x72, 0x50c8e00, 0x6eae6e8, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc008ccdd00, 0xc003f93d40, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
        net.(*netFD).Read(0xc008ccdd00, 0xc003f93d40, 0x205, 0x205, 0x203002, 0x58, 0xc000488800)
        	/usr/local/go/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc002c5b658, 0xc003f93d40, 0x205, 0x205, 0x0, 0x0, 0x0)
        	/usr/local/go/src/net/net.go:182 +0x8e
        crypto/tls.(*atLeastReader).Read(0xc00a2323c0, 0xc003f93d40, 0x205, 0x205, 0xc003f93d40, 0x0, 0xc004f49688)
        	/usr/local/go/src/crypto/tls/conn.go:779 +0x62
        bytes.(*Buffer).ReadFrom(0xc006535780, 0x50bd080, 0xc00a2323c0, 0x426965, 0x3eb5280, 0x43b29e0)
        	/usr/local/go/src/bytes/buffer.go:204 +0xb1
        crypto/tls.(*Conn).readFromUntil(0xc006535500, 0x50c5760, 0xc002c5b658, 0x5, 0xc002c5b658, 0x10000042f9d60)
        	/usr/local/go/src/crypto/tls/conn.go:801 +0xf3
        crypto/tls.(*Conn).readRecordOrCCS(0xc006535500, 0xc00892b200, 0x114, 0x120)
        	/usr/local/go/src/crypto/tls/conn.go:608 +0x115
        crypto/tls.(*Conn).readRecord(...)
        	/usr/local/go/src/crypto/tls/conn.go:576
        crypto/tls.(*Conn).readHandshake(0xc006535500, 0xc004a39516, 0xc004a39500, 0x10f, 0x180)
        	/usr/local/go/src/crypto/tls/conn.go:992 +0x6d
        crypto/tls.(*Conn).clientHandshake(0xc006535500, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/handshake_client.go:170 +0x2a7
        crypto/tls.(*Conn).Handshake(0xc006535500, 0x0, 0x0)
        	/usr/local/go/src/crypto/tls/conn.go:1362 +0xc9
        net/http.(*persistConn).addTLS.func2(0x0, 0xc006535500, 0x0, 0xc00656c960)
        	/usr/local/go/src/net/http/transport.go:1512 +0x45
        created by net/http.(*persistConn).addTLS
        	/usr/local/go/src/net/http/transport.go:1508 +0x177
--- FAIL: TestTelemetrySQLStatsIndependence (11.49s)
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestTelemetrySQLStatsIndependence PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@solongordon
Copy link
Contributor

Closing this since the test has been skipped and the test issue is being tracked in #63851.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@solongordon solongordon

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@knz @solongordon @cockroach-teamcity