Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/sqlproxyccl/tenant: TestWatchTenants failed #103494

Closed
cockroach-teamcity opened this issue May 17, 2023 · 4 comments · Fixed by #103625
Closed

ccl/sqlproxyccl/tenant: TestWatchTenants failed #103494

cockroach-teamcity opened this issue May 17, 2023 · 4 comments · Fixed by #103625
Assignees
@jaylim-crl
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 17, 2023 •
edited by cockroach-jira-scripts
Loading

ccl/sqlproxyccl/tenant.TestWatchTenants failed with artifacts on master @ 76078816b69ee6d1f8dacd581ef464660fb1b697:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4e2bd31]

Stack:

goroutine 16050 [running]:
testing.tRunner.func1.2({0x56a4da0, 0xa7138e0})
	GOROOT/src/testing/testing.go:1396 +0x24e
testing.tRunner.func1()
	GOROOT/src/testing/testing.go:1399 +0x39f
panic({0x56a4da0, 0xa7138e0})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func2()
	github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:122 +0x2a9
panic({0x56a4da0, 0xa7138e0})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00017f398?, {0x75a3c70, 0xc0000c6058})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x6a
panic({0x56a4da0, 0xa7138e0})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc000dff080, {0x75a3c70, 0xc0000c6058})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:507 +0x21d
panic({0x56a4da0, 0xa7138e0})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test.TestWatchTenants(0xc002448340)
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:221 +0xed1
testing.tRunner(0xc002448340, 0x63403a0)
	GOROOT/src/testing/testing.go:1446 +0x10b
created by testing.(*T).Run
	GOROOT/src/testing/testing.go:1493 +0x35f
Log preceding fatal error 

=== RUN   TestWatchTenants
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/f215c8f665df7837d22d3ffe15cbecee/logTestWatchTenants985868698
    test_log_scope.go:79: use -show-logs to present logs inline
*
* ERROR: a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:507
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | runtime.panicmem
*   | 	GOROOT/src/runtime/panic.go:260
*   | runtime.sigpanic
*   | 	GOROOT/src/runtime/signal_unix.go:835
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test.TestWatchTenants
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:221
*   | testing.tRunner
*   | 	GOROOT/src/testing/testing.go:1446
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*
    stopper.go:229: -- test log scope end --

ERROR: a panic has occurred!
Details cannot be printed yet because we are still unwinding.
Hopefully the test harness prints the panic below, otherwise check the test logs.

test logs left over in: /artifacts/tmp/_tmp/f215c8f665df7837d22d3ffe15cbecee/logTestWatchTenants985868698
    stopper.go:229: panic: runtime error: invalid memory address or nil pointer dereference
--- FAIL: TestWatchTenants (0.08s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-28029
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels May 17, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone May 17, 2023
@rafiss
Copy link
Collaborator

rafiss commented May 18, 2023

cc @jeffswenson for triaging

@jaylim-crl
Copy link
Collaborator

Thanks - will take a look.

@jeffswenson
Copy link
Collaborator

I assigned this to Jay as it is likely related to #103070.

@jaylim-crl jaylim-crl mentioned this issue May 18, 2023
Merged
@jaylim-crl
Copy link
Collaborator

See #103625.

craig bot pushed a commit that referenced this issue May 18, 2023
@otan @rharding6373
@yuzefovich @jaylim-crl
Merge #102299 #103078 #103588 #103625
b62c5f7 
102299: tree: fix decimal strings for 0 with exponents < -6 r=rafiss a=otan

Release note (bug fix): Fix a bug where 0 with exponents < -6 would display
as `0E(exponent)` instead of printing all 0s, e.g. `0E-7` should be `0.0000000`.

Informs #102217

103078: sql: resolve values types after building scalars in optbuilder r=rharding6373 a=rharding6373

Previously, we would build scalars in VALUES clauses after resolving the value column types. However, for UDFs, we modify the type if it's a record-returning function during the build. In this change we reverse the order so that we build scalars and then resolve types.

This bug was introduced by #98162.

Epic: None
Fixes: #102718

Release note: This fixes a bug in VALUES clauses containing a call to a record-returning UDF that could manifest as an internal error in some queries.

103588: kvstreamer: add non-negative float validation to a cluster setting r=yuzefovich a=yuzefovich

We forgot to add non-negative validation function to private `sql.distsql.streamer.avg_response_size_multiple` cluster setting. If this were set to a negative value, it would result in an undefined behavior of the streamer (we could try setting negative `TargetBytes` limit on `BatchRequest`s). I don't think anyone ever used it though so far.

Epic: None

Release note: None

103625: ccl/sqlproxyccl: fix flake on TestWatchTenants r=JeffSwenson a=jaylim-crl

Fixes #103494.

This commit fixes a flake on TestWatchTenants. There's a possibility where the cache invalidation logic races with the watcher termination logic in the test. This commit updates the test to wait for the cache invalidation before proceeding.

Release note: None

Epic: none

Co-authored-by: Oliver Tan <otan@cockroachlabs.com>
Co-authored-by: rharding6373 <rharding6373@users.noreply.github.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Jay <jay@cockroachlabs.com>
@craig craig bot closed this as completed in d8a7b11 May 18, 2023
jaylim-crl added a commit to jaylim-crl/cockroach that referenced this issue May 22, 2023
@jaylim-crl
ccl/sqlproxyccl: fix flake on TestWatchTenants
768d496 
Fixes cockroachdb#103494.

This commit fixes a flake on TestWatchTenants. There's a possibility where
the cache invalidation logic races with the watcher termination logic in the
test. This commit updates the test to wait for the cache invalidation before
proceeding.

Release note: None

Epic: none
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jaylim-crl jaylim-crl

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

ccl/sqlproxyccl: fix flake on TestWatchTenants jaylim-crl/cockroach
4 participants
@rafiss @jeffswenson @cockroach-teamcity @jaylim-crl