Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv/kvserver/batcheval: TestDBAddSSTable failed #61393

Closed
cockroach-teamcity opened this issue Mar 3, 2021 · 2 comments · Fixed by #62095
Closed

kv/kvserver/batcheval: TestDBAddSSTable failed #61393

cockroach-teamcity opened this issue Mar 3, 2021 · 2 comments · Fixed by #62095
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

(kv/kvserver/batcheval).TestDBAddSSTable failed on master@6de4313ec216161c79fe725fcc31fc87ef1804ea:

      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_decoder.go:59 +0x10f
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).Decode()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:123 +0x671
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:619 +0x61f
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:459 +0x177
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:523 +0x1c8
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:284 +0x3e5
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:229 +0x55
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0x149

Goroutine 2309 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1168 +0x5bb
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval_test.TestDBAddSSTable()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_add_sstable_test.go:76 +0x16b
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1123 +0x202

Goroutine 1613 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0x107
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:205 +0x24c
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaft()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:609 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:1525 +0xd31
  github.com/cockroachdb/cockroach/pkg/server.(*Node).start()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:382 +0xbc4
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1567 +0x338f
  github.com/cockroachdb/cockroach/pkg/server.(*Server).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1096 +0x50
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:459 +0x92
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:255 +0xbd
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval_test.TestDBAddSSTable.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_add_sstable_test.go:83 +0x23a
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1123 +0x202
==================
    testing.go:1038: race detected during execution of test
    --- FAIL: TestDBAddSSTable/store=on-disk (1.55s)

More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=4
make stressrace TESTS=TestDBAddSSTable PKG=./pkg/kv/kvserver/batcheval TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 3, 2021
@tbg tbg assigned irfansharif and unassigned tbg Mar 3, 2021
@tbg
Copy link
Member

tbg commented Mar 3, 2021

@irfansharif would you mind taking this?

==================
WARNING: DATA RACE
Write at 0x00c0091dabf8 by goroutine 2309:
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).TestingIncludeAsyncSpansInRecordings()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/tracer.go:682 +0x5c
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval_test.runTestDBAddSSTable()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_add_sstable_test.go:103 +0x4f
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval_test.TestDBAddSSTable.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_add_sstable_test.go:95 +0x4a9
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:459 +0x92
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:255 +0xbd
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval_test.TestDBAddSSTable.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_add_sstable_test.go:83 +0x23a
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1123 +0x202

Previous read at 0x00c0091dabf8 by goroutine 1613:
  github.com/cockroachdb/cockroach/pkg/util/tracing.ForkSpan()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/tracer.go:706 +0xc4
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).createTracingSpans()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_decoder.go:142 +0x244
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).DecodeAndBind()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_decoder.go:59 +0x10f
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).Decode()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:123 +0x671
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:619 +0x61f
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:459 +0x177
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:523 +0x1c8
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:284 +0x3e5
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:229 +0x55
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0x149

Goroutine 2309 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1168 +0x5bb
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval_test.TestDBAddSSTable()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_add_sstable_test.go:76 +0x16b
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1123 +0x202

Goroutine 1613 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0x107
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:205 +0x24c
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaft()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:609 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:1525 +0xd31
  github.com/cockroachdb/cockroach/pkg/server.(*Node).start()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:382 +0xbc4
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1567 +0x338f
  github.com/cockroachdb/cockroach/pkg/server.(*Server).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1096 +0x50
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:459 +0x92
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      /go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:255 +0xbd
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval_test.TestDBAddSSTable.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_add_sstable_test.go:83 +0x23a
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1123 +0x202
==================
    testing.go:1038: race detected during execution of test

@irfansharif
Copy link
Contributor

Looking.

craig bot pushed a commit that referenced this issue Mar 17, 2021
61585: sql: stop duplicating regions on the database descriptor r=arulajmani a=arulajmani

Previously, we were duplicating regions both on the type descriptor and
the database descriptor. Theres a verbose explanation about why this is
bad on the linked issue, but the crux of the problem with this approach
is the added complexity it requires keeping these two lists in sync as
regions are added/dropped from a database in the face of possible
failure scenarios.

This patch removes the `Regions` field from the database descriptors's
region config. Instead, the regions stored on the multi-region enum
serve as the sole source of truth. To accomplish this, we introduce a
new `RegionConfig` struct which is synthesized from the state (primary
region, survival goal) stored on the database and the state (available
regions) stored on the type descriptor. All the places that constructed
zone configurations from the raw region config proto now use this new
struct. This patch also introduces a `InitializationRegionConfig`
struct which wraps a `RegionConfig` with initilization specific
meta-data.

I'm not too thrilled by the fact that survival goal and primary region
is still stored on the database descriptor. Ideally, the database
descriptor would contain just the multi-region enum ID and all user
configured parameters would be in one place (on the type descriptor's
region config). The level of indirection introduced by this patch sets
us on the path to that future.

Closes #60620

Release justification: low risk updates to new functionality
Release note: None

61763: sql: disallow adding OIDVECTOR and INT2VECTOR columns r=mgartner a=mgartner

Creating columns of type `OIDVECTOR` or `INT2VECTOR` is not allowed in
`CREATE TABLE` statements. This commit disallows these types in
`ALTER TABLE ... ADD COLUMN` statements for consistency.

Fixes #61762

Release justification: This is a low-risk bug fix.

Release note (bug fix): Adding columns of type `OIDVECTOR` or
`INT2VECTOR` to a table in `ALTER TABLE ... ADD COLUMN` statements is no
longer allowed. These types are not allowed in user-created tables via
`CREATE TABLE ` and were erroneously allowed previously in
`ALTER TABLE ... ADD COLUMN`.

62095: tracing: lock around testing knobs r=irfansharif a=irfansharif

Fixes #61393. There was a race condition in our tests by not doing so.

Release note: None

Co-authored-by: arulajmani <arulajmani@gmail.com>
Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com>
Co-authored-by: irfan sharif <irfanmahmoudsharif@gmail.com>
@craig craig bot closed this as completed in 399db7c Mar 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants