Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv/kvnemesis: TestKVNemesisSingleNode failed #48046

Closed
cockroach-teamcity opened this issue Apr 26, 2020 · 1 comment · Fixed by #48245
Closed

kv/kvnemesis: TestKVNemesisSingleNode failed #48046

cockroach-teamcity opened this issue Apr 26, 2020 · 1 comment · Fixed by #48245
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

(kv/kvnemesis).TestKVNemesisSingleNode failed on master@60c9e055e970bd7f150ebcfad266929b2638d635:


goroutine 2153 [select, 19 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/idalloc.(*Allocator).start.func1(0x637f640, 0xc00136c840)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/idalloc/id_alloc.go:119 +0x58a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000328c00, 0xc0001d2dc0, 0xc000328bd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x1ba
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xc4

goroutine 320859 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).heartbeatLoop(0xc0cf2f53d0, 0x637f640, 0xc0cfb43770)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:257 +0x246
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0001d2dc0, 0x637f640, 0xc0cfb43770, 0xc0cfac0fc0, 0x29, 0x63ee900, 0xc0cf58f500, 0xc0cfafa9e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x163
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x14b

goroutine 320832 [select]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc001546120, 0x637f5c0, 0xc000074108, 0x20, 0x63e4480, 0xc0cf8638c0, 0x0, 0x1, 0x637f5c0, 0xc000074108)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:248 +0x255
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).gcTxnRecord(0xc0007a42d0, 0x637f5c0, 0xc000074108, 0x20, 0xc0cfc67680, 0x0, 0xc0cfb28780)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:699 +0x563
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x637f5c0, 0xc000074108)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:738 +0x91
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0001d2dc0, 0x637f5c0, 0xc000074108, 0xc0cfd18a80, 0x33, 0x0, 0x0, 0xc0cfc90ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x163
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x14b

goroutine 318412 [select]:
github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1(0x637f640, 0xc0017ca540)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:274 +0x1d0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0001d2dc0, 0x637f640, 0xc0017ca540, 0xc0c26e2ff0, 0x2a, 0x0, 0x0, 0xc0c2d20570)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x163
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x14b

goroutine 320885 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).heartbeatLoop(0xc0cfb90650, 0x637f640, 0xc0cfe014d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:257 +0x246
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0001d2dc0, 0x637f640, 0xc0cfe014d0, 0xc0cfd7b980, 0x29, 0x63ee900, 0xc0cfcd3ea0, 0xc0cfa856d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x163
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x14b


ERROR: exit status 2

83 runs completed, 1 failures, over 23m13s
context canceled

More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestKVNemesisSingleNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 26, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Apr 26, 2020
@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisSingleNode failed on master@3b612692db93aa7c87493705e1fad85c9c664f6c:

Fatal error:

F200430 05:44:39.222082 2101 kv/kvclient/kvcoord/txn_coord_sender.go:809  [n1,s1,r32/1:/Table/{36-50/"84ad2…},txn=ec86e982] unexpected error with ABORTED txn: (*roachpb.internalError) cannot write undeclared span /Local/RangeID/32/r/AbortSpan/"ec86e982-fee8-4550-8c6b-2b745d4c25ff"
declared:
read local: /Local/RangeID/32/r/AbortSpan/"ec86e982-fee8-4550-8c6b-2b745d4c25ff" at 0,0
read local: /Local/Range/Table/36/RangeDescriptor at 0,0
write global: /Meta2/Table/50/"84ad26e7" at 1588225478.924505396,0
write global: /Meta2/Max at 1588225478.924505396,0
write global: /Table/13/1/2020-04-30T05:44:38.924505Z/551010563341254657/0 at 1588225478.924505396,0
write global: /Table/13/1/2020-04-30T05:44:38.924505Z/551010563341254657/2/1 at 1588225478.924505396,0
write global: /Table/13/1/2020-04-30T05:44:38.924505Z/551010563341254657/3/1 at 1588225478.924505396,0
write global: /Table/13/1/2020-04-30T05:44:38.924505Z/551010563341254657/4/1 at 1588225478.924505396,0
write global: /Table/13/1/2020-04-30T05:44:38.924505Z/551010563341254657/5/1 at 1588225478.924505396,0
write global: /Table/13/1/2020-04-30T05:44:38.924505Z/551010563341254657/6/1 at 1588225478.924505396,0
write local: /Local/Range/Table/36/RangeDescriptor at 0,0
write local: /Local/Range/Table/36/Transaction/"ec86e982-fee8-4550-8c6b-2b745d4c25ff" at 0,0
write local: /Local/Range/Table/50/"84ad26e7"/RangeDescriptor at 0,0

stack:

Stack:

goroutine 2101 [running]:
runtime/debug.Stack(0xc002080220, 0x16, 0x20)
	/usr/local/go/src/runtime/debug/stack.go:24 +0xab
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).checkAllowed(0xc0014ac540, 0x1, 0xc000c88be0, 0x1b, 0x20, 0x0, 0x0, 0x0, 0x55908a0, 0xc000ae70f0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:293 +0x3ad
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).CheckAllowed(0xc0014ac540, 0x1, 0xc000c88be0, 0x1b, 0x20, 0x0, 0x0, 0x0, 0xc001414b90, 0x15468a27697e)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:224 +0xb5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.spanSetWriter.Clear(0x15469be78368, 0xc0014ac6c0, 0xc0014ac540, 0x1, 0x0, 0x0, 0xc000c88be0, 0x1b, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/batch.go:386 +0xdd
github.com/cockroachdb/cockroach/pkg/storage.mvccPutInternal(0x63ee280, 0xc0006d5fb0, 0x15469bed4670, 0xc0010f1680, 0x646ab00, 0xc001414b90, 0xc0007ad960, 0xc000c88be0, 0x1b, 0x20, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1463 +0x6dd
github.com/cockroachdb/cockroach/pkg/storage.mvccPutUsingIter(0x63ee280, 0xc0006d5fb0, 0x15469bed4670, 0xc0010f1680, 0x646ab00, 0xc001414b90, 0xc0007ad960, 0xc000c88be0, 0x1b, 0x20, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1254 +0x18f
github.com/cockroachdb/cockroach/pkg/storage.MVCCDelete(0x63ee280, 0xc0006d5fb0, 0x15469bed45d8, 0xc0010f1680, 0xc0007ad960, 0xc000c88be0, 0x1b, 0x20, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1224 +0x26c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/abortspan.(*AbortSpan).Del(0xc0003f19c0, 0x63ee280, 0xc0006d5fb0, 0x15469bed45d8, 0xc0010f1680, 0xc0007ad960, 0x5045e8fe82e986ec, 0xff254c5d742b6b8c, 0x1, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/abortspan/abortspan.go:125 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.UpdateAbortSpan(0x63ee280, 0xc0006d5fb0, 0x647b4c0, 0xc00182ea10, 0x15469bed45d8, 0xc0010f1680, 0xc0007ad960, 0x5045e8fe82e986ec, 0xff254c5d742b6b8c, 0xc00036e8f0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/transaction.go:102 +0x466
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.resolveLocalLocks(0x63ee280, 0xc0006d5fb0, 0xc00101e420, 0x15469bed45d8, 0xc0010f1680, 0xc0007ad960, 0xc0010f15f0, 0xc001a838c0, 0x647b4c0, 0xc00182ea10, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_end_transaction.go:556 +0x5c4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.EndTxn(0x63ee280, 0xc0006d5fb0, 0x15469bed45d8, 0xc0010f1680, 0x647b4c0, 0xc00182ea10, 0x160a82a892fc2e3f, 0x1, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_end_transaction.go:342 +0xc7d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand(0x63ee280, 0xc0006d5fb0, 0xc00091c138, 0x8, 0x0, 0x15469bed45d8, 0xc0010f1680, 0x647b4c0, 0xc00182ea10, 0xc0007ad960, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:471 +0x2bf
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch(0x63ee280, 0xc0006d5fb0, 0xc00091c138, 0x8, 0x15469bed45d8, 0xc0010f1680, 0x647b4c0, 0xc00182ea10, 0xc0007ad960, 0xc0014a4600, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:241 +0x492
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper(0xc00080f800, 0x63ee280, 0xc0006d5fb0, 0xc00091c138, 0x8, 0x647b4c0, 0xc00182ea10, 0xc0007ad960, 0xc0014a4600, 0xc0014ac540, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:555 +0x155
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes(0xc00080f800, 0x63ee280, 0xc0006d5fb0, 0xc00091c138, 0x8, 0x647b4c0, 0xc00182ea10, 0xc0007ad960, 0xc0014a4600, 0xc0014ac540, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:524 +0x177
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch(0xc00080f800, 0x63ee280, 0xc0006d5fb0, 0xc00091c138, 0x8, 0xc0014a4600, 0xc0014ac540, 0x0, 0x8, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:347 +0x1f0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal(0xc00080f800, 0x63ee280, 0xc0006d5fb0, 0xc00091c138, 0x8, 0xc0014a4600, 0xc0014ac540, 0x0, 0xc000aedc00, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:731 +0x13c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal(0xc00080f800, 0x63ee280, 0xc0006d5fb0, 0xc00091c138, 0x8, 0xc0014a4600, 0xc0014ac540, 0xc000c88b60, 0xc000aedd90)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:851 +0x9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc00080f800, 0x63ee280, 0xc0006d5fb0, 0xc0014a4600, 0xc001b85650, 0xc000aeeae0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:73 +0x10b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc00080f800, 0x63ee280, 0xc0006d5fb0, 0xc0014a4600, 0x0, 0x0, 0x0, 0x100000001, 0x1, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:133 +0xa5b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00080f800, 0x63ee280, 0xc0006d5fb0, 0xc0014a4600, 0x55903b0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:247 +0x5b7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00080f800, 0x63ee280, 0xc0006d5fb0, 0x20, 0xc0014a4600, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:94 +0xa8c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00080f800, 0x63ee280, 0xc0006d5f50, 0x160a82a892fc2e3f, 0x1, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0xce
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0006d9500, 0x63ee280, 0xc0006d5ec0, 0x160a82a892fc2e3f, 0x1, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:202 +0x6c4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc000682900, 0x63ee280, 0xc0006d5ec0, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:177 +0x206
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x63ee280, 0xc0006d5e60, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:850 +0x26b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc001168460, 0x63ee280, 0xc0006d5e60, 0x542fda4, 0x10, 0xc000aefef8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x151
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0002b4a00, 0x63ee280, 0xc0006d5e60, 0xc0014a4400, 0xc0006d5e60, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:838 +0x2af
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0002b4a00, 0x63ee280, 0xc0006d5e30, 0xc0014a4400, 0x7db0ec, 0xc0006d5d70, 0x6d4ddc)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:876 +0xb0
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:537
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc0016abe40, 0x63ee280, 0xc0006d5e30, 0x1, 0x63b1c40, 0xc0003bb1b0, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:199 +0x1d7
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc0016abe40, 0x63ee280, 0xc0006d5d70, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:168 +0x21c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext(0x640df00, 0xc0016abe40, 0x63ee280, 0xc0006d5d70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x3fe
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc0014187e0, 0x63ee280, 0xc0006d5d70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1672 +0x348
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendRPC(0xc0014187e0, 0x63ee280, 0xc0006d5d70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:466 +0x32c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendSingleRange(0xc0014187e0, 0x63ee280, 0xc0006d5d70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:548 +0x3e1
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc0014187e0, 0x63ee280, 0xc0006d5d70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1429 +0x546
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc0014187e0, 0x63ee280, 0xc0006d5d70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1089 +0x1d90
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc0014187e0, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:740 +0xa99
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0xc00135c8d0, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:86 +0x195
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked(0xc00135c898, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:46 +0xf4
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked(0xc00135c868, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:190 +0x8ba
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc00135c7d0, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:247 +0x108
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked(0xc00135c7d0, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:182 +0x2cb
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked(0xc00135c710, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:252 +0x1ea
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked(0xc00135c6f0, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:105 +0x2c3
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked(0xc00135c650, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:172 +0xff
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0xc00135c480, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:503 +0x6b7
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc000162d80, 0x63ee280, 0xc001765740, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:739 +0x174
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0xc001366000, 0x63ee280, 0xc001765740, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:911 +0x1d2
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).rollback(0xc001366000, 0x63ee280, 0xc001765740, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:709 +0x5fc
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).CleanupOnError(0xc001366000, 0x63ee280, 0xc001765740, 0x636f8e0, 0xc000e752c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:603 +0x9e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).AdminMerge(0xc00080f800, 0x63ee280, 0xc001765740, 0xc000392330, 0xc, 0x10, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_command.go:773 +0x3a6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeAdminBatch(0xc00080f800, 0x63ee280, 0xc001765740, 0xc00181d400, 0xa, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:466 +0xe60
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00080f800, 0x63ee280, 0xc001765740, 0x20, 0xc00181d400, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:101 +0x8de
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00080f800, 0x63ee280, 0xc0017656e0, 0x160a82a87d9e7645, 0x0, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0xce
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0006d9500, 0x63ee280, 0xc001765650, 0x160a82a87d9e7645, 0x0, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:202 +0x6c4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc000682900, 0x63ee280, 0xc001765650, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:177 +0x206
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x63ee280, 0xc001765620, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:850 +0x26b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc001168460, 0x63ee280, 0xc001765620, 0x542fda4, 0x10, 0xc0016106b0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x151
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0002b4a00, 0x63ee280, 0xc001765620, 0xc00181d280, 0xc001765620, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:838 +0x2af
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0002b4a00, 0x63ee280, 0xc0017655f0, 0xc00181d280, 0x7db0ec, 0xc001765500, 0x6d4ddc)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:876 +0xb0
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:537
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc000c036c0, 0x63ee280, 0xc0017655f0, 0x1, 0x63b1c40, 0xc0003bb1b0, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:199 +0x1d7
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc000c036c0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:168 +0x21c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext(0x640df00, 0xc000c036c0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x3fe
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc0014187e0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1672 +0x348
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendRPC(0xc0014187e0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:466 +0x32c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendSingleRange(0xc0014187e0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:548 +0x3e1
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc0014187e0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1429 +0x546
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc0014187e0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1089 +0x1d90
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc0014187e0, 0x63ee280, 0xc0017654a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:740 +0xa99
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc000162de0, 0x63ee280, 0xc001765440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:223 +0xe0
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc000162d80, 0x63ee280, 0xc001765440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:739 +0x174
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:721
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill(0x63ee280, 0xc001765440, 0xc001613778, 0xc00133cc00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:657 +0x165
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run(0xc000162d80, 0x63ee280, 0xc001765440, 0xc00133cc00, 0xc0016137e8, 0x67bbdd)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:680 +0xd2
github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminMerge(0xc000162d80, 0x63ee280, 0xc001765440, 0x4e11c80, 0xc000345a80, 0x15, 0xa)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:510 +0xce
github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.applyOp(0x63ee280, 0xc001765440, 0xc000162d80, 0xc001a27420)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/applier.go:78 +0x793
github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*Applier).Apply(0xc0003b2240, 0x63ee280, 0xc001765440, 0xc001a27420, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/applier.go:58 +0x179
github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis.func2(0x63ee1c0, 0xc00158fb80, 0x1, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:64 +0x2d5
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers.func1(0x63ee1c0, 0xc00158fb80, 0xc001168460, 0x6a06aa)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:175 +0x5e
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0xc00009ff68, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4f
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000cef6e0, 0xc000cef770)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x72
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x74
. ba: [txn: ec86e982], EndTxn(commit:false tsflex:true) [/Local/Range/Table/36/RangeDescriptor] . txn: "merge" meta={id=ec86e982 key=/Local/Range/Table/36/RangeDescriptor pri=0.02359217 epo=1 ts=1588225479.133638207,1 min=1588225478.924505396,0 seq=1} lock=true stat=ABORTED rts=1588225479.133638207,1 wto=false max=1588225478.924505396,0.
goroutine 2101 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x94a0801, 0x0, 0x0, 0xc000af3918)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xc6
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0x8ad5ba0, 0xc000000004, 0x8486d2e, 0x27, 0x329, 0xc001c26000, 0x5647)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:212 +0xe04
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x63ee280, 0xc0006d5d10, 0xc000000004, 0x2, 0x54ce14e, 0x3c, 0xc000af4120, 0x4, 0x4)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x291
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x63ee280, 0xc0006d5d10, 0x1, 0xc000000004, 0x54ce14e, 0x3c, 0xc000af4120, 0x4, 0x4)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:44 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:155
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked(0xc00135c480, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:809 +0x720
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0xc00135c480, 0x63ee280, 0xc0006d5d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:505 +0x74d
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc000162d80, 0x63ee280, 0xc001765740, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:739 +0x174
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0xc001366000, 0x63ee280, 0xc001765740, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:911 +0x1d2
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).rollback(0xc001366000, 0x63ee280, 0xc001765740, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:709 +0x5fc
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).CleanupOnError(0xc001366000, 0x63ee280, 0xc001765740, 0x636f8e0, 0xc000e752c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:603 +0x9e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).AdminMerge(0xc00080f800, 0x63ee280, 0xc001765740, 0xc000392330, 0xc, 0x10, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_command.go:773 +0x3a6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeAdminBatch(0xc00080f800, 0x63ee280, 0xc001765740, 0xc00181d400, 0xa, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:466 +0xe60
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00080f800, 0x63ee280, 0xc001765740, 0x20, 0xc00181d400, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:101 +0x8de
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00080f800, 0x63ee280, 0xc0017656e0, 0x160a82a87d9e7645, 0x0, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0xce
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0006d9500, 0x63ee280, 0xc001765650, 0x160a82a87d9e7645, 0x0, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:202 +0x6c4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc000682900, 0x63ee280, 0xc001765650, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:177 +0x206
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x63ee280, 0xc001765620, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:850 +0x26b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc001168460, 0x63ee280, 0xc001765620, 0x542fda4, 0x10, 0xc0016106b0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x151
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0002b4a00, 0x63ee280, 0xc001765620, 0xc00181d280, 0xc001765620, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:838 +0x2af
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0002b4a00, 0x63ee280, 0xc0017655f0, 0xc00181d280, 0x7db0ec, 0xc001765500, 0x6d4ddc)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:876 +0xb0
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:537
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc000c036c0, 0x63ee280, 0xc0017655f0, 0x1, 0x63b1c40, 0xc0003bb1b0, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:199 +0x1d7
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc000c036c0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:168 +0x21c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext(0x640df00, 0xc000c036c0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x3fe
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc0014187e0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1672 +0x348
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendRPC(0xc0014187e0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:466 +0x32c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendSingleRange(0xc0014187e0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:548 +0x3e1
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc0014187e0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1429 +0x546
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc0014187e0, 0x63ee280, 0xc001765500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1089 +0x1d90
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc0014187e0, 0x63ee280, 0xc0017654a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:740 +0xa99
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc000162de0, 0x63ee280, 0xc001765440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:223 +0xe0
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc000162d80, 0x63ee280, 0xc001765440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:739 +0x174
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:721
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill(0x63ee280, 0xc001765440, 0xc001613778, 0xc00133cc00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:657 +0x165
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run(0xc000162d80, 0x63ee280, 0xc001765440, 0xc00133cc00, 0xc0016137e8, 0x67bbdd)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:680 +0xd2
github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminMerge(0xc000162d80, 0x63ee280, 0xc001765440, 0x4e11c80, 0xc000345a80, 0x15, 0xa)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:510 +0xce
github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.applyOp(0x63ee280, 0xc001765440, 0xc000162d80, 0xc001a27420)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/applier.go:78 +0x793
github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*Applier).Apply(0xc0003b2240, 0x63ee280, 0xc001765440, 0xc001a27420, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/applier.go:58 +0x179
github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis.func2(0x63ee1c0, 0xc00158fb80, 0x1, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:64 +0x2d5
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers.func1(0x63ee1c0, 0xc00158fb80, 0xc001168460, 0x6a06aa)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:175 +0x5e
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0xc00009ff68, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4f
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000cef6e0, 0xc000cef770)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x72
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x74

Log preceding fatal error

  3 OP      b.Get(ctx, /Table/50/"d6a8af78") // ("v-30", nil)
  3 OP      txn.Run(ctx, b) // nil
  3 OP    }
  3 OP    {
  3 OP      b := &Batch{}
  3 OP      b.Get(ctx, /Table/50/"fe5f4295") // ("v-31", nil)
  3 OP      b.Get(ctx, /Table/50/"fe5f4295") // ("v-31", nil)
  3 OP      txn.Run(ctx, b) // nil
  3 OP    }
  3 OP    b := &Batch{}
  3 OP    b.Put(ctx, /Table/50/"47180bf9", v-34) // nil
  3 OP    txn.CommitInBatch(ctx, b) // nil
  3 OP    return nil
  3 OP  }) // nil txnpb:("unnamed" meta={id=08b25139 key=/Table/50/"47180bf9" pri=0.04817643 epo=0 ts=1588225478.860734864,0 min=1588225478.860734864,0 seq=0} lock=true stat=COMMITTED rts=1588225478.860734864,0 wto=false max=1588225479.360734864,0)
  after: 1588225478.870499882,0
I200430 05:44:38.934642 2101 kv/kvserver/replica_command.go:663  [n1,s1,r32/1:/Table/{36-50/"84ad2…}] initiating a merge of r33:/{Table/50/"84ad26e7"-Max} [(n1,s1):1, next=2, gen=1, sticky=9223372036.854775807,2147483647] into this range (manual)
I200430 05:44:38.942071 2100 kv/kvnemesis/kvnemesis.go:78  
  before: 1588225478.941230848,0
  0 OP  db0.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
  0 OP    b := &Batch{}
  0 OP    txn.CommitInBatch(ctx, b) // nil
  0 OP    return nil
  0 OP  }) // nil txnpb:("unnamed" meta={id=2b5a5db2 key=/Min pri=0.02027838 epo=0 ts=1588225478.941242508,0 min=1588225478.941242508,0 seq=0} lock=false stat=COMMITTED rts=1588225478.941242508,0 wto=false max=1588225479.441242508,0)
  after: 1588225478.941397657,0
I200430 05:44:39.129343 2102 kv/kvserver/replica_command.go:663  [n1,s1,r32/1:/Table/{36-50/"84ad2…}] initiating a merge of r33:/{Table/50/"84ad26e7"-Max} [(n1,s1):1, next=2, gen=1, sticky=9223372036.854775807,2147483647] into this range (manual)
I200430 05:44:39.136049 2103 kv/kvnemesis/kvnemesis.go:78  
  before: 1588225479.116716853,0
  3 OP  db0.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
  3 OP    txn.Get(ctx, /Table/50/"b15a9666") // (nil, nil)
  3 OP    {
  3 OP      b := &Batch{}
  3 OP      b.Get(ctx, /Table/50/"c94d8bbe") // (nil, nil)
  3 OP      b.Get(ctx, /Table/50/"47180bf9") // ("v-34", nil)
  3 OP      txn.Run(ctx, b) // nil
  3 OP    }
  3 OP    {
  3 OP      b := &Batch{}
  3 OP      b.Put(ctx, /Table/50/"82df536e", v-35) // nil
  3 OP      txn.Run(ctx, b) // nil
  3 OP    }
  3 OP    return errors.New("rollback")
  3 OP  }) // rollback
  after: 1588225479.130335295,0
I200430 05:44:39.212509 1854 kv/kvnemesis/watcher.go:165  rangefeed Put /Table/50/"9dd3a04d" 1588225478.759675732,0 -> /BYTES/v-18 (prev /<empty>)
I200430 05:44:39.212980 1854 kv/kvnemesis/watcher.go:165  rangefeed Put /Table/50/"9dd3a04d" 1588225478.845873164,0 -> /BYTES/v-29 (prev /BYTES/v-18)
I200430 05:44:39.213297 1854 kv/kvnemesis/watcher.go:165  rangefeed Put /Table/50/"ab718c4b" 1588225478.759675732,0 -> /BYTES/v-17 (prev /<empty>)
I200430 05:44:39.213704 1854 kv/kvnemesis/watcher.go:165  rangefeed Put /Table/50/"b54dc9fe" 1588225478.845873164,0 -> /BYTES/v-33 (prev /<empty>)
I200430 05:44:39.215904 1854 kv/kvnemesis/watcher.go:165  rangefeed Put /Table/50/"d6a8af78" 1588225478.845873164,0 -> /BYTES/v-30 (prev /<empty>)
I200430 05:44:39.216283 1854 kv/kvnemesis/watcher.go:165  rangefeed Put /Table/50/"f491d173" 1588225478.765659040,0 -> /BYTES/v-22 (prev /<empty>)
I200430 05:44:39.216568 1854 kv/kvnemesis/watcher.go:165  rangefeed Put /Table/50/"fe5f4295" 1588225478.845873164,0 -> /BYTES/v-31 (prev /<empty>)

More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestKVNemesisSingleNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

craig bot pushed a commit that referenced this issue Apr 30, 2020
48190: sql: inject tenant ID in sqlServerArgs, pass through ExecutorConfig r=nvanbenschoten a=nvanbenschoten

Fixes #47903.
Informs #48123.

Also known as "the grand plumbing", this change replaces a few instances of `TODOSQLCodec` in `pkg/sql/sqlbase/index_encoding.go` and watches the house of cards fall apart. It then glues the world back together, this time using a properly injected tenant-bound SQLCodec to encode and decode all SQL table keys.

A tenant ID field is added to `sqlServerArgs`. This is used to construct a tenant-bound `keys.SQLCodec` during server creation. This codec morally lives on the `sql.ExecutorConfig`. In practice, it is also copied onto `tree.EvalContext` and `execinfra.ServerConfig` to help carry it around. SQL code is adapted to use this codec whenever it needs to encode or decode keys.

If all tests pass after this refactor, there is a good chance it got things right. This is because any use of an uninitialized SQLCodec will panic immediately when the codec is first used. This was helpful in ensuring that it was properly plumbed everywhere.

48245: kv: declare write access to AbortSpan on all aborting EndTxn reqs r=nvanbenschoten a=nvanbenschoten

Fixes #43707.
Fixes #48046.
Fixes #48189.

Part of the change made by #42765 was to clear AbortSpan entries on
non-poisoning, aborting EndTxn requests. Specifically, this change was
made in 1328787. The change forgot to update the corresponding span
declaration logic to reflect the fact that we were now writing to the
AbortSpan in cases where we previously weren't.

This was triggering an assertion in race builds that tried to catch this
kind of undeclared span access. The assertion failure was very rare
because it required the following conditions to all be met:
1. running a test with the race detector enabled
2. a txn (A) must have been aborted by another txn (B)
3. txn B must have cleared an intent on txn A's transaction record range
4. txn A must have noticed and issued a non-poisoning EndTxn(ABORT)

We should backport this when we get a change (once v20.1.0 has
stabilized), but I don't expect that this could actually cause any
issues. The AbortSpan update was strictly a matter of performance and we
should never be racing with another request that is trying to read the
same AbortSpan entry.

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
@craig craig bot closed this as completed in 18f8836 Apr 30, 2020
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue May 4, 2020
Fixes cockroachdb#43707.
Fixes cockroachdb#48046.
Fixes cockroachdb#48189.

Part of the change made by cockroachdb#42765 was to clear AbortSpan entries on
non-poisoning, aborting EndTxn requests. Specifically, this change was
made in 1328787. The change forgot to update the corresponding span
declaration logic to reflect the fact that we were now writing to the
AbortSpan in cases where we previously weren't.

This was triggering an assertion in race builds that tried to catch this
kind of undeclared span access. The assertion failure was very rare
because it required the following conditions to all be met:
1. running a test with the race detector enabled
2. a txn (A) must have been aborted by another txn (B)
3. txn B must have cleared an intent on txn A's transaction record range
4. txn A must have noticed and issued a non-poisoning EndTxn(ABORT)

We should backport this when we get a change (once v20.1.0 has
stabilized), but I don't expect that this could actually cause any
issues. The AbortSpan update was strictly a matter of performance and we
should never be racing with another request that is trying to read the
same AbortSpan entry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants