Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccl/importccl: TestImportPgDump failed #48189

Closed
cockroach-teamcity opened this issue Apr 30, 2020 · 3 comments · Fixed by #48245
Closed

ccl/importccl: TestImportPgDump failed #48189

cockroach-teamcity opened this issue Apr 30, 2020 · 3 comments · Fixed by #48245
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

(ccl/importccl).TestImportPgDump failed on master@3b612692db93aa7c87493705e1fad85c9c664f6c:

Fatal error:

F200430 05:26:11.212651 205329 kv/kvclient/kvcoord/txn_coord_sender.go:809  [n3,merge,s3,r36/2:/Table/53/{1/0-2},txn=f97e0cbc] unexpected error with ABORTED txn: (*roachpb.internalError) cannot write undeclared span /Local/RangeID/33/r/AbortSpan/"f97e0cbc-cb3e-4838-a10c-68bc40c32a02"
declared:
read local: /Local/RangeID/33/r/AbortSpan/"f97e0cbc-cb3e-4838-a10c-68bc40c32a02" at 0,0
read local: /Local/Range/Table/53/1/RangeDescriptor at 0,0
write global: /Meta2/Table/53/2 at 1588224370.690895377,0
write global: /Meta2/Table/53/2/NULL/NULL/10 at 1588224370.690895377,0
write global: /Table/13/1/2020-04-30T05:26:10.690895Z/551006932014989315/0 at 1588224370.690895377,0
write global: /Table/13/1/2020-04-30T05:26:10.690895Z/551006932014989315/2/1 at 1588224370.690895377,0
write global: /Table/13/1/2020-04-30T05:26:10.690895Z/551006932014989315/3/1 at 1588224370.690895377,0
write global: /Table/13/1/2020-04-30T05:26:10.690895Z/551006932014989315/4/1 at 1588224370.690895377,0
write global: /Table/13/1/2020-04-30T05:26:10.690895Z/551006932014989315/5/1 at 1588224370.690895377,0
write global: /Table/13/1/2020-04-30T05:26:10.690895Z/551006932014989315/6/1 at 1588224370.690895377,0
write local: /Local/Range/Table/53/1/0/RangeDescriptor at 0,0
write local: /Local/Range/Table/53/1/0/Transaction/"f97e0cbc-cb3e-4838-a10c-68bc40c32a02" at 0,0
write local: /Local/Range/Table/53/2/RangeDescriptor at 0,0

stack:

Stack:

goroutine 205784 [running]:
runtime/debug.Stack(0xc0002c1e60, 0xb, 0x10)
	/usr/local/go/src/runtime/debug/stack.go:24 +0xab
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).checkAllowed(0xc002f47800, 0x1, 0xc0014be020, 0x1b, 0x20, 0x0, 0x0, 0x0, 0x5a96a48, 0xc0057c44a0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:293 +0x3ad
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).CheckAllowed(0xc002f47800, 0x1, 0xc0014be020, 0x1b, 0x20, 0x0, 0x0, 0x0, 0xc00522f4f0, 0x1460400975ae)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:224 +0xb5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.spanSetWriter.Clear(0x14607ae5a110, 0xc002524c00, 0xc002f47800, 0x1, 0x0, 0x0, 0xc0014be020, 0x1b, 0x20, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/batch.go:386 +0xdd
github.com/cockroachdb/cockroach/pkg/storage.mvccPutInternal(0x68d1860, 0xc001ddbb00, 0x14607ae5c960, 0xc0028eb830, 0x6956440, 0xc00522f4f0, 0xc0017fdc70, 0xc0014be020, 0x1b, 0x20, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1463 +0x6dd
github.com/cockroachdb/cockroach/pkg/storage.mvccPutUsingIter(0x68d1860, 0xc001ddbb00, 0x14607ae5c960, 0xc0028eb830, 0x6956440, 0xc00522f4f0, 0xc0017fdc70, 0xc0014be020, 0x1b, 0x20, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1254 +0x18f
github.com/cockroachdb/cockroach/pkg/storage.MVCCDelete(0x68d1860, 0xc001ddbb00, 0x14607ae5c8c8, 0xc0028eb830, 0xc0017fdc70, 0xc0014be020, 0x1b, 0x20, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1224 +0x26c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/abortspan.(*AbortSpan).Del(0xc000328bf0, 0x68d1860, 0xc001ddbb00, 0x14607ae5c8c8, 0xc0028eb830, 0xc0017fdc70, 0x38483ecbbc0c7ef9, 0x22ac340bc680ca1, 0x1, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/abortspan/abortspan.go:125 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.UpdateAbortSpan(0x68d1860, 0xc001ddbb00, 0x69679c0, 0xc00072bb90, 0x14607ae5c8c8, 0xc0028eb830, 0xc0017fdc70, 0x38483ecbbc0c7ef9, 0x22ac340bc680ca1, 0xc0002c1e20, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/transaction.go:102 +0x466
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.resolveLocalLocks(0x68d1860, 0xc001ddbb00, 0xc002ba1500, 0x14607ae5c8c8, 0xc0028eb830, 0xc0017fdc70, 0xc0028eb7a0, 0xc0038d26c0, 0x69679c0, 0xc00072bb90, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_end_transaction.go:556 +0x5c4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.EndTxn(0x68d1860, 0xc001ddbb00, 0x14607ae5c8c8, 0xc0028eb830, 0x69679c0, 0xc00072bb90, 0x160a81a68100edd3, 0x1, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_end_transaction.go:342 +0xc7d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand(0x68d1860, 0xc001ddbb00, 0xc0002c1f18, 0x8, 0x0, 0x14607ae5c8c8, 0xc0028eb830, 0x69679c0, 0xc00072bb90, 0xc0017fdc70, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:471 +0x2bf
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch(0x68d1860, 0xc001ddbb00, 0xc0002c1f18, 0x8, 0x14607ae5c8c8, 0xc0028eb830, 0x69679c0, 0xc00072bb90, 0xc0017fdc70, 0xc002b76280, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:241 +0x492
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper(0xc001daf000, 0x68d1860, 0xc001ddbb00, 0xc0002c1f18, 0x8, 0x69679c0, 0xc00072bb90, 0xc0017fdc70, 0xc002b76280, 0xc002f47800, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:555 +0x155
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes(0xc001daf000, 0x68d1860, 0xc001ddbb00, 0xc0002c1f18, 0x8, 0x69679c0, 0xc00072bb90, 0xc0017fdc70, 0xc002b76280, 0xc002f47800, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:524 +0x177
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch(0xc001daf000, 0x68d1860, 0xc001ddbb00, 0xc0002c1f18, 0x8, 0xc002b76280, 0xc002f47800, 0x0, 0x10, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:347 +0x1f0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal(0xc001daf000, 0x68d1860, 0xc001ddbb00, 0xc0002c1f18, 0x8, 0xc002b76280, 0xc002f47800, 0x0, 0xc00440b000, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:731 +0x13c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal(0xc001daf000, 0x68d1860, 0xc001ddbb00, 0xc0002c1f18, 0x8, 0xc002b76280, 0xc002f47800, 0x8f381e3, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:851 +0x9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc001daf000, 0x68d1860, 0xc001ddbb00, 0xc002b76280, 0xc00072b960, 0xc0057cbe90, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:73 +0x10b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc001daf000, 0x68d1860, 0xc001ddbb00, 0xc002b76280, 0x0, 0x0, 0xc000329200, 0x100000001, 0x1, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:133 +0xa5b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc001daf000, 0x68d1860, 0xc001ddbb00, 0xc002b76280, 0x5a96558, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:247 +0x5b7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc001daf000, 0x68d1860, 0xc001ddbb00, 0x21, 0xc002b76280, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:94 +0xa8c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc001daf000, 0x68d1860, 0xc001ddbad0, 0x160a81a68100edd3, 0x1, 0x100000001, 0x1, 0x0, 0x21, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0xce
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc00347c700, 0x68d1860, 0xc001ddba10, 0x160a81a68100edd3, 0x1, 0x100000001, 0x1, 0x0, 0x21, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:202 +0x6c4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc001801050, 0x68d1860, 0xc001ddba10, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x21, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:177 +0x206
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x68d1860, 0xc001ddb9e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:850 +0x26b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc003509540, 0x68d1860, 0xc001ddb9e0, 0x591e929, 0x10, 0xc00440d2a8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x151
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc001016500, 0x68d1860, 0xc001ddb9e0, 0xc00119dc00, 0xc001ddb9e0, 0x716e1b, 0xc00004e000)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:838 +0x2af
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc001016500, 0x68d1860, 0xc001ddb920, 0xc00119dc00, 0xc001016500, 0xc00029ffc0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:876 +0xb0
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0x68d1860, 0xc001ddb920, 0x58d3c60, 0xc00119dc00, 0x58d3c60, 0xc00119dc00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:10201 +0xa2
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x68d1860, 0xc001ddb920, 0x58d3c60, 0xc00119dc00, 0xc00473ef40, 0xc00473ef60, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:44 +0xbd9
github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x68d1860, 0xc001ddb920, 0x58d3c60, 0xc00119dc00, 0xc00473ef40, 0xc00473ef60, 0xc00473ef60, 0x9210c60, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:224 +0x10f
github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func4(0x68d1860, 0xc001ddb920, 0x58d3c60, 0xc00119dc00, 0xc00473ef40, 0xc00473ef60, 0xc001fd7500, 0xc002382080, 0xc0049ad870, 0x71a728)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:254 +0xea
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x58872a0, 0xc001016500, 0x68d1860, 0xc001ddb920, 0xc002f47680, 0xc000c2e130, 0x68d1860, 0xc001ddb920, 0xc002467800, 0x1a5)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:10203 +0x1d9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc00170a480, 0x692ebc0, 0xc000ee8d80, 0xc000c59e00, 0xc000d594d0, 0x8ee4270, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:999 +0x995
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00170a480, 0x692ebc0, 0xc000ee8d80, 0xc000c59e00, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1279 +0x1344
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0042ccaf0, 0xc00170a480, 0x692ebc0, 0xc000ee8d80, 0xc000c59e00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0xc9
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xb9
. ba: [txn: f97e0cbc], EndTxn(commit:false tsflex:true) [/Local/Range/Table/53/1/0/RangeDescriptor] . txn: "merge" meta={id=f97e0cbc key=/Local/Range/Table/53/1/0/RangeDescriptor pri=0.01623891 epo=1 ts=1588224370.730397139,1 min=1588224370.690895377,0 seq=1} lock=true stat=ABORTED rts=1588224370.730397139,1 wto=false max=1588224370.807037830,0.
goroutine 205329 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x9be4201, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xc6
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0x9213120, 0xc000000004, 0x8b2b8b7, 0x27, 0x329, 0xc0046e5000, 0x2d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:212 +0xe04
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x68d1860, 0xc004218570, 0xc000000004, 0x2, 0x59c9775, 0x3c, 0xc003eea2a0, 0x4, 0x4)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x291
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x68d1860, 0xc004218570, 0x1, 0xc000000004, 0x59c9775, 0x3c, 0xc003eea2a0, 0x4, 0x4)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:44 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:155
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked(0xc00234e000, 0x68d1860, 0xc004218570, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:809 +0x720
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0xc00234e000, 0x68d1860, 0xc004218570, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:505 +0x74d
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc00119ca80, 0x68d1820, 0xc003f90060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:739 +0x174
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0xc003ef67e0, 0x68d1820, 0xc003f90060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:911 +0x1d2
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).rollback(0xc003ef67e0, 0x68d1820, 0xc003f90060, 0x4)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:709 +0x5fc
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).CleanupOnError(0xc003ef67e0, 0x68d1820, 0xc003f90060, 0x6849c60, 0xc003837400)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:603 +0x9e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).AdminMerge(0xc0026c9000, 0x68d1820, 0xc003f90060, 0xc0034f3ca0, 0x3, 0x8, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_command.go:773 +0x3a6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*mergeQueue).process(0xc0002d9560, 0x68d1820, 0xc003f90060, 0xc0026c9000, 0xc005b68e10, 0x5916a82, 0xd)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/merge_queue.go:326 +0x171c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1(0x68d1820, 0xc003f90060, 0xdf8475800, 0x68d1820)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:955 +0x24e
github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout(0x68d1820, 0xc003f90060, 0xc001f8c960, 0x1e, 0xdf8475800, 0xc001a6ed98, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:135 +0xde
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica(0xc0012b4580, 0x68d1860, 0xc005489230, 0x693c1c0, 0xc0026c9000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:914 +0x3aa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func1.2(0x68d1860, 0xc005488fc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:842 +0x116
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000797a40, 0x68d1860, 0xc005488fc0, 0xc002dcc270, 0x29, 0x0, 0x0, 0xc0054890e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x163
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x14b

Log preceding fatal error

=== RUN   TestImportPgDump/single_table_dump
I200430 05:26:10.392608 203630 sql/table.go:992  [n1,client=127.0.0.1:49508,hostssl,user=root] queued new schema change job for table 53, mutation 0
I200430 05:26:10.392959 203630 sql/sqlbase/structured.go:1559  [n1,client=127.0.0.1:49508,hostssl,user=root] publish: descID=53 (simple) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200430 05:26:10.534465 203630 sql/table.go:992  [n1,client=127.0.0.1:49508,hostssl,user=root] queued new schema change job for table 53, mutation 0
I200430 05:26:10.534681 203630 sql/sqlbase/structured.go:1559  [n1,client=127.0.0.1:49508,hostssl,user=root] publish: descID=53 (simple) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200430 05:26:10.605970 203630 sql/event_log.go:132  [n1,client=127.0.0.1:49508,hostssl,user=root] Event: "drop_table", target: 53, info: {TableName:foo.public.simple Statement:DROP TABLE IF EXISTS simple, second User:root CascadeDroppedViews:[]}
I200430 05:26:10.606178 203630 sql/event_log.go:132  [n1,client=127.0.0.1:49508,hostssl,user=root] Event: "drop_table", target: 53, info: {TableName:foo.public.simple Statement:DROP TABLE IF EXISTS simple, second User:root CascadeDroppedViews:[]}
I200430 05:26:10.606467 203630 jobs/registry.go:288  [n1,client=127.0.0.1:49508,hostssl,user=root] scheduled jobs [551006930586796033]
I200430 05:26:10.733173 199704 jobs/registry.go:1187  job 551006930586796033: resuming execution
I200430 05:26:10.750398 205275 kv/kvserver/replica_command.go:663  [n1,merge,s1,r33/1:/Table/53/1{-/0}] initiating a merge of r36:/Table/53/{1/0-2} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=7] into this range (lhs+rhs has (size=0 B+19 KiB=19 KiB qps=5.95+1.68=7.62qps) below threshold (size=128 MiB, qps=1250.00))
I200430 05:26:10.755039 205392 jobs/registry.go:806  [n1] job 551006930586796033: stepping through state running with error <nil>
I200430 05:26:10.758700 205392 sql/schema_changer.go:418  [n1,job=551006930586796033,scExec] schema change on simple (53 v3) mutation 0 starting execution...
I200430 05:26:10.812259 205329 kv/kvserver/replica_command.go:663  [n3,merge,s3,r36/2:/Table/53/{1/0-2}] initiating a merge of r34:/Table/53/2{-/NULL/NULL/10} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=8] into this range (lhs+rhs has (size=19 KiB+0 B=19 KiB qps=1.68+4.95=6.63qps) below threshold (size=128 MiB, qps=1250.00))
I200430 05:26:10.821823 205392 sql/sqlbase/structured.go:1559  [n1,job=551006930586796033,scExec] publish: descID=53 (simple) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200430 05:26:11.058963 205392 sql/schema_changer.go:706  [n1,job=551006930586796033,scExec] waiting for a single version of table 53...
I200430 05:26:11.086592 205502 jobs/registry.go:806  [n1] job 551006932006076417: stepping through state running with error <nil>
I200430 05:26:11.086988 205392 sql/schema_changer.go:708  [n1,job=551006930586796033,scExec] waiting for a single version of table 53... done (at v 4)
I200430 05:26:11.087217 205392 jobs/registry.go:806  [n1] job 551006930586796033: stepping through state succeeded with error <nil>
I200430 05:26:11.105978 199517 kv/kvserver/store_remove_replica.go:122  [n1,s1,r33/1:/Table/53/1{-/0}] removing replica r36/1
I200430 05:26:11.108713 200831 kv/kvserver/store_remove_replica.go:122  [n3,s3,r33/2:/Table/53/1{-/0}] removing replica r36/2
I200430 05:26:11.112678 200445 kv/kvserver/store_remove_replica.go:122  [n2,s2,r33/3:/Table/53/1{-/0}] removing replica r36/3
I200430 05:26:11.160185 205392 jobs/registry.go:961  [n1] job 551006930586796033: status succeeded after adoption finished

More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestImportPgDump PKG=./pkg/ccl/importccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 30, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Apr 30, 2020
@maddyblue maddyblue assigned andreimatei and unassigned maddyblue Apr 30, 2020
@maddyblue
Copy link
Contributor

@andreimatei is this also you?

@andreimatei
Copy link
Contributor

I guess so... A bunch of things seem to go wrong here; not sure what to make of it yet.

@nvanbenschoten
Copy link
Member

I see what's going wrong here. This is fallout from #42765. I'll send a patch.

craig bot pushed a commit that referenced this issue Apr 30, 2020
48190: sql: inject tenant ID in sqlServerArgs, pass through ExecutorConfig r=nvanbenschoten a=nvanbenschoten

Fixes #47903.
Informs #48123.

Also known as "the grand plumbing", this change replaces a few instances of `TODOSQLCodec` in `pkg/sql/sqlbase/index_encoding.go` and watches the house of cards fall apart. It then glues the world back together, this time using a properly injected tenant-bound SQLCodec to encode and decode all SQL table keys.

A tenant ID field is added to `sqlServerArgs`. This is used to construct a tenant-bound `keys.SQLCodec` during server creation. This codec morally lives on the `sql.ExecutorConfig`. In practice, it is also copied onto `tree.EvalContext` and `execinfra.ServerConfig` to help carry it around. SQL code is adapted to use this codec whenever it needs to encode or decode keys.

If all tests pass after this refactor, there is a good chance it got things right. This is because any use of an uninitialized SQLCodec will panic immediately when the codec is first used. This was helpful in ensuring that it was properly plumbed everywhere.

48245: kv: declare write access to AbortSpan on all aborting EndTxn reqs r=nvanbenschoten a=nvanbenschoten

Fixes #43707.
Fixes #48046.
Fixes #48189.

Part of the change made by #42765 was to clear AbortSpan entries on
non-poisoning, aborting EndTxn requests. Specifically, this change was
made in 1328787. The change forgot to update the corresponding span
declaration logic to reflect the fact that we were now writing to the
AbortSpan in cases where we previously weren't.

This was triggering an assertion in race builds that tried to catch this
kind of undeclared span access. The assertion failure was very rare
because it required the following conditions to all be met:
1. running a test with the race detector enabled
2. a txn (A) must have been aborted by another txn (B)
3. txn B must have cleared an intent on txn A's transaction record range
4. txn A must have noticed and issued a non-poisoning EndTxn(ABORT)

We should backport this when we get a change (once v20.1.0 has
stabilized), but I don't expect that this could actually cause any
issues. The AbortSpan update was strictly a matter of performance and we
should never be racing with another request that is trying to read the
same AbortSpan entry.

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
@craig craig bot closed this as completed in 18f8836 Apr 30, 2020
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue May 4, 2020
Fixes cockroachdb#43707.
Fixes cockroachdb#48046.
Fixes cockroachdb#48189.

Part of the change made by cockroachdb#42765 was to clear AbortSpan entries on
non-poisoning, aborting EndTxn requests. Specifically, this change was
made in 1328787. The change forgot to update the corresponding span
declaration logic to reflect the fact that we were now writing to the
AbortSpan in cases where we previously weren't.

This was triggering an assertion in race builds that tried to catch this
kind of undeclared span access. The assertion failure was very rare
because it required the following conditions to all be met:
1. running a test with the race detector enabled
2. a txn (A) must have been aborted by another txn (B)
3. txn B must have cleared an intent on txn A's transaction record range
4. txn A must have noticed and issued a non-poisoning EndTxn(ABORT)

We should backport this when we get a change (once v20.1.0 has
stabilized), but I don't expect that this could actually cause any
issues. The AbortSpan update was strictly a matter of performance and we
should never be racing with another request that is trying to read the
same AbortSpan entry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants