Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv: TestRangeLookupWithOpenTransaction failed under stress #25511

Closed
cockroach-teamcity opened this issue May 15, 2018 · 3 comments
Closed

kv: TestRangeLookupWithOpenTransaction failed under stress #25511

cockroach-teamcity opened this issue May 15, 2018 · 3 comments
Assignees
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/9376ca50f874e475189b1fe1f7ef8f4e0f6c18b4

Parameters:

TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=658872&tab=buildLog


I180515 10:29:54.039640 23765 server/config.go:539  [n?] 1 storage engine initialized
I180515 10:29:54.039667 23765 server/config.go:542  [n?] RocksDB cache size: 128 MiB
I180515 10:29:54.039673 23765 server/config.go:542  [n?] store 0: in-memory, size 0 B
I180515 10:29:54.042374 23765 server/node.go:376  [n?] **** cluster b9b15181-73f3-4a28-b20f-6d2421fb098c has been created
I180515 10:29:54.042402 23765 server/server.go:1351  [n?] **** add additional nodes by specifying --join=127.0.0.1:39509
I180515 10:29:54.043093 23765 storage/store.go:1431  [n1,s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180515 10:29:54.043437 23765 server/node.go:506  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=6.9 KiB), ranges=1, leases=0, writes=0.00, bytesPerReplica={p10=7043.00 p25=7043.00 p50=7043.00 p75=7043.00 p90=7043.00 pMax=7043.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I180515 10:29:54.043486 23765 server/node.go:354  [n1] node ID 1 initialized
I180515 10:29:54.043552 23765 gossip/gossip.go:333  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:39509" > attrs:<> locality:<> ServerVersion:<major_val:2 minor_val:0 patch:0 unstable:3 > 
I180515 10:29:54.043623 23765 storage/stores.go:222  [n1] read 0 node addresses from persistent storage
I180515 10:29:54.043762 23765 server/node.go:647  [n1] connecting to gossip network to verify cluster ID...
I180515 10:29:54.043797 23765 server/node.go:672  [n1] node connected via gossip and verified as part of cluster "b9b15181-73f3-4a28-b20f-6d2421fb098c"
I180515 10:29:54.043833 23765 server/node.go:440  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180515 10:29:54.043951 23765 server/server.go:1476  [n1] starting https server at 127.0.0.1:36557
I180515 10:29:54.043977 23765 server/server.go:1477  [n1] starting grpc/postgres server at 127.0.0.1:39509
I180515 10:29:54.049678 23765 server/server.go:1478  [n1] advertising CockroachDB node at 127.0.0.1:39509
W180515 10:29:54.049725 23765 sql/jobs/registry.go:286  [n1] unable to get node liveness: node not in the liveness table
I180515 10:29:54.058840 24325 sql/event_log.go:124  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I180515 10:29:54.069103 24204 sql/event_log.go:124  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:$1 User:root}
I180515 10:29:54.078389 24328 sql/event_log.go:124  [n1,intExec=disableNetTrace] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:root}
I180515 10:29:54.089010 24338 sql/event_log.go:124  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:gen_random_uuid()::STRING User:root}
I180515 10:29:54.091563 23765 server/server.go:1555  [n1] done ensuring all necessary migrations have run
I180515 10:29:54.091600 23765 server/server.go:1558  [n1] serving sql connections
I180515 10:29:54.094752 24346 sql/event_log.go:124  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:39509} Attrs: Locality: ServerVersion:2.0-3} ClusterID:b9b15181-73f3-4a28-b20f-6d2421fb098c StartedAt:1526380194043802611 LastUp:1526380194043802611}
I180515 10:29:54.095214 23765 util/stop/stopper.go:471  quiescing; tasks left:
1      storage.intentResolver: processing intents
1      node.Node: batch
1      [async] auto-upgrade
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x103c915]

goroutine 24209 [running]:
github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*DString).Size(0x0, 0xc420d423e8)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/datum.go:933 +0x5
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*RowContainer).rowSize(0xc420dea3f0, 0xc420d424c8, 0x1, 0x1, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/row_container.go:247 +0x66
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*RowContainer).AddRow(0xc420dea3f0, 0x24d8c60, 0xc42118c450, 0xc420d424c8, 0x1, 0x1, 0x7, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/row_container.go:272 +0x7c
github.com/cockroachdb/cockroach/pkg/sql.(*planner).ShowClusterSetting.func1(0x24d8c60, 0xc42118c450, 0xc421240420, 0x0, 0x0, 0x22c2500, 0xc4201224a8)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/show_cluster_setting.go:161 +0x186
github.com/cockroachdb/cockroach/pkg/sql.doExpandPlan(0x24d8c60, 0xc42118c450, 0xc421240420, 0x7fffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x24ce2a0, 0xc420264380, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/expand_plan.go:320 +0x2c51
github.com/cockroachdb/cockroach/pkg/sql.(*planner).expandPlan(0xc421240420, 0x24d8c60, 0xc42118c450, 0x24ce2a0, 0xc420264380, 0x24ce2a0, 0xc420264380, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/expand_plan.go:35 +0xce
github.com/cockroachdb/cockroach/pkg/sql.(*planner).optimizePlan(0xc421240420, 0x24d8c60, 0xc42118c450, 0x24ce2a0, 0xc420264380, 0xc42039a5d2, 0x1, 0x1, 0x24ce2a0, 0xc420264380, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/optimize.go:47 +0xeb
github.com/cockroachdb/cockroach/pkg/sql.(*planner).makePlan(0xc421240420, 0x24d8c60, 0xc42118c450, 0x24db220, 0xc420c220e0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:324 +0x2ea
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc421240000, 0x24d8c60, 0xc42118c450, 0x24db220, 0xc420c220e0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:578 +0xb28
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc421240000, 0x24d8c60, 0xc42118c450, 0x24db220, 0xc420c220e0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:397 +0xb33
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc421240000, 0x24d8c60, 0xc42118c450, 0x24db220, 0xc420c220e0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:91 +0x358
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc421240000, 0x24d8c60, 0xc42118c120, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1051 +0x2039
github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).initConnEx.func1(0xc421240000, 0x24d8c60, 0xc42118c120, 0xc420366180, 0xc42039a5e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:211 +0x86
created by github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).initConnEx
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:200 +0x268

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4206df770, 0x21d63bd, 0x22, 0x22bf638, 0x883a01)
	/usr/local/go/src/testing/testing.go:825 +0x301
testing.runTests.func1(0xc420236960)
	/usr/local/go/src/testing/testing.go:1063 +0x64
testing.tRunner(0xc420236960, 0xc420631dd8)
	/usr/local/go/src/testing/testing.go:777 +0xd0
testing.runTests(0xc4204f0f00, 0x3215120, 0x80, 0x80, 0x7fdc7c7bf51b)
	/usr/local/go/src/testing/testing.go:1061 +0x2c4
testing.(*M).Run(0xc42002af80, 0x0)
	/usr/local/go/src/testing/testing.go:978 +0x171
github.com/cockroachdb/cockroach/pkg/kv_test.TestMain(0xc42002af80)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/main_test.go:51 +0x51
main.main()
	_testmain.go:300 +0x151

goroutine 20 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 11 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1158 +0xf1
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:591 +0x110

goroutine 12 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:598 +0xab
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x128

goroutine 50 [select, locked to thread]:
runtime.gopark(0x22c43d0, 0x0, 0x21a3730, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc42059a750, 0xc420594060)
	/usr/local/go/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 24291 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x24d8c60, 0xc420337920)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:580 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203e2120, 0xc420b18090, 0xc4204f8300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24294 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x24d8c60, 0xc420afc000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:580 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203e2160, 0xc420b18090, 0xc4204f83e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24244 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc40000003f)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4207ce3f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4207ce3f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fee40, 0xc420b18090, 0xc4202fee30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24275 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x24d8c60, 0xc4210d0c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1535 +0x3ae
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff660, 0xc420b18090, 0xc420d31e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24253 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x5c)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d07e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d07e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff190, 0xc420b18090, 0xc4202ff120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24241 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc40000003c)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4207ce390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4207ce390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fedc0, 0xc420b18090, 0xc4202fedb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24311 [select]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x24d8c60, 0xc420f04090)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:125 +0x14c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203e28f0, 0xc420b18090, 0xc4203e2880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24240 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc40000003a)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4207ce360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4207ce360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202feda0, 0xc420b18090, 0xc4202fed70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24233 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc400000030)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4207ce180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4207ce180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202febc0, 0xc420b18090, 0xc4202febb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24085 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x24d8c60, 0xc4203362a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x162
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420436570, 0xc420b18090, 0xc420436560)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24080 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x44)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d01b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d01b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fe680, 0xc420b18090, 0xc4202fe630)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24082 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x24d8c60, 0xc420336180)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x162
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204364f0, 0xc420b18090, 0xc4204364e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 23887 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x24d8c60, 0xc4203360f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x162
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420436490, 0xc420b18090, 0xc420436480)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24299 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc4202928f0, 0x24d8c60, 0xc4210d0d50, 0xc420b18090, 0x293e2e6, 0xed28cada2, 0x0, 0xc420ba0000, 0x868c89)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:212 +0x212
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc420ba0000, 0xc420336a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:274 +0x81
github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc420336a20, 0xc42062a5c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:322 +0x1a6
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x24d8c60, 0xc4210d0d50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:272 +0x1c0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203e2260, 0xc420b18090, 0xc4204f84a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24227 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc40000002a)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4207ce000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4207ce000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fea50, 0xc420b18090, 0xc4202fea40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24111 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x24d8c60, 0xc420f52180)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1190 +0x241
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420436ee0, 0xc420b18090, 0xc420436ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24249 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x57)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d06f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d06f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202feee0, 0xc420b18090, 0xc4202feed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24303 [select]:
github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc421010000, 0xc4206fc310, 0xc4203e26f0, 0x22c4758, 0xc4200b4ae0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:884 +0x198
github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x24d8c60, 0xc4210d1380)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:958 +0x6a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203e2700, 0xc420b18090, 0xc4210d10b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24232 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc40000002f)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4207ce150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4207ce150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202feb90, 0xc420b18090, 0xc4202feb80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24093 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420af85a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab

goroutine 24297 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x24d8c60, 0xc420337a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:580 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203e21f0, 0xc420b18090, 0xc4204f8460)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24215 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x4b)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d03c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d03c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fe850, 0xc420b18090, 0xc4202fe840)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24252 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x5a)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff030, 0xc420b18090, 0xc4202fef90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24083 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x24d8c60, 0xc420336000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x162
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420436530, 0xc420b18090, 0xc420436500)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24256 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x53)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff290, 0xc420b18090, 0xc4202ff280)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24091 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x24d8c60, 0xc4203361b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x162
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420436690, 0xc420b18090, 0xc420436680)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24214 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x4a)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fe830, 0xc420b18090, 0xc4202fe7e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24255 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x5e)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0840)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0840)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff270, 0xc420b18090, 0xc4202ff1c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24312 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x24d8c60, 0xc420f040c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:296 +0x11f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203e2970, 0xc420b18090, 0xc4204f8fe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24102 [chan receive]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func6(0x24d8c60, 0xc420f52060)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1112 +0xcf
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420436a00, 0xc420b18090, 0xc420d1bd40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24287 [select]:
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).Start.func2(0x24d8c60, 0xc4204eaf60)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:269 +0x127
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42058a4c0, 0xc420b18090, 0xc420df4330)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24239 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc400000037)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4207ce300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4207ce300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fed60, 0xc420b18090, 0xc4202fed50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24251 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x59)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fef40, 0xc420b18090, 0xc4202fef30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24258 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x62)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d08d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d08d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff2f0, 0xc420b18090, 0xc4202ff2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24273 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x24d8c60, 0xc4210d0bd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1535 +0x3ae
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff640, 0xc420b18090, 0xc420d31d60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24248 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x56)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d06c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d06c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202feec0, 0xc420b18090, 0xc4202feeb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24344 [chan receive]:
github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).execInternal(0xc420ad44b0, 0x24d8c60, 0xc42118c120, 0x21ad981, 0xc, 0x0, 0xc4202641c0, 0x21caccf, 0x1d, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:490 +0x7e2
github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).queryInternal(0xc420ad44b0, 0x24d8ba0, 0xc4212df7c0, 0x21ad981, 0xc, 0x0, 0xc4202641c0, 0x21caccf, 0x1d, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:240 +0x101
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).Query(0xc420ad44b0, 0x24d8ba0, 0xc4212df7c0, 0x21ad981, 0xc, 0x0, 0x21caccf, 0x1d, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:226 +0x123
github.com/cockroachdb/cockroach/pkg/server.(*Server).clusterVersion(0xc42025ee00, 0x24d8ba0, 0xc4212df7c0, 0xc420b07c80, 0x11802c4, 0xc420b13a80, 0xc420c57d3a)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server_update.go:168 +0xa5
github.com/cockroachdb/cockroach/pkg/server.(*Server).upgradeStatus(0xc42025ee00, 0x24d8ba0, 0xc4212df7c0, 0x6fc23ac00, 0x4000000000000000, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server_update.go:103 +0x5d
github.com/cockroachdb/cockroach/pkg/server.(*Server).startAttemptUpgrade.func1(0x24d8ba0, 0xc4212df7c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server_update.go:62 +0x1b7
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc420b18090, 0x24d8ba0, 0xc4212df7c0, 0xc421165cc0, 0x14, 0x0, 0x0, 0xc420505ea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:264 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:259 +0x133

goroutine 24265 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc400000036)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0a80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0a80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff420, 0xc420b18090, 0xc4202ff410)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24295 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x24d8c60, 0xc4203379b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:580 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203e2180, 0xc420b18090, 0xc4204f8420)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 23882 [select]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc420b18090, 0xc4207ee150)
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:122 +0x108
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:119 +0x49

goroutine 24307 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x24d8c60, 0xc420f04030)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:775 +0x1e1
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203e2810, 0xc420b18090, 0xc4204f8f80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24245 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x5b)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc420f53f20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc420f53f20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fee60, 0xc420b18090, 0xc4202fee50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24219 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x4f)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0510)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0510)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fe910, 0xc420b18090, 0xc4202fe900)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24076 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x24d8c60, 0xc4210d00f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:156 +0x49
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fe580, 0xc420b18090, 0xc4204f0460)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24277 [select]:
github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1(0x24d8c60, 0xc4203378f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:108 +0x1b0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff6f0, 0xc420b18090, 0xc420337890)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24213 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x49)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0330)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0330)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fe7d0, 0xc420b18090, 0xc4202fe7c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24259 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc400000024)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff310, 0xc420b18090, 0xc4202ff300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24246 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x54)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0660)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0660)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fee80, 0xc420b18090, 0xc4202fee70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24264 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x5f)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff3f0, 0xc420b18090, 0xc4202ff3b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24260 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc400000025)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff330, 0xc420b18090, 0xc4202ff320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24301 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x24d8c60, 0xc420afc060)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:738 +0x13b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203e2690, 0xc420b18090, 0xc4204f8e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24267 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc400000039)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff480, 0xc420b18090, 0xc4202ff470)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24247 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x55)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0690)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0690)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202feea0, 0xc420b18090, 0xc4202fee90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24257 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x61)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d08a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d08a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff2b0, 0xc420b18090, 0xc4202ff2a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24268 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc40000003b)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0b10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0b10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff5a0, 0xc420b18090, 0xc4202ff520)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24210 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x40)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d0090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d0090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fe6d0, 0xc420b18090, 0xc4202fe6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24211 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0x47)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4210d02d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4210d02d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fe790, 0xc420b18090, 0xc4202fe780)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24110 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x24d8c60, 0xc4203363c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1152 +0x2fc
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420436ec0, 0xc420b18090, 0xc420436eb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24228 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc40000002b)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4207ce030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4207ce030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fea90, 0xc420b18090, 0xc4202fea60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 23885 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x24d8c60, 0xc420336030)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x162
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204363c0, 0xc420b18090, 0xc4204363b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24269 [semacquire]:
sync.runtime_Semacquire(0xc42119e084)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc42119e078)
	/usr/local/go/src/sync/waitgroup.go:129 +0x72
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc42119e000, 0x24d8c60, 0xc4210d0b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:172 +0x31
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x24d8c60, 0xc4210d0b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3581 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff5c0, 0xc420b18090, 0xc4202ff5b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24235 [semacquire]:
sync.runtime_notifyListWait(0xc420d8e590, 0xc400000032)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420d8e580)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42119e000, 0x24d8c60, 0xc4207ce1e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x24d8c60, 0xc4207ce1e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fec20, 0xc420b18090, 0xc4202fec00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24270 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc420ffc000, 0x24d8c60, 0xc4210d0a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3597 +0x1e7
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.raftTickLoop)-fm(0x24d8c60, 0xc4210d0a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3583 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff5e0, 0xc420b18090, 0xc4202ff5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24271 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc420ffc000, 0x24d8c60, 0xc4210d0c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3638 +0x159
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.coalescedHeartbeatsLoop)-fm(0x24d8c60, 0xc4210d0c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3584 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff600, 0xc420b18090, 0xc4202ff5f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 23889 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x24d8c60, 0xc420336150)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x162
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204364d0, 0xc420b18090, 0xc4204364c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xad

goroutine 24108 [semacquire]:
sync.runtime_notifyListWait(0xc420e8ae28, 0xc40000002b)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc420e8ae18)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc420e8ad00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:647 +0x9c
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:634 +0x418

goroutine 24353 [runnable]:
sync.runtime_SemacquireMutex(0xc420b180c4, 0x0)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420b180c0)
	/usr/local/go/src/sync/mutex.go:134 +0x108
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Lock(0xc420b180c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:32 +0x2d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).runPostlude(0xc420b18090, 0x21b3729, 0x10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x3a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc420b18090, 0x24d8c60, 0xc4210aa300, 0x21b3729, 0x10, 0xc420aa25a8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:243 +0x102
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc421162f00, 0x24d8c60, 0xc4210aa300, 0xc421246150, 0xc4210aa300, 0xfae33e, 0x24d8c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:908 +0x201
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc421162f00, 0x24d8c60, 0xc4210aa300, 0xc421246150, 0x1f, 0x3486700, 0x20cdf01)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:949 +0x99
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).send.func1(0xc4216560e0, 0xc421246140, 0x24d8c60, 0xc4210aa270, 0x0, 0xc4210eaa98, 0x8069f8)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:242 +0x6df
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).send(0xc4216560e0, 0x24d8c60, 0xc4210aa270, 0xc4203b2020, 0xf, 0x0, 0x0, 0x100000001, 0x1, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:273 +0x90
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc4216560e0, 0x24d8c60, 0xc4210aa270, 0xc4213b25a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:207 +0x2bb
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc4206de870, 0x24d8c60, 0xc4210aa270, 0xc4206de8c0, 0x1, 0xc42044c160, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1289 +0x386
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc4206de870, 0x24d8c60, 0xc4210aa270, 0x1, 0xc42044c160, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:393 +0x242
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc4206de870, 0x24d8c60, 0xc4210aa270, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x21a
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc4206de870, 0x24d8c60, 0xc4210aa270, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1067 +0x325
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc4206de870, 0x24d8c60, 0xc4210aa270, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:739 +0x15e6
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc4206de870, 0x24d8c60, 0xc4210aa270, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:655 +0x415
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc420182540, 0x24d8c60, 0xc4210aa240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:490 +0x1ec
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc421042e10, 0x24d8c60, 0xc420337170, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:553 +0x118
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc421042e10, 0x24d8c60, 0xc420337170, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:535 +0xa5
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).(github.com/cockroachdb/cockroach/pkg/internal/client.send)-fm(0x24d8c60, 0xc420337170, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:491 +0x76
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x24d8c60, 0xc420337170, 0xc42111eab0, 0xc42123c000, 0x0, 0xc420456040)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:468 +0xf8
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc421042e10, 0x24d8c60, 0xc420337170, 0xc42123c000, 0x1, 0xc420456040)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:491 +0x94
github.com/cockroachdb/cockroach/pkg/storage.(*intentResolver).maybePushTransactions(0xc4203366f0, 0x24d8c60, 0xc420337170, 0xc420b19680, 0x1, 0x1, 0x152ecab30d38371c, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/intent_resolver.go:246 +0xb85
github.com/cockroachdb/cockroach/pkg/storage.(*intentResolver).cleanupIntents(0xc4203366f0, 0x24d8c60, 0xc420337170, 0xc420b19680, 0x1, 0x1, 0x152ecab30d38371c, 0xc400000000, 0xc400000002, 0x81f899, ...)
@cockroach-teamcity cockroach-teamcity added this to the 2.1 milestone May 15, 2018
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels May 15, 2018
@cockroach-teamcity
Copy link
Member Author

@cockroach-teamcity
Copy link
Member Author

@cockroach-teamcity
Copy link
Member Author

craig bot pushed a commit that referenced this issue May 17, 2018
24883: dep: Bump grpc-go version r=a-robinson a=a-robinson

And pull in new packages -- in particular, the encoding/proto package
isn't needed for compilation but is needed at runtime.

Release note: None

---------------------

We should wait to merge this until they've cut the 1.12 release tag so that we aren't just at an arbitrary commit in their git history but I'm sending out the PR now so that I (or whoever would have done this) don't have to deal with debugging the missing encoding/proto package when it comes time to merge this.

As tested in #17370 (comment), this gives a 5-10% boost in whole-cluster throughput and improved tail latencies when run with a highly concurrent workload. It appears to have little performance effect for lower-concurrency workloads.

25410:  sql: run schema changes after CREATE TABLE in txn r=vivekmenezes a=vivekmenezes

Includes a commit from #25362 and should be reviewed after that change.

25612: util: fix `retry.WithMaxAttempts` context cancelled before run. r=windchan7 a=windchan7

If context gets cancelled right after `retry.WithMaxAttempts` runs, the function
passed to it will never gets run. Now `retry.WithMaxAttempts` will at least run
the function once otherwise an error will be returned.

Making this change because there are places such as `show_cluster_setting.go`
require the passed in function to be run at least once. Otherwise there will
be seg fault.

Fixes: #25600. 
Fixes: #25603.
Fixes: #25570. 
Fixes: #25567.
Fixes: #25566.
Fixes: #25511.
Fixes: #25485.
Release note: None

25625: storage: Adding testing knob to disable automatic lease renewals r=a-robinson a=a-robinson

In order to fix the test flakes caused by automatic lease renewals

Fixes #25537
Fixes #25540
Fixes #25568
Fixes #25573
Fixes #25576
Fixes #25589
Fixes #25594
Fixes #25599
Fixes #25605
Fixes #25620

Release note: None

Co-authored-by: Alex Robinson <alexdwanerobinson@gmail.com>
Co-authored-by: Vivek Menezes <vivek@cockroachlabs.com>
Co-authored-by: Victor Chen <victor@cockroachlabs.com>
@craig craig bot closed this as completed in #25612 May 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

No branches or pull requests

3 participants