perf: optimize the insert performance

[a6802739]

QUESTION
here is handle raft ready warning log.

W170307 17:45:10.071231 502 storage/store.go:3426  [n1,s1,r937/1:/Table/51/1/92{80000_-90000_},@c42a7c4e00] handle raft ready: 0.119s [processed=0]
W170307 17:45:10.073804 396 storage/store.go:3426  [n1,s1,r459/1:/Table/51/1/45{10000_-20000_},@c421ad6e00] handle raft ready: 0.059s [processed=0]
W170307 17:45:10.074350 672 storage/store.go:3426  [n1,s1,r491/1:/Table/51/1/48{30000_-40000_},@c423739500] handle raft ready: 0.027s [processed=0]
W170307 17:45:10.074658 531 storage/store.go:3426  [n1,s1,r186/1:/Table/51/1/17{80000_-90000_},@c421336a80] handle raft ready: 0.028s [processed=0]
W170307 17:45:10.078208 411 storage/store.go:3426  [n1,s1,r1975/1:/Table/51/1/196{60000_-70000_},@c435bec700] handle raft ready: 0.026s [processed=0]
W170307 17:45:10.097807 493 storage/store.go:3426  [n1,s1,r1602/1:/Table/51/1/159{30000_-40000_},@c432904700] handle raft ready: 0.088s [processed=0]
W170307 17:45:10.108872 466 storage/store.go:3426  [n1,s1,r311/1:/Table/51/1/30{30000_-40000_},@c4231c8000] handle raft ready: 0.028s [processed=0]
W170307 17:45:10.115758 387 storage/store.go:3426  [n1,s1,r1727/1:/Table/51/1/171{80000_-90000_},@c43bac8700] handle raft ready: 0.028s [processed=0]
W170307 17:45:10.139689 489 storage/store.go:3426  [n1,s1,r222/1:/Table/51/1/21{40000_-50000_},@c422ae3880] handle raft ready: 0.040s [processed=0]
W170307 17:45:10.143264 425 storage/store.go:3426  [n1,s1,r177/1:/Table/51/1/1{690000_-700000_},@c421657180] handle raft ready: 0.032s [processed=0]
W170307 17:45:10.143953 646 storage/store.go:3426  [n1,s1,r897/1:/Table/51/1/88{80000_-90000_},@c42a70b500] handle raft ready: 0.032s [processed=0]
W170307 17:45:10.144841 476 storage/store.go:3426  [n1,s1,r560/1:/Table/51/1/55{10000_-20000_},@c4250f7c00] handle raft ready: 0.041s [processed=0]
W170307 17:45:10.150204 423 storage/store.go:3426  [n1,s1,r985/1:/Table/51/1/97{60000_-70000_},@c42c41ee00] handle raft ready: 0.042s [processed=0]
W170307 17:45:10.152715 669 storage/store.go:3426  [n1,s1,r1345/1:/Table/51/1/133{60000_-70000_},@c429ce0700] handle raft ready: 0.022s [processed=0]
W170307 17:45:10.158739 401 storage/store.go:3426  [n1,s1,r784/1:/Table/51/1/77{50000_-60000_},@c42a7c4000] handle raft ready: 0.033s [processed=0]
W170307 17:45:10.159426 403 storage/store.go:3426  [n1,s1,r459/1:/Table/51/1/45{10000_-20000_},@c421ad6e00] handle raft ready: 0.022s [processed=0]
W170307 17:45:10.176708 525 storage/store.go:3426  [n1,s1,r1123/1:/Table/51/1/111{40000_-50000_},@c42666ee00] handle raft ready: 0.042s [processed=0]
W170307 17:45:10.180951 517 storage/store.go:3426  [n1,s1,r330/1:/Table/51/1/32{20000_-30000_},@c423a18e00] handle raft ready: 0.062s [processed=0]
W170307 17:45:10.182812 407 storage/store.go:3426  [n1,s1,r888/1:/Table/51/1/8{790000_-800000_},@c4245fa000] handle raft ready: 0.030s [processed=0]
W170307 17:45:10.187220 523 storage/store.go:3426  [n1,s1,r1041/1:/Table/51/1/103{20000_-30000_},@c42a70ae00] handle raft ready: 0.029s [processed=0]
W170307 17:45:10.193273 432 storage/store.go:3426  [n1,s1,r1121/1:/Table/51/1/111{20000_-30000_},@c42667f500] handle raft ready: 0.091s [processed=0]
W170307 17:45:10.207636 447 storage/store.go:3426  [n1,s1,r595/1:/Table/51/1/58{60000_-70000_},@c4224a6a80] handle raft ready: 0.026s [processed=0]
W170307 17:45:10.209479 553 storage/store.go:3426  [n1,s1,r1948/1:/Table/51/1/19{39000_-40000_},@c43e658000] handle raft ready: 0.024s [processed=0]
W170307 17:45:10.214800 454 storage/store.go:3426  [n1,s1,r1765/1:/Table/51/1/175{60000_-70000_},@c4373b8700] handle raft ready: 0.039s [processed=0]
W170307 17:45:10.245556 611 storage/store.go:3426  [n1,s1,r1827/1:/Table/51/1/181{80000_-90000_},@c4271ef180] handle raft ready: 0.031s [processed=0]
W170307 17:45:10.246420 455 storage/store.go:3426  [n1,s1,r1010/1:/Table/51/1/100{10000_-20000_},@c42f0fca80] handle raft ready: 0.024s [processed=0]
W170307 17:45:10.255797 527 storage/store.go:3426  [n1,s1,r299/1:/Table/51/1/29{10000_-20000_},@c4223b6380] handle raft ready: 0.024s [processed=0]
W170307 17:45:10.257514 379 storage/store.go:3426  [n1,s1,r1073/1:/Table/51/1/106{40000_-50000_},@c42624d180] handle raft ready: 0.033s [processed=0]
W170307 17:45:10.258676 483 storage/store.go:3426  [n1,s1,r651/1:/Table/51/1/64{20000_-30000_},@c425f9aa80] handle raft ready: 0.026s [processed=0]
W170307 17:45:10.259466 443 storage/store.go:3426  [n1,s1,r1616/1:/Table/51/1/160{70000_-80000_},@c42b0a7880] handle raft ready: 0.025s [processed=0]

we could found handle raft ready takes a log time.

suggestions:

• process handle raft ready and proposeRaftRequest in a concurrent way
• in the handle raft ready, apply the commited raft log in a asynchronous way

I post the trace for some insert statement:

170301 23:40:52.250997 68815 sql/session.go:382  [client=10.111.65.251:5114,user=root,n1] SQL trace:
     0.030ms      0.000ms    event:[n3] 1 CPut, 1 BeginTxn, 1 EndTxn
     0.049ms      0.018ms    event:[n3,s3] executing 3 requests
     0.069ms      0.039ms    event:[n3,s3,r461/5:/Table/51/1/4{390000[7m<80>00000[7m<80>@c421284e00] read-write path
     0.080ms      0.049ms    event:[n3,s3,r461/5:/Table/51/1/4{390000[7m<80>00000[7m<80>@c421284e00] command queue
     0.524ms      0.494ms    event:[n3,s3,r461/5:/Table/51/1/4{390000[7m<80>00000[7m<80>@c421284e00] raft
     1.553ms      1.523ms    event:[client=10.111.65.251:5114,user=root,n1] sending request to gzns-dba-sz-ndbtest-03.gzns.baidu.com:26257
   188.411ms    188.381ms    event:[n3,s3,r461/5:/Table/51/1/4{390000[7m<80>00000[7m<80>@c421284e00] applying command

the schema is :

CREATE TABLE sbtest.sbtest (id INT NOT NULL, 
k INT NOT NULL DEFAULT 0,
c char(120) NOT NULL DEFAULT '', 
pad char(60) NOT NULL DEFAULT '', 
PRIMARY KEY (id));

the insert statement like:

INSERT INTO sbtest (id, k, c, pad) VALUES (1405408086, 45722668, '76270487008-13829930298-41780013278-00878975514-50642785846-87993477247-12309897943-95575276496-45701800236-69025727556', '23633624876-23203686979-52109353109-09160349760-07018326299')

[petermattis]

Per discussion on other channels, the testing is being done on a table pre-split with 2000 ranges using 600 threads to INSERT data. Given these numbers, I don't think improving the concurrency of writes to a particular range will provide a significant benefit as there will be very little per-range concurrency.

Two thoughts:

• It is possible that tweaking COCKROACH_SCHEDULER_CONCURRENCY will help. The default is 2*NumCPU which might be low for this workload.
• The number of insert threads is quite high. I imagine reducing that number would improve the latency of any individual operation. For a 6-node cluster, that is ~100 concurrent writes per node. Is that the expected number of concurrent operations for the application or just what is being tested with?

[petermattis]

I did a bit of investigation this morning and managed to reproduce poor latencies using the kv load generator. No analysis yet of what is going on. Hopefully tomorrow.

[a6802739]

@petermattis forgotten to say, in our test, we set COCKROACH_SCHEDULER_CONCURRENCY to 300.

it's our benchmark test, we found the cpu usage is not very high(about 60%, 24core), so we try to increase the concurrent writes to increase insert QPS.

[petermattis]

Getting back to this today, a mutex profile showed high contention for the raftEntryCache mutex via the getEntries method. We could probably convert that to a RWMutex and remove the contention, but a quick experiment to doing so showed no improvement.

I too have had the observation that latencies seem to be rising faster than I would expect given a significant amount of idle CPU on my test machines.

[petermattis]

Apparently I can't read a profile: both getEntries and addEntries show up as contention points. We'd need to shard raftEntryCache to fix the addEntries contention. I'm taking a look at that.

[a6802739]

@petermattis, how do you detect the contention for the raftEntryCache mutex, through cpu profile?

[petermattis]

@a6802739 I set COCKROACH_MUTEX_PROFILE_RATE=1 and then retrieved <cockroach-host>/debug/pprof/mutex' and used go tool pprof `. The env var controls runtime.SetMutexProfileFraction.

[a6802739]

@petermattis about a mutex profile showed high contention for the raftEntryCache, how do you find it?
After I set COCKROACH_MUTEX_PROFILE_RATE = 5, and use go tool pprof to profile the mutex contention, I got this:

root@st01-dba-sz-cr01.st01:/home/yanlong03/golang/src/github.com/cockroachdb/cockroach$ go tool pprof http://cq01-dba-cockroachtest2-007.cq01:8090/debug/pprof/mutex?debug=1
Fetching profile from http://cq01-dba-cockroachtest2-007.cq01:8090/debug/pprof/mutex?debug=1
Saved profile in /root/pprof/pprof.cq01-dba-cockroachtest2-007.cq01:8090.contentions.delay.009.pb.gz
Entering interactive mode (type "help" for commands)
(pprof) list
Total: 1.09hrs
No source information for bufio.(*Reader).ReadByte
No source information for bufio.(*Reader).fill
No source information for context.WithCancel
No source information for context.propagateCancel
No source information for crypto/rand.(*devReader).Read
No source information for crypto/rand.Read
No source information for fmt.(*pp).doPrintf
No source information for fmt.(*pp).free
No source information for fmt.(*pp).handleMethods
No source information for fmt.(*pp).printArg
No source information for fmt.Fprintf
No source information for fmt.Sprintf
No source information for fmt.newPrinter
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).AddInfo
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).AddInfoProto
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).GetNodeDescriptor
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).tightenNetwork
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*client).handleResponse
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*client).sendGossip
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*infoStore).addInfo
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*infoStore).combine
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*infoStore).processCallbacks
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*infoStore).runCallbacks
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*infoStore).runCallbacks.func1
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*infoStore).runCallbacks.func2
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip.func3.1
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*server).gossipReceiver
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func1
No source information for github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func2
No source information for github.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*Batch).CPut
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).PutInline
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run.func1
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).(github.com/cockroachdb/cockroach/pkg/internal/client.send)-fm
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).CommitInBatch
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Exec
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Get
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).GetProto
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).send
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).sendInternal
No source information for github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill
No source information for github.com/cockroachdb/cockroach/pkg/keys.PrettyPrint
No source information for github.com/cockroachdb/cockroach/pkg/keys.decodeKeyPrint
No source information for github.com/cockroachdb/cockroach/pkg/keys.prettyPrintInternal
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).RangeLookup
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).getDescriptor
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).updateLeaseHolderCache
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*RangeIterator).Seek
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send.func1
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).heartbeatLoop
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).maybeBeginTxn
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).printStatsLoop
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).tryAsyncAbort
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).updateState
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).updateState.func2
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext.func1
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext.func1.1
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*leaseHolderCache).Lookup
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*leaseHolderCache).Update
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*rangeDescriptorCache).LookupRangeDescriptor
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*rangeDescriptorCache).lookupRangeDescriptorInternal
No source information for github.com/cockroachdb/cockroach/pkg/kv.(*rangeDescriptorCache).performRangeLookup
No source information for github.com/cockroachdb/cockroach/pkg/kv.NewReplicaSlice
No source information for github.com/cockroachdb/cockroach/pkg/kv.NewTxnCoordSender.func1
No source information for github.com/cockroachdb/cockroach/pkg/kv.grpcTransportFactory
No source information for github.com/cockroachdb/cockroach/pkg/kv.grpcTransportFactoryImpl
No source information for github.com/cockroachdb/cockroach/pkg/roachpb.(*RKey).String
No source information for github.com/cockroachdb/cockroach/pkg/roachpb.(*Value).InitChecksum
No source information for github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch
No source information for github.com/cockroachdb/cockroach/pkg/roachpb.Key.String
No source information for github.com/cockroachdb/cockroach/pkg/roachpb.MakePriority
No source information for github.com/cockroachdb/cockroach/pkg/roachpb.NewConditionalPut
No source information for github.com/cockroachdb/cockroach/pkg/roachpb.NewTransaction
No source information for github.com/cockroachdb/cockroach/pkg/roachpb.RKey.String
No source information for github.com/cockroachdb/cockroach/pkg/roachpb.Value.Verify
No source information for github.com/cockroachdb/cockroach/pkg/roachpb.Value.computeChecksum
No source information for github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler
No source information for github.com/cockroachdb/cockroach/pkg/rpc.(*Context).ConnHealth
No source information for github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial
No source information for github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.2.1
No source information for github.com/cockroachdb/cockroach/pkg/rpc.(*Context).heartbeat
No source information for github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat
No source information for github.com/cockroachdb/cockroach/pkg/rpc.(*HeartbeatService).Ping
No source information for github.com/cockroachdb/cockroach/pkg/rpc.(*RemoteClockMonitor).UpdateOffset
No source information for github.com/cockroachdb/cockroach/pkg/rpc.(*heartbeatClient).Ping
No source information for github.com/cockroachdb/cockroach/pkg/rpc._Heartbeat_Ping_Handler
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1.1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).computePeriodicMetrics
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).computePeriodicMetrics.func1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).gossipStores
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).gossipStores.func1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).setupSpanForIncomingRPC
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).setupSpanForIncomingRPC.func1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteSummaries.func1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).writeSummaries
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Node).writeSummaries.func1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func8.1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1
No source information for github.com/cockroachdb/cockroach/pkg/server.(*adminServer).Health
No source information for github.com/cockroachdb/cockroach/pkg/server.nodeMetrics.callComplete
No source information for github.com/cockroachdb/cockroach/pkg/server/serverpb._Admin_Health_Handler
No source information for github.com/cockroachdb/cockroach/pkg/server/status.(*MetricsRecorder).GetStatusSummary
No source information for github.com/cockroachdb/cockroach/pkg/server/status.(*MetricsRecorder).GetTimeSeriesData
No source information for github.com/cockroachdb/cockroach/pkg/server/status.(*MetricsRecorder).WriteStatusSummary
No source information for github.com/cockroachdb/cockroach/pkg/server/status.(*RuntimeStatSampler).SampleEnvironment
No source information for github.com/cockroachdb/cockroach/pkg/server/status.eachRecordableValue
No source information for github.com/cockroachdb/cockroach/pkg/server/status.eachRecordableValue.func1
No source information for github.com/cockroachdb/cockroach/pkg/server/status.registryRecorder.record
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*Executor).ExecuteStatements
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*Executor).Prepare
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execClassic
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execRequest
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execRequest.func2
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execStmt
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execStmtInOpenTxn
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execStmtsInCurrentTxn
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).Acquire
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).AcquireByName
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).Release
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).findTableState
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).resolveName
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*LeaseState).hasSomeLifeLeft
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*RowContainer).AddRow
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*RowInserter).InsertRow
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*databaseCache).getID
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*deleteNode).Start
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*deleteNode).fastDelete
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*editNodeRun).startEditNode
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*insertNode).Next
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*insertNode).Start
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).(github.com/cockroachdb/cockroach/pkg/sql.getTableLease)-fm
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).Insert
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).Split
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).exec
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).getDataSource
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).getDatabaseID
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).getSources
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).getTableDesc
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).getTableLease
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).getTableScanOrViewPlan
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeEditNode
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).makePlan
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).mustGetTableDesc
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).newPlan
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).prepare
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).query
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).releaseLeases
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).removeLeaseIfExpiring
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*planner).startPlan
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*renderNode).initFrom
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*tableDeleter).fastDelete
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*tableDeleter).finalize
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*tableInserter).finalize
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*tableInserter).row
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*tableNameCache).get
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*tableState).acquire
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*tableState).release
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*tableState).release.func1
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*tableState).removeLease.func1
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*traceResult).String
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*txnState).finishSQLTxn
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*txnState).resetForNewSQLTxn
No source information for github.com/cockroachdb/cockroach/pkg/sql.(*valuesNode).Start
No source information for github.com/cockroachdb/cockroach/pkg/sql.InternalExecutor.ExecuteStatementInTransaction
No source information for github.com/cockroachdb/cockroach/pkg/sql.LeaseStore.Release
No source information for github.com/cockroachdb/cockroach/pkg/sql.LeaseStore.Release.func1
No source information for github.com/cockroachdb/cockroach/pkg/sql.MustGetDatabaseDesc
No source information for github.com/cockroachdb/cockroach/pkg/sql.PreparedStatements.New
No source information for github.com/cockroachdb/cockroach/pkg/sql.ProcessDefaultColumns
No source information for github.com/cockroachdb/cockroach/pkg/sql.countRowsAffected
No source information for github.com/cockroachdb/cockroach/pkg/sql.getDatabaseDesc
No source information for github.com/cockroachdb/cockroach/pkg/sql.getDescriptor
No source information for github.com/cockroachdb/cockroach/pkg/sql.getTableDesc
No source information for github.com/cockroachdb/cockroach/pkg/sql.getTableOrViewDesc
No source information for github.com/cockroachdb/cockroach/pkg/sql.insertCPutFn
No source information for github.com/cockroachdb/cockroach/pkg/sql.makeDefaultExprs
No source information for github.com/cockroachdb/cockroach/pkg/sql.mustGetTableDesc
No source information for github.com/cockroachdb/cockroach/pkg/sql.runTxnAttempt
No source information for github.com/cockroachdb/cockroach/pkg/sql/mon.(*BoundAccount).Grow
No source information for github.com/cockroachdb/cockroach/pkg/sql/mon.(*MemoryMonitor).GrowAccount
No source information for github.com/cockroachdb/cockroach/pkg/sql/mon.(*MemoryMonitor).Stop
No source information for github.com/cockroachdb/cockroach/pkg/sql/mon.(*MemoryMonitor).increaseBudget
No source information for github.com/cockroachdb/cockroach/pkg/sql/mon.(*MemoryMonitor).releaseBudget
No source information for github.com/cockroachdb/cockroach/pkg/sql/mon.(*MemoryMonitor).reserveMemory
No source information for github.com/cockroachdb/cockroach/pkg/sql/parser.ParseExprsTraditional
No source information for github.com/cockroachdb/cockroach/pkg/sql/parser.parseExprs
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).IsDraining
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).IsDraining-fm
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readBuffer).readTypedMsg
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*v3Conn).executeStatements
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*v3Conn).handleExecute
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*v3Conn).handleParse
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*v3Conn).serve
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*v3Conn).serve.func2
No source information for github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*v3Conn).serve.func2.1
No source information for github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*TableDescriptor).Validate
No source information for github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*TableDescriptor).validateCrossReferences
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Allocator).ComputeAction
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Allocator).ShouldTransferLease
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Allocator).TransferLeaseTarget
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).(github.com/cockroachdb/cockroach/pkg/storage.livenessGossipUpdate)-fm
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).GetIsLiveMap
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).GetLiveness
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).IsLive
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).heartbeatInternal
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).livenessGossipUpdate
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).updateLiveness
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).updateLiveness.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*ProposalData).finish
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).GetCircuitBreaker
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendAsync
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).handleRaftRequest
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).ChangeReplicas
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).ChangeReplicas.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).CheckConsistency
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Entries
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).GetSnapshot
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).IsLeaseValid
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Snapshot
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Term
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).acquireSplitLock.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).addReadOnlyCmd
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).addWriteCmd
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).append
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applyRaftCommand
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applyRaftCommandInBatch
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applyTimestampCache
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).assertState
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).assertStateRLocked
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).beginCmds
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).checkBatchRange
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).computeChecksumDone
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).computeChecksumPostApply.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).evaluateProposal
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeBatch
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeCmd
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeWriteBatch
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).getChecksum
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).getEstimatedBehindCountRLocked
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).getLastReplicaGCTimestamp
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleEvalResult
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReady
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked.func3
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleReplicatedEvalResult
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).init
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).initLocked
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).isLeaseValidRLocked
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).leaseGoodToGo
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).leaseStatus
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).maybeCoalesceHeartbeat
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).maybeInitializeRaftGroup
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).metrics
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).ownsValidLease
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).processRaftCommand
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).processRaftCommand.func3
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).propose
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).refreshProposalsLocked
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendRaftMessage
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendRaftMessageRequest
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendSnapshot
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).setLastReplicaDescriptors
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).setReplicaID
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).submitProposalLocked
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tick
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tryAddWriteCmd
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).withRaftGroup
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Replica).withRaftGroupLocked
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Server).CollectChecksum
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).Capacity
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).ComputeMetrics
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).Descriptor
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).GossipStore
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleRaftRequest
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleRaftResponse
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleRaftUncoalescedRequest
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).LeaseCount
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).LeaseCount.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).RemoveReplica
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).SplitRange
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).canCampaignIdleReplica
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).enqueueRaftUpdateCheck
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).getOrCreateReplica
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).insertRangeLogEvent
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).logChange
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).maybeGossipOnCapacityChange.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest.func3
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequest.func4
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).processReady
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).processTick
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).removeReplicaImpl
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).sendQueuedHeartbeats
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).sendQueuedHeartbeatsToNode
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).startCoalescedHeartbeatsLoop.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).tryGetOrCreateReplica
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).uncoalesceBeats
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).updateReplicationGauges
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Store).updateReplicationGauges.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*StoreMetrics).addMVCCStats
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*StorePool).deadReplicas
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*StorePool).getStoreList
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*Stores).VisitStores
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).Add
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).MaybeAdd
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).pop
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*consistencyQueue).process
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*consistencyQueue).shouldQueue
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*endCmds).done
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*pendingLeaseRequest).requestLeaseAsync.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftEntryCache).addEntries
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftEntryCache).clearTo
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftEntryCache).delEntries
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftEntryCache).getEntries
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftLogQueue).process
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftLogQueue).shouldQueue
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).EnqueueRaftReady
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).EnqueueRaftRequest
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).EnqueueRaftTick
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).enqueue1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).enqueueN
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicaGCQueue).process
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicaGCQueue).shouldQueue
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicaStats).record
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicateQueue).addReplica
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicateQueue).process
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicateQueue).processOneChange
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicateQueue).removeReplica
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicateQueue).shouldQueue
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*replicateQueue).transferLease
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*storeDetail).isDead
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*storeDetail).status
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit
No source information for github.com/cockroachdb/cockroach/pkg/storage.(*timedMutex).Unlock
No source information for github.com/cockroachdb/cockroach/pkg/storage.Allocator.RebalanceTarget
No source information for github.com/cockroachdb/cockroach/pkg/storage.MakeStorePoolNodeLivenessFunc.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.Server.CollectChecksum
No source information for github.com/cockroachdb/cockroach/pkg/storage.Server.CollectChecksum.func1
No source information for github.com/cockroachdb/cockroach/pkg/storage.Server.execStoreCommand
No source information for github.com/cockroachdb/cockroach/pkg/storage._Consistency_CollectChecksum_Handler
No source information for github.com/cockroachdb/cockroach/pkg/storage.defaultSubmitProposalLocked
No source information for github.com/cockroachdb/cockroach/pkg/storage.defaultSubmitProposalLocked.func2
No source information for github.com/cockroachdb/cockroach/pkg/storage.entries
No source information for github.com/cockroachdb/cockroach/pkg/storage.evalConditionalPut
No source information for github.com/cockroachdb/cockroach/pkg/storage.evalTruncateLog
No source information for github.com/cockroachdb/cockroach/pkg/storage.getTruncatableIndexes
No source information for github.com/cockroachdb/cockroach/pkg/storage.makeIDKey
No source information for github.com/cockroachdb/cockroach/pkg/storage.newTimestampCache
No source information for github.com/cockroachdb/cockroach/pkg/storage.replicaStateLoader.load
No source information for github.com/cockroachdb/cockroach/pkg/storage.replicaStateLoader.loadLease
No source information for github.com/cockroachdb/cockroach/pkg/storage.replicaStateLoader.setAppliedIndexBlind
No source information for github.com/cockroachdb/cockroach/pkg/storage.replicaStateLoader.setHardState
No source information for github.com/cockroachdb/cockroach/pkg/storage.replicaStateLoader.setLastIndex
No source information for github.com/cockroachdb/cockroach/pkg/storage.replicaStateLoader.setMVCCStats
No source information for github.com/cockroachdb/cockroach/pkg/storage.snapshot
No source information for github.com/cockroachdb/cockroach/pkg/storage.splitPostApply
No source information for github.com/cockroachdb/cockroach/pkg/storage.term
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).NewIterator
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.(*putBuffer).release
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCBlindPut
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCConditionalPut
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCDeleteRange
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCGet
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCGetProto
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCIterate
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCPut
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCPutProto
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.mvccConditionalPutUsingIter
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.mvccGetInternal
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.mvccGetUsingIter
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.mvccPutUsingIter
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.newGetBuffer
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.newPutBuffer
No source information for github.com/cockroachdb/cockroach/pkg/storage/engine.newRocksDBIterator
No source information for github.com/cockroachdb/cockroach/pkg/ts.(*DB).Query
No source information for github.com/cockroachdb/cockroach/pkg/ts.(*Server).Query.func1.1
No source information for github.com/cockroachdb/cockroach/pkg/ts.(*poller).poll
No source information for github.com/cockroachdb/cockroach/pkg/ts.(*poller).poll.func1
No source information for github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1
No source information for github.com/cockroachdb/cockroach/pkg/util/encoding.PrettyPrintValue
No source information for github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).Now
No source information for github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).PhysicalNow
No source information for github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).PhysicalTime
No source information for github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).Update
No source information for github.com/cockroachdb/cockroach/pkg/util/log.(*AmbientContext).AnnotateCtxWithSpan
No source information for github.com/cockroachdb/cockroach/pkg/util/log.Eventf
No source information for github.com/cockroachdb/cockroach/pkg/util/log.VEventf
No source information for github.com/cockroachdb/cockroach/pkg/util/log.eventInternal
No source information for github.com/cockroachdb/cockroach/pkg/util/metric.(*Histogram).Inspect
No source information for github.com/cockroachdb/cockroach/pkg/util/metric.(*Histogram).RecordValue
No source information for github.com/cockroachdb/cockroach/pkg/util/metric.(*Histogram).Windowed
No source information for github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each
No source information for github.com/cockroachdb/cockroach/pkg/util/metric.(*Registry).Each.func1
No source information for github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1
No source information for github.com/cockroachdb/cockroach/pkg/util/shuffle.Shuffle
No source information for github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
No source information for github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
No source information for github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask.func1
No source information for github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
No source information for github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
No source information for github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1
No source information for github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).runPostlude
No source information for github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).runPrelude
No source information for github.com/cockroachdb/cockroach/pkg/util/stop.taskKey.String
No source information for github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Unlock
No source information for github.com/cockroachdb/cockroach/pkg/util/syncutil.(*RWMutex).Unlock
No source information for github.com/cockroachdb/cockroach/pkg/util/timeutil.(*Timer).Reset
No source information for github.com/cockroachdb/cockroach/pkg/util/timeutil.(*Timer).Stop
No source information for github.com/cockroachdb/cockroach/pkg/util/timeutil.NewTimer
No source information for github.com/cockroachdb/cockroach/pkg/util/tracing.ChildSpan
No source information for github.com/cockroachdb/cockroach/pkg/util/tracing.FinishSpan
No source information for github.com/cockroachdb/cockroach/pkg/util/tracing.ForkCtxSpan
No source information for github.com/cockroachdb/cockroach/pkg/util/tracing.JoinRemoteTrace
No source information for github.com/cockroachdb/cockroach/pkg/util/tracing.netTraceIntegrator.func1
No source information for github.com/cockroachdb/cockroach/pkg/util/uuid.MakeV4
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*RawNode).ApplyConfChange
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*RawNode).Campaign
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*RawNode).Propose
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*RawNode).Ready
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*RawNode).Step
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*RawNode).newReady
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raft).Step
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raft).bcastAppend
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raft).campaign
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raft).handleAppendEntries
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raft).maybeCommit
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raft).removeNode
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raft).sendAppend
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).entries
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).findConflict
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).lastTerm
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).matchTerm
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).maybeAppend
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).maybeCommit
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).nextEnts
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).slice
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).snapshot
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).term
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.newReady
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.stepCandidate
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.stepFollower
No source information for github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.stepLeader
No source information for github.com/cockroachdb/cockroach/vendor/github.com/opentracing/basictracer-go.(*spanImpl).Finish
No source information for github.com/cockroachdb/cockroach/vendor/github.com/opentracing/basictracer-go.(*spanImpl).FinishWithOptions
No source information for github.com/cockroachdb/cockroach/vendor/github.com/opentracing/basictracer-go.(*spanImpl).onCreate
No source information for github.com/cockroachdb/cockroach/vendor/github.com/opentracing/basictracer-go.(*spanImpl).onFinish
No source information for github.com/cockroachdb/cockroach/vendor/github.com/opentracing/basictracer-go.(*tracerImpl).StartSpan
No source information for github.com/cockroachdb/cockroach/vendor/github.com/opentracing/basictracer-go.(*tracerImpl).StartSpanWithOptions
No source information for github.com/cockroachdb/cockroach/vendor/github.com/opentracing/basictracer-go.(*tracerImpl).startSpanInternal
No source information for github.com/cockroachdb/cockroach/vendor/github.com/opentracing/basictracer-go.randomID
No source information for github.com/cockroachdb/cockroach/vendor/github.com/opentracing/basictracer-go.randomID2
No source information for github.com/cockroachdb/cockroach/vendor/github.com/satori/go%2euuid.NewV4
No source information for github.com/cockroachdb/cockroach/vendor/github.com/satori/go%2euuid.safeRandom
No source information for github.com/cockroachdb/cockroach/vendor/golang.org/x/net/context.WithCancel
No source information for github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.(*eventLog).Printf
No source information for github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.(*eventLog).printf
No source information for github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.(*trace).Finish
No source information for github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.(*traceBucket).Add
No source information for github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.(*traceSet).Add
No source information for github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.(*traceSet).Remove
No source information for github.com/cockroachdb/cockroach/vendor/golang.org/x/net/trace.New
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).sendResponse
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.Invoke
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvResponse
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.sendRequest
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).write
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).CloseStream
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).NewStream
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).getStream
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).handleData
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).handleWindowUpdate
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).newStream.func1
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).operateHeaders
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).updateWindow
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).Write
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).WriteStatus
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).closeStream
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).getStream
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).handleData
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).operateHeaders
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).operateHeaders.func1
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).updateWindow
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*inFlow).onData
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*inFlow).onRead
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*quotaPool).add
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBuffer).load
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBuffer).put
No source information for github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read
No source information for io.ReadAtLeast
No source information for io.ReadFull
No source information for math/rand.(*Rand).ExpFloat64
No source information for math/rand.(*Rand).Float64
No source information for math/rand.(*Rand).Int31
No source information for math/rand.(*Rand).Int31n
No source information for math/rand.(*Rand).Int63
No source information for math/rand.(*Rand).Intn
No source information for math/rand.(*Rand).Uint32
No source information for math/rand.(*lockedSource).Int63
No source information for math/rand.ExpFloat64
No source information for math/rand.Int63
No source information for math/rand.Intn
No source information for runtime.goexit
No source information for sync.(*Cond).Wait
No source information for sync.(*Mutex).Unlock
No source information for sync.(*Pool).Get
No source information for sync.(*Pool).Put
No source information for sync.(*Pool).getSlow
No source information for sync.(*Pool).pin
No source information for sync.(*Pool).pinSlow
No source information for sync.(*RWMutex).Unlock

And I couldn't quite understand the debug info when I open http://cq01-dba-cockroachtest2-007.cq01:8090/debug/pprof/mutex?debug=1, could you help me explain it?

Thanks very much.

[petermattis]

You'll likely need to build cockroach from source to view the mutex profile. Also, see this blog post about how to use go tool pprof. In particular, you probably want to visualize the profile using the web command, or list the top methods using top10.

[a6802739]

@petermattis thanks very much.

[petermattis]

@a6802739 Are your requirements for max latency? 99th-%tile latency? Some other percentile?

[petermattis]

I ran the following experiment against a 6-node cluster where I had pre-split the test.kv table into 1000 ranges:

for i in 1 6 12 24 48 96; do 
  ./kv --duration 5m --read-percent 0 --tolerate-errors --concurrency $i --splits 0 'postgresql://root@localhost:27183?sslrootcert=certs/ca.crt&sslcert=certs/root.client.crt&sslkey=certs/root.client.key'
done

The kv load generator was run on a separate machine from the cluster and was also running haproxy.

SQL latencies started in the 5-6ms range for the 1 and 6 client runs, and rose 45-50ms for the 96 client run.

There is a similar rise in the Exec latency graph. This graph measures the time it takes to perform a Node.Batch operation. Somewhat interesting that the latencies here are consistently 2/3 of the SQL latencies. I would have expected the SQL processing to add a fixed latency overhead.

The Exec Success graph shows that the Node operations are fairly evenly distributed across the nodes in the cluster.

Our internal measure of network latency calculated using the Heartbeat.Ping RPC shows a steady climb. Note that I removed 2 nodes from this graph which had outliers. The rise in latencies here is interesting as Heartbeat.Ping does nothing.

CPU usage climbed over the course of the test, but these are 8 CPU machines so even at the peak usage the machines were 50% idle.

Network usage is higher than I would expect at 32 MiB/sec on the 96 client run. The corresponding SQL traffic:

That's a hefty network traffic blowup. Are we doing something excessive somewhere?

Disk utilization is negligible:

No conclusions from this yet.

[petermattis]

Oh, wow! The Network Traffic graph was for a single node. All of the nodes show similar traffic, so the cluster was pumping 180 MiB/sec at the peak.

[petermattis]

Ah, that's Mib, not MiB. So 22.5 MiB/sec. Phew!

[a6802739]

@petermattis, you run test from 1~96, But I only can see 6 nodes result from the grafana, any comparison between different amount of clients.
By the way, Our latency should be limited within 20~40ms(P99).

[petermattis]

@a6802739 I ran the test with 1-96 clients. Each test always used 6 nodes. The purpose was to see how latency was affected by the number of clients and to try and identify the bottleneck.

[petermattis]

FYI, I'm still investigating this. Specifically I'm trying to figure out why increasing load doesn't eventually drive CPU utilization close to 100%. There seems to be something going on with too many Go runtime threads being stuck in Cgo calls. I have an experimental branch which replaces rocksDBBatch.Commit with a version which combines the batches from concurrent calls in Go rather than letting RocksDB do this work internally and that small change boosts performance in a write-only workload for a single-node cluster by 50%. And it also allows bumps CPU utilization from 50% to 80%. This isn't expected and I'm currently at a loss to explain what is going on. Perhaps my experimental branch has a bug.

[petermattis]

The first run is using a binary built from my pmattis/rocksdb-batch branch. The second is the same branch, but with batch combining disabled. These runs were on a 6-node cluster running kv --read-percent 0 --concurrency 600 --splits 1000. The nodes were configured with COCKROACH_SCHEDULER_CONCURRENCY=128. I'm still unable to explain what is going on here.

[a6802739]

@petermattis how do you run you test to 6 nodes, I mean how to config haproxy and how to specify the URL in kv?

[petermattis]

We use the following haproxy.cfg:

global
  maxconn 4096
  pidfile /tmp/cockroach-loadbalancer.pid

defaults
    mode                tcp
    log                 global
    option              dontlognull
    retries             2
    timeout connect     10s
    timeout client      5s
    timeout server      2m
    maxconn             4096

listen psql
    bind 0.0.0.0:27183
    mode tcp
    balance roundrobin
    server cockroach1 cockroach-denim-0001:26257
    server cockroach2 cockroach-denim-0002:26257
    server cockroach3 cockroach-denim-0003:26257
    server cockroach4 cockroach-denim-0004:26257
    server cockroach5 cockroach-denim-0005:26257
    server cockroach6 cockroach-denim-0006:26257

I then I run the kv command as:

./kv --duration 10m --read-percent 0 --tolerate-errors --concurrency $i --splits 0 'postgresql://root@localhost:27183?sslrootcert=certs/ca.crt&sslcert=certs/root.client.crt&sslkey=certs/root.client.key'

So haproxy and kv are running on the same machine and the 6 cockroach nodes are running on a different set of machines.

[dianasaur323]

@petermattis looks like this might have gotten addressed by #14138? I'm going to close for now, but please re-open if this is still an issue.