executor: implement disk-based sort (Part 2) #14279

wshwsh12 · 2019-12-30T06:35:35Z

What problem does this PR solve?

part of #12431

Implement externalSorting .

What is changed and how it works?

In this pr, I do the following things.

Read as mush as rows into memory.
If memory quota is triggered, sort these rows in memory and put them into disk as partition 1, then reset the memory quota trigger and return to step 1
If memory quota is not triggered and child is consumed, sort these rows in memory as parition N
merge sort if the count of partitions is larger than 1.
If there is only one partition in step 4, it works just like in-memory sort before.

Check List

Tests

Unit test
Integration test

Code changes

Side effects

Increased code complexity

Related changes

Release note

Benchstat

In memory, master vs this pr.

name                                                     old time/op  new time/op  delta
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[0_0])-8      192ms ± 7%   204ms ± 3%    ~     (p=0.095 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[1_0])-8      267ms ± 2%   275ms ± 4%    ~     (p=0.222 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[1_0])-8       20.4ms ± 1%  21.3ms ± 0%  +4.75%  (p=0.008 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[1_0])-8        191ms ± 1%   198ms ± 6%    ~     (p=0.095 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[2_0])-8      270ms ± 1%   269ms ± 3%    ~     (p=0.421 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[2_0])-8       28.7ms ± 5%  29.9ms ± 4%    ~     (p=0.151 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[2_0])-8        195ms ± 2%   197ms ± 2%    ~     (p=0.222 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[4_0])-8      270ms ± 5%   270ms ± 4%    ~     (p=0.690 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[4_0])-8       37.2ms ± 6%  39.1ms ± 4%    ~     (p=0.151 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[4_0])-8        192ms ± 1%   203ms ± 4%  +5.47%  (p=0.008 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[8_0])-8      269ms ± 2%   271ms ± 2%    ~     (p=0.690 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[8_0])-8       48.7ms ± 7%  51.7ms ± 2%    ~     (p=0.056 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[8_0])-8        193ms ± 2%   200ms ± 3%  +3.50%  (p=0.032 n=5+4)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[16_0])-8     271ms ± 7%   286ms ± 9%    ~     (p=0.310 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[16_0])-8      62.5ms ± 1%  65.2ms ± 7%  +4.37%  (p=0.008 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[16_0])-8       196ms ± 3%   204ms ± 4%    ~     (p=0.095 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[32_0])-8     270ms ± 2%   272ms ± 3%    ~     (p=0.548 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[32_0])-8      77.7ms ± 2%  78.6ms ± 2%    ~     (p=0.548 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[32_0])-8       194ms ± 2%   201ms ± 7%    ~     (p=0.548 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[64_0])-8     262ms ± 3%   282ms ±22%    ~     (p=0.222 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[64_0])-8      91.9ms ± 3%  96.0ms ± 5%    ~     (p=0.095 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[64_0])-8       194ms ± 1%   202ms ± 4%  +4.27%  (p=0.008 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[128_0])-8    262ms ± 2%   284ms ±24%    ~     (p=0.421 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[128_0])-8      105ms ± 2%   110ms ± 6%    ~     (p=0.151 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[128_0])-8      192ms ± 1%   194ms ± 2%    ~     (p=0.222 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[256_0])-8    254ms ± 2%   256ms ± 4%    ~     (p=0.548 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[256_0])-8      118ms ± 3%   123ms ± 5%  +4.49%  (p=0.016 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[256_0])-8      192ms ± 1%   204ms ± 4%  +6.13%  (p=0.008 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[512_0])-8    254ms ± 4%   263ms ± 8%    ~     (p=0.151 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[512_0])-8      130ms ± 3%   133ms ± 6%    ~     (p=0.548 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[512_0])-8      192ms ± 2%   196ms ± 1%    ~     (p=0.095 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[1024_0])-8   252ms ± 2%   241ms ± 4%    ~     (p=0.056 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[1024_0])-8     140ms ± 2%   143ms ± 2%    ~     (p=0.222 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[1024_0])-8     191ms ± 1%   196ms ± 5%    ~     (p=0.056 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[2048_0])-8   244ms ± 4%   238ms ± 3%    ~     (p=0.222 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[2048_0])-8     152ms ± 3%   158ms ± 8%    ~     (p=0.095 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[2048_0])-8     192ms ± 2%   195ms ± 3%    ~     (p=0.310 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[4096_0])-8   237ms ± 2%   235ms ± 2%    ~     (p=0.548 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[4096_0])-8     161ms ± 2%   160ms ± 2%    ~     (p=0.841 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[4096_0])-8     193ms ± 4%   196ms ± 9%    ~     (p=0.841 n=5+5)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[8192_0])-8   231ms ± 1%   235ms ± 4%    ~     (p=0.222 n=5+5)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[8192_0])-8     171ms ± 1%   182ms ±16%    ~     (p=1.000 n=5+5)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[8192_0])-8     198ms ± 3%   192ms ± 3%    ~     (p=0.095 n=5+5)

Master in memory vs this ps in disk.

name                                                     old time/op  new time/op    delta
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[0_0])-8      192ms ± 7%    1510ms ± 2%   +685.99%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[1_0])-8      267ms ± 2%    1574ms ± 2%   +489.57%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[1_0])-8       20.4ms ± 1%   259.4ms ± 0%  +1173.63%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[1_0])-8        191ms ± 1%    1519ms ± 1%   +694.35%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[2_0])-8      270ms ± 1%    1574ms ± 3%   +483.40%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[2_0])-8       28.7ms ± 5%   339.7ms ± 0%  +1084.61%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[2_0])-8        195ms ± 2%    1497ms ± 1%   +666.35%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[4_0])-8      270ms ± 5%    1562ms ± 1%   +479.18%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[4_0])-8       37.2ms ± 6%   357.2ms ± 1%   +859.32%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[4_0])-8        192ms ± 1%    1503ms ± 2%   +682.86%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[8_0])-8      269ms ± 2%    1564ms ± 1%   +482.00%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[8_0])-8       48.7ms ± 7%   407.7ms ±23%   +736.85%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[8_0])-8        193ms ± 2%    1484ms ± 1%   +669.76%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[16_0])-8     271ms ± 7%    1533ms ± 1%   +464.74%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[16_0])-8      62.5ms ± 1%   523.8ms ± 1%   +738.54%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[16_0])-8       196ms ± 3%    1474ms ± 1%   +651.60%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[32_0])-8     270ms ± 2%    1522ms ± 0%   +463.74%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[32_0])-8      77.7ms ± 2%  1018.1ms ± 1%  +1211.04%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[32_0])-8       194ms ± 2%    1480ms ± 1%   +662.06%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[64_0])-8     262ms ± 3%    1525ms ± 1%   +481.10%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[64_0])-8      91.9ms ± 3%  1090.3ms ± 1%  +1086.54%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[64_0])-8       194ms ± 1%    1484ms ± 1%   +665.99%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[128_0])-8    262ms ± 2%    1523ms ± 1%   +480.56%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[128_0])-8      105ms ± 2%    1169ms ± 0%  +1011.15%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[128_0])-8      192ms ± 1%    1475ms ± 1%   +670.14%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[256_0])-8    254ms ± 2%    1507ms ± 1%   +492.98%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[256_0])-8      118ms ± 3%    1248ms ± 2%   +959.31%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[256_0])-8      192ms ± 1%    1484ms ± 2%   +671.74%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[512_0])-8    254ms ± 4%    1510ms ± 1%   +494.94%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[512_0])-8      130ms ± 3%    1323ms ± 1%   +915.77%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[512_0])-8      192ms ± 2%    1480ms ± 0%   +669.33%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[1024_0])-8   252ms ± 2%    1513ms ± 1%   +501.58%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[1024_0])-8     140ms ± 2%    1378ms ± 2%   +882.65%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[1024_0])-8     191ms ± 1%    1476ms ± 1%   +672.55%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[2048_0])-8   244ms ± 4%    1504ms ± 1%   +515.85%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[2048_0])-8     152ms ± 3%    1436ms ± 1%   +843.61%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[2048_0])-8     192ms ± 2%    1481ms ± 1%   +672.22%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[4096_0])-8   237ms ± 2%    1495ms ± 1%   +530.49%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[4096_0])-8     161ms ± 2%    1436ms ± 0%   +792.24%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[4096_0])-8     193ms ± 4%    1462ms ± 1%   +656.35%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0_1],_ndvs:_[8192_0])-8   231ms ± 1%    1497ms ± 1%   +546.76%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[0],_ndvs:_[8192_0])-8     171ms ± 1%    1470ms ± 1%   +758.94%  (p=0.036 n=5+3)
SortExec/(rows:300000,_orderBy:[1],_ndvs:_[8192_0])-8     198ms ± 3%    1485ms ± 2%   +651.97%  (p=0.036 n=5+3)

wshwsh12 · 2019-12-30T06:52:10Z

/run-all-tests

executor/sort.go

sre-bot · 2020-01-01T08:58:37Z

@SunRunAway, @qw4990, PTAL.

SunRunAway

LGTM

sre-bot · 2020-02-06T07:48:56Z

/run-all-tests

sre-bot · 2020-02-06T07:51:24Z

@wshwsh12 merge failed.

SunRunAway · 2020-02-06T08:05:14Z

DATA RACE:

[2020-02-06T07:51:04.154Z] ==================
[2020-02-06T07:51:04.154Z] WARNING: DATA RACE
[2020-02-06T07:51:04.154Z] Read at 0x00c00cabe730 by goroutine 85:
[2020-02-06T07:51:04.154Z]   github.com/pingcap/tidb/util/chunk.(*SpillDiskAction).Action()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/chunk/row_container.go:219 +0x42
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/util/memory.(*Tracker).Consume()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/memory/tracker.go:194 +0x240
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/util/memory.(*Tracker).AttachTo()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/memory/tracker.go:125 +0x1ae
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*SortExec).fetchRowChunks()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/sort.go:268 +0x61f
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*SortExec).Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/sort.go:128 +0xa6f
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*WindowExec).fetchChild()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:154 +0x12b
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*WindowExec).consumeOneGroup()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:78 +0xdd3
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*WindowExec).Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:56 +0xb4
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*shuffleWorker).run()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:342 +0x273
[2020-02-06T07:51:04.155Z] 
[2020-02-06T07:51:04.155Z] Previous write at 0x00c00cabe730 by goroutine 707:
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*SortExec).fetchRowChunks()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/chunk/row_container.go:245 +0x7b4
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*SortExec).Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/sort.go:128 +0xa6f
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*WindowExec).fetchChild()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:154 +0x12b
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*WindowExec).consumeOneGroup()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:78 +0xdd3
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*WindowExec).Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:56 +0xb4
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*shuffleWorker).run()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:342 +0x273
[2020-02-06T07:51:04.155Z] 
[2020-02-06T07:51:04.155Z] Goroutine 85 (running) created at:
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*ShuffleExec).prepare4ParallelExec()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:164 +0x163
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*ShuffleExec).Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:179 +0x77f
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*projectionInputFetcher).run()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:368 +0x41c
[2020-02-06T07:51:04.155Z] 
[2020-02-06T07:51:04.155Z] Goroutine 707 (running) created at:
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*ShuffleExec).prepare4ParallelExec()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:164 +0x163
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*ShuffleExec).Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:179 +0x77f
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.155Z]   github.com/pingcap/tidb/executor.(*projectionInputFetcher).run()
[2020-02-06T07:51:04.155Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:368 +0x41c
[2020-02-06T07:51:04.155Z] ==================
[2020-02-06T07:51:04.155Z] ==================
[2020-02-06T07:51:04.156Z] WARNING: DATA RACE
[2020-02-06T07:51:04.156Z] Read at 0x00c00cabe720 by goroutine 85:
[2020-02-06T07:51:04.156Z]   sync/atomic.LoadInt32()
[2020-02-06T07:51:04.156Z]       /usr/local/go/src/runtime/race_amd64.s:206 +0xb
[2020-02-06T07:51:04.156Z]   sync.(*Once).Do()
[2020-02-06T07:51:04.156Z]       /usr/local/go/src/sync/once.go:55 +0x3a
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/util/chunk.(*SpillDiskAction).Action()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/chunk/row_container.go:224 +0xa5
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/util/memory.(*Tracker).Consume()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/memory/tracker.go:194 +0x240
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/util/memory.(*Tracker).AttachTo()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/memory/tracker.go:125 +0x1ae
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*SortExec).fetchRowChunks()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/sort.go:268 +0x61f
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*SortExec).Next()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/sort.go:128 +0xa6f
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*WindowExec).fetchChild()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:154 +0x12b
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*WindowExec).consumeOneGroup()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:78 +0xdd3
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*WindowExec).Next()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:56 +0xb4
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*shuffleWorker).run()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:342 +0x273
[2020-02-06T07:51:04.156Z] 
[2020-02-06T07:51:04.156Z] Previous write at 0x00c00cabe720 by goroutine 707:
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*SortExec).fetchRowChunks()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/chunk/row_container.go:240 +0x80d
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*SortExec).Next()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/sort.go:128 +0xa6f
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*WindowExec).fetchChild()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:154 +0x12b
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*WindowExec).consumeOneGroup()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:78 +0xdd3
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*WindowExec).Next()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/window.go:56 +0xb4
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*shuffleWorker).run()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:342 +0x273
[2020-02-06T07:51:04.156Z] 
[2020-02-06T07:51:04.156Z] Goroutine 85 (running) created at:
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*ShuffleExec).prepare4ParallelExec()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:164 +0x163
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*ShuffleExec).Next()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:179 +0x77f
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*projectionInputFetcher).run()
[2020-02-06T07:51:04.156Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:368 +0x41c
[2020-02-06T07:51:04.156Z] 
[2020-02-06T07:51:04.156Z] Goroutine 707 (running) created at:
[2020-02-06T07:51:04.156Z]   github.com/pingcap/tidb/executor.(*ShuffleExec).prepare4ParallelExec()
[2020-02-06T07:51:04.157Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:164 +0x163
[2020-02-06T07:51:04.157Z]   github.com/pingcap/tidb/executor.(*ShuffleExec).Next()
[2020-02-06T07:51:04.157Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/shuffle.go:179 +0x77f
[2020-02-06T07:51:04.157Z]   github.com/pingcap/tidb/executor.Next()
[2020-02-06T07:51:04.157Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:212 +0x167
[2020-02-06T07:51:04.157Z]   github.com/pingcap/tidb/executor.(*projectionInputFetcher).run()
[2020-02-06T07:51:04.157Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:368 +0x41c
[2020-02-06T07:51:04.157Z] ==================

wshwsh12 · 2020-02-06T08:48:25Z

/rebuild

wshwsh12 · 2020-02-06T09:18:25Z

/merge

sre-bot · 2020-02-06T09:20:56Z

/run-all-tests

sre-bot · 2020-02-06T09:24:23Z

@wshwsh12 merge failed.

wshwsh12 · 2020-02-07T01:34:16Z

/merge

sre-bot · 2020-02-07T01:36:27Z

/run-all-tests

wshwsh12 added the sig/execution SIG execution label Dec 30, 2019

wshwsh12 requested a review from SunRunAway December 30, 2019 06:35

wshwsh12 added the type/new-feature label Dec 30, 2019

wshwsh12 added the status/all tests passed label Dec 30, 2019

wshwsh12 requested a review from qw4990 December 30, 2019 07:12

wshwsh12 added the status/PTAL label Dec 30, 2019

SunRunAway reviewed Dec 30, 2019

View reviewed changes

executor/sort.go Outdated Show resolved Hide resolved

executor/sort.go Outdated Show resolved Hide resolved

executor/sort.go Outdated Show resolved Hide resolved

SunRunAway reviewed Dec 30, 2019

View reviewed changes

executor/sort.go Outdated Show resolved Hide resolved

SunRunAway added the status/ReqChange label Dec 30, 2019

wshwsh12 added 9 commits December 31, 2019 14:04

merge sort

063f0a4

split partition

d823392

mem track

aca5777

comments

ef80a75

address heap comments

b22f83b

add action

2ccadc6

add comments

f6c4ebe

address comments

d83e9af

add some test

525d779

wshwsh12 force-pushed the sort_part2 branch from dcc1673 to 525d779 Compare December 31, 2019 07:44

wshwsh12 added 4 commits December 31, 2019 15:58

Merge branch 'master' into sort_part2

c461ec7

remove useless field

19ec95a

addres comments

76af1d8

fix

556a858

address comments

90e8081

wshwsh12 removed the status/PTAL label Jan 2, 2020

fix

6483d74

wshwsh12 force-pushed the sort_part2 branch from e27c31a to 6483d74 Compare January 2, 2020 03:16

optimize

bd11fd7

add ut

dae9cdf

wshwsh12 force-pushed the sort_part2 branch from 6212d2f to dae9cdf Compare February 5, 2020 13:10

wshwsh12 requested a review from SunRunAway February 5, 2020 13:11

SunRunAway previously approved these changes Feb 6, 2020

View reviewed changes

SunRunAway added status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Feb 6, 2020

Merge branch 'master' into sort_part2

2aeb4a6

SunRunAway linked an issue Feb 6, 2020 that may be closed by this pull request

Disk-based sort executor #12431

Closed

fix rowcontainer data race

36ecd38

wshwsh12 dismissed SunRunAway’s stale review via 36ecd38 February 6, 2020 08:28

wshwsh12 requested a review from SunRunAway February 6, 2020 08:56

fix rowcontainer data race

09b7253

SunRunAway approved these changes Feb 6, 2020

View reviewed changes

Merge branch 'master' into sort_part2

f85c0f5

Merge branch 'master' into sort_part2

31a0909

Merge branch 'master' into sort_part2

3f62898

sre-bot merged commit 13bf6cc into pingcap:master Feb 7, 2020

XuHuaiyu mentioned this pull request Feb 7, 2020

use external storage to reduce the memory space usage of some executors #14392

Open

hsqlu pushed a commit to hsqlu/tidb that referenced this pull request Feb 8, 2020

executor: implement disk-based sort (Part 2) (pingcap#14279)

79a9c04

wshwsh12 deleted the sort_part2 branch April 21, 2020 05:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: implement disk-based sort (Part 2) #14279

executor: implement disk-based sort (Part 2) #14279

wshwsh12 commented Dec 30, 2019 •

edited

Loading

wshwsh12 commented Dec 30, 2019

sre-bot commented Jan 1, 2020

SunRunAway left a comment

sre-bot commented Feb 6, 2020

sre-bot commented Feb 6, 2020

SunRunAway commented Feb 6, 2020

wshwsh12 commented Feb 6, 2020

wshwsh12 commented Feb 6, 2020

sre-bot commented Feb 6, 2020

sre-bot commented Feb 6, 2020

wshwsh12 commented Feb 7, 2020

sre-bot commented Feb 7, 2020

executor: implement disk-based sort (Part 2) #14279

executor: implement disk-based sort (Part 2) #14279

Conversation

wshwsh12 commented Dec 30, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Benchstat

wshwsh12 commented Dec 30, 2019

sre-bot commented Jan 1, 2020

SunRunAway left a comment

Choose a reason for hiding this comment

sre-bot commented Feb 6, 2020

sre-bot commented Feb 6, 2020

SunRunAway commented Feb 6, 2020

wshwsh12 commented Feb 6, 2020

wshwsh12 commented Feb 6, 2020

sre-bot commented Feb 6, 2020

sre-bot commented Feb 6, 2020

wshwsh12 commented Feb 7, 2020

sre-bot commented Feb 7, 2020

wshwsh12 commented Dec 30, 2019 •

edited

Loading