Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor: implement disk-based sort (Part 1) #13718

Merged
merged 14 commits into from
Dec 18, 2019
Merged

executor: implement disk-based sort (Part 1) #13718

merged 14 commits into from
Dec 18, 2019

Conversation

wshwsh12
Copy link
Contributor

What problem does this PR solve?

part of #12431
This introduces some ListInDisk field and function readPartition to Sort Executor. So that in further pull request, I can implement externalSorting.

What is changed and how it works?

In this pr, I do the following thins.

  1. Spill all data to disk.
  2. Implement readPartition function, and use it to read all data to memory.
  3. Use memory sorting algorithm to sort the data.
  4. Restore the data to disk.
  5. Read data from disk and return.

Check List

Tests

  • Unit test
  • Integration test

Code changes

  • Has exported function/method change
  • Has exported variable/fields change
  • Has interface methods change
  • Has persistent data change

Side effects

  • Possible performance regression
  • Increased code complexity

Related changes

Release note

  • Write release note for bug-fix or new feature.

Support external_sorting.

@wshwsh12
Copy link
Contributor Author

/run-all-tests

@codecov
Copy link

codecov bot commented Nov 25, 2019 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (master@9b7b2c0). Click here to learn what that means.
The diff coverage is 74.6153%.

@@             Coverage Diff             @@
##             master     #13718   +/-   ##
===========================================
  Coverage          ?   80.1468%           
===========================================
  Files             ?        483           
  Lines             ?     121633           
  Branches          ?          0           
===========================================
  Hits              ?      97485           
  Misses            ?      16369           
  Partials          ?       7779

@wshwsh12 wshwsh12 requested a review from qw4990 December 3, 2019 08:19
qw4990
qw4990 reviewed Dec 4, 2019
View reviewed changes
Copy link
Contributor

@qw4990 qw4990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qw4990 qw4990 added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 4, 2019
SunRunAway
SunRunAway reviewed Dec 9, 2019
View reviewed changes
executor/sort.go Outdated Show resolved Hide resolved
executor/sort.go Outdated Show resolved Hide resolved
executor/sort.go Outdated Show resolved Hide resolved
executor/sort.go Outdated Show resolved Hide resolved
executor/sort.go Outdated Show resolved Hide resolved
@wshwsh12 wshwsh12 requested a review from SunRunAway December 17, 2019 07:29
@wshwsh12 wshwsh12 requested a review from SunRunAway December 18, 2019 04:15
SunRunAway
SunRunAway reviewed Dec 18, 2019
View reviewed changes
Copy link
Contributor

@SunRunAway SunRunAway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SunRunAway SunRunAway added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 18, 2019
@SunRunAway SunRunAway added the status/can-merge Indicates a PR has been approved by a committer. label Dec 18, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Dec 18, 2019

Your auto merge job has been accepted, waiting for 14088, 14111

@sre-bot
Copy link
Contributor

sre-bot commented Dec 18, 2019

/run-all-tests

@sre-bot sre-bot merged commit fc603b6 into pingcap:master Dec 18, 2019
XiaTianliang pushed a commit to XiaTianliang/tidb that referenced this pull request Dec 21, 2019
@wshwsh12 @XiaTianliang
executor: implement disk-based sort (Part 1) (pingcap#13718)
2e8e89c
@SunRunAway SunRunAway mentioned this pull request Jan 21, 2020
Closed
55 tasks
@XuHuaiyu XuHuaiyu mentioned this pull request Feb 7, 2020
Open
@wshwsh12 wshwsh12 deleted the sort branch April 21, 2020 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qw4990 qw4990 qw4990 left review comments

@SunRunAway SunRunAway SunRunAway approved these changes

@XuHuaiyu XuHuaiyu Awaiting requested review from XuHuaiyu

Assignees
No one assigned
Labels
sig/execution SIG execution status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/new-feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wshwsh12 @sre-bot @SunRunAway @qw4990