Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

global sort: test the global sort whole process in ut #48952

Open
ywqzzy opened this issue Nov 28, 2023 · 0 comments
Open

global sort: test the global sort whole process in ut #48952

ywqzzy opened this issue Nov 28, 2023 · 0 comments
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@ywqzzy
Copy link
Contributor

ywqzzy commented Nov 28, 2023

Enhancement

Currently, global sort contains 3 steps:

  1. write sorted files to s3.
  2. merge sort sorted files and write to s3.
  3. read files by key range and sort them.

But the whole process only run in integration test in ci with only 1 tidb node.
We need a way to mimic the dist execution process in UT to make the planning process and execution process more reliable and easier to debug during developing.
@ywqzzy ywqzzy added the type/enhancement The issue or PR belongs to an enhancement. label Nov 28, 2023
@ywqzzy ywqzzy changed the title global sort: test the global sort whole process in ut. global sort: test the global sort whole process in ut Nov 28, 2023
@lance6716 lance6716 mentioned this issue Nov 30, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Nov 30, 2023
@lance6716
global sort: fix a bug and prepare large scale data to s3 (#49024)
f39e9bf 
ref #48952
@lance6716 lance6716 mentioned this issue Nov 30, 2023
Closed
13 tasks
@ywqzzy ywqzzy mentioned this issue Dec 12, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 12, 2023
@ywqzzy
global sort: bench merge step (#49376)
1b199c2 
ref #48952
@ywqzzy ywqzzy mentioned this issue Dec 13, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 13, 2023
@ywqzzy
global sort: refine and add tests for readAllData (#49389)
fc2e10b 
ref #48952
@ywqzzy ywqzzy mentioned this issue Dec 14, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 14, 2023
@ywqzzy
global sort: global sort whole process in ut (#49419)
d076bc1 
ref #48952
@lance6716 lance6716 mentioned this issue Dec 14, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 15, 2023
@lance6716
storage: try to detect credentials first in unit tests (#49478)
47d460f 
ref #48952
@ywqzzy ywqzzy mentioned this issue Dec 18, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 18, 2023
@ywqzzy
global sort: enable concurrent read test (#49539)
7fd5460 
ref #48952, close #49537
@ywqzzy ywqzzy mentioned this issue Dec 20, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 20, 2023
@ywqzzy
global sort: bench new merge step (#49595)
d84df2b 
ref #48952
@lance6716 lance6716 mentioned this issue Jan 10, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jan 11, 2024
@lance6716
test: use fgprof to show off-CPU profile in external package (#50179)
82744b3 
ref #48952
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[WIP] global sort: test the performance of merge step lance6716/tidb
1 participant
@ywqzzy