Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

import with global sort is slow, both on encode&sort and write&ingest step #47572

Closed
D3Hunter opened this issue Oct 12, 2023 · 1 comment
Closed

import with global sort is slow, both on encode&sort and write&ingest step #47572

D3Hunter opened this issue Oct 12, 2023 · 1 comment
Labels
affects-7.5 component/ddl This issue is related to DDL of TiDB. feature/developing the related feature is in development severity/major type/enhancement The issue or PR belongs to an enhancement.

Comments

@D3Hunter
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

we can see the encode speed is about 90m/s, but without global sort, it's around 210m/s.
write tikv speed is slow too. the total import time is 495m(global sort) vs 23m(local sort)
64035cef-82c0-4d87-a56b-fb6820a903a1

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

master/7.4
@D3Hunter D3Hunter added the type/bug The issue is confirmed as a bug. label Oct 12, 2023
@seiya-annie seiya-annie added severity/major feature/developing the related feature is in development component/ddl This issue is related to DDL of TiDB. labels Oct 12, 2023
@ti-chi-bot ti-chi-bot bot added may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 labels Oct 12, 2023
@seiya-annie seiya-annie removed may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 labels Oct 12, 2023
This was referenced Oct 16, 2023
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Oct 17, 2023
@D3Hunter
importinto: add more logs to external backend, disable parallel sort (#…
1978479 
…47680)

ref #47572
@ti-chi-bot ti-chi-bot mentioned this issue Oct 17, 2023
Merged
13 tasks
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue Oct 17, 2023
@D3Hunter @ti-chi-bot
importinto: add more logs to external backend, disable parallel sort (p…
5cf9476 
…ingcap#47680)

ref pingcap#47572
ti-chi-bot bot pushed a commit that referenced this issue Oct 17, 2023
@ti-chi-bot
importinto: add more logs to external backend, disable parallel sort (#…
ffe13ab 
…47680) (#47719)

ref #47572
@D3Hunter D3Hunter mentioned this issue Oct 19, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 23, 2023
@D3Hunter
external: pre-alloc mem for writer, use offset to avoid golang slice (#…
05b50a1 
…47793)

ref #47572
@ti-chi-bot ti-chi-bot mentioned this issue Oct 23, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 23, 2023
@ti-chi-bot
external: pre-alloc mem for writer, use offset to avoid golang slice (#…
f738b86 
…47793) (#47929)

ref #47572
@D3Hunter D3Hunter mentioned this issue Oct 27, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 31, 2023
@D3Hunter
importinto: optimize write ingest step and fix oom (#48047)
99a4f35 
ref #47572, close #47643
@ti-chi-bot ti-chi-bot mentioned this issue Oct 31, 2023
Merged
13 tasks
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue Oct 31, 2023
@D3Hunter @ti-chi-bot
importinto: optimize write ingest step and fix oom (pingcap#48047)
db9cb4e 
ref pingcap#47572, close pingcap#47643
ti-chi-bot bot pushed a commit that referenced this issue Nov 1, 2023
@ti-chi-bot
importinto: optimize write ingest step and fix oom (#48047) (#48144)
c2c8d6b 
ref #47572, close #47643
@GMHDBJD GMHDBJD mentioned this issue Nov 1, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Nov 3, 2023
@GMHDBJD
importinto: calculate subtask count for global sort (#48169)
e392766 
ref #47572
@ti-chi-bot ti-chi-bot mentioned this issue Nov 3, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Nov 3, 2023
@ti-chi-bot
importinto: calculate subtask count for global sort (#48169) (#48258)
aa93091 
ref #47572
@Benjamin2037 Benjamin2037 removed the type/bug The issue is confirmed as a bug. label Dec 22, 2023
@D3Hunter
Copy link
Contributor Author

in version 7.5, we release global sort as a experionmental feature

wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@D3Hunter @wuhuizuo
importinto: add more logs to external backend, disable parallel sort (p…
413d1a7 
…ingcap#47680)

ref pingcap#47572
@jebter jebter added the type/enhancement The issue or PR belongs to an enhancement. label May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-7.5 component/ddl This issue is related to DDL of TiDB. feature/developing the related feature is in development severity/major type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@D3Hunter @jebter @Benjamin2037 @seiya-annie @ti-chi-bot