Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support spill to disk for memory intensive operator #6528

Open
windtalker opened this issue Dec 22, 2022 · 0 comments
Open

Support spill to disk for memory intensive operator #6528

windtalker opened this issue Dec 22, 2022 · 0 comments
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@windtalker
Copy link
Contributor

windtalker commented Dec 22, 2022 •
edited
Loading

Enhancement

Support spill to disk for memory intensive operator like join/aggregation/sort/topN/window.

Pre-work

Sub-tasks

  • Framework
    • Memory tracker for MPP task/query/query group
    • Spill framework
  • Spill algorithm for memory intensive operators
    • Sort/TopN
    • (Hash)Aggregation
    • (Hash)Join
    • Window
@windtalker windtalker added the type/enhancement The issue or PR belongs to an enhancement. label Dec 22, 2022
This was referenced Jan 3, 2023
Merged
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Jan 6, 2023
@windtalker
support scan concurrency hint for mock table (#6591)
20c5a07 
ref #6528
ti-chi-bot pushed a commit that referenced this issue Jan 6, 2023
@windtalker
add spiller (#6568)
b930bd8 
ref #6528
@windtalker windtalker mentioned this issue Jan 9, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Jan 10, 2023
@windtalker
move non joined block stream into HashJoinProbeBlockInputStream (#6576)
104f678 
ref #6528
This was referenced Jan 12, 2023
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Jan 17, 2023
@windtalker
Aggregator use spiller (#6643)
d477e2c 
ref #6528
@windtalker windtalker mentioned this issue Jan 20, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Jan 21, 2023
@windtalker
Remove useless code in join (#6673)
b88b2aa 
ref #6528
@windtalker windtalker mentioned this issue Jan 30, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Jan 30, 2023
@windtalker
Fix potential hang in right join (#6690)
bca56b9 
ref #6528, ref #6692
@windtalker windtalker mentioned this issue Feb 1, 2023
Merged
12 tasks
@windtalker windtalker mentioned this issue Feb 8, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Feb 8, 2023
@windtalker
Refine spiller (#6780)
dc0460c 
ref #6528
ti-chi-bot pushed a commit that referenced this issue Feb 9, 2023
@windtalker
Refine the spill threshold for aggregation/sort, refine the memory us…
8942a86 
…age calculation for aggregation (#6708)

ref #6528
guo-shaoge pushed a commit to guo-shaoge/tiflash that referenced this issue Feb 10, 2023
@windtalker @guo-shaoge
Refine spiller (pingcap#6780)
6a1bee4 
ref pingcap#6528
guo-shaoge pushed a commit to guo-shaoge/tiflash that referenced this issue Feb 10, 2023
@windtalker @guo-shaoge
Refine the spill threshold for aggregation/sort, refine the memory us…
ff2949f 
…age calculation for aggregation (pingcap#6708)

ref pingcap#6528
ywqzzy pushed a commit to ywqzzy/tiflash_1 that referenced this issue Feb 13, 2023
@windtalker @ywqzzy
Remove useless code in join (pingcap#6673)
a468cb2 
ref pingcap#6528

Signed-off-by: ywqzzy <592838129@qq.com>
ywqzzy pushed a commit to ywqzzy/tiflash_1 that referenced this issue Feb 13, 2023
@windtalker @ywqzzy
Fix potential hang in right join (pingcap#6690)
4ab534a 
ref pingcap#6528, ref pingcap#6692

Signed-off-by: ywqzzy <592838129@qq.com>
ywqzzy pushed a commit to ywqzzy/tiflash_1 that referenced this issue Feb 13, 2023
@windtalker @ywqzzy
Refine spiller (pingcap#6780)
ed22397 
ref pingcap#6528

Signed-off-by: ywqzzy <592838129@qq.com>
This was referenced Feb 15, 2023
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Feb 15, 2023
@windtalker
refine spill log (#6821)
c4d519e 
ref #6528
ti-chi-bot pushed a commit that referenced this issue Feb 16, 2023
@windtalker
add support for auxiliary memory bound in mpmc_queue (#6820)
f2abf0c 
ref #6528
@windtalker windtalker mentioned this issue Feb 20, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Feb 22, 2023
@windtalker
Refine Spiller to support append write (#6862)
94425db 
ref #6528
mengxin9014 pushed a commit to mengxin9014/tiflash that referenced this issue Feb 22, 2023
@windtalker
merge master to join spill branch (#1)
03f1874 
* Refine `Spiller` to support append write (pingcap#6862)

ref pingcap#6528

* fix
 build

Signed-off-by: xufei <xufei@pingcap.com>

* fix

Signed-off-by: xufei <xufei@pingcap.com>

* fix error

Signed-off-by: xufei <xufei@pingcap.com>

---------

Signed-off-by: xufei <xufei@pingcap.com>
@windtalker windtalker mentioned this issue Feb 24, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Feb 24, 2023
@windtalker
avoid spill empty block in agg (#6891)
52eb53c 
ref #6528
@windtalker windtalker mentioned this issue Feb 28, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Mar 1, 2023
@windtalker
fix potential deadlock in spiller (#6915)
3bcfa83 
ref #6528, close #6902
@windtalker windtalker mentioned this issue Mar 8, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Mar 8, 2023
@windtalker
release block earlier in SpillHandler (#6992)
5f96256 
ref #6528
@windtalker windtalker mentioned this issue Mar 9, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Mar 9, 2023
@windtalker
mock exchange receiver with fine grained enable support different max…
2444393 
…_streams (#6999)

ref #6528
@windtalker windtalker mentioned this issue Mar 15, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Mar 15, 2023
@windtalker
Add more spill tests and fix bug (#7073)
35e2f8d 
ref #6528
This was referenced Mar 16, 2023
Merged
Closed
ti-chi-bot pushed a commit that referenced this issue Mar 16, 2023
@windtalker
TiDB use session variable to set the memory threshold for TiFlash spi…
c58c1fc 
…ll (#7085)

ref #6528
@windtalker windtalker mentioned this issue Apr 6, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Apr 7, 2023
@windtalker
Fix data race in join when Join::cancel() is called (#7228)
2da4538 
ref #6528
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@windtalker