executor: support read global indexes in IndexMergeReader and index join #20350
Conversation
ldeng-ustc
changed the title
| if e.hasGlobalIndex && !e.skipGlobalIndex { | ||
| e.handleMaps = make(map[int64]*kv.HandleMap) | ||
| for _, p := range e.table.Meta().GetPartitionInfo().Definitions { | ||
| e.handleMaps[p.ID] = kv.NewHandleMap() |
There was a problem hiding this comment.
Now the maximum partitions count is 4096, so there is a potential performance regression here.
There was a problem hiding this comment.
But if we make the may lazy initialized, maybe there would be a data race
There was a problem hiding this comment.
I found that handles are unique in whole partitioned table. How about use a single global handleMap to store all handles and not clear when call nextPartition. But memory resource can't immediately recycle after processed every partitions.
There was a problem hiding this comment.
Or use two handle maps, one store handles read from global index, another store handles read from local index, so that local handle map could free after process every partition. But we must check twice for every handles.
|
LGTM |
|
PTAL @XuHuaiyu |
|
I'm going to close this PR since it hasn't been updated for a long time. feel free to reopen if you want to continue with it. thank you for your contribution. |
What problem does this PR solve?
Problem Summary: support read global indexes in
IndexMergeReader.What is changed and how it works?
Proposal: #18982
When
IndexMergeReaderis processing first partition of partitioned table, global indexes will be read. handles of different partitions will temporarily store in differentHandleMap. When certain partition processing, worker first process the handles has stored inHandleMap, then process the other handles from local indexes as before.Add test case for index join with global index.
Check List
Tests
Release note