Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] partition/file pruning by indexing #1552

Closed
BohuTANG opened this issue Aug 21, 2021 · 4 comments
Closed

[feature] partition/file pruning by indexing #1552

BohuTANG opened this issue Aug 21, 2021 · 4 comments
Assignees
Labels
C-cloud-re-architecture Category: cloud architecture C-feature Category: feature community-take

Comments

@BohuTANG
Copy link
Member

Summary

Impl the apply_index against the expression.

@BohuTANG BohuTANG added the C-feature Category: feature label Aug 21, 2021
@BohuTANG BohuTANG added this to the v0.5 milestone Aug 21, 2021
@leiysky
Copy link
Contributor

leiysky commented Aug 22, 2021

I have a plan on expressions normalization in #1539 , in purpose of providing ability to do complicated partition pruning(e.g. rewrite a + 1 < 2 to a < 1 so we can directly apply the comparison on partition key).

By the way, it's a very big challeng to do partition prune in a data warehouse, since there may be a large amout of partitions in a single table(says millions), which can bring serious burden to optimizer. The mainstream data warehouse(e.g. Snowflake) has a mechanism like intermidiate partition to reduce the metadata should be read during query optimization.

We can have a more robust design on this in the future.

@BohuTANG
Copy link
Member Author

BohuTANG commented Aug 22, 2021

@leiysky
That's interesting.
Datafuse supports 3-level partition pruning to fast query(Which ClickHouse did, simply and efficiently), there is an introduce in:
https://github.com/datafuselabs/datafuse/blob/master/common/indexing/README.md

This is the pruning after the optimizer,apply index patch will be launched next week.

@zhyass
Copy link
Member

zhyass commented Aug 23, 2021

/assignme

@BohuTANG BohuTANG removed this from the v0.5 milestone Oct 29, 2021
@zhyass zhyass closed this as completed Jan 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-cloud-re-architecture Category: cloud architecture C-feature Category: feature community-take
Projects
None yet
Development

No branches or pull requests

4 participants