Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lightning: Support Custom File Iteration Algorithm when Constructing MDLoader #38598

Closed
dsdashun opened this issue Oct 21, 2022 · 1 comment · Fixed by #38599
Closed

Lightning: Support Custom File Iteration Algorithm when Constructing MDLoader #38598

dsdashun opened this issue Oct 21, 2022 · 1 comment · Fixed by #38599
Labels
component/lightning This issue is related to Lightning of TiDB. type/enhancement The issue or PR belongs to an enhancement.

Comments

@dsdashun
Copy link
Contributor

Enhancement

The MDLoader is the abstraction that stores the result of scanning the data source with file route rules and filters. However, in some situations, we don't need to iterate ALL the data source files to construct a complete MDLoader object. Instead, we just need to pick a few of key files in the data source to construct the information we want in MDLoader, so that the performance for generating an MDLoader can be boosted.

Currently, the file iteration logic cannot be customized, all we have is using the ExternalStorage interface, which provides a WalkDir method to iterate ALL the files in the data source. We want to make the iteration logic customizable.
@dsdashun dsdashun added the type/enhancement The issue or PR belongs to an enhancement. label Oct 21, 2022
@dsdashun
Copy link
Contributor Author

/component lightning

@ti-chi-bot ti-chi-bot added the component/lightning This issue is related to Lightning of TiDB. label Oct 21, 2022
@dsdashun dsdashun mentioned this issue Oct 21, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Oct 30, 2022
@dsdashun
lightning: support custom file listing policy (#38599)
bc0b419 
close #38598
@niubell niubell mentioned this issue Nov 9, 2022
Merged
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/lightning This issue is related to Lightning of TiDB. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

lightning: support custom file listing policy dsdashun/tidb
2 participants
@dsdashun @ti-chi-bot