Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: Self-Optimizing scan files from metadata instead of from file info cache #1093

Closed
2 tasks done
wangtaohz opened this issue Feb 9, 2023 · 0 comments · Fixed by #1100
Closed
2 tasks done

[Feature]: Self-Optimizing scan files from metadata instead of from file info cache #1093

wangtaohz opened this issue Feb 9, 2023 · 0 comments · Fixed by #1100
Labels
priority:blocker security, data-loss, correctness, etc. type:feature Feature Requests
Milestone
Release 0.4.1

Comments

@wangtaohz
Copy link
Contributor

Description

Self-Optimizing scan files from metadata with TableScan API.

Use case/motivation

Now, Self-Optimizing of KeyedTable and UnkeyedTable scan files from the file info cache, making the data consistency rely too much on the correctness of the file info cache, and affecting the stability of Self-Optimizing.

Describe the solution

For KeyedTable, use Table.newScan() API to get all data files and delete files.

For ArcticKeyed, use KeyedTable.newScan() API to get all insert files, delete files and base files.

Subtasks

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority:blocker security, data-loss, correctness, etc. type:feature Feature Requests
Projects
None yet
Milestone
Release 0.4.1
2 participants
@zhoujinsong @wangtaohz