-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
feat(chunk): add ChunkPrefix and ChunkLargeFileOnly options #1321
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds support for configurable chunk folder prefixes in the chunk driver, allowing users to customize the prefix used for chunk folders instead of being hardcoded to "[openlist_chunk]".
- Introduces two new configuration fields:
ChunkLargeFileOnlyandChunkPrefix - Updates all hardcoded chunk prefix references to use the configurable prefix
- Moves chunk validation logic from Get to Link method
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| drivers/chunk/meta.go | Adds new configuration fields and sets default chunk prefix |
| drivers/chunk/driver.go | Updates prefix usage throughout and adds validation logic |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
突然发现List的时候原来会把每个子chunk文件夹都List一遍,这样不会有性能问题吗, 不对,disk usage 可以 goroutine 是因为每次请求的是不同的驱动,这里请求的是相同的驱动,会被驱动的 api 限速机制拦住,不能提升性能 |
做成选项吧 |
我有个比较极端的想法,我觉得可以考虑List方法直接不统计大小和哈希,这样chunk文件夹可以直接不List |
不显示大小吗? |
显示大小的话就得每个chunk文件夹都List一遍,如果你觉得这样性能可以接受的话那就现在这样也行 |
不然给chunk一个选项? |
list有缓存,goroutine可以整 |
|
完蛋,新选项忘记初始化了 |
文件夹前缀吗,我试了一下,旧版本的驱动直接升级以后能默认成那个,我本来还写过一个patch,后来发现能默认出来就没提交 |
|
|
好吧,原来是这样默认上的 |
因为default这个标签不是给json反序列化用的,是oplist内部的 OpenList/drivers/chunk/meta.go Line 15 in 9d3da44
OpenList/internal/op/driver.go Lines 170 to 177 in 9d3da44
|
…Team#1321) * fix(chunk): move chunk existence check to Link method * feat(chunk): add chunk prefix configuration * feat(chunk): add chunk_large_file_only configuration * feat(chunk): concurrently list chunk folder * refactor(chunk): remove unnecessary mutex for result handling in List method --------- Co-authored-by: KirCute <kircute@foxmail.com>
Close #1311
Close #1307
Close #1312