Skip to content

Conversation

@j2rong4cn
Copy link
Member

@j2rong4cn j2rong4cn commented Sep 21, 2025

Close #1311
Close #1307
Close #1312

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for configurable chunk folder prefixes in the chunk driver, allowing users to customize the prefix used for chunk folders instead of being hardcoded to "[openlist_chunk]".

  • Introduces two new configuration fields: ChunkLargeFileOnly and ChunkPrefix
  • Updates all hardcoded chunk prefix references to use the configurable prefix
  • Moves chunk validation logic from Get to Link method

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
drivers/chunk/meta.go Adds new configuration fields and sets default chunk prefix
drivers/chunk/driver.go Updates prefix usage throughout and adds validation logic

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

KirCute
KirCute previously approved these changes Sep 22, 2025
@KirCute
Copy link
Member

KirCute commented Sep 22, 2025

突然发现List的时候原来会把每个子chunk文件夹都List一遍,这样不会有性能问题吗,需不需要也改成 goroutine

不对,disk usage 可以 goroutine 是因为每次请求的是不同的驱动,这里请求的是相同的驱动,会被驱动的 api 限速机制拦住,不能提升性能

@j2rong4cn j2rong4cn changed the title feat(chunk): support configurable chunk folder prefix feat(chunk): add ChunkPrefix and ChunkLargeFileOnly options Sep 22, 2025
@j2rong4cn
Copy link
Member Author

突然发现List的时候原来会把每个子chunk文件夹都List一遍,这样不会有性能问题吗,需不需要也改成 goroutine

做成选项吧

@KirCute
Copy link
Member

KirCute commented Sep 22, 2025

做成选项吧

我有个比较极端的想法,我觉得可以考虑List方法直接不统计大小和哈希,这样chunk文件夹可以直接不List

@j2rong4cn
Copy link
Member Author

做成选项吧

我有个比较极端的想法,我觉得可以考虑List方法直接不统计大小和哈希,这样chunk文件夹可以直接不List

不显示大小吗?

@KirCute
Copy link
Member

KirCute commented Sep 22, 2025

不显示大小吗?

显示大小的话就得每个chunk文件夹都List一遍,如果你觉得这样性能可以接受的话那就现在这样也行

@ILoveScratch2
Copy link
Member

不显示大小吗?

显示大小的话就得每个chunk文件夹都List一遍,如果你觉得这样性能可以接受的话那就现在这样也行

不然给chunk一个选项?

@j2rong4cn
Copy link
Member Author

不显示大小吗?

显示大小的话就得每个chunk文件夹都List一遍,如果你觉得这样性能可以接受的话那就现在这样也行

list有缓存,goroutine可以整

@j2rong4cn j2rong4cn merged commit 8f17d35 into OpenListTeam:main Sep 30, 2025
12 checks passed
@j2rong4cn
Copy link
Member Author

完蛋,新选项忘记初始化了

@KirCute
Copy link
Member

KirCute commented Sep 30, 2025

完蛋,新选项忘记初始化了

文件夹前缀吗,我试了一下,旧版本的驱动直接升级以后能默认成那个,我本来还写过一个patch,后来发现能默认出来就没提交

@j2rong4cn
Copy link
Member Author

完蛋,新选项忘记初始化了

文件夹前缀吗,我试了一下,旧版本的驱动直接升级以后能默认成那个,我本来还写过一个patch,后来发现能默认出来就没提交

26fee97

@KirCute
Copy link
Member

KirCute commented Sep 30, 2025

26fee97

好吧,原来是这样默认上的

@j2rong4cn
Copy link
Member Author

j2rong4cn commented Sep 30, 2025

26fee97

好吧,原来是这样默认上的

因为default这个标签不是给json反序列化用的,是oplist内部的

NumListWorkers int `json:"num_list_workers" required:"true" type:"number" default:"5"`

item := driver.Item{
Name: name,
Type: strings.ToLower(field.Type.Name()),
Default: tag.Get("default"),
Options: tag.Get("options"),
Required: tag.Get("required") == "true",
Help: tag.Get("help"),
}

lyy2005a2 pushed a commit to lyy2005a2/OpenList that referenced this pull request Oct 3, 2025
…Team#1321)

* fix(chunk): move chunk existence check to Link method

* feat(chunk): add chunk prefix configuration

* feat(chunk): add chunk_large_file_only configuration

* feat(chunk): concurrently list chunk folder

* refactor(chunk): remove unnecessary mutex for result handling in List method

---------

Co-authored-by: KirCute <kircute@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants