Skip to content

Conversation

@huoxingdawang
Copy link
Contributor

根据 #724 中的现象,我推测阿里云盘的上游对API调用都有限速。但是现有的限速给的太极限了,导致出现很多几毫秒几十毫秒的TooManyRequest,我适当降低了ListLink两个API的限速。

同时我注意到RemoveMakeDir也会出现TooManyRequest,大概都在3300ms左右。我给这两个API也进行了限速。

现在看起来工作的比较稳定,但是很慢,一直在限速。。。。

@ILoveScratch2 ILoveScratch2 requested a review from Copilot August 8, 2025 14:47

This comment was marked as outdated.

huoxingdawang and others added 2 commits August 8, 2025 22:52
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: 火星大王 <34576789+huoxingdawang@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: 火星大王 <34576789+huoxingdawang@users.noreply.github.com>
@xrgzs xrgzs linked an issue Aug 9, 2025 that may be closed by this pull request
8 tasks
xrgzs
xrgzs previously approved these changes Aug 9, 2025
@xrgzs
Copy link
Member

xrgzs commented Aug 9, 2025

@huoxingdawang
Copy link
Contributor Author

huoxingdawang commented Aug 9, 2025

https://www.yuque.com/aliyundrive/zpfszx/mqocg38hlxzc5vcd

感谢,所以实际上是其他所有的API共享15QPS的限流?

我再下去修一轮,把其他所有API共享限流的逻辑给写了再合并还是怎么说?
如果我们要共享limit,确实需要一个新的类似于LimitFnCtx的东西,但是这个函数在另一个仓库里,https://github.com/OpenListTeam/rateg

是直接向那个仓库交PR,还是考虑撤掉提交 7e9cdd8 中的修改重新把LimitFnLimitFnCtx移动回 pkg/utils下?

亦或者我看LimitFnCtx里面核心就是limiter.Wait(ctx)及其err处理,我感觉可以拿掉LimitFnCtx这一层wrapper,直接在各个函数里面处理限流的逻辑,如果我们顺着前面的逻辑接着改的话,每一个函数都要包上这一层限流, 整个 AliyundriveOpen 里面会全都是限流函数,而且Put这种方法有四个参数的还要更多的wrapper实现或者更多的辅助struct来传参,因为现在的LimitFnCtx只能包两个参数。

@xrgzs

@huoxingdawang
Copy link
Contributor Author

另外我注意到AliyundriveShare里也有类似的逻辑,这个driver是什么状态,是否一并修改?

@huoxingdawang
Copy link
Contributor Author

huoxingdawang commented Aug 9, 2025

另外我测试的时候还注意到,这个限流在阿里云盘那边应该是针对同一个账户共享请求频率限制,即便token或者挂载的rootid不一样,只要是同一个账户就会有相同的配额。

我实际测试下来如果用多个driver挂载了同一用户的多个不同的路径,这些路径之间会相互触发limit。。。。

如果要改的话,应该需要启动的时候从 /oauth/users/info先把阿里云盘的userid拿下来,然后搞一个全局的map,把limit的本体存成全局的让多个driver共享的。

@huoxingdawang huoxingdawang changed the title fix(aliyundrive_open): limit rate for Remove and MakeDir; reduce limit for List and Link (close #724) fix(aliyundrive_open): limit rate for every request (close #724) Aug 9, 2025
@huoxingdawang
Copy link
Contributor Author

我根据文档,从user/getDriveInfo接口拿到了用户ID,并根据用户ID为用户分配limiter。
在获取用户ID之前(Init阶段)所有用户共享同一个全局limiter。

我注意到所有的请求都经过base.RestyClient发起,同时,1. requestReturnErrResp包含了refreshToken等重试机制,我个人的理解,重试的部分应该也要遵守QPS的限制;2. 部分操作,如put,内部本身就包含了多次request;所以按照原来的实现在最外层接口中做限制不能完全实现文档中关于QPS的要求,为此,我将所有的limit操作放在base.RestyClient请求前以对每一次请求进行QPS限制。

  1. 添加了新的类limiter用于统一管理所有limiter。
  2. 添加了新的枚举limiterType用于传递limiter的类型。
  3. 新的方法wait用于等待limter就绪。
  4. 新的全局函数getLimiterForUser和全局变量limiterslimitersLock用于在不同driver之间给同一个用户分配limiter。

@huoxingdawang huoxingdawang requested a review from Copilot August 9, 2025 04:42

This comment was marked as outdated.

@huoxingdawang huoxingdawang requested a review from Copilot August 9, 2025 05:08

This comment was marked as outdated.

@huoxingdawang huoxingdawang requested a review from Copilot August 9, 2025 05:14

This comment was marked as outdated.

@huoxingdawang huoxingdawang requested review from Copilot and xrgzs August 9, 2025 05:20

This comment was marked as outdated.

@xrgzs
Copy link
Member

xrgzs commented Aug 9, 2025

另外我注意到AliyundriveShare里也有类似的逻辑,这个driver是什么状态,是否一并修改?

这个也改造一下吧,然后 go mod 里面就可以删了,辛苦啦!

@huoxingdawang
Copy link
Contributor Author

另外我注意到AliyundriveShare里也有类似的逻辑,这个driver是什么状态,是否一并修改?

这个也改造一下吧,然后 go mod 里面就可以删了,辛苦啦!

彳亍口巴

@huoxingdawang huoxingdawang requested a review from Copilot August 9, 2025 08:15

This comment was marked as outdated.

@huoxingdawang huoxingdawang requested a review from Copilot August 9, 2025 08:17
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements rate limiting for Aliyun Drive Open API requests to address TooManyRequest errors mentioned in issue #724. The implementation reduces API call rates from the documented limits to provide a safety margin and adds rate limiting to previously unprotected operations.

  • Replaces the rateg library with a custom rate limiting solution using golang.org/x/time/rate
  • Implements per-user rate limiting with global fallback for initial requests
  • Adds rate limiting to Remove and MakeDir operations that were experiencing TooManyRequest errors

Reviewed Changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
go.mod Removes rateg dependency and moves gopsutil from indirect to direct dependency
drivers/aliyundrive_share/limiter.go Adds new rate limiter implementation for share driver with fixed rate limits
drivers/aliyundrive_share/util.go Updates methods to accept context and use new rate limiting
drivers/aliyundrive_share/driver.go Integrates new rate limiter and removes rateg usage
drivers/aliyundrive_open/limiter.go Adds sophisticated per-user rate limiter with cleanup mechanism
drivers/aliyundrive_open/util.go Updates request methods to use new rate limiting system
drivers/aliyundrive_open/upload.go Adds rate limiting to upload-related operations
drivers/aliyundrive_open/driver.go Integrates per-user rate limiting and removes rateg dependencies

@huoxingdawang
Copy link
Contributor Author

另外我注意到AliyundriveShare里也有类似的逻辑,这个driver是什么状态,是否一并修改?

这个也改造一下吧,然后 go mod 里面就可以删了,辛苦啦!

已处理

@xrgzs xrgzs merged commit df479ba into OpenListTeam:main Aug 10, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] 阿里云盘单文件夹内文件过多会无法打开该文件夹

2 participants