Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

当从local存储copy大文件至其他Alistv3存储时,内存暴增甚至会导致crash #6613

Closed
4 tasks done
TookeX opened this issue Jun 12, 2024 · 2 comments · Fixed by #6616
Closed
4 tasks done
Labels
bug Something isn't working

Comments

@TookeX
Copy link

TookeX commented Jun 12, 2024

Please make sure of the following things

  • I have read the documentation.
    我已经阅读了文档

  • I'm sure there are no duplicate issues or discussions.
    我确定没有重复的issue或讨论。

  • I'm sure it's due to AList and not something else(such as Network ,Dependencies or Operational).
    我确定是AList的问题,而不是其他原因(例如网络依赖操作)。

  • I'm sure this issue is not fixed in the latest version.
    我确定这个问题在最新版本中没有被修复。

AList Version / AList 版本

v3.35.0

Driver used / 使用的存储驱动

Local & Alist v3

Describe the bug / 问题描述

同局域网下多个不同类型的设备部署了Alist,当尝试从local copy一些大文件去到远端的alist v3的时候,会观察到alist内存占用暴增,甚至超过正在拷贝的文件大小的增幅。在一些内存较小的设备上会导致进程crash,进而copy失败。为提issue在台式机上windows平台的alist做演示,目标端为android的alist_flutter。同时验证过crash会出现在以android的alist和docker的alist等内存较小的设备上作为源端向远端alist v3的storage copy时会crash。

Reproduction / 复现链接

https://alist.nn.ci/zh/guide/api/fs.html#post-%E5%A4%8D%E5%88%B6%E6%96%87%E4%BB%B6

Config / 配置

image

Logs / 日志

image
image

@TookeX TookeX added the bug Something isn't working label Jun 12, 2024
Copy link

welcome bot commented Jun 12, 2024

Thanks for opening your first issue here! Be sure to follow the issue template!

@j2rong4cn
Copy link
Contributor

j2rong4cn commented Jun 13, 2024

问题出现在Alist v3驱动,用的 https://github.com/go-resty/resty 这个库实现的上传
一番debug后定位到go-resty/resty/middleware.go#L490
go-resty/resty这个库为了支持重试会缓存io.Reader类型,执行go-resty/resty/middleware.go#L490之后内存就会暴增
想了2种解决方法

  • 1 改用 go 自带的 net/http ,但不支持重试(需要支持重试的话,要预先缓存到文件,不需要计算哈希值感觉没必要缓存)
  • 2 继续使用 github.com/go-resty/resty 然后改为分片上传,需要alist支持

我试了第1个方法,内存不会暴增
@xhofe 有更好的解决方法吗?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants