Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(archive): archive manage #7817

Merged
merged 10 commits into from
Jan 18, 2025
Merged

feat(archive): archive manage #7817

merged 10 commits into from
Jan 18, 2025

Conversation

KirCute
Copy link
Contributor

@KirCute KirCute commented Jan 13, 2025

对压缩文件的在线预览/在线解压支持,在线预览需要驱动支持 range 请求。
尚未解决的问题:

  1. mholt/archives实现了对加密rar7z的支持,但internal/archive/archives/utils.go#L28这种用法似乎是错误的。
  2. zip 压缩文件的文件名存在编码问题,现使用saintfish/chardet推测文件名的编码,但对于文件名较短的情况效果不尽如人意,可能删除一些不常用的编码效果会更好。

内容:

  1. 新增以下九个 API:
    1. /api/public/archive_extensions:获取后端支持的压缩文件后缀。
    2. /api/fs/archive/meta:获取压缩文件的元信息,包括是否被加密、注释内容、和可选的完整目录结构。
    3. /api/fs/archive/list:当/meta没有返回完整目录结构时,用于获取压缩文件内部的目录结构。
    4. /api/fs/archive/decompress:解压。
    5. /ad/*path?inner=<内部路径>pass=<解压密码>:通过驱动提取压缩文件内部的一个文件。
    6. /ap/*path?inner=<内部路径>pass=<解压密码>:通过驱动本地代理上述文件。
    7. /ae/*path?inner=<内部路径>pass=<解压密码>:通过内部工具提取压缩文件内部的一个文件。
    8. /api/task/decompress/*:管理下载和解压压缩文件任务。
    9. /api/task/decompress_upload/*:管理上传解压后文件的任务。
  2. 新增decompressdecompress_upload两种任务,分别用于下载待解压的压缩文件和上传解压后的内容,支持使用 HttpRange 访问压缩文件的部分内容以实现不下载压缩文件的完整内容就解压其一部分。
  3. 新增以下驱动接口,用于管理压缩文件:
    1. GetArchiveMeta(ctx context.Context, obj model.Obj, args model.ArchiveArgs) (model.ArchiveMeta, error):获取压缩文件的元信息。
    2. ListArchive(ctx context.Context, obj model.Obj, args model.ArchiveInnerArgs) ([]model.Obj, error):列举压缩文件内部的目录结构。
    3. Extract(ctx context.Context, obj model.Obj, args model.ArchiveInnerArgs) (*model.Link, error):提取压缩文件内的文件。
    4. ArchiveGet(ctx context.Context, obj model.Obj, args model.ArchiveInnerArgs) (model.Obj, error):直接获取压缩文件内的文件信息。
    5. ArchiveDecompress(ctx context.Context, srcObj, dstDir model.Obj, args model.ArchiveDecompressArgs) ([]model.Obj, error):解压。
  4. 新增压缩文件管理模块internal/archive,用于在驱动没有实现压缩文件相关接口,或进行跨驱动解压时,使用内部工具操作压缩文件,管理工具接口:
    1. AcceptedExtensions() []string:返回工具支持的压缩文件后缀。
    2. GetMeta(ss *stream.SeekableStream, args model.ArchiveArgs) (model.ArchiveMeta, error):获取元信息。
    3. List(ss *stream.SeekableStream, args model.ArchiveInnerArgs) ([]model.Obj, error):列举目录结构。
    4. Extract(ss *stream.SeekableStream, args model.ArchiveInnerArgs) (io.ReadCloser, int64, error):提取。
    5. Decompress(ss *stream.SeekableStream, outputPath string, args model.ArchiveInnerArgs, up model.UpdateProgress) error:解压(到临时目录)。
  5. 基于以下库支持了以下类型的压缩文件:
    1. mholt/archives.br, .bz2, .gz, .lz4, .lz, .sz, .s2, .xz, .zz, .zst, .tar, .rar, .7z
    2. yeka/zip.zipmholt/archives不支持加密的.zip,所以用了这个)
    3. kdomanski/iso9660.iso
  6. 解压和预览功能的前端部分。
  7. 新增两个用户权限:读取压缩文件、解压。
  8. 一些与该功能关系不大的小修改:
    1. 新增stream.SStreamReadAtSeeker用于将stream.SeekableStream封装为io.ReaderAtio.Seeker使用。
    2. 新增stream.ReaderUpdatingProgress用于在读取一个流的同时基于driver.UpdateProgress(现移至model包)提交进度。
    3. model.FileStreamer新增CacheFullInTempFileAndUpdateProgress(up UpdateProgress)用于在将流缓存到本地的过程中提交进度。
    4. 新增命令kill,与原stop指令相同,stop指令终止服务端进程的方式从发送SIGKILL信号改为发送SIGTERM信号。这一改动是因为decompress_upload为了实现重试,会在失败的时候不删除本地的临时文件,直到任务本身被删除时才同时删除临时文件,为了避免服务端退出时还有剩余失败的decompress_upload任务,需要在退出时将它们删除,这就必须保证stop指令的效果是在cmd/server.go#L160处让程序继续执行,而不是直接杀死进程。
  9. 阿里云盘Open又拍云存储本地存储SFTP上测试了不缓存模式的解压,在又拍云存储本地存储上通过了预览相关各项功能的测试,在百度网盘阿里云盘Open上没有通过预览相关功能的测试

Closes #7573
Closes #5114

@KirCute
Copy link
Contributor Author

KirCute commented Jan 13, 2025

Front-end part AlistGo/alist-web#234

@KirCute KirCute marked this pull request as ready for review January 16, 2025 19:36
@xhofe xhofe merged commit bb40e2e into AlistGo:main Jan 18, 2025
6 checks passed
@Clouddark75
Copy link

Clouddark75 commented Jan 18, 2025

Cool. Just testing the latest beta, and I don't see the archive support anywhere (it will make my backup process slot better). Offline downloading only shows simple HTTP; it does not matter if it's a supported host or not. I don't see any new options to configure or change things. What am I missing? 🤔

@KirCute
Copy link
Contributor Author

KirCute commented Jan 19, 2025

Cool. Just testing the latest beta, and I don't see the archive support anywhere (it will make my backup process slot better). Offline downloading only shows simple HTTP; it does not matter if it's a supported host or not. I don't see any new options to configure or change things. What am I missing? 🤔

Check whether there are two additional permissions in the user management. If so, you need to enable them to use this new feature. If there are no new permissions, you may have encountered the familiar issue of inconsistent front-end and back-end versions.

@KirCute
Copy link
Contributor Author

KirCute commented Jan 19, 2025

Cool. Just testing the latest beta, and I don't see the archive support anywhere (it will make my backup process slot better). Offline downloading only shows simple HTTP; it does not matter if it's a supported host or not. I don't see any new options to configure or change things. What am I missing? 🤔

After enabling the two permissions, you can see the new "Decompress" option by right-clicking the compressed file. Clicking on the compressed file will reveal the new preview method.

@Clouddark75
Copy link

Clouddark75 commented Jan 19, 2025 via email

@KirCute
Copy link
Contributor Author

KirCute commented Jan 19, 2025

Same problem, then. Hahaha. But this time those new options aren't there, at all. Can you tell me the code for the db to have them all enabled? El sáb., 18 de ene. de 2025 11:02 PM, KirCute_ECT @.> escribió:

Cool. Just testing the latest beta, and I don't see the archive support anywhere (it will make my backup process slot better). Offline downloading only shows simple HTTP; it does not matter if it's a supported host or not. I don't see any new options to configure or change things. What am I missing? 🤔 After enabling the two permissions, you can see the new "Decompress" option by right-clicking the compressed file. Clicking on the compressed file will reveal the new preview method. — Reply to this email directly, view it on GitHub <#7817 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUE6MR7AEGLSGEKJPHCONMD2LMIVRAVCNFSM6AAAAABVDN47BSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBQGUZTEMRZGY . You are receiving this because you commented.Message ID: @.
>

You can change the permission to 16383. But simply enabling permissions in the database may not solve the problem as it did last time, because this new feature also requires support from new front-end code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

希望可以增加在线解压的功能 能否支持zip、tar等压缩包的在线内容预览
3 participants