-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
feat(drivers/mediafire): support concurrent upload #1387
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
- Implement complete MediaFire storage driver with session token authentication - Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir - Include thumbnail generation for media files - Handle MediaFire's resumable upload with intelligent and multi-unit transfers - Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files - Optimize memory usage with adaptive buffer sizing (10MB-100MB (default)) - Include rate limiting and retry logic for API requests - Add proper error handling and progress reporting - Handle MediaFire's bitmap-based resumable upload protocol Closes PR #1322
- Implement complete MediaFire storage driver with session token authentication - Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir - Include thumbnail generation for media files - Handle MediaFire's resumable upload with intelligent and multi-unit transfers - Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files - Optimize memory usage with adaptive buffer sizing (10MB-100MB (default)) - Include rate limiting and retry logic for API requests - Add proper error handling and progress reporting - Handle MediaFire's bitmap-based resumable upload protocol - Implement automatic session token renewal Closes PR #1322
…andling and error management
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
|
咱们自己改的话,我觉得不如直接合并,然后后续工作都在#1322 进行 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR implements concurrent upload support for the MediaFire driver, transforming sequential unit uploads into parallel processing for improved performance. The changes include adding thread-safe upload operations, intelligent buffer sizing, and automatic session token renewal.
- Replaces sequential upload loop with concurrent processing using ordered groups and retry logic
- Adds configurable upload thread count and intelligent buffer sizing based on file size
- Implements automatic session token renewal with randomized intervals to prevent expiry
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/stream/util.go | Adds DiscardSection method for skipping data sections in stream processing |
| drivers/mediafire/util.go | Major refactor to implement concurrent upload with thread-safe operations and retry logic |
| drivers/mediafire/meta.go | Adds UploadThreads configuration parameter for concurrent upload control |
| drivers/mediafire/driver.go | Implements automatic session token renewal with randomized cron scheduling |
| README_nl.md | Updates documentation to include MediaFire in supported storage providers list |
Comments suppressed due to low confidence (2)
drivers/mediafire/util.go:1
- [nitpick] The comment has a formatting issue with '(6- 9)' - it should be '(6-9)' without the extra space.
package mediafire
drivers/mediafire/util.go:1
- [nitpick] The comment 'Crazy, but working way' is unprofessional and doesn't explain why this approach is necessary. Consider replacing with a more descriptive explanation of the token refresh strategy.
package mediafire
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| d.cron = cron.NewCron(time.Minute * time.Duration(num)) | ||
| d.cron.Do(func() { | ||
| // Crazy, but working way to refresh session token | ||
| d.renewToken(ctx) |
Copilot
AI
Sep 29, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The context ctx from Init method is captured in the cron closure, but this context may be canceled or expired when the cron job runs later. The cron job should use a background context or create a new context for token renewal.
| d.renewToken(ctx) | |
| d.renewToken(context.Background()) |
| if _, err := d.getSessionToken(ctx); err != nil { | ||
| return d.renewToken(ctx) | ||
|
|
||
| d.renewToken(ctx) |
Copilot
AI
Sep 29, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The error returned by d.renewToken(ctx) is ignored. If token renewal fails, the Init method should return the error instead of continuing execution.
| d.renewToken(ctx) | |
| if err := d.renewToken(ctx); err != nil { | |
| return fmt.Errorf("Init :: [MediaFire] failed to renew token: %w", err) | |
| } |
|
Now confusion is solved, if you need something, just ask me. I found out you are good mates. |
* feat(drivers): add MediaFire driver support (#9319) - Implement complete MediaFire storage driver - Add authentication via session_token and cookie - Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir - Include thumbnail generation for media files - Handle MediaFire's resumable upload API with multi-unit transfers - Add proper error handling and progress reporting Co-authored-by: Da3zKi7 <da3zki7@duck.com> * fix(mediafire): fix code errors in mediafire * fix(mediafire): fix code errors in mediafire * fix(drivers): add session renewal cron for MediaFire driver (#9321) - Implement automatic session token renewal every 6-9 minutes - Add validation for required SessionToken and Cookie fields in Init - Handle session expiration by calling renewToken on validation failure - Prevent storage failures due to MediaFire session timeouts Fixes session closure issues that occur after server restarts or extended periods. Co-authored-by: Da3zKi7 <da3zki7@duck.com> * docs: restore README changes Signed-off-by: ILoveScratch <ilovescratch@foxmail.com> * fix * fix * fix: add stream upload limit * fix * fix: clear action token on drop and refactor header setting * feat(drivers/mediafire): optimize file caching - support direct stream processing - Remove forced caching to *os.File type - Support generic model.File interface for better flexibility - Improve upload efficiency by avoiding unnecessary file conversions - Fix return type to use model.Object instead of model.ObjThumb * feat(drivers/mediafire): improve global rate limiting - Ensure all API methods properly use context for rate limiting - Fix context parameter usage in getDirectDownloadLink, getActionToken, getFileByHash - Maintain consistent rate limiting across all MediaFire API calls * feat(drivers/mediafire): unify return types - remove unnecessary ObjThumb - Change MakeDir, Rename, Copy methods to return model.Object instead of model.ObjThumb - Remove empty Thumbnail fields where not meaningful - Keep ObjThumb only for fileToObj (List operations) which provides actual thumbnail URLs - Improve code consistency and reduce unnecessary wrapper objects * refactor(drivers/mediafire): extract common error handling logic - Add checkAPIResult helper function to reduce code duplication - Replace repetitive MediaFire API error checks with centralized function - Maintain specific error messages for unique cases (token, upload, search) - Improve code maintainability and consistency * enhance(drivers/mediafire): improve quick upload implementation - Add null check for existingFile to prevent potential issues - Improve error handling in quick upload - continue normal upload if search fails - Add detailed comments explaining quick upload logic - Optimize getExistingFileInfo with clearer fallback strategy - Ensure upload reliability even when file search encounters issues * refactor(drivers/mediafire): optimize request method reusability - Extract common HTTP request logic into apiRequest method - Reduce code duplication between getForm and postForm methods - Maintain backward compatibility with existing method signatures - Centralize rate limiting and header management - Support extensible HTTP method handling * docs(drivers/mediafire): add comprehensive English comments - Add function-level comments for all major driver methods - Document Init, List, Link, MakeDir, Move, Rename, Copy, Remove, Put methods - Add comments for key utility functions including session token management - Improve code readability and maintainability for community collaboration - Follow Go documentation conventions with clear, concise descriptions * perf(mediafire): optimize memory allocation and type assertion performance - Pre-allocate slice capacity in getFiles and bitmap conversion to reduce reallocations - Cache file type check in uploadUnits to avoid repeated type assertions - Add uploadSingleUnitOptimized for os.File to eliminate redundant type checks - Optimize string to int conversion with proper error handling - Improve memory efficiency in file upload operations * fix(mediafire): upload without cache * feat(mediafire): add rate limiting to all API methods - Add WaitLimit(ctx) calls to all driver methods: List, Link, MakeDir, Move, Rename, Copy, Remove, Put - Ensure consistent rate limiting across all MediaFire API interactions - Follow project standard pattern used by other drivers * feat(mediafire): improve error handling consistency - Add context parameter to all HTTP API functions for proper context propagation - Update getForm, postForm and apiRequest to accept context parameter - Fix rate limiting to use caller context instead of background context - Ensure consistent error handling patterns across all API calls - Improve cancellation and timeout support * feat(mediafire): refactor resumableUpload to use io.ReadSeeker and improve upload handling * fix(mediafire): release section reader * feat: add disk usage * feat(drivers/mediafire): support concurrent upload (#1387) * feat(drivers): add MediaFire driver with concurrent upload support - Implement complete MediaFire storage driver with session token authentication - Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir - Include thumbnail generation for media files - Handle MediaFire's resumable upload with intelligent and multi-unit transfers - Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files - Optimize memory usage with adaptive buffer sizing (10MB-100MB (default)) - Include rate limiting and retry logic for API requests - Add proper error handling and progress reporting - Handle MediaFire's bitmap-based resumable upload protocol Closes PR #1322 * feat(stream): add DiscardSection method to StreamSectionReader for skipping data * feat(mediafire): refactor resumableUpload logic for improved upload handling and error management * fix(mediafire): stop cron job and clear action token in Drop method * . * fix(mediafire): optimize buffer sizing logic in uploadUnits method * fix(docs): remove duplicate MediaFire * fix(mediafire): revert 'optimization', large files should not be fully chached. --------- Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com> Co-authored-by: Da3zKi7 <da3zki7@duck.com> Co-authored-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com> Co-authored-by: j2rong4cn <j2rong@qq.com> Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com> * fix(mediafire): optimize hash calculation in uploadUnits function * feat(drivers/mediafire): support concurrent upload (#1366) * feat(drivers): add MediaFire driver with concurrent upload support - Implement complete MediaFire storage driver with session token authentication - Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir - Include thumbnail generation for media files - Handle MediaFire's resumable upload with intelligent and multi-unit transfers - Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files - Optimize memory usage with adaptive buffer sizing (10MB-100MB (default)) - Include rate limiting and retry logic for API requests - Add proper error handling and progress reporting - Handle MediaFire's bitmap-based resumable upload protocol Closes PR #1322 * feat(stream): add DiscardSection method to StreamSectionReader for skipping data * feat(mediafire): refactor resumableUpload logic for improved upload handling and error management * fix(mediafire): stop cron job and clear action token in Drop method * . * fix(mediafire): optimize buffer sizing logic in uploadUnits method * fix(docs): remove duplicate MediaFire * fix(mediafire): revert 'optimization', large files should not be fully chached. --------- Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com> Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com> Co-authored-by: j2rong4cn <j2rong@qq.com> Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com> --------- Signed-off-by: ILoveScratch <ilovescratch@foxmail.com> Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com> Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com> Co-authored-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com> Co-authored-by: Da3zKi7 <da3zki7@duck.com> Co-authored-by: KirCute <951206789@qq.com> Co-authored-by: Suyunmeng <Susus0175@proton.me> Co-authored-by: j2rong4cn <j2rong@qq.com> Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Reopened #1366