Skip to content

Conversation

@ILoveScratch2
Copy link
Member

Reopened #1366

Da3zKi7 and others added 10 commits September 26, 2025 08:49
- Implement complete MediaFire storage driver with session token authentication
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload with intelligent and multi-unit transfers
- Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files
- Optimize memory usage with adaptive buffer sizing (10MB-100MB (default))
- Include rate limiting and retry logic for API requests
- Add proper error handling and progress reporting
- Handle MediaFire's bitmap-based resumable upload protocol

Closes PR #1322
- Implement complete MediaFire storage driver with session token authentication
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload with intelligent and multi-unit transfers
- Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files
- Optimize memory usage with adaptive buffer sizing (10MB-100MB (default))
- Include rate limiting and retry logic for API requests
- Add proper error handling and progress reporting
- Handle MediaFire's bitmap-based resumable upload protocol
- Implement automatic session token renewal

Closes PR #1322
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
@ILoveScratch2 ILoveScratch2 changed the base branch from main to mediafire September 29, 2025 12:13
@KirCute
Copy link
Member

KirCute commented Sep 29, 2025

咱们自己改的话,我觉得不如直接合并,然后后续工作都在#1322 进行

@ILoveScratch2 ILoveScratch2 merged commit ca91490 into mediafire Sep 29, 2025
@j2rong4cn j2rong4cn requested a review from Copilot September 29, 2025 13:36
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements concurrent upload support for the MediaFire driver, transforming sequential unit uploads into parallel processing for improved performance. The changes include adding thread-safe upload operations, intelligent buffer sizing, and automatic session token renewal.

  • Replaces sequential upload loop with concurrent processing using ordered groups and retry logic
  • Adds configurable upload thread count and intelligent buffer sizing based on file size
  • Implements automatic session token renewal with randomized intervals to prevent expiry

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
internal/stream/util.go Adds DiscardSection method for skipping data sections in stream processing
drivers/mediafire/util.go Major refactor to implement concurrent upload with thread-safe operations and retry logic
drivers/mediafire/meta.go Adds UploadThreads configuration parameter for concurrent upload control
drivers/mediafire/driver.go Implements automatic session token renewal with randomized cron scheduling
README_nl.md Updates documentation to include MediaFire in supported storage providers list
Comments suppressed due to low confidence (2)

drivers/mediafire/util.go:1

  • [nitpick] The comment has a formatting issue with '(6- 9)' - it should be '(6-9)' without the extra space.
package mediafire

drivers/mediafire/util.go:1

  • [nitpick] The comment 'Crazy, but working way' is unprofessional and doesn't explain why this approach is necessary. Consider replacing with a more descriptive explanation of the token refresh strategy.
package mediafire

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

d.cron = cron.NewCron(time.Minute * time.Duration(num))
d.cron.Do(func() {
// Crazy, but working way to refresh session token
d.renewToken(ctx)
Copy link

Copilot AI Sep 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The context ctx from Init method is captured in the cron closure, but this context may be canceled or expired when the cron job runs later. The cron job should use a background context or create a new context for token renewal.

Suggested change
d.renewToken(ctx)
d.renewToken(context.Background())

Copilot uses AI. Check for mistakes.
if _, err := d.getSessionToken(ctx); err != nil {
return d.renewToken(ctx)

d.renewToken(ctx)
Copy link

Copilot AI Sep 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error returned by d.renewToken(ctx) is ignored. If token renewal fails, the Init method should return the error instead of continuing execution.

Suggested change
d.renewToken(ctx)
if err := d.renewToken(ctx); err != nil {
return fmt.Errorf("Init :: [MediaFire] failed to renew token: %w", err)
}

Copilot uses AI. Check for mistakes.
@Da3zKi7
Copy link
Contributor

Da3zKi7 commented Sep 29, 2025

@KirCute @ILoveScratch2

Now confusion is solved, if you need something, just ask me. I found out you are good mates.

ILoveScratch2 added a commit that referenced this pull request Sep 30, 2025
* feat(drivers): add MediaFire driver support (#9319)

- Implement complete MediaFire storage driver
- Add authentication via session_token and cookie
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload API with multi-unit transfers
- Add proper error handling and progress reporting

Co-authored-by: Da3zKi7 <da3zki7@duck.com>

* fix(mediafire): fix code errors in mediafire

* fix(mediafire): fix code errors in mediafire

* fix(drivers): add session renewal cron for MediaFire driver (#9321)

- Implement automatic session token renewal every 6-9 minutes
- Add validation for required SessionToken and Cookie fields in Init
- Handle session expiration by calling renewToken on validation failure
- Prevent storage failures due to MediaFire session timeouts

Fixes session closure issues that occur after server restarts or extended periods.

Co-authored-by: Da3zKi7 <da3zki7@duck.com>

* docs: restore README changes

Signed-off-by: ILoveScratch <ilovescratch@foxmail.com>

* fix

* fix

* fix: add stream upload limit

* fix

* fix: clear action token on drop and refactor header setting

* feat(drivers/mediafire): optimize file caching - support direct stream processing

- Remove forced caching to *os.File type
- Support generic model.File interface for better flexibility
- Improve upload efficiency by avoiding unnecessary file conversions
- Fix return type to use model.Object instead of model.ObjThumb

* feat(drivers/mediafire): improve global rate limiting

- Ensure all API methods properly use context for rate limiting
- Fix context parameter usage in getDirectDownloadLink, getActionToken, getFileByHash
- Maintain consistent rate limiting across all MediaFire API calls

* feat(drivers/mediafire): unify return types - remove unnecessary ObjThumb

- Change MakeDir, Rename, Copy methods to return model.Object instead of model.ObjThumb
- Remove empty Thumbnail fields where not meaningful
- Keep ObjThumb only for fileToObj (List operations) which provides actual thumbnail URLs
- Improve code consistency and reduce unnecessary wrapper objects

* refactor(drivers/mediafire): extract common error handling logic

- Add checkAPIResult helper function to reduce code duplication
- Replace repetitive MediaFire API error checks with centralized function
- Maintain specific error messages for unique cases (token, upload, search)
- Improve code maintainability and consistency

* enhance(drivers/mediafire): improve quick upload implementation

- Add null check for existingFile to prevent potential issues
- Improve error handling in quick upload - continue normal upload if search fails
- Add detailed comments explaining quick upload logic
- Optimize getExistingFileInfo with clearer fallback strategy
- Ensure upload reliability even when file search encounters issues

* refactor(drivers/mediafire): optimize request method reusability

- Extract common HTTP request logic into apiRequest method
- Reduce code duplication between getForm and postForm methods
- Maintain backward compatibility with existing method signatures
- Centralize rate limiting and header management
- Support extensible HTTP method handling

* docs(drivers/mediafire): add comprehensive English comments

- Add function-level comments for all major driver methods
- Document Init, List, Link, MakeDir, Move, Rename, Copy, Remove, Put methods
- Add comments for key utility functions including session token management
- Improve code readability and maintainability for community collaboration
- Follow Go documentation conventions with clear, concise descriptions

* perf(mediafire): optimize memory allocation and type assertion performance

- Pre-allocate slice capacity in getFiles and bitmap conversion to reduce reallocations
- Cache file type check in uploadUnits to avoid repeated type assertions
- Add uploadSingleUnitOptimized for os.File to eliminate redundant type checks
- Optimize string to int conversion with proper error handling
- Improve memory efficiency in file upload operations

* fix(mediafire): upload without cache

* feat(mediafire): add rate limiting to all API methods

- Add WaitLimit(ctx) calls to all driver methods: List, Link, MakeDir, Move, Rename, Copy, Remove, Put
- Ensure consistent rate limiting across all MediaFire API interactions
- Follow project standard pattern used by other drivers

* feat(mediafire): improve error handling consistency

- Add context parameter to all HTTP API functions for proper context propagation
- Update getForm, postForm and apiRequest to accept context parameter
- Fix rate limiting to use caller context instead of background context
- Ensure consistent error handling patterns across all API calls
- Improve cancellation and timeout support

* feat(mediafire): refactor resumableUpload to use io.ReadSeeker and improve upload handling

* fix(mediafire): release section reader

* feat: add disk usage

* feat(drivers/mediafire): support concurrent upload (#1387)

* feat(drivers): add MediaFire driver with concurrent upload support

- Implement complete MediaFire storage driver with session token authentication
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload with intelligent and multi-unit transfers
- Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files
- Optimize memory usage with adaptive buffer sizing (10MB-100MB (default))
- Include rate limiting and retry logic for API requests
- Add proper error handling and progress reporting
- Handle MediaFire's bitmap-based resumable upload protocol

Closes PR #1322

* feat(stream): add DiscardSection method to StreamSectionReader for skipping data

* feat(mediafire): refactor resumableUpload logic for improved upload handling and error management

* fix(mediafire): stop cron job and clear action token in Drop method

* .

* fix(mediafire): optimize buffer sizing logic in uploadUnits method

* fix(docs): remove duplicate MediaFire

* fix(mediafire): revert 'optimization', large files should not be fully chached.

---------

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: Da3zKi7 <da3zki7@duck.com>
Co-authored-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

* fix(mediafire): optimize hash calculation in uploadUnits function

* feat(drivers/mediafire): support concurrent upload  (#1366)

* feat(drivers): add MediaFire driver with concurrent upload support

- Implement complete MediaFire storage driver with session token authentication
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload with intelligent and multi-unit transfers
- Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files
- Optimize memory usage with adaptive buffer sizing (10MB-100MB (default))
- Include rate limiting and retry logic for API requests
- Add proper error handling and progress reporting
- Handle MediaFire's bitmap-based resumable upload protocol

Closes PR #1322

* feat(stream): add DiscardSection method to StreamSectionReader for skipping data

* feat(mediafire): refactor resumableUpload logic for improved upload handling and error management

* fix(mediafire): stop cron job and clear action token in Drop method

* .

* fix(mediafire): optimize buffer sizing logic in uploadUnits method

* fix(docs): remove duplicate MediaFire

* fix(mediafire): revert 'optimization', large files should not be fully chached.

---------

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

---------

Signed-off-by: ILoveScratch <ilovescratch@foxmail.com>
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: Da3zKi7 <da3zki7@duck.com>
Co-authored-by: KirCute <951206789@qq.com>
Co-authored-by: Suyunmeng <Susus0175@proton.me>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
@jyxjjj jyxjjj deleted the Da3zKi7-feat/mediafire branch October 14, 2025 08:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants