feat(drivers/mediafire): support concurrent upload #1366

Da3zKi7 · 2025-09-26T14:36:41Z

Implement complete MediaFire storage driver with session token authentication
Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
Include thumbnail generation for media files
Handle MediaFire's resumable upload with intelligent and multi-unit transfers
Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files
Optimize memory usage with adaptive buffer sizing (10MB-100MB (default))
Include rate limiting and retry logic for API requests
Add proper error handling and progress reporting
Handle MediaFire's bitmap-based resumable upload protocol

Closes PR #1322

Description / 描述

MediaFire driver

Motivation and Context / 背景

Originally asked at AList repo by @ILoveScratch2

Relates to PR #1322

How Has This Been Tested? / 测试

Adding my account session values and uploading files of different sizes. Also this driver is currently released at AList repo.

Checklist / 检查清单

I have read the CONTRIBUTING document.
我已阅读 CONTRIBUTING 文档。
I have formatted my code with go fmt or prettier.
我已使用 go fmt 或 prettier 格式化提交的代码。
I have added appropriate labels to this PR (or mentioned needed labels in the description if lacking permissions).
我已为此 PR 添加了适当的标签（如无权限或需要的标签不存在，请在描述中说明，管理员将后续处理）。
I have requested review from relevant code authors using the "Request review" feature when applicable.
我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
I have updated the repository accordingly (If it’s needed).
我已相应更新了相关仓库（若适用）。
- OpenList-Frontend #XXXX
- OpenList-Docs #XXXX

NOTES:

Sorry for creating another PR, I have no permissions to push against original mediafire branch here.
Maybe a cron job is required to continuously new MediaFire token, in fact, token expires every 10 mins.

Thanks @ILoveScratch2 @suyunjing @j2rong4cn @KirCute for improvements of original AList implementation.

KirCute · 2025-09-26T14:45:44Z

We didn't know the MediaFire token expired so fast. You can consider reverting this part of the change, but please remember to stop the cron job in the Drop method.

- Implement complete MediaFire storage driver with session token authentication - Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir - Include thumbnail generation for media files - Handle MediaFire's resumable upload with intelligent and multi-unit transfers - Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files - Optimize memory usage with adaptive buffer sizing (10MB-100MB (default)) - Include rate limiting and retry logic for API requests - Add proper error handling and progress reporting - Handle MediaFire's bitmap-based resumable upload protocol Closes PR OpenListTeam#1322

Da3zKi7 · 2025-09-26T15:02:13Z

We didn't know the MediaFire token expired so fast. You can consider reverting this part of the change, but please remember to stop the cron job in the Drop method.

Yeah, that happens because since MediaFire does not provide API_KEY nor APP support for new projects, basically we are using API, but without the "classical" auth.

And... if OpenList server is running then MediaFire is constantly refreshing the token, but once server or device is shutdown, you have to configure session again, which is very disgusting, but this is how it goes, not even rclone has MediaFire support now, no more official dev support :(

Ok, I'll revert cron code.

- Implement complete MediaFire storage driver with session token authentication - Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir - Include thumbnail generation for media files - Handle MediaFire's resumable upload with intelligent and multi-unit transfers - Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files - Optimize memory usage with adaptive buffer sizing (10MB-100MB (default)) - Include rate limiting and retry logic for API requests - Add proper error handling and progress reporting - Handle MediaFire's bitmap-based resumable upload protocol - Implement automatic session token renewal Closes PR OpenListTeam#1322

Da3zKi7 · 2025-09-26T15:10:02Z

@KirCute done, I followed threaded logic with chunks, without fully caching files.

ILoveScratch2 · 2025-09-26T15:12:18Z

Closes PR #1322

并不会，因为可以修改PR目标分支。目标基本分支已改为mediafire 合并后修改将会直接出现在#1322内

Da3zKi7 · 2025-09-26T15:35:16Z

@ILoveScratch2, yes, but once merged it mediafire branch, could be then merged into main.

ILoveScratch2 · 2025-09-26T16:16:49Z

@Da3zKi7 Yep

Da3zKi7 · 2025-09-26T16:18:44Z

@ILoveScratch2 thank your for your work mate

ILoveScratch2 · 2025-09-26T16:24:40Z

@Da3zKi7 You're welcome. Actually, you were the one who handled most of the APIs, and without your initial modifications, this wouldn't have been possible..

Da3zKi7 · 2025-09-26T16:49:50Z

Yeah, some day, just for curiosity I tried to add MediaFire account to AList and figured out was not there. Then I saw your issue requesting the driver. And I did it, maybe bad code :) but actually working )))

You are always welcome!

drivers/mediafire/driver.go

…ipping data

…andling and error management

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

…y chached.

Da3zKi7 · 2025-09-28T07:50:41Z

@j2rong4cn

Small files (< 10MB): Uses the MediaFire API's unitSize (typically 4MB) as the buffer. This may cause "stream not cached" errors with concurrent access since the buffer might be smaller than conf.MaxBufferLimit. That error already happened me!

Medium files (10MB-100MB): Uses the entire file size as buffer, which triggers full file caching in NewStreamSectionReader, enabling concurrent chunk access.

Large files (>100MB): Caps the buffer at 100MB to prevent excessive memory usage while still likely triggering the caching mechanism for concurrent access.

For example, testing using go run main.go server

max buffer limit: i.e. 1024MB

If a large file is uploaded, i.e. 2GB, will exceed the avail buffer limit, also as @KirCute stated before fully caching a file is not a good practice, and obviously not a optimized way.

Also, as I already told all, MediaFire resumable upload already manages chunks (yes, that endpoint is uploading each part of file and could be paused, and then resumed), so,

What you did:

ss, err := stream.NewStreamSectionReader(file, int(unitSize), &up)

This fully caches the file, and for example a 1GB file will use 1GB in memory, and a file of 5GB need to be cached in temp storage, which is actually not optimization.

Medium files (10MB-100MB): Uses full file size, triggering file.CacheFullAndWriter() since it exceeds typical unit sizes

Large files (>100MB): Uses 100MB buffer, which exceeds i.e 1024MB (or whatever in your machine) MaxBufferLimit threshold, also triggering full caching

Small files: Uses unit size for minimal memory footprint

MediaFire commonly uses 4MB units, but it depends of how big the file is, and assuming it is an error.

If you have Pro / Ultra plan you can upload up to 50GB files, obviously using 50GB in temp storage has no sense

I know doing ss, err := stream.NewStreamSectionReader(file, int(unitSize), &up) is used in another drivers, but I explained why at least for MediaFire is not optimized and does not even has sense doing it.

The intelligent buffer sizing approach is the best way because:

Enables concurrent uploads for better performance
Prevents memory issues through automatic disk caching
Follows OpenList patterns used across other drivers
Scales efficiently from small to very large files
The simple int(unitSize) approach only works for sequential uploads and will cause errors with the concurrent upload implementation. Remember, the upload I coded is actually using a concurrent upload, I even clarified it on commit title.

Ordered Group with Context: Creates multiple goroutines (up to d.UploadThreads, default 3) that can upload different chunks simultaneously

Stream Section Reader: Uses intelligent buffer sizing to enable concurrent access to different file sections, forcing file caching when needed for large files

threadG, uploadCtx := errgroup.NewOrderedGroupWithContext(ctx, thread,
		retry.Attempts(3),
		retry.Delay(time.Second),
		retry.DelayType(retry.BackOffDelay))

This implements the most common upload in drivers, but remember, MediaFire resumable upload is actually uploading file chunks.

The concurrent implementation processes multiple MediaFire upload units simultaneously (up to 3 threads by default), significantly improving upload performance compared to sequential processing of each unit one by one.

So my aproach is using AList / OpenList threaded upload over MediaFire chunked upload

For more context, please read MediaFire documentation.

But for simplicity, a short paragraph:

Definition: Resumes an upload that was interrupted. Upload a unit/chunk of a file as part of a resumable upload to the user's account. Uploading large files through upload/simple can take a long time, and if it fails, re-uploading the file starts from the beginning--unlike a resumable upload, which allows you to upload the file in small units or chunks. So if one unit fails to upload, only that unit is re-uploaded. upload/resumable can only be called after a call to upload/check which initiates the resumable upload. This API returns the upload key when successful. After you upload all the units, you can pass the key to upload/poll_upload to get the quickkey. Please refer to the documentation about the API upload/poll_upload for more details.

Here, is another proof why fully caching has no logic explanation. The upload need it in chunks, not the full file!

j2rong4cn · 2025-09-28T14:49:41Z

/upload/check.php需要提供SHA256，而stream只能读取一次，在stream没携带SHA256时会触发缓存，这无法避免

OpenList/drivers/mediafire/driver.go

Lines 359 to 368 in a0ebdcd

    
           fileHash := file.GetHash().GetHash(utils.SHA256) 
        
           var err error 
        
           // Try to use existing hash first, cache only if necessary 
        
           if len(fileHash) != utils.SHA256.Width { 
        
           	_, fileHash, err = stream.CacheFullAndHash(file, &up, utils.SHA256) 
        
           	if err != nil { 
        
           		return nil, err 
        
           	} 
        
           }

而stream.NewStreamSectionReader是为了某个块上传出错时可以重新读取而设计的，如果mediafire不需要，直接使用for+io.LimitReader最合适

Da3zKi7 · 2025-09-29T01:17:32Z

Caching and uploading is per chunk, not per full file LoL

你说得对，但unitSize大于bufferSize GetSectionReader会报错

OpenList/internal/stream/util.go

Line 212 in d465da4

buf = tempBuf[:length]

你应该开个新的pr修改StreamSectionReader实现块缓存

Yes, but MediaFire's unitSize is only 4MB by default, so even for big files will never exceed the buffer.

``
// Intelligent buffer sizing for large files
bufferSize := int(unitSize)
fileSize := file.GetSize()

// Split in chunks
if fileSize > d.ChunkSize*1024*1024 {

	// Large, use ChunkSize (default = 100MB)
	bufferSize = min(int(fileSize), int(d.ChunkSize)*1024*1024)
} else if fileSize > 10*1024*1024 {
	// Medium, use full file size for concurrent access
	bufferSize = int(fileSize)
}

// Create stream section reader for efficient chunking
ss, err := stream.NewStreamSectionReader(file, bufferSize, &up)

``

By default bufferSize = 4MB

If file is bigger than 100MB, bufferSize is 100MB
If file is bigger than 10MB, just set a bufferSize like 58MB
If file is smaller than 10MB just set bufferSize to likely 4MB

That means a 8MB file will use a 4MB buffer without problems, because MediaFire expects only 4MB chunk, not a 8MB chunk. If MediaFire determines that due the file is so big, for example 20MB for every unitSize, MediaFire will accept only 20MB chunks.

Dynamic bufferSize approach is the solution:

Ensures proper file caching for concurrent uploads
Optimizes memory usage based on file size
Prevents the "stream not cached" errors. Yeah, that happened me when I did modifications to use GThreads...
Follows established patterns in the OpenList codebase, but respects MediaFire resumable upload logic at max stability, speed and resource usage.

Remember, unitSize logic is handled by MediaFire, not by the driver, once MediaFire defines unitSize is used in buffer logic.

The fixed int(unitSize) approach works only for sequential uploads but fails with concurrent access patterns, the implemented upload is concurrent, not sequential!

I really appreciate your suggestions, but this is the way MediaFire's API works.

Da3zKi7 · 2025-09-29T01:33:51Z

Also...

When MediaFire's unitSize by default is 4MB and you use it directly as the buffer size, this creates several issues based on the OpenList caching logic.

Since your system max buffer limit could be .i.e. 1024MB, the 4MB unitSize is much smaller than the MaxBufferLimit. This means:

No file caching occurs - The condition maxBufferSize > conf.MaxBufferLimit (4MB > 1024MB) is false
Sequential reading only - Without caching, concurrent access fails with "stream not cached" errors
Poor concurrent performance - Your concurrent goroutines can't access different file offsets simultaneously

I have an example, that happened me while testing the upload...

I experienced exactly this issue:

Using int(unitSize) with 4MB caused "stream not cached: request offset 4194304 != current offset 0"
The offset 4194304 is exactly 4MB (4 * 1024 * 1024), showing the second chunk couldn't be accessed concurrently

MediaFire upload is concurrent, not sequential... that causes errors

j2rong4cn · 2025-09-29T03:40:23Z

Using int(unitSize) with 4MB caused "stream not cached: request offset 4194304 != current offset 0"
The offset 4194304 is exactly 4MB (4 * 1024 * 1024), showing the second chunk couldn't be accessed concurrently

errgroup.Lifecycle的Before在errgroup.NewOrderedGroupWithContext是顺序执行的

OpenList/pkg/errgroup/errgroup.go

Lines 55 to 63 in d465da4

    
           type Lifecycle struct { 
        
           	// Before在OrderedGroup是线程安全的。 
        
           	// 只会被调用一次 
        
           	Before func(ctx context.Context) error 
        
           	// 如果Before返回err就不调用Do 
        
           	Do func(ctx context.Context) error 
        
           	// 最后调用一次After 
        
           	After func(err error) 
        
           }

OpenList/pkg/errgroup/errgroup.go

Lines 32 to 36 in d465da4

    
           func NewOrderedGroupWithContext(ctx context.Context, limit int, retryOpts ...retry.Option) (*Group, context.Context) { 
        
           	group, ctx := NewGroupWithContext(ctx, limit, retryOpts...) 
        
           	group.startChan = make(chan token, 1) 
        
           	return group, ctx 
        
           }

OpenList/pkg/errgroup/errgroup.go

Lines 65 to 108 in d465da4

    
           func (g *Group) GoWithLifecycle(lifecycle Lifecycle) { 
        
           	if g.startChan != nil { 
        
           		select { 
        
           		case <-g.ctx.Done(): 
        
           			return 
        
           		case g.startChan <- token{}: 
        
           		} 
        
           	} 
        
           	if g.sem != nil { 
        
           		select { 
        
           		case <-g.ctx.Done(): 
        
           			return 
        
           		case g.sem <- token{}: 
        
           		} 
        
           	} 
        
           	g.wg.Add(1) 
        
           	go func() { 
        
           		defer g.done() 
        
           		var err error 
        
           		if lifecycle.Before != nil { 
        
           			err = lifecycle.Before(g.ctx) 
        
           		} 
        
           		if err == nil { 
        
           			if g.startChan != nil { 
        
           				<-g.startChan 
        
           			} 
        
           			err = retry.Do(func() error { return lifecycle.Do(g.ctx) }, g.opts...) 
        
           		} 
        
           		if lifecycle.After != nil { 
        
           			lifecycle.After(err) 
        
           		} 
        
           		if err != nil { 
        
           			select { 
        
           			case <-g.ctx.Done(): 
        
           				return 
        
           			default: 
        
           				g.cancel(err) 
        
           			} 
        
           		} 
        
           	}() 
        
           }

按理说不会出现这种情况，请你重新测试

Da3zKi7 · 2025-09-29T08:58:37Z

@ILoveScratch2 I am tired of this disgusting guy with god complex. Please do not take my work again, I'll only contribute to AList, this is only another fork. I have no time and I do not take a cent for coding drivers. So sorry, I will not support your needs anymore. If you need something, ask for it at AList repo. Open source is supposed to be collaborative, looks like a random guy is pretending to be the good and the best developer, I am really tired, he does not even know how MediaFire API works, I am the first guy how developed a driver, rclone does not even support MediaFire yet.

So, keep acting as gods, and develop your own code without taking others job.

I do not even see you commited a driver at AList, you are not AList experts, you did nothing for AList, just being disgusting and selfish people, instead of being thankful about what others code for free.

Do not tag me anymore, I do not talk with selfish people

@ILoveScratch2 MediaFire is already available at AList, leave this sh!t and disgusting people. Nobody is perfect, but at least I read all MediaFire documentation, and what you did apart of just treating others as dummies, ok gods, I need you code drivers and really commit to AList instead of trashy forks!

Do not tag me here anymore!

Da3zKi7 · 2025-09-29T09:06:10Z

I am done here

j2rong4cn · 2025-09-29T09:19:02Z

我只想说
unitSize大于bufferSize时在stream没缓存时GetSectionReader会panic
errgroup.Lifecycle的Before在errgroup.NewOrderedGroupWithContext是顺序执行的

I just want to point out that

when unitSize is greater than bufferSize and the stream has no cache, GetSectionReader will panic.

The Before events in errgroup.Lifecycle are executed sequentially in errgroup.NewOrderedGroupWithContext.

If you're okay with Google Translate's machine translation, I can use it to translate the Chinese into English and then reply.

ILoveScratch2 · 2025-09-29T12:31:42Z

@ILoveScratch2 I am tired of this disgusting guy with god complex. Please do not take my work again, I'll only contribute to AList, this is only another fork. I have no time and I do not take a cent for coding drivers. So sorry, I will not support your needs anymore. If you need something, ask for it at AList repo. Open source is supposed to be collaborative, looks like a random guy is pretending to be the good and the best developer, I am really tired, he does not even know how MediaFire API works, I am the first guy how developed a driver, rclone does not even support MediaFire yet.

So, keep acting as gods, and develop your own code without taking others job.

I do not even see you commited a driver at AList, you are not AList experts, you did nothing for AList, just being disgusting and selfish people, instead of being thankful about what others code for free.

Do not tag me anymore, I do not talk with selfish people

@ILoveScratch2 MediaFire is already available at AList, leave this sh!t and disgusting people. Nobody is perfect, but at least I read all MediaFire documentation, and what you did apart of just treating others as dummies, ok gods, I need you code drivers and really commit to AList instead of trashy forks!

Do not tag me here anymore!

For did nothing for AList, most of the people here are former contributors of AList
We never tagged you, and references to the original PR are inevitable
Open source really should be collaborative, so we never asked you to do anything. You can choose to help us to complete this driver, or not.

https://github.com/AlistGo/alist/commits/main/?author=KirCute
https://github.com/AlistGo/alist/commits/main/?author=j2rong4cn
https://github.com/AlistGo/alist/commits/main/?author=xrgzs
https://github.com/AlistGo/alist/graphs/contributors

Da3zKi7 · 2025-09-29T12:51:58Z

j2rong4cn never coded a driver, that was my point. You can criticize a lot others code, but, if you never developed an AList driver why being so mad. This is not about which codes better or who has more commits at AList or OpenList repos, is about why being hard with someone just tried to help!

Anyway ILoveScratch2, if do you think my code is crap, I expect you never cherry-pick or fork what I do, for you I am just a dummy who codes trash. I also read a comment in another PR that "looks like I do not know what I do", okay, so just ignore the drivers I code at AList. I am not the best, I am not even good maybe, but at least I tried to help, especially when you opened issue at AList repo, but okay, everything I do is with good intentions to help a little.

Best regards ))

ILoveScratch2 · 2025-09-29T12:59:00Z

j2rong4cn never coded a driver, that was my point. You can criticize a lot others code, but, if you never developed an AList driver why being so mad. This is not about which codes better or who has more commits at AList or OpenList repos, is about why being hard with someone just tried to help!

Anyway ILoveScratch2, if do you think my code is crap, I expect you never cherry-pick or fork what I do, for you I am just a dummy who codes trash. I also read a comment in another PR that "looks like I do not know what I do", okay, so just ignore the drivers I code at AList. I am not the best, I am not even good maybe, but at least I tried to help, especially when you opened issue at AList repo, but okay, everything I do is with good intentions to help a little.

Best regards ))

I opened issue in AList repo?

Anyway, You may be partly right, we shouldn't get too much into the driving problem, it feels like we've gone a little too far
He just felt that there was a potential bug in the logic, and that's it...

Da3zKi7 · 2025-09-29T13:02:57Z

j2rong4cn never coded a driver, that was my point. You can criticize a lot others code, but, if you never developed an AList driver why being so mad. This is not about which codes better or who has more commits at AList or OpenList repos, is about why being hard with someone just tried to help!
Anyway ILoveScratch2, if do you think my code is crap, I expect you never cherry-pick or fork what I do, for you I am just a dummy who codes trash. I also read a comment in another PR that "looks like I do not know what I do", okay, so just ignore the drivers I code at AList. I am not the best, I am not even good maybe, but at least I tried to help, especially when you opened issue at AList repo, but okay, everything I do is with good intentions to help a little.
Best regards ))

I opened issue in AList repo?

Anyway, You may be partly right, we shouldn't get too much into the driving problem, it feels like we've gone a little too far He just felt that there was a potential bug in the logic, and that's it...

I think you are right, too far. But I just needed you understand me, I explained and explained why even looks crazy, the code in that part had sense, due to the way MediaFire takes the chunks. But is okay, If you say is incorrect, is wrong...

KirCute · 2025-09-29T13:04:57Z

j2rong4cn never coded a driver, that was my point. You can criticize a lot others code, but, if you never developed an AList driver why being so mad. This is not about which codes better or who has more commits at AList or OpenList repos, is about why being hard with someone just tried to help!

Anyway ILoveScratch2, if do you think my code is crap, I expect you never cherry-pick or fork what I do, for you I am just a dummy who codes trash. I also read a comment in another PR that "looks like I do not know what I do", okay, so just ignore the drivers I code at AList. I am not the best, I am not even good maybe, but at least I tried to help, especially when you opened issue at AList repo, but okay, everything I do is with good intentions to help a little.

Best regards ))

Nobody say your code is crap. @j2rong4cn kept asking questions simply because he didn't fully understand what you were saying. To be honest, I didn't completely grasp it either, even though I used a translation tool. @j2rong4cn developed nearly the entire stream package and also worked on drivers after OpenList separated from AList. It is not appropriate to equate his workload in AList directly with his familiarity with the OpenList code.

Understanding how file streams should be properly handled is crucial, and we are all simply working towards ensuring the drivers function correctly, rather than competing to see who knows more about MediaFire.

KirCute · 2025-09-29T13:16:52Z

j2rong4cn never coded a driver, that was my point. You can criticize a lot others code, but, if you never developed an AList driver why being so mad. This is not about which codes better or who has more commits at AList or OpenList repos, is about why being hard with someone just tried to help!

Anyway ILoveScratch2, if do you think my code is crap, I expect you never cherry-pick or fork what I do, for you I am just a dummy who codes trash. I also read a comment in another PR that "looks like I do not know what I do", okay, so just ignore the drivers I code at AList. I am not the best, I am not even good maybe, but at least I tried to help, especially when you opened issue at AList repo, but okay, everything I do is with good intentions to help a little.

Best regards ))

"looks like I do not know what I do" is what I said. However, this is an inaccurate translation of my original statement. What I actually said was that 'the original author did not seem sufficiently familiar with the AList source code'. I had solid reasons for making this judgment. For example, you returned an ObjThumb with an empty Thumbnail and an object with an unchanged Path after moving it—these are common mistakes often made by first-time drive contributors, and it does not mean that I completely dismiss your work.

KirCute · 2025-09-29T13:39:08Z

I think you are right, too far. But I just needed you understand me, I explained and explained why even looks crazy, the code in that part had sense, due to the way MediaFire takes the chunks. But is okay, If you say is incorrect, is wrong...

All in all, I hope you don’t interpret the discussion as one-sided criticism. In my view, j2rong4cn’s comments were more about confirming and inquiring why the code you wrote is correct, rather than unilaterally accusing you of writing it wrong.

All the best.

Da3zKi7 · 2025-09-29T17:19:33Z

j2rong4cn never coded a driver, that was my point. You can criticize a lot others code, but, if you never developed an AList driver why being so mad. This is not about which codes better or who has more commits at AList or OpenList repos, is about why being hard with someone just tried to help!
Anyway ILoveScratch2, if do you think my code is crap, I expect you never cherry-pick or fork what I do, for you I am just a dummy who codes trash. I also read a comment in another PR that "looks like I do not know what I do", okay, so just ignore the drivers I code at AList. I am not the best, I am not even good maybe, but at least I tried to help, especially when you opened issue at AList repo, but okay, everything I do is with good intentions to help a little.
Best regards ))

Nobody say your code is crap. @j2rong4cn kept asking questions simply because he didn't fully understand what you were saying. To be honest, I didn't completely grasp it either, even though I used a translation tool. @j2rong4cn developed nearly the entire stream package and also worked on drivers after OpenList separated from AList. It is not appropriate to equate his workload in AList directly with his familiarity with the OpenList code.

Understanding how file streams should be properly handled is crucial, and we are all simply working towards ensuring the drivers function correctly, rather than competing to see who knows more about MediaFire.

Hello, sorry. Seems the context was misinterpreted and is due to translation faults. I know you committed crucial and util things for AList, discredit your work was wrong.

Instead of being stressed maybe I should appreciate you do not want to merge possible unstable code and having a strict quality control is not bad at all.

Yes, I do not see this as competition, is not about who codes the best drivers or things, is about supporting a community and many people who does not have a GitHub account and is a "standard" user who downloaded OpenList, most people would never thank you for the driver, first of all because is not about fame, and because they do not know how to do it.

Da3zKi7 · 2025-09-29T17:30:00Z

j2rong4cn never coded a driver, that was my point. You can criticize a lot others code, but, if you never developed an AList driver why being so mad. This is not about which codes better or who has more commits at AList or OpenList repos, is about why being hard with someone just tried to help!
Anyway ILoveScratch2, if do you think my code is crap, I expect you never cherry-pick or fork what I do, for you I am just a dummy who codes trash. I also read a comment in another PR that "looks like I do not know what I do", okay, so just ignore the drivers I code at AList. I am not the best, I am not even good maybe, but at least I tried to help, especially when you opened issue at AList repo, but okay, everything I do is with good intentions to help a little.
Best regards ))

Nobody say your code is crap. @j2rong4cn kept asking questions simply because he didn't fully understand what you were saying. To be honest, I didn't completely grasp it either, even though I used a translation tool. @j2rong4cn developed nearly the entire stream package and also worked on drivers after OpenList separated from AList. It is not appropriate to equate his workload in AList directly with his familiarity with the OpenList code.

Understanding how file streams should be properly handled is crucial, and we are all simply working towards ensuring the drivers function correctly, rather than competing to see who knows more about MediaFire.

There is no doubt @j2rong4cn supported AList, and I know he has a strong knowledge, I interpreted his comments as criticism instead of a suggestion or pure question because context or examples were not clear.

I comprehend translation did not help us too!

Yes, now I understand he was trying to ask me instead of pointing the code was wrong.

Having a quality control is good

Da3zKi7 · 2025-09-29T17:35:05Z

j2rong4cn never coded a driver, that was my point. You can criticize a lot others code, but, if you never developed an AList driver why being so mad. This is not about which codes better or who has more commits at AList or OpenList repos, is about why being hard with someone just tried to help!
Anyway ILoveScratch2, if do you think my code is crap, I expect you never cherry-pick or fork what I do, for you I am just a dummy who codes trash. I also read a comment in another PR that "looks like I do not know what I do", okay, so just ignore the drivers I code at AList. I am not the best, I am not even good maybe, but at least I tried to help, especially when you opened issue at AList repo, but okay, everything I do is with good intentions to help a little.
Best regards ))

"looks like I do not know what I do" is what I said. However, this is an inaccurate translation of my original statement. What I actually said was that 'the original author did not seem sufficiently familiar with the AList source code'. I had solid reasons for making this judgment. For example, you returned an ObjThumb with an empty Thumbnail and an object with an unchanged Path after moving it—these are common mistakes often made by first-time drive contributors, and it does not mean that I completely dismiss your work.

That wrong translation changed all panorama. Yes, and I do not get offended by that statement. In fact MediaFire driver was my first commit and my first driver, also Golang is not my prior nor my favorite language, not even a language I have a solid experience.

Yes, if you do not update path, thumb fails. Also I forgot to add cron shutdown in Drop()

My code is not perfect and will never be the perfect code

Da3zKi7 · 2025-09-29T17:36:16Z

I think you are right, too far. But I just needed you understand me, I explained and explained why even looks crazy, the code in that part had sense, due to the way MediaFire takes the chunks. But is okay, If you say is incorrect, is wrong...

All in all, I hope you don’t interpret the discussion as one-sided criticism. In my view, j2rong4cn’s comments were more about confirming and inquiring why the code you wrote is correct, rather than unilaterally accusing you of writing it wrong.

All the best.

Got it, know I understand

All the best

Da3zKi7 · 2025-09-29T17:42:12Z

If you consider it appropriate, you could merge it or not, I tried to explain why works, and I tested uploading really small, small, medium and also big files. I know you have worries about if the driver is working, but you could help me testing the driver, if you find is not working we could fix it

Da3zKi7 · 2025-09-29T17:46:44Z

@ILoveScratch2 @j2rong4cn @KirCute

If you felt disrespected, I offer an apologize for you.

Despite what you decide, giving a sorry is always good :)

Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>

ILoveScratch2 · 2025-09-30T05:23:41Z

I understand you, it doesn't matter much.

Da3zKi7 · 2025-09-30T05:56:53Z

I understand you, it doesn't matter much.

Thank you, If you need something just ask me, I am not the best, but I always try to help.

* feat(drivers): add MediaFire driver support (#9319) - Implement complete MediaFire storage driver - Add authentication via session_token and cookie - Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir - Include thumbnail generation for media files - Handle MediaFire's resumable upload API with multi-unit transfers - Add proper error handling and progress reporting Co-authored-by: Da3zKi7 <da3zki7@duck.com> * fix(mediafire): fix code errors in mediafire * fix(mediafire): fix code errors in mediafire * fix(drivers): add session renewal cron for MediaFire driver (#9321) - Implement automatic session token renewal every 6-9 minutes - Add validation for required SessionToken and Cookie fields in Init - Handle session expiration by calling renewToken on validation failure - Prevent storage failures due to MediaFire session timeouts Fixes session closure issues that occur after server restarts or extended periods. Co-authored-by: Da3zKi7 <da3zki7@duck.com> * docs: restore README changes Signed-off-by: ILoveScratch <ilovescratch@foxmail.com> * fix * fix * fix: add stream upload limit * fix * fix: clear action token on drop and refactor header setting * feat(drivers/mediafire): optimize file caching - support direct stream processing - Remove forced caching to *os.File type - Support generic model.File interface for better flexibility - Improve upload efficiency by avoiding unnecessary file conversions - Fix return type to use model.Object instead of model.ObjThumb * feat(drivers/mediafire): improve global rate limiting - Ensure all API methods properly use context for rate limiting - Fix context parameter usage in getDirectDownloadLink, getActionToken, getFileByHash - Maintain consistent rate limiting across all MediaFire API calls * feat(drivers/mediafire): unify return types - remove unnecessary ObjThumb - Change MakeDir, Rename, Copy methods to return model.Object instead of model.ObjThumb - Remove empty Thumbnail fields where not meaningful - Keep ObjThumb only for fileToObj (List operations) which provides actual thumbnail URLs - Improve code consistency and reduce unnecessary wrapper objects * refactor(drivers/mediafire): extract common error handling logic - Add checkAPIResult helper function to reduce code duplication - Replace repetitive MediaFire API error checks with centralized function - Maintain specific error messages for unique cases (token, upload, search) - Improve code maintainability and consistency * enhance(drivers/mediafire): improve quick upload implementation - Add null check for existingFile to prevent potential issues - Improve error handling in quick upload - continue normal upload if search fails - Add detailed comments explaining quick upload logic - Optimize getExistingFileInfo with clearer fallback strategy - Ensure upload reliability even when file search encounters issues * refactor(drivers/mediafire): optimize request method reusability - Extract common HTTP request logic into apiRequest method - Reduce code duplication between getForm and postForm methods - Maintain backward compatibility with existing method signatures - Centralize rate limiting and header management - Support extensible HTTP method handling * docs(drivers/mediafire): add comprehensive English comments - Add function-level comments for all major driver methods - Document Init, List, Link, MakeDir, Move, Rename, Copy, Remove, Put methods - Add comments for key utility functions including session token management - Improve code readability and maintainability for community collaboration - Follow Go documentation conventions with clear, concise descriptions * perf(mediafire): optimize memory allocation and type assertion performance - Pre-allocate slice capacity in getFiles and bitmap conversion to reduce reallocations - Cache file type check in uploadUnits to avoid repeated type assertions - Add uploadSingleUnitOptimized for os.File to eliminate redundant type checks - Optimize string to int conversion with proper error handling - Improve memory efficiency in file upload operations * fix(mediafire): upload without cache * feat(mediafire): add rate limiting to all API methods - Add WaitLimit(ctx) calls to all driver methods: List, Link, MakeDir, Move, Rename, Copy, Remove, Put - Ensure consistent rate limiting across all MediaFire API interactions - Follow project standard pattern used by other drivers * feat(mediafire): improve error handling consistency - Add context parameter to all HTTP API functions for proper context propagation - Update getForm, postForm and apiRequest to accept context parameter - Fix rate limiting to use caller context instead of background context - Ensure consistent error handling patterns across all API calls - Improve cancellation and timeout support * feat(mediafire): refactor resumableUpload to use io.ReadSeeker and improve upload handling * fix(mediafire): release section reader * feat: add disk usage * feat(drivers/mediafire): support concurrent upload (#1387) * feat(drivers): add MediaFire driver with concurrent upload support - Implement complete MediaFire storage driver with session token authentication - Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir - Include thumbnail generation for media files - Handle MediaFire's resumable upload with intelligent and multi-unit transfers - Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files - Optimize memory usage with adaptive buffer sizing (10MB-100MB (default)) - Include rate limiting and retry logic for API requests - Add proper error handling and progress reporting - Handle MediaFire's bitmap-based resumable upload protocol Closes PR #1322 * feat(stream): add DiscardSection method to StreamSectionReader for skipping data * feat(mediafire): refactor resumableUpload logic for improved upload handling and error management * fix(mediafire): stop cron job and clear action token in Drop method * . * fix(mediafire): optimize buffer sizing logic in uploadUnits method * fix(docs): remove duplicate MediaFire * fix(mediafire): revert 'optimization', large files should not be fully chached. --------- Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com> Co-authored-by: Da3zKi7 <da3zki7@duck.com> Co-authored-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com> Co-authored-by: j2rong4cn <j2rong@qq.com> Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com> * fix(mediafire): optimize hash calculation in uploadUnits function * feat(drivers/mediafire): support concurrent upload (#1366) * feat(drivers): add MediaFire driver with concurrent upload support - Implement complete MediaFire storage driver with session token authentication - Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir - Include thumbnail generation for media files - Handle MediaFire's resumable upload with intelligent and multi-unit transfers - Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files - Optimize memory usage with adaptive buffer sizing (10MB-100MB (default)) - Include rate limiting and retry logic for API requests - Add proper error handling and progress reporting - Handle MediaFire's bitmap-based resumable upload protocol Closes PR #1322 * feat(stream): add DiscardSection method to StreamSectionReader for skipping data * feat(mediafire): refactor resumableUpload logic for improved upload handling and error management * fix(mediafire): stop cron job and clear action token in Drop method * . * fix(mediafire): optimize buffer sizing logic in uploadUnits method * fix(docs): remove duplicate MediaFire * fix(mediafire): revert 'optimization', large files should not be fully chached. --------- Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com> Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com> Co-authored-by: j2rong4cn <j2rong@qq.com> Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com> --------- Signed-off-by: ILoveScratch <ilovescratch@foxmail.com> Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com> Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com> Co-authored-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com> Co-authored-by: Da3zKi7 <da3zki7@duck.com> Co-authored-by: KirCute <951206789@qq.com> Co-authored-by: Suyunmeng <Susus0175@proton.me> Co-authored-by: j2rong4cn <j2rong@qq.com> Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

KirCute changed the base branch from main to mediafire September 26, 2025 14:49

Da3zKi7 force-pushed the feat/mediafire branch from 7e49493 to 5d22bda Compare September 26, 2025 14:49

Da3zKi7 force-pushed the feat/mediafire branch from 5d22bda to d68b1da Compare September 26, 2025 14:49

Da3zKi7 force-pushed the feat/mediafire branch from 217519a to 06abed9 Compare September 26, 2025 15:08

ILoveScratch2 requested a review from KirCute September 27, 2025 13:30

KirCute requested changes Sep 27, 2025

View reviewed changes

drivers/mediafire/driver.go Show resolved Hide resolved

j2rong4cn added 4 commits September 27, 2025 23:16

feat(stream): add DiscardSection method to StreamSectionReader for sk…

7beeb51

…ipping data

feat(mediafire): refactor resumableUpload logic for improved upload h…

9b33d93

…andling and error management

fix(mediafire): stop cron job and clear action token in Drop method

c3031e1

.

94bc043

j2rong4cn changed the title ~~feat(drivers): add MediaFire driver with concurrent upload support~~ feat(mediafire): support concurrent upload Sep 27, 2025

ILoveScratch2 requested a review from KirCute September 28, 2025 05:06

j2rong4cn and others added 4 commits September 28, 2025 13:23

Merge branch 'mediafire' into feat/mediafire

0b24556

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

fix(mediafire): optimize buffer sizing logic in uploadUnits method

21c6ea9

fix(docs): remove duplicate MediaFire

e756a67

fix(mediafire): revert 'optimization', large files should not be full…

7081986

…y chached.

jyxjjj changed the title ~~feat(mediafire): support concurrent upload~~ feat(drivers/mediafire): support concurrent upload Sep 28, 2025

Da3zKi7 closed this Sep 29, 2025

Da3zKi7 deleted the feat/mediafire branch September 29, 2025 09:06

ILoveScratch2 mentioned this pull request Sep 29, 2025

feat(drivers/mediafire): support concurrent upload #1387

Merged

Da3zKi7 restored the feat/mediafire branch September 29, 2025 17:42

Da3zKi7 reopened this Sep 29, 2025

Da3zKi7 force-pushed the feat/mediafire branch from 438fc72 to 7081986 Compare September 29, 2025 17:44

Merge branch 'mediafire' into feat/mediafire

8d598b6

Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>

ILoveScratch2 merged commit db05222 into OpenListTeam:mediafire Sep 30, 2025

Uh oh!

feat(drivers/mediafire): support concurrent upload #1366

feat(drivers/mediafire): support concurrent upload #1366

Uh oh!

Conversation

Da3zKi7 commented Sep 26, 2025

Description / 描述

Motivation and Context / 背景

How Has This Been Tested? / 测试

Checklist / 检查清单

Uh oh!

KirCute commented Sep 26, 2025

Uh oh!

Da3zKi7 commented Sep 26, 2025

Uh oh!

Da3zKi7 commented Sep 26, 2025

Uh oh!

ILoveScratch2 commented Sep 26, 2025

Uh oh!

Da3zKi7 commented Sep 26, 2025

Uh oh!

ILoveScratch2 commented Sep 26, 2025

Uh oh!

Da3zKi7 commented Sep 26, 2025

Uh oh!

ILoveScratch2 commented Sep 26, 2025

Uh oh!

Da3zKi7 commented Sep 26, 2025

Uh oh!

Uh oh!

Da3zKi7 commented Sep 28, 2025

Uh oh!

j2rong4cn commented Sep 28, 2025

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

j2rong4cn commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

j2rong4cn commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ILoveScratch2 commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

ILoveScratch2 commented Sep 29, 2025

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

KirCute commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KirCute commented Sep 29, 2025

Uh oh!

KirCute commented Sep 29, 2025

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

Da3zKi7 commented Sep 29, 2025

Uh oh!

ILoveScratch2 commented Sep 30, 2025

Uh oh!

Da3zKi7 commented Sep 30, 2025

Uh oh!

j2rong4cn commented Sep 29, 2025 •

edited

Loading

j2rong4cn commented Sep 29, 2025 •

edited

Loading

ILoveScratch2 commented Sep 29, 2025 •

edited

Loading

KirCute commented Sep 29, 2025 •

edited

Loading