Tracking issues of Buffer migration

[Xuanwo]

Recently, OpenDAL has transitioned its I/O traits to a Buffer-based approach, which is optimized for both contiguous and non-contiguous bytes input. Additionally, the new Buffer facilitates vectored I/O and is io_uring compatible, enhancing prospects for future development.

However, some code paths convert Buffer back into Bytes, resulting in additional allocations, unnecessary copying, and increased memory usage, thereby negating all the advantages of using Buffer.

This tracking issue addresses all Buffer-related issues. Please comment if you identify any other cases that might cause regression.

All tasks are easy and good-first-issues, please jump in for details.

Tasks

• Use Buffer as cache in RangeWrite #4453
• Use Buffer as cache in BlockWrite #4454
• Use Buffer as cache in MultipartWrite #4455
• Use Buffer as cache in OneshotWrite #4456
• Reader should return Buffer #4464
• Writer should accept Buffer #4465
• feat: Optimize buffer implementation and add stream support #4458
• feat(core): Implement FromIterator for Buffer #4459
• refactor(core): Use Buffer in http request #4460
• kv::adapter should use Buffer #4479
• typed_kv::adapter should use Buffer #4480
• final: Polish the API naming
• Docs for read & write related API
• Upgrade Notes

Related

• close writer: support passing ChunkedBytes in Writer::write #4414

[George-Miao]

Is there any particular reason why implementing Iterator directly for Buffer instead of IntoIterator, iter and iter_mut?

[Xuanwo]

Is there any particular reason why implementing Iterator directly for Buffer instead of IntoIterator, iter and iter_mut?

No, this is just for easy implementation. I can't expose to expose too many types to user. We can refine this part later if users find it unsatisfactory.

[Xuanwo]

All task has been finised. Closing...