feat(core): sets default chunk_size and sends buffer > chunk_size directly

[evenyag]

What's changed

Currently, CompleteLayer sanitizes the chunk size when users call with_chunk() explicitly.
https://docs.rs/opendal/latest/opendal/raw/struct.OpWrite.html#method.with_chunk

If users don't provide a chunk size, the writer won't guarantee the body size is larger than the minimum multi-part upload size. It's easy to forget to set the correct chunk size for the writer.

This PR changes this behavior:

• Uses write_multi_min_size or write_multi_align_size as the chunk size if users don't provide one.
• The ChunkedWriter attempts to send the bs directly if it is larger than the chunk size
  • It adds a new argument exact to the ChunkedWriter. If exact is true, the writer still ensures the upload body size is exactly the chunk size.
  • Backends with write_multi_align_size will set exact to true.
  • If users provide a chunk size, exact will also be true.

There is a breaking change to ChunkedWriter::new().

Reference

GreptimeTeam/greptimedb#4098 (comment)

[evenyag]

The docs CI for the main branch is broken.
https://github.com/apache/opendal/actions/workflows/docs.yml?query=branch%3Amain

I guess the failure of this check is unrelated to this PR.
https://github.com/apache/opendal/actions/runs/9419578266/job/25949653713?pr=4710

---- behavior::test_rename_nested ----
test panicked: read must succeed: Unexpected (persistent) at  => reader got too little data

Context:
   expect: 64912
   actual: 0

[Xuanwo]

Thanks for you PR, I will take a review this weekend 🙏

[Xuanwo]

Thanks a lot for your PR! This change affects our behavior of chunk in write process. Would you like to update the comments for chunk too? Like:

opendal/core/src/types/operator/operator.rs

Lines 918 to 932 in 78692ce

	/// ## `chunk`
	///
	/// Set `chunk` for the writer.
	///
	/// OpenDAL is designed to write files directly without chunking by default, giving users
	/// control over the exact size of their writes and helping avoid unnecessary costs.
	///
	/// This is not efficient for cases when users write small chunks of data. Some storage services
	/// like `s3` could even return hard errors like `EntityTooSmall`. Besides, cloud storage services
	/// will cost more money if we write data in small chunks.
	///
	/// Set a good chunk size might improve the performance, reduce the API calls and save money.
	///
	/// The following example will set the writer chunk to 8MiB. Only one API call will be sent at
	/// `close` instead.

[Xuanwo]

Nice idea about introducing exact.

[Xuanwo]

We can calculate if bs.len() + self.buffer.len() >= self.chunk_size and write them at once. This could be helpful for cases like: write(1KiB), write(16MiB), write(1KiB).

[evenyag]

Would you like to update the comments for chunk too

Fixed in d6a0449

[evenyag]

Some tests failed. I'll take a look.

[Xuanwo]

Hi, I just noticed that exact doesn't behave as I expected. If user has configured one or write_multi_align_size is some, we should use exact. Some storage services requires to use the exactly the same chunk for uploading, like R2

Object part sizes must be at least 5MiB but no larger than 5GiB. All parts except the last one must be the same size.

[Xuanwo]

Write returning 0 often been treated as an error WriteZero. It's better not to do this.

We can push only 1B in the buffer.

This case is extremely unusual, and almost no other service behaves this way. Those rules are just a safety measure. We can make it clear in the comments.

[evenyag]

What about pushing written bytes to the buffer queue? We already sent written bytes so the queue should have "written" bytes space left.

I'll push a commit later.

[Xuanwo]

Makes sense to me.

[Xuanwo]

Thanks a lot!