Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(raw/oio): Use Buffer as cache in OneshotWrite #4477

Merged
merged 5 commits into from
Apr 13, 2024

Conversation

reswqa
Copy link
Member

@reswqa reswqa commented Apr 12, 2024

This closes #4456.

@reswqa
Copy link
Member Author

reswqa commented Apr 12, 2024

Oops, too many impl should also be changed. Give me a minute 😉

@Xuanwo
Copy link
Member

Xuanwo commented Apr 12, 2024

Oops, too many impl should also be changed. Give me a minute 😉

Thank you!

let req = self.core.dbfs_create_file_request(&self.path, bs)?;
let req = self
.core
.dbfs_create_file_request(&self.path, bs.to_bytes())?;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function would call BASE64_STANDARD.encode(body), which require a AsRef[u8], this is not implements for Buffer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add a helper function that uses base64::write::EncoderWriter. But I believe it's out of the scope of current PR, maybe worth a new one.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tracked at #4483

let size = bs.len();

if size <= Self::MAX_SIMPLE_SIZE {
self.write_simple(bs).await?;
} else {
self.write_chunked(bs).await?;
self.write_chunked(bs.to_bytes()).await?;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

write_chunked splits continuous data in chunks. The migration to Buffer should not be trivial. WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do want to refactor OneDrive entirely. Its implementation doesn't fit well with our current design.

@reswqa reswqa marked this pull request as ready for review April 12, 2024 15:56
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect, thanks!

@Xuanwo Xuanwo merged commit 275224d into apache:main Apr 13, 2024
217 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use Buffer as cache in OneshotWrite
2 participants