Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(nodejs): add WriteOptions for write methods #4785

Merged
merged 4 commits into from
Jun 24, 2024
Merged

feat(nodejs): add WriteOptions for write methods #4785

merged 4 commits into from
Jun 24, 2024

Conversation

bxb100
Copy link
Contributor

@bxb100 bxb100 commented Jun 22, 2024

close: #4782

@bxb100

This comment was marked as outdated.

@bxb100

This comment was marked as off-topic.

@syhily
Copy link

syhily commented Jun 22, 2024

hi, @syhily the code is implemented like this, any suggestion?

async function main() {
  const op = new Operator('s3', {
    root: '/test_opendal',
    bucket: 'your bucket name',
    region: 'your bucket region',
    endpoint: 'your endpoint',
    access_key_id: 'your access key id',
    secret_access_key: 'your secret access key',
  })
  
  await op.writeWith('test', 'Hello, World!')
    .contentType('text/plain')
    .apply()
}

Cool. I think this code snippet satisfied my needs.

@Xuanwo
Copy link
Member

Xuanwo commented Jun 23, 2024

hi, @Xuanwo @sundy-li add devbox1 (nix wrapper) is a good idea or not?

This might be a good idea but how about moving them to a seperate PR?

bindings/nodejs/generated.d.ts Outdated Show resolved Hide resolved
bindings/nodejs/generated.d.ts Outdated Show resolved Hide resolved
@bxb100

This comment was marked as outdated.

@bxb100 bxb100 marked this pull request as ready for review June 23, 2024 12:23
@bxb100
Copy link
Contributor Author

bxb100 commented Jun 23, 2024

remove writeWith instead enhance original write methods

await operator.write(filename, c, {
      contentType,
      contentDisposition,
    })
// or
const writer = await operator.writer(filename, {
      contentType,
      contentDisposition,
    })
// or
operator.writeSync(filename, c, {
      contentType,
      contentDisposition,
    })
// or
const writer = operator.writerSync(filename, {
      contentType,
      contentDisposition,
    })

bindings/nodejs/generated.d.ts Outdated Show resolved Hide resolved
bindings/nodejs/.prettierignore Show resolved Hide resolved
bindings/nodejs/generated.d.ts Outdated Show resolved Hide resolved
bindings/nodejs/generated.d.ts Outdated Show resolved Hide resolved
bindings/nodejs/src/lib.rs Show resolved Hide resolved
bindings/nodejs/src/lib.rs Outdated Show resolved Hide resolved
bindings/nodejs/tests/testcontainers.test.mjs Outdated Show resolved Hide resolved
@bxb100 bxb100 changed the title feat(nodejs): Expose writeWith method feat(nodejs): add OpWriteOptions for write methods Jun 23, 2024
@bxb100
Copy link
Contributor Author

bxb100 commented Jun 23, 2024

hi, @suyanhanx , should I manually update the package.json version

@suyanhanx
Copy link
Member

hi, @suyanhanx , should I manually update the package.json version

Please add those new features into upgrade.md as unreleased. That's enough. We will update the version when we release it.

@bxb100
Copy link
Contributor Author

bxb100 commented Jun 23, 2024

hi, @suyanhanx , should I manually update the package.json version

Please add those new features into upgrade.md as unreleased. That's enough. We will update the version when we release it.

Thanks replied, I will add it lately

@suyanhanx
Copy link
Member

hi, @suyanhanx , should I manually update the package.json version

Please add those new features into upgrade.md as unreleased. That's enough. We will update the version when we release it.

Thanks replied, I will add it lately

This is not something you need to finish in this PR.😊A seperate PR is okay.

@@ -32,6 +32,35 @@ export interface ListOptions {
limit?: number
recursive?: boolean
}
export interface OpWriteOptions {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for you effort. But OpXxx is not an existing concept at the binding side. How about using WriteOptions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO, the OpWriteOptions concept corresponds to OpWrite, so I think it's better to use one object not OpWriteOptions, OpWriterOptions

the name starts with Op also because of this, but WriteOptions is ok for me

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO, the OpWriteOptions concept corresponds to OpWrite, so I think it's better to use one object not OpWriteOptions, OpWriterOptions

Your understanding is correct. But there is a gap between rust core and nodejs binding.

Rust Core users have a concept called OpXxx like OpWrite and OpRead. But we never expose those APIs to nodejs binding. So from the nodejs users view, they don't know about OpXxx thus they can't understand what's OpXxx represents for.

So I think it's better to use WriteOptions without the Op prefix.

And for WriteOptions and WriterOptions, it's possible for the future to introduce different options inside write and writer options. So I want to seperate them for now instead of another breaking changes in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that's a comprehensive explanation, I will fix it ASAP

/**
* Write multiple bytes into path synchronously.
*
* It could be used to write large file in a streaming way.
*/
writerSync(path: string): BlockingWriter
writerSync(path: string, options?: OpWriteOptions | undefined | null): BlockingWriter
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we need WriterOptions too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as I mentioned above, I think WriteOptions is enough

@@ -30,6 +29,11 @@ use futures::AsyncReadExt;
use futures::TryStreamExt;
use napi::bindgen_prelude::*;

use opendal::operator_functions::{FunctionWrite, FunctionWriter};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better not to play with those APIs. What do you think about adding bunch of if-else directly?


This PR also makes me thinking about exporting WriteOptions from the rust core side. We can have a discussion there.

Copy link
Contributor Author

@bxb100 bxb100 Jun 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you mean this?

         let mut writer = self.0.write_with(&path, c);
         if let Some(options) = options {
-            writer = writer.with(options);
+            if let Some(append) = options.append {
+                writer = writer.append(append);
+            }
+            if let Some(chunk) = options.chunk {
+                writer = writer.chunk(chunk.get_u64().1 as usize);
+            }
+            if let Some(ref content_type) = options.content_type {
+                writer = writer.content_type(content_type);
+            }
+            if let Some(ref content_disposition) = options.content_disposition {
+                writer = writer.content_disposition(content_disposition);
+            }
+            if let Some(ref cache_control) = options.cache_control {
+                writer = writer.cache_control(cache_control);
+            }
         }
         writer.await.map_err(format_napi_error)
     }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

Copy link
Member

@Xuanwo Xuanwo Jun 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can polish here in the future if rust core exposes WriteOptions directly.

@Xuanwo Xuanwo changed the title feat(nodejs): add OpWriteOptions for write methods feat(nodejs): add WriteOptions for write methods Jun 24, 2024
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot!

@Xuanwo Xuanwo merged commit 591d034 into apache:main Jun 24, 2024
58 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Expose write_with method for Node.js binding
4 participants