Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quest: N-API Support #444

Closed
26 tasks done
dherman opened this issue Sep 24, 2019 · 22 comments
Closed
26 tasks done

Quest: N-API Support #444

dherman opened this issue Sep 24, 2019 · 22 comments
Labels
Milestone

Comments

@dherman
Copy link
Collaborator

dherman commented Sep 24, 2019

Prepare, once and future Neon contributors, for our noblest quest yet!

Pippin: Great! Where are we going?

We are going to port Neon to Node's new N-API!

Pippin: Huh?

I'll explain. N-API brings to Neon the promise of a stable, backwards-compatible ABI—binary compatibility across all future versions of Node.

This is a big deal.

Portability across Node versions means Neon will finally be practical for publishing libraries, not just apps: a few prebuilt binaries should be sufficient for all downstream customers to use your native library without ever knowing the difference.

Pippin: Oh, I get it!

The stuff of legend, no?

Our Quest

Step 1. Create the feature flag

  • Create a cargo feature flag to allow us to concurrently maintain the main Neon codebase along with the experimental N-API support in the same master branch. (Merged!)
  • Set up a test suite specifically for the N-API backend so each task can easily include adding tests (Basic N-API tests #449)

Step 2. Implement the port

  • Module contexts and initialization: Implement the neon::context::ModuleContext type and pass it to the module initialization function inside the register_module! macro defined in /src/lib.rs. The context struct will likely need to encapsulate the underlying napi_env and napi_value as private fields. This can be implemented before we implement functions, with an unimplemented export_function() method for now.
  • Functions: This is probably one of the subtler tasks. See the implementation of neon::types::JsFunction::new(). The Rust callback can be stored as the extra void* data passed to napi_create_function.
  • Function arguments: Implement CallContext::len() and CallContext::argument().
  • Function returns: Implement function return values.
  • this: Implement CallContext::this().
  • Call kinds: Implement CallContext::kind().
  • Function exports: Once we have module contexts and functions implemented, we can implement the ModuleContext::export_function() shorthand method.
  • Objects: See neon::types::JsObject::new() and the neon::object::Object methods.
  • Arrays: See neon::types::JsArray.
  • ArrayBuffers and Buffers: See neon::types::binary and the N-API functions for working with binary data, such as napi_create_arraybuffer, napi_create_buffer, etc.
  • Uninitialized and null: These should be pretty bite-sized. See neon::types::JsUndefined and neon::types::JsNull. @goto-bus-stop
  • Booleans: See neon::types::JsBoolean. @goto-bus-stop
  • Numbers: See neon::types::JsNumber.
  • Strings: See neon::types::JsString. We'll need to explore what binary string representations can be used between the NAN vs N-API runtimes for constructing JS strings.
  • Classes: This will require us to figure out how to do unique branding with N-API, but I believe napi_define_class supports this. (Here is one pure C example we can look to for inspiration.) <== not needed for functional completeness; see Transition guide for porting to N-API backend #596
  • Errors: See neon::types::error. We'll need to explore how N-API does throwing and catching errors. - @anshulrgoyal 🔒
  • Conversions: See the uses of neon_runtime::convert::* and the napi_coerce_* functions.
  • Scopes: Luckily, the N-API HandleScope mechanim matches V8's mechanism very closely. See neon::context and the uses of various HandleScope internal types.
  • Tag checks: See uses of neon_runtime::tag::*.
  • Task scheduling: See neon::task and neon::context::TaskContext, and the N-API "simply asynchronous operations" API, which uses the same underlying libuv thread pool as Neon's current backend, but with N-API's stable ABI. <== not needed for functional completeness; see Transition guide for porting to N-API backend #596
  • Thread-safe callbacks: This can be implemented for N-API once we've merged an implementation for RFC 25, using napi_make_callback. <== not needed for functional completeness; see Transition guide for porting to N-API backend #596
  • Windows Support: Windows requires linking against node.lib and win_delay_load_hook. Create a custom build script to link these on windows.

We have just a couple remaining items to finish up:

Step 3. Deprecate the legacy runtime

Once we finish the complete port, we can switch the default feature flags to use the new runtime and publish a new 0.x minor version. Eventually after a few releases we can remove the old runtime completely.

How to Contribute

Building N-API-based projects

To experiment with the N-API runtime or do manual testing, you can create a Neon project that uses the right feature flags. To try it out, you can run:

neon new --no-default-features --features=napi-latest --neon=path/to/neon my-project

where path/to/neon is the path on your local filesystem to a local clone of the Neon repo.

Manual Steps

The output of neon new executed above will produce a project that fails to build. When using the neon backend, either neon-build should be used with a simple cargo build or neon-cli should be used and neon-build should be removed. If both are used, the project will fail to build.

There is an RFC (neon-bindings/rfcs#36) to replace neon new which will correctly generate a project. The simplest change is to edit native/Cargo.toml:

  • Remove the neon-build dependency
  • Remove build = "build.rs"
  • delete native/build.rs

Note: If you create a Neon project nested inside the directory tree of a clone of the Neon repo, you'll need to add the line

[workspace]

to your Neon project's native/Cargo.toml manifest in order to build the project.

Adding an N-API primitive

To add an N-API primitive, you should implement it in pure Rust (using unsafe as necessary, but only as necessary!) in crates/neon-runtime/napi, and call out to the N-API backend exposed through nodejs-sys.

When the Neon runtime needs to pass around a data structure, you can make two different definitions of the type, separated by testing the feature flag with #[cfg(feature = "...")]. You may sometimes need to refactor the types in the Neon runtime to accommodate differences between the legacy and N-API runtimes.

Adding a test

The test/napi directory is the space for adding N-API acceptance tests. You can add native Rust logic to test/napi/native/src and JS logic to test/napi/lib. You can get examples of existing acceptance tests in our existing backend in test/dynamic, which has the same structure.

Will You Join Us?

As you can see, the quest ahead of us will be no small feat!

Pippin: Anyway you'll need people of intelligence on this... thing

Indeed, but fear not: we're here to help you if you get stuck. And many of these tasks can be a great way to get started with contributing to Neon and even learning Rust.

Claim one of the tasks today by leaving a comment below or pinging @dherman or @kjvalencik on Slack!

Pippin: I'm getting one

@dherman dherman added the quest label Sep 24, 2019
@ciscojeremy
Copy link

I'm interested.

@dherman
Copy link
Collaborator Author

dherman commented Oct 24, 2019

@ciscojeremy Awesome! Are you on the community Slack? @kjvalencik is just now working on some refactoring so you might want to sync up with a chat.

@ciscojeremy
Copy link

ciscojeremy commented Oct 24, 2019 via email

@kjvalencik
Copy link
Member

The Rust Bindings community Slack is open to all; use the Slackin app to receive an invitation.

@Brooooooklyn
Copy link

Maybe this repo could save your time. https://github.com/Brooooooklyn/napi-rs

@kjvalencik
Copy link
Member

kjvalencik commented Oct 25, 2019

👋 I'm working on ModuleContext and initialization. This work also covers some of scopes.

#458

@jrop
Copy link

jrop commented Oct 25, 2019

This is the best dev-related news today! (came from Twitter)

@dio
Copy link

dio commented Oct 25, 2019

Wow. I’m so excited! I hope I can contribute something! joining the community slack

@qwelias
Copy link

qwelias commented Oct 25, 2019

I'd like to learn rust, but I have no experience yet, would like to contribute and have lots of experience with node tho. So if you think a total rust noob can be trusted with a certain task -- hit me :)

@kjvalencik
Copy link
Member

@qwelias definitely! There will be some beginner friendly Rust tasks and likely some pure javascript as well. And, of course, testing and vetting of ideas are always needed!

@goto-bus-stop
Copy link
Member

I'm looking at undefined/null for a start! Possibly bools too if they are as similar as I suspect, so all the global singleton values could be done in one go.

@timsuchanek
Copy link

What are some of the next possible steps here?

@goto-bus-stop
Copy link
Member

i was planning on working on either objects or functions this Friday

@goto-bus-stop
Copy link
Member

ah i thought numbers got in together with bools but i had them in a separate branch: #492

started working on objects and as i was writing a test case, i wanted to use tag checks like val.is_a::<JsNumber>() as well, so that's where i'm looking. the tag checks need lots of boring mechanical changes to thread the Context value through a dozen layers 🙃

@timsuchanek
Copy link

Nice! So it sounds like a first prototype-ish version would already be usable quite soon?

@goto-bus-stop
Copy link
Member

goto-bus-stop commented Feb 28, 2020

I also started working on function calls (from JS into Rust for now) so parts of the existing test suite can be run on n-api. In-progress branch here: https://github.com/goto-bus-stop/neon/tree/functions

@timsuchanek once both objects and function calls are in, some very limited use cases would be possible. I expect many real-world cases would want the Class stuff from Neon so you can pass Rust structs between JS and Rust, so they can live for longer than a single call. that would probably come next.

@goto-bus-stop
Copy link
Member

Functions, function arguments, and function return values can be checked off now that #507 is merged 🎉

I would like to look at ArrayBuffers support next, since #507 already ported a few of the buffer tests that are currently being skipped.

@goto-bus-stop
Copy link
Member

Looking into getting the N-API runtime going on Windows today!

@dherman
Copy link
Collaborator Author

dherman commented Sep 18, 2020

We're almost there! I've struck out the items that are not needed for functional completeness and noted them in the new #596 issue for the deprecation plan. All that's left is to review the implementation of ComputeContext and ExecuteContext to make sure they properly create nested scopes, and we'll be able to close this issue!

@wspl
Copy link

wspl commented Nov 25, 2020

Would neon n-api support Electron without electron-rebuild or electron-build-env?

@kjvalencik
Copy link
Member

@wspl Correct, they would not be needed. There would be a minimum n-api version that we might need to provide to neon with alternate means if we use APIs that aren't in the Electron node version yet, but we could probably do that with feature flags.

@dherman
Copy link
Collaborator Author

dherman commented Jan 13, 2021

With the merging of #664 and #666, the last two items on the list are done, and the quest is officially complete! Congratulations and a huge thank you to everyone who contributed to making it happen!

200-5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

10 participants