Enable Node.js to run with Microsoft's ChakraCore engine #4765
Conversation
ChakraShim implements essential V8 APIs needed by Node.js on top of the Chakra runtime hosting API (JSRT). This enables Node.js and other native addon modules written for V8 to build and run with the Chakra JavaScript engine.
Source code of [chakracore](https://github.com/Microsoft/ChakraCore.git) that lights up Node.js for Chakra. Building Node.js with Chakra produces chakracore.dll along with other binaries that is needed by node.exe to function.
Fishrock123
added
vm
|
@kunalspathak Thanks for the PR! This has a very large impact so please expect review to take a considerable amount of time. :) |
|
VERY cool! Don't envy the code reviewers here though! |
|
This is by far one of the coolest things I've seen happen to Node.js since the convergence of io.js and Node.js. |
|
Legit. Thanks, cant wait until this happens 👍 |
|
The review will really take a lot of time and scrutiny! 🍴 |
|
+:one: guys, really good job! :boom: |
| @@ -10,6 +10,8 @@ | |||
| 'component%': 'static_library', # NB. these names match with what V8 expects | |||
| 'msvs_multi_core_compile': '0', # we do enable multicore compiles, but not using the V8 way | |||
| 'python%': 'python', | |||
| 'node_engine%': 'v8', | |||
There was a problem hiding this comment.
Nit: is node_engine established elsewhere? Maybe js_engine? This line and everywhere else for consistency
There was a problem hiding this comment.
We usually just prefix node stuff with node_ in here.
|
This is incredibly cool, I have a request. Please do not post comments on the GH issue unless you have something important to add. These issues gain a lot of attention and it makes it incredibly hard for collaborators to communicate. Locking the issue to collaborators means other people from the outside who have a significant ***** contribution or want to help can't do that. Comments like +1 -1 and such create a significant amount of noise. Support open source, keep the discussion clean. EDIT(@trevnorris): Possible accidental use of inappropriate word? Has been removed. |
|
@kunalspathak What happens when v8 needs to be updated and the API doesn't match the Chakra shim? Will we need to hold off until the shim also has an update? |
|
@kunalspathak ... thank you for submitting this. As you can imagine, it's going to be a big review and could take some time to settle out. @trevnorris ... I have separately reached out to each of the V8 and Chakra teams and invited both to sit down face to face to work through the API/ABI impact of this change and figure out how we can make the ABI layer more robust. I'm working out the logistics for that face to face now and want to make sure to extend the invite to all the @nodejs/ctc members as well. There are a ton of questions this brings up and I think sitting down for an afternoon to hash things out would be quite productive. |
jasnell
added
semver-major
| @@ -1101,6 +1112,10 @@ def configure_intl(o): | |||
| pprint.pformat(icu_config, indent=2) + '\n') | |||
| return # end of configure_intl | |||
|
|
|||
| def configure_engine(o): | |||
| engine = options.engine or 'v8' | |||
There was a problem hiding this comment.
Prefer function default parameter values def configure_engine(o = 'v8'):
There was a problem hiding this comment.
But it's not a function parameter.
There was a problem hiding this comment.
Then default options.engine at nodejs/node-chakracore@12cc5d9#diff-e2d5a00791bce9a01f99bc6fd613a39dR380
dest='engine',
+ default='v8',
help='Use specified JS engine (default is V8)')There was a problem hiding this comment.
@pritambaral , @benjamingr - Default is v8 engine as seen in vcbuild.bat
There was a problem hiding this comment.
@kunalspathak Yes, but the line commented-upon here is in configure and not vcbuild.bat. It would be better if the or-set in this line were replaced with the very first assignment of options.engine, which is what I think @benjamingr intended.
|
Marking it as semver-major for now (just to be safe), although semver-minor might be appropriate once we get further along in the review. Very happy to see this tho. |
| @@ -0,0 +1,54 @@ | |||
| ## Contributing Code | |||
There was a problem hiding this comment.
Is there any need for these files here?
Why isn't ChakraCore brought as a dependency?
There was a problem hiding this comment.
It's because this is the shim, the Core is brought in as a dep, but this is closely linked to node.
There was a problem hiding this comment.
Thats right. The sources in core folder are from Microsoft/ChakraCore. It is just easier to update the chakracore sources in node deps when they are aligned with their origin in github counterpart.
There was a problem hiding this comment.
Why not use a git submodule?
This follows the convention of existing deps -- source copy instead of submodule. It allows more flexibility such as making changes here before upstream when necessary (Not familiar with submodule, don't know if that's also supported).
There was a problem hiding this comment.
@panuhorsmalahti node currently does not use submodules for dependencies. It may be a good idea to document how dependencies are updated.
Currently for dependencies such as v8, libuv, or npm we take version updates as a single commit updating the state of the current directory, generally from a collaborator of the project we are downstream from. In between large updates we will sometimes backport changes as individual patch's.
There was a problem hiding this comment.
@jianchun: okay, I didn't know about that convention. I disagree with it, but I guess changing that convention is another discussion. To answer your question, if the upstream needs to be modified without changing e.g. the master branch, a new branch should be created upstream.
There was a problem hiding this comment.
@panuhorsmalahti git submodules don't allow the thirdparty source to be patched, or work with thirdparty deps that are not available via git.
|
Maybe this is a stupid question, but what is the advantage to this PR? To use Chakra instead of V8, right? Is that a runtime specification or a build specification? |
|
Am I correct that this currently works only on Windows platform? Initial Linux support is still on the roadmap of ChakraCore: https://github.com/Microsoft/ChakraCore/wiki/Roadmap. And that is decribed as «make it link, make it run, no JIT». |
|
Yes, its currently Windows only. For cross-platform support, the key target for next six months on the roadmap is to get the interpreter & runtime working. JIT would come after that (don't read it as no JIT forever - it was just a breakdown of what we need enable and its ordering for next 6 mos). |
|
Will ChakraCore be treated as a first-class citizen? That is:
|
|
@ChALkeR I think those questions can't be answered until after this PR lands. We have support in Node.js for architectures that we don't have in CI and pretty much all the ones we do have were added some time after support for the architecture was landed. I don't think anyone wants to add another vector to CI until we know there is some stability here and we're only going to figure that out after it lands. |
|
@ChALkeR ... at least in my opinion, it depends on what is meant by "first-class citizen"
2 and 3: I think it's too early to determine that. Given the dependency on the V8 APIs currently, and given how quickly those could change, there will need to be dedicated effort by Microsoft and other interested collaborators to ensure that the Chakra shim continues to work. It is entirely possible that semver-major update of V8 could land that could break these. Until we're sure that the shim is stable, we likely won't want to commit to ensuring that things would continue to work. +1 to what @mikeal just said as I was typing this :-) What I'd personally like to see is getting this landed but clearly indicating that it is experimental and unsupported for the time being. |
|
Yup, this is sort of a classic "let's put this behind a flag and see what happens" situation :) |
As a note, this is not a runtime feature, so it is explicitly behind a build-time flag. |
|
It seems like a nightmare to support. Imagine the following: someone make a PR that uses v8 API. CI breaks on Chakra build, because shim doesn't support said API. What we are going to do? wait until Chakra implements them? abandon the PR? What if next week someone PRs support for JSC or SpiderMonkey? |
|
And here is node-chakracore: https://github.com/nodejs/node-chakracore ! Hoping to get lots of contributions 😄 |
Testing this wasn't really useful, besides Object.observe is going to be deprecated. Also this test fails with Chakra (#4765) for obvious reason. PR-URL: #4769 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Roman Reiss <me@silverwind.io>
Testing this wasn't really useful, besides Object.observe is going to be deprecated. Also this test fails with Chakra (#4765) for obvious reason. PR-URL: #4769 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Roman Reiss <me@silverwind.io>
|
As @orangemocha mentioned these changes are ported to new repo https://github.com/nodejs/node-chakracore |
|
@orangemocha Does this still need |
|
Removed ctc-agenda. We'll ping the CTC periodically to provide updates and get feedback on the direction of the project. Thank you. |
Testing this wasn't really useful, besides Object.observe is going to be deprecated. Also this test fails with Chakra (#4765) for obvious reason. PR-URL: #4769 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Roman Reiss <me@silverwind.io>
|
Abstracting Node from the underlying engine sounds good, but we should avoid incompatibilities, otherwise we may start to see packages only working for a specific engine that will not be reusable in the other, so here comes the potential fragmentation. |
|
@mjolnic I agree, but I will note that Browserify/Webpack isn't helping that too much. I do feel that NAN might help alleviate much of that concern, though, but it'll have to go through a major version change to account for all the additional abstraction boilerplate it'll need. I'm also staring intently on WebAssembly, which will also be runnable on future versions of Node, with Chakra and V8 both supporting it, and that may have an even more significant impact on native addons (more than even supporting Chakra IMHO, because of the language difference). |
|
From Node's POV WebAssembly will look like an opaque blob of JS. You can call into it from JS, it can call out to JS, but the API surface is the same as an opaque JS blob. Eventually WebAssembly may gain access to extra APIs, but the MVP won't have anything special. If / when WebAssembly gains new APIs I think they should be designed with Node in mind, not just browsers. |
Testing this wasn't really useful, besides Object.observe is going to be deprecated. Also this test fails with Chakra (nodejs#4765) for obvious reason. PR-URL: nodejs#4769 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Roman Reiss <me@silverwind.io>
Microsoft's chakracore engine is dependent on Windows SDK, and build tools should know the version installed on user machine. This change adds those dependencies in node-gyp tools. Below is the summary: * Configure msvs_windows_target_platform_version to use the right Windows SDK. * Configure msvs_use_library_dependency_inputs to export symbols correctly (otherwise functions not used by node.exe but might be needed by native addon modules could be optimized away by linker). These changes were originally made in nodejs/node#4765, but as @shigeki mentioned, it was more sensible to send these changes as PR to node-gyp repo.
Microsoft's chakracore engine is dependent on Windows SDK, and build tools should know the version installed on user machine. This change adds those dependencies in node-gyp tools. Below is the summary: * Configure msvs_windows_target_platform_version to use the right Windows SDK. * Configure msvs_use_library_dependency_inputs to export symbols correctly (otherwise functions not used by node.exe but might be needed by native addon modules could be optimized away by linker). These changes were originally made in nodejs/node#4765, but as @shigeki mentioned, it was more sensible to send these changes as PR to node-gyp repo.
Microsoft's chakracore engine is dependent on Windows SDK, and build tools should know the version installed on user machine. This change adds those dependencies in node-gyp tools. Below is the summary: * Configure msvs_windows_target_platform_version to use the right Windows SDK. * Configure msvs_use_library_dependency_inputs to export symbols correctly (otherwise functions not used by node.exe but might be needed by native addon modules could be optimized away by linker). These changes were originally made in nodejs/node#4765, but as @shigeki mentioned, it was more sensible to send these changes as PR to node-gyp repo. PR-URL: #873 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
|
Just echoing @davidmurdoch, What are the benefits of adding this? this question is specially important when you consider the complexity that is introduced not only to the core of node.js but also the code that will be written against node, having to worry about the version of node.js is enough, don't need to compound that into a matrix of backends and versions. @kunalspathak Can you please give some context? |
(Note from the CTC (Fishrock123): This thread is expected to garner a lot of attention. Comments that are not productive to discussing the technical aspects may be removed.)
Enable Node.js to optionally build and run on Microsoft's ChakraCore JavaScript engine.
We implemented a V8 API shim (aka chakrashim) which takes advantage of ChakraCore runtime hosting APIs (JSRT). This shim implements most essential V8 APIs so that the underlying JavaScript engine changes are transparent to Node.js and native addon modules written for V8.
Here is the summary of commits :
1 node-gyp changes won't directly be submitted via PR, but will be first submitted to upstream branch as pointed out in nodejs/node-gyp#873. They are included in PR so reviewers can build and test Node.js with chakracore. Node-gyp changes are divided in two commits, first is based on changes needed to build node.js with chakracore for x86 and x64 for Windows OS and other based on changes need to support building it for ARM architecture for Windows OS.
2 gyp changes are also divided in two commits. One for x86/x64 and other for ARM.