Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support rebuild and build for cross-compiling Node-API module to wasm on Windows #2974

Merged
merged 15 commits into from
Jun 28, 2024

Conversation

toyobayashi
Copy link
Contributor

Checklist
  • npm install && npm run lint && npm test passes
  • commit message follows commit guidelines
Description of change

node-gyp rebuild and node-gyp build doesn't work on windows if using -f make to perform cross-compiling, this PR add support for it.

Related:

@cclauss
Copy link
Contributor

cclauss commented Mar 8, 2024

The lint failing here will be resolved by:

@toyobayashi
Copy link
Contributor Author

toyobayashi commented Apr 3, 2024

@cclauss Branch updated. Once nodejs/gyp-next#222 and nodejs/gyp-next#240 land in gyp-next 0.17.0 and this PR get merged, excited to see the next release of node-gyp can support build wasm!

@toyobayashi
Copy link
Contributor Author

gyp 0.17.0 released, anyone can review this PR?

lib/configure.js Outdated Show resolved Hide resolved
Copy link
Member

@legendecas legendecas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % a nit.

lib/build.js Outdated Show resolved Hide resolved
@legendecas
Copy link
Member

Also I think it will be helpful to add an integration test like https://github.com/nodejs/node-gyp/blob/main/test/test-addon.js#L45-L55.

@toyobayashi
Copy link
Contributor Author

toyobayashi commented Jun 6, 2024

But integration test for wasm requires latest gyp. Should I include the commit of update-gyp in this PR?

The test with this change is here https://github.com/toyobayashi/emnapi-node-gyp-test/actions/runs/9403090113

@toyobayashi
Copy link
Contributor Author

Or this PR should wait latest gyp get merged and then add the integration test?

@toyobayashi toyobayashi mentioned this pull request Jun 6, 2024
3 tasks
@legendecas
Copy link
Member

@toyobayashi would you mind adding integration tests to this PR? In this way, we can merge #3039 just for gyp updating first. Thanks!

@toyobayashi
Copy link
Contributor Author

@legendecas OK, cherry picked b4cad44

package.json Outdated
"bindings": "^1.5.0",
"cross-env": "^7.0.3",
"emnapi": "^1.2.0",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should use clang or gcc that is also available on github action https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md instead of introducing these dependencies.

Copy link
Contributor Author

@toyobayashi toyobayashi Jun 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you can see https://github.com/nodejs/node-gyp/actions/runs/9483655441, the make generator does not support native windows, I just do the minimum work in gyp repo to make it work for wasm (require emnapi's common.gypi and some sources code).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Short to say, to let the windows && make && rebuild test work, we need to build it to wasm (require emnapi as dependency)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My local native test shows: clang: error: unsupported option '-fPIC' for target 'x86_64-pc-windows-msvc'
gyp info it worked if it ends with ok
gyp verb cli [
gyp verb cli 'C:\\Users\\toyobayashi\\app\\nodejs\\node.exe',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\bin\\node-gyp.js',
gyp verb cli 'configure',
gyp verb cli '-C',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\',
gyp verb cli '--verbose',
gyp verb cli '--',
gyp verb cli '-f',
gyp verb cli 'make-linux'
gyp verb cli ]
gyp info using node-gyp@10.1.0
gyp info using node@16.18.1 | win32 | x64
gyp info chdir C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\
gyp verb download using dist-url https://npm.taobao.org/mirrors/node
gyp verb find Python Python is not set from command line or npm configuration
gyp verb find Python Python is not set from environment variable PYTHON
gyp verb find Python checking if the py launcher can be used to find Python 3
gyp verb find Python - executing "py.exe" to get Python 3 executable path
gyp verb find Python - executable path is "C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe"
gyp verb find Python - executing "C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe" to get version
gyp verb find Python - version is "3.10.6"
gyp info find Python using Python version 3.10.6 found at "C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe"
gyp verb get node dir no --target version specified, falling back to host node version: 16.18.1

gyp verb download using dist-url https://npm.taobao.org/mirrors/node
gyp verb install input version string "16.18.1"
gyp verb install installing version: 16.18.1
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version is already installed, need to check "installVersion"
gyp verb got "installVersion" 11
gyp verb needs "installVersion" 11
gyp verb install version is good
gyp verb on Windows; need to check node.lib
gyp verb get node dir target node version installed: 16.18.1
gyp verb build dir attempting to create "build" dir: C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\build
gyp verb build dir "build" dir needed to be created? No
gyp verb build/config.gypi creating config file
gyp verb build/config.gypi writing out config file: C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\build\config.gypi
gyp verb config.gypi checking for gypi file: C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\config.gypi
gyp verb common.gypi checking for gypi file: C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\common.gypi
gyp info spawn C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe
gyp info spawn args [
gyp info spawn args 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args 'binding.gyp',
gyp info spawn args '-f',
gyp info spawn args 'make-linux',
gyp info spawn args '-I',
gyp info spawn args 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\build\\config.gypi',
gyp info spawn args '-I',
gyp info spawn args 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\addon.gypi',
gyp info spawn args '-I',
gyp info spawn args 'C:\\Users\\toyobayashi\\AppData\\Local\\node-gyp\\Cache\\16.18.1\\include\\node\\common.gypi',
gyp info spawn args '-Dlibrary=shared_library',
gyp info spawn args '-Dvisibility=default',
gyp info spawn args '-Dnode_root_dir=C:\\Users\\toyobayashi\\AppData\\Local\\node-gyp\\Cache\\16.18.1',
gyp info spawn args '-Dnode_gyp_dir=C:\\Users\\toyobayashi\\Projects\\node-gyp',
gyp info spawn args '-Dnode_lib_file=C:\\\\Users\\\\toyobayashi\\\\AppData\\\\Local\\\\node-gyp\\\\Cache\\\\16.18.1\\\\<(target_arch)\\\\node.lib',
gyp info spawn args '-Dmodule_root_dir=C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi',
gyp info spawn args '-Dnode_engine=v8',
gyp info spawn args '--depth=.',
gyp info spawn args '--no-parallel',
gyp info spawn args '--generator-output',
gyp info spawn args 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\build',
gyp info spawn args '-Goutput_dir=.'
gyp info spawn args ]
gyp info ok
gyp info it worked if it ends with ok
gyp verb cli [
gyp verb cli 'C:\\Users\\toyobayashi\\app\\nodejs\\node.exe',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\bin\\node-gyp.js',
gyp verb cli 'build',
gyp verb cli '-C',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\',
gyp verb cli '--verbose'
gyp verb cli ]
gyp info using node-gyp@10.1.0
gyp info using node@16.18.1 | win32 | x64
gyp info chdir C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\
gyp verb build type Release
gyp verb architecture x64
gyp verb node dev dir C:\Users\toyobayashi\AppData\Local\node-gyp\Cache\16.18.1
gyp verb python C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe
gyp verb `which` succeeded for `make` C:\Users\toyobayashi\app\make\make.EXE
gyp info spawn make
gyp info spawn args [ 'V=1', 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory 'C:/Users/toyobayashi/Projects/node-gyp/test/node_modules/hello_napi/build'
  "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/Llvm/x64/bin/clang.exe" -o Release/obj.target/hello/hello.o ../hello.c '-DNODE_GYP_MODULE_NAME=hello' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_GLIBCXX_USE_CXX11_ABI=1' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DBUILDING_NODE_EXTENSION' -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/include/node -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/src -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/openssl/config -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/openssl/openssl/include -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/uv/include -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/zlib -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/v8/include  -fPIC -pthread -Wall -Wextra -Wno-unused-parameter -m64 -O3 -fno-omit-frame-pointer  -MMD -MF ./Release/.deps/Release/obj.target/hello/hello.o.d.raw   -c
clang: error: unsupported option '-fPIC' for target 'x86_64-pc-windows-msvc'
make: *** [hello.target.mk:111: Release/obj.target/hello/hello.o] Error 1
make: Leaving directory 'C:/Users/toyobayashi/Projects/node-gyp/test/node_modules/hello_napi/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (C:\Users\toyobayashi\Projects\node-gyp\lib\build.js:216:23)
gyp ERR! stack at ChildProcess.emit (node:events:513:28)
gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)
gyp ERR! System Windows_NT 10.0.22631
gyp ERR! command "C:\\Users\\toyobayashi\\app\\nodejs\\node.exe" "C:\\Users\\toyobayashi\\Projects\\node-gyp\\bin\\node-gyp.js" "build" "-C" "C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\" "--verbose"
gyp ERR! cwd C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi
gyp ERR! node -v v16.18.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok
After I remove -fPIC manually in makefile then output some link errors
gyp info it worked if it ends with ok
gyp verb cli [
gyp verb cli 'C:\\Users\\toyobayashi\\app\\nodejs\\node.exe',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\bin\\node-gyp.js',
gyp verb cli 'build',
gyp verb cli '-C',
gyp verb cli 'C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\',
gyp verb cli '--verbose'
gyp verb cli ]
gyp info using node-gyp@10.1.0
gyp info using node@16.18.1 | win32 | x64
gyp info chdir C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi\
gyp verb build type Release
gyp verb architecture x64
gyp verb node dev dir C:\Users\toyobayashi\AppData\Local\node-gyp\Cache\16.18.1
gyp verb python C:\Users\toyobayashi\AppData\Local\Programs\Python\Python310\python.exe
gyp verb `which` succeeded for `make` C:\Users\toyobayashi\app\make\make.EXE
gyp info spawn make
gyp info spawn args [ 'V=1', 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory 'C:/Users/toyobayashi/Projects/node-gyp/test/node_modules/hello_napi/build'
  "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/Llvm/x64/bin/clang.exe" -o Release/obj.target/hello/hello.o ../hello.c '-DNODE_GYP_MODULE_NAME=hello' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_GLIBCXX_USE_CXX11_ABI=1' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DBUILDING_NODE_EXTENSION' -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/include/node -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/src -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/openssl/config -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/openssl/openssl/include -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/uv/include -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/zlib -IC:/Users/toyobayashi/AppData/Local/node-gyp/Cache/16.18.1/deps/v8/include  -pthread -Wall -Wextra -Wno-unused-parameter -m64 -O3 -fno-omit-frame-pointer  -MMD -MF ./Release/.deps/Release/obj.target/hello/hello.o.d.raw   -c
  "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/Llvm/x64/bin/clang++.exe" -o Release/obj.target/hello.node -shared -pthread -rdynamic -m64  -Wl,-soname=hello.node -Wl,--start-group Release/obj.target/hello/hello.o -Wl,--end-group
clang++: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang++: warning: argument unused during compilation: '-rdynamic' [-Wunused-command-line-argument]
LINK : warning LNK4044: 无法识别的选项“/soname=hello.node”;已忽略
LINK : warning LNK4044: 无法识别的选项“/-start-group”;已忽略
LINK : warning LNK4044: 无法识别的选项“/-end-group”;已忽略
  正在创建库 Release/obj.target/hello.lib 和对象 Release/obj.target/hello.exp
hello.o : error LNK2019: 无法解析的外部符号 __imp_napi_module_register,函数 _register_hello 中引用了该符号
hello.o : error LNK2019: 无法解析的外部符号 __imp_napi_create_function,函数 napi_register_module_v1 中引用了该符号
hello.o : error LNK2019: 无法解析的外部符号 __imp_napi_set_named_property,函数 napi_register_module_v1 中引用了该符号
hello.o : error LNK2019: 无法解析的外部符号 __imp_napi_create_string_utf8,函数 hello 中引用了该符号
Release\obj.target\hello.node : fatal error LNK1120: 4 个无法解析的外部命令
clang++: error: linker command failed with exit code 1120 (use -v to see invocation)
make: *** [hello.target.mk:137: Release/obj.target/hello.node] Error 96
make: Leaving directory 'C:/Users/toyobayashi/Projects/node-gyp/test/node_modules/hello_napi/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (C:\Users\toyobayashi\Projects\node-gyp\lib\build.js:216:23)
gyp ERR! stack at ChildProcess.emit (node:events:513:28)
gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)
gyp ERR! System Windows_NT 10.0.22631
gyp ERR! command "C:\\Users\\toyobayashi\\app\\nodejs\\node.exe" "C:\\Users\\toyobayashi\\Projects\\node-gyp\\bin\\node-gyp.js" "build" "-C" "C:\\Users\\toyobayashi\\Projects\\node-gyp\\test\\node_modules\\hello_napi\\" "--verbose"
gyp ERR! cwd C:\Users\toyobayashi\Projects\node-gyp\test\node_modules\hello_napi
gyp ERR! node -v v16.18.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok

More effort need to be done to the gyp's make generator, it's out of scope. I personally don't care whether it work for native, it's enough that work for wasm.

Copy link
Member

@legendecas legendecas Jun 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since gyp can generate correct makefiles on windows now, could emnapi call gyp directly if it has it's own gypi and configurations?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think most of the config in https://github.com/nodejs/node-gyp/blob/main/addon.gypi would not be applicable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since gyp can generate correct makefiles on windows now, could emnapi call gyp directly if it has it's own gypi and configurations?

Yes, indeed can do this. However, all my efforts on gyp are ultimately aimed at making it easier for Node.js native module developers to use node-gyp to compile to both native and wasm at the same time, instead of introducing another gyp wrapper or something like additional way to reuse there binding.gyp configuration.

I added a minimal wasm gypi config to make the tests work, only targeting the change of this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you take another look at these changes, they are the final piece of the puzzle

@toyobayashi toyobayashi changed the title Support rebuild and build when configure with -- -f make on Windows Support rebuild and build for cross-compiling Node-API module to wasm on Windows Jun 12, 2024
}

if (ext === '.js') {
const emnapi = require('@emnapi/runtime')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this section be removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, it has been removed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants