tfjs-node-gpu 0.2.3 fails to install on Windows

[iacovlev-pavel]

TensorFlow.js version

tensorflow/tfjs-node-gpu: 0.2.3

Browser version

node: v8.15.0

Describe the problem or feature request

npm i @tensorflow/tfjs-node-gpu results in an error.
0.2.2 also fails, while 0.2.1 works, this means the breaking change was added in 0.2.2.

If I download the .zip of 0.2.3 manually and then run node scripts/install.js it works.

Code to reproduce the bug / link to feature request

D:\tmp\tfjs_0_2_3>npm i @tensorflow/tfjs-node-gpu

> @tensorflow/tfjs-node-gpu@0.2.3 install D:\tmp\tfjs_0_2_3\node_modules\@tensorflow\tfjs-node-gpu
> node scripts/install.js gpu download

* Downloading libtensorflow
[==============================] 11080854/bps 100% 0.0s
* Building TensorFlow Node.js bindings
D:\tmp\tfjs_0_2_3\node_modules\@tensorflow\tfjs-node-gpu\scripts\install.js:171
      throw new Error('node-gyp rebuild failed with: ' + err);
      ^

Error: node-gyp rebuild failed with: Error: Command failed: node-gyp rebuild
gyp ERR! build error
gyp ERR! stack Error: `C:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Users\pavel\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\build.js:262:23)
gyp ERR! stack     at emitTwo (events.js:126:13)
gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\pavel\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd D:\tmp\tfjs_0_2_3\node_modules\@tensorflow\tfjs-node-gpu
gyp ERR! node -v v8.12.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok

    at cp.exec (D:\tmp\tfjs_0_2_3\node_modules\@tensorflow\tfjs-node-gpu\scripts\install.js:171:13)
    at ChildProcess.exithandler (child_process.js:283:5)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:915:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
npm WARN tfjs_0_2_3@1.0.0 No description
npm WARN tfjs_0_2_3@1.0.0 No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @tensorflow/tfjs-node-gpu@0.2.3 install: `node scripts/install.js gpu download`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @tensorflow/tfjs-node-gpu@0.2.3 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\pavel\AppData\Roaming\npm-cache\_logs\2019-01-23T14_41_47_959Z-debug.log

2019-01-23T14_41_47_959Z-debug.log

[kangyizhang]

Hi @iacovlev-pavel

Can you install windows-build-tools (https://github.com/tensorflow/tfjs-node/blob/master/WINDOWS_TROUBLESHOOTING.md) and try again?

[iacovlev-pavel]

Hello @kangyizhang

Fresh VM from https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/
Installed node v8.15.0
Ran npm --vs2015 install --global windows-build-tools
Ran npm i @tensorflow/tfjs-node-gpu
and got the same error.

I am doing 2 non standard things here: node 8 and vs2015. Will do additional tests and see if using the latest versions of these applications change anything.

[iacovlev-pavel]

Tried with Node 10 and vs 2017, got the same result.

[nkreeger]

The tool for building native Node.js plugins node-gyp currently has a variety of issues with Visual Studio 2017. I'd recommend following this issue for some solutions - nodejs/node-gyp#1056 - not a great option. I feel your pain because I have two Windows workstations and each one required a different solution.

[iacovlev-pavel]

@nkreeger Thank you for the reply, will look into the recommended issue. I also forgot to restart the VM after installing vs2015 and later vs2017, will give another go Tomorrow.

[iacovlev-pavel]

I still can't seem to install latest tfjs-node-gpu. Tried fresh VMs with both vs2015 and vs2017 and I always end up with the above mentioned error.

I looked at it a bit more Today, and here is the actual node-gyp error:

D:\tmp\tfjs-install\tensorflow-tfjs-node-gpu-0.2.3\package>if not defined npm_config_node_gyp (node "C:\Users\pavel\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Users\pavel\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
Warning: Missing input files:
D:\tmp\tfjs-install\tensorflow-tfjs-node-gpu-0.2.3\package\deps\include\tensorflow\c\eager\c_api.h
D:\tmp\tfjs-install\tensorflow-tfjs-node-gpu-0.2.3\package\deps\include\tensorflow\c\c_api.h
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  deps-stage
  (node:3480) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, rename 'D:\tmp\tfjs-install\tensorflow-tfjs-node-gpu-0.2.3\package\deps\lib\tensorflow.dll' -> 'D:\tmp\tfjs-install\tensorflow-tfjs-node-gpu-0.2.3\package\build\Release\tensorflow.dll'
  (node:3480) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
  (node:3480) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
  generate_def
  fs.js:646
    return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                   ^

CUSTOMBUILD : error : ENOENT: no such file or directory, open 'D:\tmp\tfjs-install\tensorflow-tfjs-node-gpu-0.2.3\package\deps\include\tensorflow\c\c_api.h' [D:\tmp\tfjs-install\tensorflow-tfjs-node-gpu-0.2.3\package\build\tfjs_binding.vcxproj]
      at Object.fs.openSync (fs.js:646:18)
      at Object.fs.readFileSync (fs.js:551:33)
      at files.map (D:\tmp\tfjs-install\tensorflow-tfjs-node-gpu-0.2.3\package\scripts\generate_defs.js:23:40)
      at Array.map (<anonymous>)
      at Object.<anonymous> (D:\tmp\tfjs-install\tensorflow-tfjs-node-gpu-0.2.3\package\scripts\generate_defs.js:23:23)
      at Module._compile (module.js:653:30)
      at Object.Module._extensions..js (module.js:664:10)
      at Module.load (module.js:566:32)
      at tryModuleLoad (module.js:506:12)
      at Function.Module._load (module.js:498:3)

I don't get a tensroflow folder in the deps directory, and the c_api.h is in the root of deps folder.

NOTE: The way I tested this I used: npm pack @tensorflow/tfjs-node-gpu to download the package locally, then modified the scripts/install.js to log the stdout output of node-gyp rebuild command.

[iacovlev-pavel]

More info:

This line checks if tensorflow.dll exist:
https://github.com/tensorflow/tfjs-node/blob/master/scripts/install.js#L186

it does not so it empties the deps directory:
https://github.com/tensorflow/tfjs-node/blob/master/scripts/install.js#L191

then it downloads:
https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-windows-x86_64-1.12.0.zip

Now the GPU zip has a different layout then CPU zip, and fails to install.
Here are the packages unzipped side by side.

[NoobTW]

Same here. Node v10, VS 2015.
Cannot install 0.2.2, 0.2.3, and 0.3.0 while 0.2.1 works.

[nkreeger]

Looks like some more Windows build fun - I had something similar and this fixed it for me:

Open an administrative prompt and run the following:

npm --add-python-to-path='true' --debug install --global windows-build-tools

Then try re-installing/yarn'ing. If that does not work - try running node-gyp configure --verbose from your node_modules or a clone of tfjs-node.

From a clone of tfjs-node you can enable gpu locally by running:

yarn enable-gpu

That will download the GPU binaries and re-compile. Let me know if this helps. More info here: tensorflow/tfjs-node#206

[nkreeger]

@iacovlev-pavel Oh OK sorry I missed the part about the GPU layout. I'll try and fix that this week. We switched to different hosted binaries for libtensorflow. I don't know why GPU Win64 builds are packaged differently...

[iacovlev-pavel]

@nkreeger Thank you

[nkreeger]

Fix in here tensorflow/tfjs-node#208

[nkreeger]

Please try with 0.3.1 (just published)

[NoobTW]

It's working. v0.3.1 on Windows 10.

[jeremy-kothe]

v0.3.1 contains the following, which surprisingly enough doesn't compile well...

copyModel: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/model_management").copyModel;
listModels: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/model_management").listModels;
moveModel: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/model_management").moveModel;
removeModel: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/model_management").removeModel;
browserFiles: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/browser_files").browserFiles;
browserHTTPRequest: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/browser_http").browserHTTPRequest;
concatenateArrayBuffers: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/io_utils").concatenateArrayBuffers;
decodeWeights: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/io_utils").decodeWeights;
encodeWeights: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/io_utils").encodeWeights;
fromMemory: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/passthrough").fromMemory;
getLoadHandlers: (url: string | string[], onProgress?: Function) => tf.io.IOHandler[];
getModelArtifactsInfoForJSON: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/io_utils").getModelArtifactsInfoForJSON;
getSaveHandlers: (url: string | string[]) => tf.io.IOHandler[];
isHTTPScheme: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/browser_http").isHTTPScheme;
loadWeights: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/weights_loader").loadWeights;
registerLoadRouter: (loudRouter: import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/router_registry").IORouter) => void;
registerSaveRouter: (loudRouter: import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/router_registry").IORouter) => void;
weightsLoaderFactory: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/weights_loader").weightsLoaderFactory;
withSaveHandler: typeof import("../../../../../Users/kreeger/workspace/tfjs-node/node_modules/@tensorflow/tfjs-core/dist/io/passthrough").withSaveHandler;

I'm guessing that's from your machine, mr kreeger?

[jeremy-kothe]

Aren't there tests for this? Something that tries to install and build from a windows machine?