build: Updates for AIX npm support - part 1 #3114

mhdawson · 2015-09-29T14:38:32Z

This PR is the first step enabling support for native modules
for AIX. The main issue is that unlike linux where all
symbols within the Node executable are available to the shared
library for a native module (npm), on AIX the symbols must
be explicitly exported. In addition, when the shared library is
built it must be linked using a list of the available symbols.

This patch covers the changes need to:

Export the symbols when building the node executable
Generate the file listing the symbols that can be used when
building the shared library.

For AIX, it breaks the build process into 2 steps. The first builds
a static library and then generates a node.exp file which contains
the symbols from that library. The second builds the node executable
and uses the node.exp file to specify which symbols should be
exported. In addition, it save the node.exp file so that it can
later be used in the creation of the shared library when building
a native module.

The following additional steps will be required in dependent projects
to fully enable AIX for native modules and are being worked
separately:

Updates to node-gyp to use node.exp when creating the
shared library for a native module
Fixes to gyp related to copying files as covered in
https://codereview.chromium.org/1368133002/patch/1/10001
Pulling in updated gyp versions to Node and node-gyp
Pulling in latest libuv

These changes were done to minimize the change to other platforms
by working within the existing structure to add the 2 step process
for AIX without changing the process for other platforms.

rvagg · 2015-09-29T14:50:10Z

tools/create_expfile.sh

+#!/bin/sh
+echo "Searching $1 to write out expfile to $2"
+echo "#!." > $2
+find $1 -name "*.a" |xargs nm -Xany -BCpg | awk '{ if ((($2 == "T") || ($2 == "D") || ($2 == "B")) && (substr($3,1,1) != ".")) {print $3} }'  | sort -u >> $2


could we clean this up a little, some missing spaces and it's quite long, maybe something like:

find $1 -name "*.a" \ | xargs nm -Xany -BCpg \ | awk '{ if ((($2 == "T") || ($2 == "D") || ($2 == "B")) && (substr($3,1,1) != ".")) { print $3 } }' \ | sort -u >> $2

Also perhaps a comment at the top of the file that this if for creating an AIX .exp file, just to give some contents to the next poor dev who goes fishing around in tools/ and stumbles on our crazy assortment of hacks

sure, once we settle any other issues I'll fix this up/

bnoordhuis · 2015-09-29T17:46:37Z

Would it make sense to split the current 'node' target in two: one that builds libnode.a, one that builds the binary? AIX-specific post-processing could then be done with less duplication.

Fishrock123 · 2015-09-30T03:26:19Z

Curious: does npm even work correctly on AIX?

S-YOU · 2015-09-30T03:38:23Z

configure

@@ -657,6 +657,10 @@ def configure_node(o):
  elif target_arch in ('mips', 'mipsel'):
    configure_mips(o)

+  if flavor in ('aix'):


Not sure what kind of python it is, but this should be if flavor in ('aix',):?

Very true. I would say change it to if flavor == 'aix'

I'd followed the style in the file just a few lines down, but given that we only expect one in the set the == 'aix' is probably better, which change

mhdawson · 2015-09-30T13:09:08Z

@Fishrock123 in response to the "does npm work for AIX" the answer is yes given the full set of changes we have in the IBM releases which is what I'm starting to contribute. There are some modules that use binaries which are platform specific but otherwise native modules can be installed/compiled.

mhdawson · 2015-09-30T13:13:50Z

@bnoordhuis the idea of modifying to split for all platforms makes sense to me but I did not want to presume we'd do this to accommodate AIX.

bnoordhuis · 2015-09-30T17:08:40Z

It seems cleaner than the current approach so I don't think anyone will stop you.

mhdawson · 2015-10-06T21:02:51Z

Thanks for all the good feedback, sorry I'm not had time to update yet but I hope to get to it in the next few days.

mhdawson · 2015-10-14T23:04:43Z

Just testing updates to address most of the comments will post tomorrow if there are no issues.

I've looked a bit at changing so all platforms build the lib and then the executable but I have a few concerns:

not sure how to keep the name of the static library the same as before as I think it is based on the target name (may just be my lack of familiarity with gyp)
there seem to be some platform specific options/configuration in the node target. I don't think we have tests that the static build case so I'm a little worried that the changes might break those platforms with a 2 step process and we don't have a good way to test. Similar issues might exist for AIX but since support for AIX is new it would not be a regression if the static build does not work.

@bnoordhuis any thoughts ?

bnoordhuis · 2015-10-16T09:16:26Z

there seem to be some platform specific options/configuration in the node target. I don't think we have tests that the static build case so I'm a little worried that the changes might break those platforms with a 2 step process and we don't have a good way to test.

The CI would catch that, wouldn't it? Maybe I misunderstand what you're worried about.

That said, while I think the two phase approach would be a little neater due to less duplication, it's not like your current approach is abjectly terrible. Go with whatever you're comfortable with.

mhdawson · 2015-10-16T21:40:35Z

Ok, at this point I think I'd like to stick with what we have instead of changing the other platforms. We want to make progress on nodejs/roadmap#9 so I think we can revisit as part of that.

I've pushed an update to address the comments and the tests pass my local CI run, @rvagg, @bnoordhuis, @S-YOU, @mscdex, @thefourtheye can you take another look.

S-YOU · 2015-10-17T00:08:24Z

tools/install.py

@@ -160,6 +160,10 @@ def headers(action):
    'src/node_version.h',
  ], 'include/node/')

+  # Add the expfile that is created on AIX
+  if 'aix' in sys.platform:


sorry again, but may be startswith like this is better than in?

Makes sense to me, updated

mhdawson · 2015-10-19T17:15:05Z

@S-YOU updated to reflect your comment.

bnoordhuis · 2015-10-20T22:22:19Z

tools/create_expfile.sh

+find $1 -name "*.a" \
+  | xargs nm -Xany -BCpg \
+  | awk '{
+      if ((($2 == "T") || ($2 == "D") || ($2 == "B")) && (substr($3,1,1) != ".")) { print $3 }


Style nit: can you break up this line?

bnoordhuis · 2015-10-20T22:22:47Z

LGTM with nit.

mhdawson · 2015-10-22T16:43:56Z

Updated to address Ben's comment and include one small change we found was needed as part of our internal testing. Given Ben's lgtm I'll plan to pull in, in the next day or so

mhdawson · 2015-10-23T13:31:37Z

CI run in advance of landing - https://ci.nodejs.org/job/node-test-pull-request/579/

This PR is the first step enabling support for native modules for AIX. The main issue is that unlike linux where all symbols within the Node executable are available to the shared library for a native module (npm), on AIX the symbols must be explicitly exported. In addition, when the shared library is built it must be linked using a list of the available symbols. This patch covers the changes need to: 1) Export the symbols when building the node executable 2) Generate the file listing the symbols that can be used when building the shared library. For AIX, it breaks the build process into 2 steps. The first builds a static library and then generates a node.exp file which contains the symbols from that library. The second builds the node executable and uses the node.exp file to specify which symbols should be exported. In addition, it save the node.exp file so that it can later be used in the creation of the shared library when building a native module. The following additional steps will be required in dependent projects to fully enable AIX for native modules and are being worked separately: - Updates to node-gyp to use node.exp when creating the shared library for a native module - Fixes to gyp related to copying files as covered in https://codereview.chromium.org/1368133002/patch/1/10001 - Pulling in updated gyp versions to Node and node-gyp - Pulling latest libuv These changes were done to minimize the change to other platforms by working within the existing structure to add the 2 step process for AIX without changing the process for other platforms.

mhdawson · 2015-10-23T14:37:19Z

CI does not look to have any related failures

Squashed down to 1 commit

This PR is the first step enabling support for native modules for AIX. The main issue is that unlike linux where all symbols within the Node executable are available to the shared library for a native module (npm), on AIX the symbols must be explicitly exported. In addition, when the shared library is built it must be linked using a list of the available symbols. This patch covers the changes need to: 1) Export the symbols when building the node executable 2) Generate the file listing the symbols that can be used when building the shared library. For AIX, it breaks the build process into 2 steps. The first builds a static library and then generates a node.exp file which contains the symbols from that library. The second builds the node executable and uses the node.exp file to specify which symbols should be exported. In addition, it save the node.exp file so that it can later be used in the creation of the shared library when building a native module. The following additional steps will be required in dependent projects to fully enable AIX for native modules and are being worked separately: - Updates to node-gyp to use node.exp when creating the shared library for a native module - Fixes to gyp related to copying files as covered in https://codereview.chromium.org/1368133002/patch/1/10001 - Pulling in updated gyp versions to Node and node-gyp - Pulling latest libuv These changes were done to minimize the change to other platforms by working within the existing structure to add the 2 step process for AIX without changing the process for other platforms. PR-URL: #3114 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

mhdawson · 2015-10-23T14:49:52Z

Landed as 15bcbf8

This PR is the first step enabling support for native modules for AIX. The main issue is that unlike linux where all symbols within the Node executable are available to the shared library for a native module (npm), on AIX the symbols must be explicitly exported. In addition, when the shared library is built it must be linked using a list of the available symbols. This patch covers the changes need to: 1) Export the symbols when building the node executable 2) Generate the file listing the symbols that can be used when building the shared library. For AIX, it breaks the build process into 2 steps. The first builds a static library and then generates a node.exp file which contains the symbols from that library. The second builds the node executable and uses the node.exp file to specify which symbols should be exported. In addition, it save the node.exp file so that it can later be used in the creation of the shared library when building a native module. The following additional steps will be required in dependent projects to fully enable AIX for native modules and are being worked separately: - Updates to node-gyp to use node.exp when creating the shared library for a native module - Fixes to gyp related to copying files as covered in https://codereview.chromium.org/1368133002/patch/1/10001 - Pulling in updated gyp versions to Node and node-gyp - Pulling latest libuv These changes were done to minimize the change to other platforms by working within the existing structure to add the 2 step process for AIX without changing the process for other platforms. PR-URL: #3114 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

rvagg reviewed Sep 29, 2015
View reviewed changes

mscdex added the build Issues and PRs related to build files or the CI. label Sep 29, 2015

S-YOU reviewed Sep 30, 2015
View reviewed changes

mhdawson mentioned this pull request Oct 2, 2015

Add support for AIX nodejs/node-gyp#753

Closed

S-YOU reviewed Oct 17, 2015
View reviewed changes

mhdawson self-assigned this Oct 20, 2015

bnoordhuis reviewed Oct 20, 2015
View reviewed changes

mhdawson force-pushed the aix2 branch from 696f842 to 3565bb5 Compare October 23, 2015 14:36

mhdawson closed this Oct 23, 2015

rvagg mentioned this pull request Oct 27, 2015

Release proposal: v5.0.0 #3466

Merged

rvagg mentioned this pull request Oct 28, 2015

Preparing v4.2.2 LTS Maintenance nodejs/Release#50

Closed

rvagg mentioned this pull request Dec 17, 2015

Release 4.2.4 Planning #4321

Closed

jasnell added the lts-watch-v4.x label Dec 17, 2015

MylesBorins added land-on-v4.x and removed lts-watch-v4.x labels Dec 29, 2015

MylesBorins mentioned this pull request Jan 19, 2016

V4.2.5 proposal #4768

Merged

mhdawson deleted the aix2 branch May 9, 2016 22:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: Updates for AIX npm support - part 1 #3114

build: Updates for AIX npm support - part 1 #3114

mhdawson commented Sep 29, 2015

rvagg Sep 29, 2015

mhdawson Sep 29, 2015

bnoordhuis commented Sep 29, 2015

Fishrock123 commented Sep 30, 2015

S-YOU Sep 30, 2015

thefourtheye Sep 30, 2015

mhdawson Sep 30, 2015

S-YOU Sep 30, 2015

mhdawson commented Sep 30, 2015

mhdawson commented Sep 30, 2015

bnoordhuis commented Sep 30, 2015

mhdawson commented Oct 6, 2015

mhdawson commented Oct 14, 2015

bnoordhuis commented Oct 16, 2015

mhdawson commented Oct 16, 2015

S-YOU Oct 17, 2015

mhdawson Oct 19, 2015

S-YOU Oct 20, 2015

mhdawson commented Oct 19, 2015

bnoordhuis Oct 20, 2015

bnoordhuis commented Oct 20, 2015

mhdawson commented Oct 22, 2015

mhdawson commented Oct 23, 2015

mhdawson commented Oct 23, 2015

mhdawson commented Oct 23, 2015

build: Updates for AIX npm support - part 1 #3114

build: Updates for AIX npm support - part 1 #3114

Conversation

mhdawson commented Sep 29, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnoordhuis commented Sep 29, 2015

Fishrock123 commented Sep 30, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhdawson commented Sep 30, 2015

mhdawson commented Sep 30, 2015

bnoordhuis commented Sep 30, 2015

mhdawson commented Oct 6, 2015

mhdawson commented Oct 14, 2015

bnoordhuis commented Oct 16, 2015

mhdawson commented Oct 16, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhdawson commented Oct 19, 2015

Choose a reason for hiding this comment

bnoordhuis commented Oct 20, 2015

mhdawson commented Oct 22, 2015

mhdawson commented Oct 23, 2015

mhdawson commented Oct 23, 2015

mhdawson commented Oct 23, 2015