Building as static library fails

[hoho]

This issue is probably not io.js-specific, and it might be the one to be reported to gyp, but I'll try here first.

I'm trying to build io.js as static library, I have just changed 'type': 'executable' to 'type': 'static_library' in node.gyp.

I get the following error in the end:

error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: unknown option character `W' in: -Wl,-force_load,/Users/hoho/iojs/out/Release/libopenssl.a
Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool -static [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-sacLT] [-no_warning_for_no_symbols]
Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-o output] [-install_name name] [-compatibility_version #] [-current_version #] [-seg1addr 0x#] [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table <filename>] [-seg_addr_table_filename <file_system_path>] [-all_load] [-noall_load]
make[1]: *** [/Users/hoho/iojs/out/Release/libiojs.a] Error 1
make: *** [iojs] Error 2

OS X Yosemite 10.10.2.

[Fishrock123]

Is this related to nodejs/roadmap#9 ?

I don't think this is currently possible.

[vinz243]

Why static and not shared? With shared I know it works.

See #537

For now, you can try patching node.gyp by hand: set the type of the 'iojs' target in node.gyp to 'shared_library' ('static_library' won't work) and drop src/node_main.cc from the sources

[hoho]

Yes, building shared library works and I'll be using it for a while. It's just shared library also requires -fPIC on Linux. And gyp is not passing it by default. And the result with -fPIC is probably not as fast.

[vinz243]

I've just seen this thread http://stackoverflow.com/q/17739095/2533082. Might help you

[Fishrock123]

Sounds like it was solved? Maybe we need a guide for this.

[hoho]

It was not solved! It is possible to build a shared library, but it is not possible to build a static one because of the error above.

[Fishrock123]

cc @bnoordhuis

(Sorry, my c++ knowledge is still pretty vague)

[hoho]

Just a few more notes.
I tried to build a static library on Linux (Ubuntu 14.04). The build process went fine, but the resulting libiojs.a is only 3 megabytes in size (that's obviously not everything) and when I tried to build something with it, I've got plenty of errors like:

string_bytes.cc:(.text._ZN4node11StringBytes5WriteEPN2v87IsolateEPcmNS1_6HandleINS1_5ValueEEENS_8encodingEPi+0x3b7): undefined reference to `v8::String::Value::~Value()'

Not just this reference, plenty of other ones too.

I tried to build a shared library on Linux too, I've got tons of other errors like:

/home/hoho/iojs/out/Release/obj.target/deps/v8/tools/gyp/libv8_base.a(debug-support.o):(.data.v8dbg_frametype_InternalFrame+0x0): multiple definition of `v8dbg_frametype_InternalFrame'
/home/hoho/iojs/out/Release/obj.target/deps/v8/tools/gyp/libv8_base.a(debug-support.o):(.data.v8dbg_frametype_InternalFrame+0x0): first defined here

Summarizing, from two platforms I'm interested in, I can only build working shared library on OSX and nothing on Linux.

I would appreciate any helpful workarounds (at least to have a shared library on both platforms).

[bnoordhuis]

@hoho You probably also need to link in the other static libraries that reside in out/Release (and specified in the right order if linking with --as-needed.)

I'm not sure why you get that error with the shared library. Maybe object files are getting linked in twice?

[hoho]

@bnoordhuis
as for the static library — I'll try to figure out what the right order is (definitely not alphabetic, because I tried it already).
as for the shared library — I'm not linking something specially and it works on OSX, here is my diff from the mainline (I checked out v1.2.0):

diff --git a/common.gypi b/common.gypi
index ea7779e..47484db 100644
--- a/common.gypi
+++ b/common.gypi
@@ -47,6 +47,7 @@
   },

   'target_defaults': {
+    'cflags': ['-fPIC'],
     'default_configuration': 'Release',
     'configurations': {
       'Debug': {
diff --git a/node.gyp b/node.gyp
index 8506b28..987df56 100644
--- a/node.gyp
+++ b/node.gyp
@@ -75,7 +75,7 @@
   'targets': [
     {
       'target_name': 'iojs',
-      'type': 'executable',
+      'type': 'shared_library',

       'dependencies': [
         'node_js2c#host',
@@ -102,7 +102,7 @@
         'src/node_file.cc',
         'src/node_http_parser.cc',
         'src/node_javascript.cc',
-        'src/node_main.cc',
+#        'src/node_main.cc',
         'src/node_os.cc',
         'src/node_v8.cc',
         'src/node_v8_platform.cc',

[bnoordhuis]

@hoho RE: linking order, if library A defines a symbol that library B needs, then B should come before A on the command line. E.g. libv8.a comes before libv8_base.a because the former uses symbols from the latter but not the other way around. You can also try linking with --no-as-needed

[hoho]

@bnoordhuis it looks like I've managed to link statically on Linux, it wasn't a trivial task though, that's what linker needs (in addition to everything else) to link with C (not C++) application on Linux (Ubuntu 14.04):

    -Wl,--whole-archive ../iojs/out/Release/libiojs.a -Wl,--no-whole-archive \
    ../iojs/out/Release/libuv.a \
    ../iojs/out/Release/libcares.a \
    ../iojs/out/Release/libopenssl.a \
    ../iojs/out/Release/libzlib.a \
    ../iojs/out/Release/libhttp_parser.a \
    ../iojs/out/Release/libv8_base.a \
    ../iojs/out/Release/libv8_libbase.a \
    ../iojs/out/Release/libv8_nosnapshot.a \
    -lstdc++ -lm -ldl

I haven't tested if the result is stable enough — just tried it with simple scripts, so I'll do further testing with many requires and native modules.

And, inspired by the elaboration above, I've managed to build static library on OSX too. It's a bit more tricky though. First of all, OTHER_LDFLAGS ('-Wl,-force_load,<(PRODUCT_DIR)/libopenssl.a' and '-Wl,-force_load,<(V8_BASE)') should be removed from node.gyp. Second of all, to link with the resulted static libraries (again, I link them into C application) the following should be added to the linker invocation

    -Wl,-force_load,../iojs/out/Release/libiojs.a \
    ../iojs/out/Release/libuv.a \
    ../iojs/out/Release/libcares.a \
    ../iojs/out/Release/libopenssl.a \
    ../iojs/out/Release/libzlib.a \
    ../iojs/out/Release/libhttp_parser.a \
    ../iojs/out/Release/libv8_base.a \
    ../iojs/out/Release/libv8_libbase.a \
    ../iojs/out/Release/libv8_nosnapshot.a \
    -lstdc++.6.0.9 -lm -ldl"

The funny thing is the necessity to explicitly add libstdc++ version because of some strangle linker bug (OSX has a symbolic link libstdc++.dylib to libstdc++.6.0.9.dylib, but the linker doesn't follow this symbolic link).

--whole-archive/-force_load is required to include native modules to the result. Otherwise everything dies at runtime with an exception like:

node.js:724
  var ContextifyScript = process.binding('contextify').ContextifyScript;
                                 ^
Error: No such module: contextify
    at Error (native)
    at node.js:724:34

Further testing might show that I need to force_load some other libraries too.

So, in the end, to fix this issue I might create a pull request to modify node.gyp the following way:

• Add a variable iojs_target_type with the default value equals to executable.
• Conditionally skip -force_loads in xcode_settings when iojs_target_type equals to static_library.

Optionally, I could describe the contents of this comment (about how to link with io.js as the static library) in some appropriate place for the future generations :). I don't know what place is the best though.

Is it ok? Does this pull request have a chance to be accepted?

And there still is an issue about errors while building shared library on Linux.

[bnoordhuis]

First of all, OTHER_LDFLAGS ('-Wl,-force_load,<(PRODUCT_DIR)/libopenssl.a' and '-Wl,-force_load,<(V8_BASE)') should be removed from node.gyp.

Can you explain why that is necessary? Was that the cause of the redefinition errors?

[hoho]

It's not about redefinition. Redefinition is a shared-library-on-linux bug. I did not dig into it.

The initial cause of this issue is in my first message:

unknown option character `W' in: -Wl,-force_load,/Users/hoho/iojs/out/Release/libopenssl.a

— it's because it's not supported to bundle one one static library into another — the result of building as a static library is actually a bunch of static libraries. -force_load should be removed in this case, it should be moved into the place where you link a binary with this bunch of static libraries.

[hoho]

By «should be removed» I meant conditionally removed. It's still required for the executable and for the shared library. But it's unsupported for the static library.

[hoho]

And back to the redefinition problem. That's how the link command looks like to build the executable:

flock /home/hoho/iojs/out/Release/linker.lock g++ -pthread -rdynamic -m64 -Wl,--whole-archive /home/hoho/iojs/out/Release/libopenssl.a -Wl,--no-whole-archive -Wl,-z,noexecstack -Wl,--whole-archive /home/hoho/iojs/out/Release/obj.target/deps/v8/tools/gyp/libv8_base.a -Wl,--no-whole-archive -pthread -o /home/hoho/iojs/out/Release/iojs -Wl,--start-group /home/hoho/iojs/out/Release/obj.target/iojs/src/debug-agent.o /home/hoho/iojs/out/Release/obj.target/iojs/src/async-wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/fs_event_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/cares_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/handle_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_buffer.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_constants.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_contextify.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_file.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_http_parser.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_javascript.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_main.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_os.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_v8.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_v8_platform.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_stat_watcher.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_watchdog.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_zlib.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_i18n.o /home/hoho/iojs/out/Release/obj.target/iojs/src/pipe_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/signal_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/smalloc.o /home/hoho/iojs/out/Release/obj.target/iojs/src/spawn_sync.o /home/hoho/iojs/out/Release/obj.target/iojs/src/string_bytes.o /home/hoho/iojs/out/Release/obj.target/iojs/src/stream_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/tcp_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/timer_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/tty_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/process_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/udp_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/uv.o /home/hoho/iojs/out/Release/obj.target/iojs/src/util.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_crypto.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_crypto_bio.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_crypto_clienthello.o /home/hoho/iojs/out/Release/obj.target/iojs/src/tls_wrap.o /home/hoho/iojs/out/Release/obj.target/deps/cares/libcares.a /home/hoho/iojs/out/Release/obj.target/deps/openssl/libopenssl.a /home/hoho/iojs/out/Release/obj.target/deps/zlib/libzlib.a /home/hoho/iojs/out/Release/obj.target/deps/http_parser/libhttp_parser.a /home/hoho/iojs/out/Release/obj.target/deps/uv/libuv.a /home/hoho/iojs/out/Release/obj.target/deps/v8/tools/gyp/libv8_base.a /home/hoho/iojs/out/Release/obj.target/deps/v8/tools/gyp/libv8_libbase.a /home/hoho/iojs/out/Release/obj.target/deps/v8/tools/gyp/libv8_nosnapshot.a -Wl,--end-group -lrt -lm -ldl

And that's the one for the shared library:

flock /home/hoho/iojs/out/Release/linker.lock g++ -shared -pthread -rdynamic -m64 -Wl,--whole-archive /home/hoho/iojs/out/Release/libopenssl.a -Wl,--no-whole-archive -Wl,-z,noexecstack -Wl,--whole-archive /home/hoho/iojs/out/Release/obj.target/deps/v8/tools/gyp/libv8_base.a -Wl,--no-whole-archive -pthread -Wl,-soname=libiojs.so -o /home/hoho/iojs/out/Release/obj.target/libiojs.so -Wl,--whole-archive /home/hoho/iojs/out/Release/obj.target/iojs/src/debug-agent.o /home/hoho/iojs/out/Release/obj.target/iojs/src/async-wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/fs_event_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/cares_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/handle_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_buffer.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_constants.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_contextify.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_file.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_http_parser.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_javascript.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_os.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_v8.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_v8_platform.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_stat_watcher.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_watchdog.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_zlib.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_i18n.o /home/hoho/iojs/out/Release/obj.target/iojs/src/pipe_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/signal_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/smalloc.o /home/hoho/iojs/out/Release/obj.target/iojs/src/spawn_sync.o /home/hoho/iojs/out/Release/obj.target/iojs/src/string_bytes.o /home/hoho/iojs/out/Release/obj.target/iojs/src/stream_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/tcp_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/timer_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/tty_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/process_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/udp_wrap.o /home/hoho/iojs/out/Release/obj.target/iojs/src/uv.o /home/hoho/iojs/out/Release/obj.target/iojs/src/util.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_crypto.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_crypto_bio.o /home/hoho/iojs/out/Release/obj.target/iojs/src/node_crypto_clienthello.o /home/hoho/iojs/out/Release/obj.target/iojs/src/tls_wrap.o /home/hoho/iojs/out/Release/obj.target/deps/cares/libcares.a /home/hoho/iojs/out/Release/obj.target/deps/openssl/libopenssl.a /home/hoho/iojs/out/Release/obj.target/deps/zlib/libzlib.a /home/hoho/iojs/out/Release/obj.target/deps/http_parser/libhttp_parser.a /home/hoho/iojs/out/Release/obj.target/deps/uv/libuv.a /home/hoho/iojs/out/Release/obj.target/deps/v8/tools/gyp/libv8_base.a /home/hoho/iojs/out/Release/obj.target/deps/v8/tools/gyp/libv8_libbase.a /home/hoho/iojs/out/Release/obj.target/deps/v8/tools/gyp/libv8_nosnapshot.a -Wl,--no-whole-archive -lrt -lm -ldl

The difference is that the executable is being linked with everything wrapped into -Wl,--start-group/-Wl,--end-group. And for the shared library it's -Wl,--whole-archive/-Wl,--no-whole-archive. This (in a combination with explicitly declared '-Wl,--whole-archive <(V8_BASE) -Wl,--no-whole-archive' and '-Wl,--whole-archive <(PRODUCT_DIR)/libopenssl.a -Wl,--no-whole-archive' from node.gyp) gives redefinitions.

Finally, I'm not sure I want to dig deep into the shared library problem right now (we could create another issue for it). But I'm definitely ready to make a pull request to fix the static library issue and make some docs about linking statically in some appropriate place.