[regression] NodeOS don't boot with Node.js v0.11.15 (musl conflict?)

[piranna]

At shown at NodeOS/NodeOS#92 and found at NodeOS/NodeOS#92 (comment), NodeOS can be build using Node.js v0.11.14 and earlier but not with Node.js v0.11.15, released last week.

When using the latest release, everything compiles fine but when running it Linux boots correctly but Node.js doesn't start nor give you any error. Since NodeOS is a Linux based system the error should be reproducible, so maybe could be related with the usage of musl as libc.

[misterdjules]

I'm not familiar with NodeOS, do you think you'd have some bandwidth to find more details about why NodeOS can't run when it uses Node.js v0.11.15?

[piranna]

NodeOS is basically Node.js compiled against musl as libc running on top of
a Linux 3.18.3 kernel, that's why I said probably is a problem related to
using musl instead of glibc. Why do you means by "bandwidth"?
El 27/01/2015 01:44, "Julien Gilli" notifications@github.com escribió:

I'm not familiar with NodeOS, do you think you'd have some bandwidth to
find more details about why NodeOS can't run when it uses Node.js v0.11.15?

—
Reply to this email directly or view it on GitHub
#9097 (comment).

[misterdjules]

@piranna By "Do you have some bandwidth?" I meant "Do you have some time?", sorry for the confusion. Is it possible to have more details about why Node.js v0.11.15 wouldn't start, like an error message, or a trace output?

[piranna]

Sorry, it gives no output :-( I could try to run the compiled binary on my
Ubuntu machine, though...
El 27/01/2015 19:08, "Julien Gilli" notifications@github.com escribió:

@piranna https://github.com/piranna By "Do you have some bandwidth?" I
meant "Do you have some time?", sorry for the confusion. Is it possible to
have more details about why Node.js v0.11.15 wouldn't start, like an error
message, or a trace output?

—
Reply to this email directly or view it on GitHub
#9097 (comment).

[misterdjules]

@piranna Is the node binary fully statically linked? If not, then running it on Ubuntu would use Ubuntu's libc and libstdc++, and thus would not reproduce the issue, or am I missing something?

[piranna]

It's compiled agains musl, so it will not use system glibc, I need to
explicit tell him where is the libc. I can be able to compile it statically
for the test, tough.
El 27/01/2015 19:41, "Julien Gilli" notifications@github.com escribió:

@piranna https://github.com/piranna Is the node binary fully statically
linked? If not, then running it on Ubuntu would use Ubuntu's libc and
libstdc++, and thus would not reproduce the issue, or am I missing
something?

—
Reply to this email directly or view it on GitHub
#9097 (comment).

[misterdjules]

@piranna As long as you're able to make it load the same C and C++ runtimes on Ubuntu than on NodeOS, the test should be valid. It doesn't need to be statically linked.

[piranna]

I've just downloaded, compiled everything from scratch and tested v0.11.15 and it works on my Ubuntu system, but inside NodeOS it doesn't show the Node.js REPL prompt. Maybe it's a problem of using /dev/console as stdio?

On my Ubuntu machine, ldd outputs:

LD_LIBRARY_PATH=node_modules/nodeos-cross-toolchain/out/i686-nodeos-linux-musl/lib ldd out/i686/node 
    linux-gate.so.1 =>  (0xf77c1000)
    libstdc++.so.6 => node_modules/nodeos-cross-toolchain/out/i686-nodeos-linux-musl/lib/libstdc++.so.6 (0xf76da000)
    libgcc_s.so.1 => node_modules/nodeos-cross-toolchain/out/i686-nodeos-linux-musl/lib/libgcc_s.so.1 (0xf76bc000)
    libc.so => node_modules/nodeos-cross-toolchain/out/i686-nodeos-linux-musl/lib/libc.so (0xf763b000)

This is the same output of v0,11,14, that's the only one that works on NodeOS.

[piranna]

I've been doing a git bisect and found the problem is at

9116b240c924d37627313416b7ee038d0580afbc is the first bad commit
commit 9116b240c924d37627313416b7ee038d0580afbc
Author: Refael Ackermann <refack@gmail.com>
Date:   Mon Sep 29 13:20:04 2014 +0400

    deps: update v8 to 3.28.73

    Reviewed-By: Fedor Indutny <fedor@indutny.com>
    PR-URL: https://github.com/joyent/node/pull/8476

:040000 040000 5d0cea5095ae0db3d7ffcb4fc28a1652318b2e78 136f6922b1d4fd413defdacae1958fa12caceff3 M  deps

... and I think that the v8 version is something that would makes sense :-/

I've also found that 8a9f263 doesn't compile at all, too.

[misterdjules]

Thank you very much @piranna for taking the time to git bisect this issue! A few issues with the V8 version currently in v0.12 have been reported already, you might want to check if they apply to NodeOS.

Regardless, it would be great if we could have a better understanding of what code fails and how. Also, how come NodeOS works when run on Ubuntu and not standalone (if that question makes sense, I'm not familiar with NodeOS at all)?

[misterdjules]

@piranna Also, does NodeOS use features behind the --harmony flag?

[piranna]

Thank you very much @piranna for taking the time to git bisect this issue!

You are welcome :-) It was easy, just do a git bisect, clean the environment, left it compiling in background, check if it works and repeat, I haven't spend too much time on this :-D

A few issues with the V8 version currently in v0.12 have been reported already, you might want to check if they apply to NodeOS.

I'm not sure, because it's not showing the REPL prompt, so I'm not be able to check more things... :-/

Regardless, it would be great if we could have a better understanding of what code fails and how. Also, how come NodeOS works when run on Ubuntu and not standalone (if that question makes sense, I'm not familiar with NodeOS at all)?

NodeOS is just a Linux kernel having Node.js running directly on top of it as init/PID1, no more. I though it would be due to usage of musl instead of glibc, but as you said, it's working on Ubuntu, so it should be other thing related to the environment (using /dev/console as stdio? lack of environment variables?). You can test it yourself, just download the proyect, change to the node_modules/nodeos-barebones dir, and from there change the 'scripts/preinstall' file to force the Node.js version and exec npm install`` and laternpm start``` (you'll need to have installed 'qemu-system-x86' package to do it), so you'll be able to see it yourself.

[piranna]

@piranna Also, does NodeOS use features behind the --harmony flag?

No, NodeOS builds use standard compilation and no flags, and it's only OpenSSL patched to allow it to compile with musl (just change TERMIO for standard TERMIOS). I'm using v0.11.x because v0.10.x is not capable to compile with musl due to some preprocesson defines.

[piranna]

I've just checked with Node.js v0.11.16 released just today and the problem is still there. I've also noticed that CPU of QEmu process is somewhat high (although not as high as when in other tests I finished PID 1 process and Linux kernel panicked, that seems it enters on a spin lock by design), so probably this is related to one of the bugs you already pointed out.

[misterdjules]

@piranna #9113 is related to the use of --harmony features, so if NodeOS isn't using them, it seems unlikely that it's related to this issue.

Did you try to pass the --log_all flag to V8 and see what's in the log files? Also, I don't know what tools are available to debug NodeOS, or if it's possible to log output during the booting process, but it would be interesting to know at least if node starts and when approximately it stops moving forward by using a debugger or by outputting debug output.

[piranna]

@piranna #9113 is related to the use of --harmony features, so if NodeOS isn't using them, it seems unlikely that it's related to this issue.

Sorry for that, didn't understood correctly that issue. No, NodeOS is not using any flag at runtine, only standard features, and it's compiled using only this flags:

        --dest-cpu=$NODE_ARCH \
        --prefix=/            \
        --dest-os=linux       \
        --without-dtrace      \
        --without-npm         \
        --without-snapshot    || exit 11

Did you try to pass the --log_all flag to V8 and see what's in the log files?

Where do I need to set this flag?

Also, I don't know what tools are available to debug NodeOS

Not too much except printf()... :-/

or if it's possible to log output during the booting process

Currently is showing the Linux kernel boot messages, is this what you means?

but it would be interesting to know at least if node starts and when approximately it stops moving forward by using a debugger or by outputting debug output.

Linux kernel don't panic, so it seems Node.js can be able to start but becames unresposive. I can be able to start it with some command line arguments as if running on my Ubuntu machine (in the end, it's just a program... :-P ), so if you tell me what arguments to give it I can be able to see if there's some success, and maybe also I would be able to exec a Javascript file, too :-) Only problem would be if it's related with output data to console (I think no), so I couldn't be able to see anything and since it's an initramfs I will not be able to store the data on a file, too, but I would do a process.exit() or try to read keys to see if it's responsive...

[piranna]

I need to test it, but I believe nodejs/node#690 could be related to this.

[piranna]

Ok, I've check it and no, they are not related. I can be able to write on the console, but it doesn't get to the stdio of the Node.js executable, like when a console is dead.

[misterdjules]

@piranna I apologize for taking so long to get back to you. Could you please check if #9185 would fix the problem?

Thank you again for your help and support!

[piranna]

Sorry for the delay, my 64 bits machine hard disk died and I wanted to test on the same environment, but this will be delayed some more days. I'm going to test on my previous 32 bits machine, if it works it could work on the 64 bits when I recover it, but if not, then it will not work on the 64 bits one or there's a new bug related to the 32 bits environment, so at least it's not a useless test :-)

[piranna]

Ok, I've checked the commit ac21a53 on 32 bits and it's not working, so seems this bug is still there, sorry... :-(

[misterdjules]

@piranna Thank you again for your support!

The commit you would want to test though is d6249d0, and it requires the other commits here: https://github.com/joyent/node/pull/9185/commits.

ac21a53 in itself wouldn't solve the issue with NodeOS, as it only brings bug fixes and new features to mdb_v8, which is a library that helps mdb to inspect node processes.

Do you have some time to try these commits?

[piranna]

I though they were chained, so doing checkout of it would be enought... how
can I be able to test it?
El 25/02/2015 02:11, "Julien Gilli" notifications@github.com escribió:

@piranna https://github.com/piranna Thank you again for your support!

The commit you would want to test though is d6249d0
d6249d0,
and it requires the other commits here:
https://github.com/joyent/node/pull/9185/commits.

ac21a53
ac21a53
in itself wouldn't solve the issue with NodeOS, as it only brings bug fixes
and new features to mdb_v8, which is a library that helps mdb to inspect
node processes.

Do you have some time to try these commits?

—
Reply to this email directly or view it on GitHub
#9097 (comment).

[misterdjules]

@piranna Yes, if you did a git checkout ac21a5390c6e02f0bc9052c7794dcabc33a0a0c3 after fetching joyent/node, then you tested the right set of changes.

I had interpreted your previous message as if you had specifically cherry-picked ac21a53, sorry for the confusion, and thank you again for your help!

[piranna]

Any update on this? Seems v0.12.7 has still the problem... Doesn't have the v8 been updated in this time? What other things could I do? Maybe check with io.js? Would the build process be the same?

[piranna]

Could this has been fixed in Node.js 4.0.0? What's the status of these issues now?

[piranna]

Thanks to some work of @heavyk we've found that this problem is still happening on Node.js 4.1.1, so NodeOS needs to still use v0.11.14. Is there any updates on this, or it got abandoned?

[bnoordhuis]

@piranna I'm afraid it's not really something we can help you with unless you have very specific information on what's causing your project to fail.

I'm going to close this issue. If you have more information, please file a new one at https://github.com/nodejs/node/issues.