Cross compiled Node.js v4.4.5 LTS for ARM7, but node threw illegal instruction error

[lijunqin]

• Version: v4.4.5
• Platform:Linux 4.1.12. Gitter chat room? #39 SMP Wed May 18 16:08:57 CDT 2016 armv7l GNU/Linux
• Subsystem: ->
  cat /proc/cpuinfo
  processor : 0
  model name : ARMv7 Processor rev 1 (v7l)
  BogoMIPS : 1594.16
  Features : half thumb fastmult edsp tls
  CPU implementer : 0x41
  CPU architecture: 7
  CPU variant : 0x4
  CPU part : 0xc09
  CPU revision : 1

processor : 1
model name : ARMv7 Processor rev 1 (v7l)
BogoMIPS : 1598.25
Features : half thumb fastmult edsp tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x4
CPU part : 0xc09
CPU revision : 1

Hardware : NPCMX50 Chip family
Revision : 0000
Serial : 0000000000000000

node gets illegal instructions
/usr/bin/node
Illegal instruction (core dumped)

The /proc/cpuinfo does not show any FPU.
Is that the cause of this issue ?
How can I cross compile V4.4.5 without FPU ?

AS=arm-poky-linux-gnueabi-as
LDFLAGS=-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed
AR=arm-poky-linux-gnueabi-ar
OECORE_TARGET_SYSROOT=/opt/poky/1.8/sysroots/armv7a-poky-linux-gnueabi
TARGET_PREFIX=arm-poky-linux-gnueabi-
XDG_SESSION_ID=7634
M4=m4
HOSTNAME=G9-dfvsmdev-CentOS7
TERM=xterm
SHELL=/bin/bash
NM=arm-poky-linux-gnueabi-nm
HISTSIZE=1000
OECORE_DISTRO_VERSION=1.8
CPPFLAGS=
SSH_CLIENT=10.238.17.91 53798 22
PERL5LIB=/home/dfvsmdev/perl5/lib/perl5:
QTDIR=/usr/lib64/qt-3.3
OLDPWD=/home/dfvsmdev/workspace/natasha2dell/delldrb
QTINC=/usr/lib64/qt-3.3/include
PERL_MB_OPT=--install_base /home/dfvsmdev/perl5
SSH_TTY=/dev/pts/1
QT_GRAPHICSSYSTEM_CHECKED=1
OECORE_SDK_VERSION=1.8
GDB=arm-poky-linux-gnueabi-gdb
USER=dfvsmdev
PKG_CONFIG_SYSROOT_DIR=/opt/poky/1.8/sysroots/armv7a-poky-linux-gnueabi
CXXFLAGS= -O2 -pipe -g -feliminate-unused-debug-types -mfloat-abi=soft
CONFIG_SITE=/opt/poky/1.8/site-config-armv7a-poky-linux-gnueabi
CPP=arm-poky-linux-gnueabi-gcc -E -march=armv7-a --sysroot=/opt/poky/1.8/sysroots/armv7a-poky-linux-gnueabi
CCACHE_PATH=/opt/poky/1.8/sysroots/x86_64-pokysdk-linux/usr/bin:/opt/poky/1.8/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi:
MAIL=/var/spool/mail/dfvsmdev
PATH=/usr/bin:/opt/poky/1.8/sysroots/x86_64-pokysdk-linux/usr/bin:/opt/poky/1.8/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi:/usr/lib64/qt-3.3/bin:/home/dfvsmdev/perl5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/dfvsmdev/.local/bin:/home/dfvsmdev/bin
LD=arm-poky-linux-gnueabi-ld --sysroot=/opt/poky/1.8/sysroots/armv7a-poky-linux-gnueabi
PWD=/home/dfvsmdev/workspace/natasha2dell/delldrb/node-v4.4.5
STRIP=arm-poky-linux-gnueabi-strip
LANG=en_US.UTF-8
MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles
LOADEDMODULES=
KDEDIRS=/usr
OECORE_NATIVE_SYSROOT=/opt/poky/1.8/sysroots/x86_64-pokysdk-linux
CONFIGURE_FLAGS=--target=arm-poky-linux-gnueabi --host=arm-poky-linux-gnueabi --build=x86_64-linux --with-libtool-sysroot=/opt/poky/1.8/sysroots/armv7a-poky-linux-gnueabi
KCFLAGS=--sysroot=/opt/poky/1.8/sysroots/armv7a-poky-linux-gnueabi
CXX=arm-poky-linux-gnueabi-g++ -march=armv7-a --sysroot=/opt/poky/1.8/sysroots/armv7a-poky-linux-gnueabi
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
HISTCONTROL=ignoredups
OBJCOPY=arm-poky-linux-gnueabi-objcopy
SHLVL=1
HOME=/home/dfvsmdev
CFLAGS= -O2 -pipe -g -feliminate-unused-debug-types -mfloat-abi=soft
PERL_LOCAL_LIB_ROOT=:/home/dfvsmdev/perl5
LOGNAME=dfvsmdev
QTLIB=/usr/lib64/qt-3.3/lib
SSH_CONNECTION=10.238.17.91 53798 10.238.25.57 22
OECORE_ACLOCAL_OPTS=-I /opt/poky/1.8/sysroots/x86_64-pokysdk-linux/usr/share/aclocal
MODULESHOME=/usr/share/Modules
PKG_CONFIG_PATH=/opt/poky/1.8/sysroots/armv7a-poky-linux-gnueabi/usr/lib/pkgconfig
LESSOPEN=||/usr/bin/lesspipe.sh %s
ARCH=arm
RANLIB=arm-poky-linux-gnueabi-ranlib
CROSS_COMPILE=arm-poky-linux-gnueabi-
CC=arm-poky-linux-gnueabi-gcc -march=armv7-a --sysroot=/opt/poky/1.8/sysroots/armv7a-poky-linux-gnueabi
XDG_RUNTIME_DIR=/run/user/1000
DISPLAY=localhost:11.0
QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
OBJDUMP=arm-poky-linux-gnueabi-objdump
SDKTARGETSYSROOT=/opt/poky/1.8/sysroots/armv7a-poky-linux-gnueabi
PERL_MM_OPT=INSTALL_BASE=/home/dfvsmdev/perl5
BASH_FUNC_module()=() { eval /usr/bin/modulecmd bash $*
}
_=/usr/bin/env

[dfvsmdev@G9-dfvsmdev-CentOS7 node-v4.4.5]$ ./configure --without-snapshot --dest-cpu=arm --dest-os=linux --with-arm-float-abi=soft --with-arm-fpu=vfp --without-dtrace --without-etw
creating ./icu_config.gypi
{ 'target_defaults': { 'cflags': [],
'default_configuration': 'Release',
'defines': [],
'include_dirs': [],
'libraries': []},
'variables': { 'arm_float_abi': 'soft',
'arm_fpu': 'vfp',
'arm_thumb': 0,
'arm_version': '7',
'asan': 0,
'gas_version': '2.24',
'host_arch': 'arm',
'icu_small': 'false',
'node_byteorder': 'little',
'node_install_npm': 'true',
'node_prefix': '/usr/local',
'node_release_urlbase': '',
'node_shared_http_parser': 'false',
'node_shared_libuv': 'false',
'node_shared_openssl': 'false',
'node_shared_zlib': 'false',
'node_tag': '',
'node_use_dtrace': 'false',
'node_use_etw': 'false',
'node_use_lttng': 'false',
'node_use_openssl': 'true',
'node_use_perfctr': 'false',
'openssl_fips': '',
'openssl_no_asm': 0,
'target_arch': 'arm',
'uv_parent_path': '/deps/uv/',
'uv_use_dtrace': 'false',
'v8_enable_gdbjit': 0,
'v8_enable_i18n_support': 0,
'v8_no_strict_aliasing': 1,
'v8_optimized_debug': 0,
'v8_random_seed': 0,
'v8_use_snapshot': 'false',
'want_separate_host_toolset': 0}}
creating ./config.gypi
creating ./config.mk

[bnoordhuis]

Can you try this?

$ ulimit -c unlimited  # turn on core dumps
$ node -e 0
$ gdb node core  # replace with actual name of core file
> disassemble $pc,$pc+32

Please post the output of the disassemble command.

[lijunqin]

Thank you so much for fast reply -
I am not sure if the issue is caused by FPU since my /proc/cpuinfo does not show any FPU
how can I configure without FPU ?
the kernel options:

Floating point emulation

At least one emulation must be selected

CONFIG_FPE_NWFPE is not set

CONFIG_FPE_FASTFPE is not set

CONFIG_VFP is not set

./configure --without-snapshot --dest-cpu=arm --dest-os=linux --with-arm-float-abi=soft --with-arm-fpu=vfp --without-dtrace --without-etw

/admin1-> ulimit -c unlimited
/admin1-> /tmp/node-test/node_test/node_v4.4.5_soft -e 0
Illegal instruction (core dumped)
/admin1-> ls
core.746 node_v4.4.5_soft
/admin1-> gdb /tmp/node-test/node_test/node_v4.4.5_soft core.746
GNU gdb (GDB) 7.8.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-poky-linux-gnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /tmp/node-test/node_test/node_v4.4.5_soft...done.

warning: core file may not match specified executable file.
[New LWP 746]
[New LWP 747]
[New LWP 750]
[New LWP 749]
[New LWP 748]

warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /lib/libthread_db-1.0.so
line to your configuration file "/home/root/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
---Type to continue, or q to quit---
Core was generated by `/tmp/node-test/node_test/node_v4.4.5_soft -e 0'.
Program terminated with signal SIGILL, Illegal instruction.

#0 0x4901e284 in ?? ()
(gdb)
(gdb)
(gdb) disassemble $pc,$pc+32
Dump of assembler code from 0x4901e284 to 0x4901e2a4:
=> 0x4901e284: vpush {d8-d15}
0x4901e288: mov r12, #0
0x4901e28c: vmov d14, r12, r12
0x4901e290: vmrs r4, fpscr
0x4901e294: tst r4, #33554432 ; 0x2000000
0x4901e298: bne 0x4901e2a4
0x4901e29c: orr r4, r4, #33554432 ; 0x2000000
0x4901e2a0: vmsr fpscr, r4
End of assembler dump.
(gdb)

[mscdex]

I believe v8 has been requiring VFP when compiling for ARM for some time now. You may be out of luck. It's actually kind of rare to find an armv7 CPU without VFP (and NEON) these days.

[bnoordhuis]

The offending instruction (vpush) is indeed a floating point instruction, so yes, looks like we've found the culprit.

Brian is correct that FPU support is mandatory now. You could try installing a kernel with FPU emulation support but it's probably going to be unbearably slow. YMMV. Good luck.

[lijunqin]

Hey Brian and Ben,
Do you know how to compile Node.js or V8 without requiring VFP ? I am running ARM7, according to the below threads, VFP emulation was removed from recent kernel.
http://www.phoronix.com/scan.php?page=news_item&px=MTM0ODA
https://lwn.net/Articles/546840/

[mscdex]

@lijunqin You cannot compile V8 without VFP. Support for compiling without VFP was removed years ago, and even then that support wasn't thoroughly tested IIRC.

[lijunqin]

Thank you so much for your time, Brian. Appreciate that -
As of now, the feasible way for me to use Node.js is to get it compiled bypasss VFP requirement -
if I tweak the source code of V8 to get it compiled without VFP requirement, any advice to for the starting point to modify the code ?
Thank you again,
Lijun

[mscdex]

@lijunqin "Tweaking" v8 as far as removing the VFP requirement is probably not as easy as you're making it out to be, unless maybe you're already a v8 expert. Even if you managed to accomplish this, it means you'd have to maintain a (presumably non-trivial) patch on top of v8 going forward. IMHO a more feasible solution is to just obtain better hardware if you want to run node on ARM.

[bnoordhuis]

@lijunqin I don't want to discourage you but you're up for a gargantuan task; it's not something you'll be able to accomplish in a few days, let alone hours.

You could try building node.js v0.10. I think that version is old enough that it nominally supports nofpu mode but, like Brian says, it was never well-tested so YMMV.

[lijunqin]

Big Thanks to Brian and Ben.
I am able to cross-compile Node.JS v0.10.x and run it on my ARMV7, but V0.10.x is close to end of life cycle ( Oct-2016). We just started a new 2-year project, so would like to try my best to use V4.x.
According to your advice,
1st option : get new HW with VFP support
2nd option: get VFP emulation in kernel working
3rd option: tweak source code of V8 to bypass VFP
3rd option is least favorite one -
Any other advice ?
Thank you again,
Lijun

[bnoordhuis]

Nope, that pretty much sums it up. IMO, the path of least resistance is to upgrade to VFP-capable hardware.

[lijunqin]

Hello Ben and Brian,
Upgrade to VFP-capable ARM7 is not an option for me. I am trying the FPU emulation right now.
I am using kernel 4.1.x for ARM7.
According to the statement at https://lwn.net/Articles/546840/ - removed the arch/arm/nwfpe code from the kernel, and the VFP code emulating the FP operations. But I am still seeing the options at kernel configuration file.
Could you please possibly offer some advice on FPU emulations ?
Can Node.js work with FPE_NWFPE or FPE_FASTFPE by specifying certain mfpu options ?

• At least one emulation must be selected

CONFIG_FPE_NWFPE is not set

CONFIG_FPE_FASTFPE is not set

CONFIG_VFP is not set

[bnoordhuis]

Could you please possibly offer some advice on FPU emulations ?

Sorry, that's a subject I don't know enough about to give informed advise. You may be able to get away with a user-space library that traps the SIGILL and emulates the instruction. I don't know of such a library for ARM, however.

Interesting side project: it's probably relatively straightforward to write a library that offloads the actual emulation to compiler-rt.

[lijunqin]

Hello Brian and Ben,
Thank you so much for your help. We might think of pursuing Node.JS V0.10.x According to LTS schedule at https://github.com/nodejs/LTS#lts_schedule. Node.JS v0.10.x is going to be LTS ended on
2016-10-1. Could you please possibly offer some advice on what support we could get due to end of life cycle if we stick to Node.JS V0.10.x ?
Your clarification on support of V0.10.X end of life will definitely help us determine the risk of pursuing V0.10.x
Thank you again,
Lijun QIN

[MylesBorins]

@lijunqin after the end of life date the project will not be offering support or security updates. one of the biggest factors here is that the openssl branch the 0.x series relies on will be end of life, and will not be receiving updates. Due to that, we cannot in good faith continue to release the 0.x series of Node.

TLDR; do not expect any support from the project after those dates

[lijunqin]

Hello Ben, Brian and Myles
Could you please possibly offer some more advice since our project tries all the possible ways to stick to Node.JS 4.x ?
Goal: Our ARMv7 does not have a FPU, but V8 requires VFP, we are seeking the possibilities to make V8 have options to bypass VFP.
#1 Could we possibly contribute funds to Node.JS development community for those changes ?
We are very much willing to pay the efforts and cost for this option. In fact, it might benefit Node,JS to grow user pool since still there are lots of users using ARM CPU without FPU out there.
Actually my company and my company's OEMs are big accounts.
#2 We also would like to seek the option to hire an V8 Engine expert from your development community to make the changes for us if possible.
if #1 is possible, could you please let me know the next step to proceed ?
if #1 is not possible, any advice on option #2 ?
Thank you again,
Lijun Qin

[bnoordhuis]

@lijunqin It's not completely out of the question but it will be an ongoing cost for your company. The upstream V8 project has no interest in maintaining nofpu support so it would have to be maintained in perpetuity by whoever steps up, possibly out of tree if upstream does not want to accept the changes.

If you're still interested, shoot me an email. My company (IBM) has the requisite expertise in-house, we maintain V8's s390 backend and co-maintain the ppc backend.

[lijunqin]

Hello Ben,
Thank you so much for your help. For the next step, I will send you message to your e-mail Ben Noordhuis info@bnoordhuis.nl. My company definitely interested in exploring this opportunity. We are willing to pay the cost to tweak V8 engine. Once you have a chance, please check your e-mail.
Thank you again,
Lijun Qin

[rench]

is there any one have way to do it?