Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SAMA5D3 and buildroot carsh with Illegal instruction with hardfloat #28906

Closed
dansmolik opened this issue Jul 31, 2019 · 21 comments
Closed

SAMA5D3 and buildroot carsh with Illegal instruction with hardfloat #28906

dansmolik opened this issue Jul 31, 2019 · 21 comments
Labels
arm Issues and PRs related to the ARM platform. build Issues and PRs related to build files or the CI.

Comments

@dansmolik
Copy link

  • v10.15.3:
  • Linux buildroot 4.19.56 armv7l:

Hello,
I found problem with nodejs on buildroot. When compile whole buildroot with softfloat nodejs works. But when enable EABIHF and VFP node always crash with Illegal instruction. All other programs work only node and npm crash. I can run node -v but simple node crash.

Dan

@dansmolik
Copy link
Author

dansmolik commented Jul 31, 2019

Hi,
I spend some time to get backtrace and there is the result:

Reading symbols from /usr/bin/node...BFD: error: /usr/bin/node(.debug_info) is too large (0x1bdcadc1 bytes)

warning: Can't read data for section '.debug_info' in file '/usr/bin/node'
(no debugging symbols found)...done.
[New LWP 645]
[New LWP 646]
[New LWP 650]
[New LWP 649]
[New LWP 648]
[New LWP 647]
Core was generated by `node'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x0073d39c in v8::internal::(anonymous namespace)::Flag::IsDefault() const
    ()
[Current thread is 1 (LWP 645)]

#0  0x0073d39c in v8::internal::(anonymous namespace)::Flag::IsDefault() const
    ()
#1  0x0073d6fc in v8::internal::ComputeFlagListHash() ()
#2  0x0092bc6c in v8::internal::V8::InitializeOncePerProcessImpl() ()
#3  0x009dff78 in v8::base::CallOnceImpl(int*, std::function<void ()>) ()
#4  0x0092bda0 in v8::internal::V8::InitializeOncePerProcess() ()
#5  0x0092bde0 in v8::internal::V8::Initialize() ()
#6  0x00518d58 in v8::V8::Initialize() ()
#7  0x0046b358 in node::Start(int, char**) ()
#8  0xb6bb05fe in __uClibc_main () from /lib//libc.so.0
#9  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

@bnoordhuis bnoordhuis added arm Issues and PRs related to the ARM platform. build Issues and PRs related to build files or the CI. labels Aug 1, 2019
@bnoordhuis
Copy link
Member

How did you build node? What options did you pass to configure and make?

@dansmolik
Copy link
Author

I use build root configured for SAMA5d3 and EABI hard float. I look to makefie and there is no special. VFP is detected and EABI and ARM too.

@bnoordhuis
Copy link
Member

Check ./configure --help, you probably need to pass some of the --with-arm-*= flags to the script.

@dansmolik
Copy link
Author

Yes I know. I test all possibilities . But still illegal instruction.

@bnoordhuis
Copy link
Member

But is it the same instruction every time? You can check with disassemble in gdb.

@dansmolik
Copy link
Author

OK I will test it. Thanks.

@dansmolik
Copy link
Author

dansmolik commented Aug 17, 2019

Dump of assembler code for function _ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv:
   0x0073dd4c <+0>:     push    {r11, lr}
   0x0073dd50 <+4>:     mov     r3, r0
   0x0073dd54 <+8>:     ldr     r2, [r0]
   0x0073dd58 <+12>:    add     r11, sp, #4
   0x0073dd5c <+16>:    cmp     r2, #8
   0x0073dd60 <+20>:    ldrls   pc, [pc, r2, lsl #2]
   0x0073dd64 <+24>:    b       0x73de48 <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+252>
   0x0073dd68 <+28>:    rsbseq  sp, r3, r12, lsl #27
   0x0073dd6c <+32>:    rsbseq  sp, r3, r8, lsr #27
   0x0073dd70 <+36>:    ldrhteq sp, [r3], #-216 ; 0xffffff28
   0x0073dd74 <+40>:    ldrhteq sp, [r3], #-216 ; 0xffffff28
   0x0073dd78 <+44>:    rsbseq  sp, r3, r8, asr #27
   0x0073dd7c <+48>:    rsbseq  sp, r3, r8, ror #27
   0x0073dd80 <+52>:    ldrhteq sp, [r3], #-216 ; 0xffffff28
   0x0073dd84 <+56>:    rsbseq  sp, r3, r8, lsl #28
   0x0073dd88 <+60>:    rsbseq  sp, r3, r12, lsr lr
   0x0073dd8c <+64>:    ldrd    r2, [r0, #8]
   0x0073dd90 <+68>:    ldrb    r0, [r2]
   0x0073dd94 <+72>:    ldrb    r3, [r3]
   0x0073dd98 <+76>:    sub     r0, r0, r3
--Type <RET> for more, q to quit, c to continue without paging--c
   0x0073dd9c <+80>:    clz     r0, r0
   0x0073dda0 <+84>:    lsr     r0, r0, #5
   0x0073dda4 <+88>:    pop     {r11, pc}
   0x0073dda8 <+92>:    ldr     r3, [r0, #8]
   0x0073ddac <+96>:    ldrb    r0, [r3]
   0x0073ddb0 <+100>:   eor     r0, r0, #1
   0x0073ddb4 <+104>:   pop     {r11, pc}
   0x0073ddb8 <+108>:   ldrd    r2, [r0, #8]
   0x0073ddbc <+112>:   ldr     r0, [r2]
   0x0073ddc0 <+116>:   ldr     r3, [r3]
   0x0073ddc4 <+120>:   b       0x73dd98 <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+76>
   0x0073ddc8 <+124>:   ldrd    r2, [r0, #8]
   0x0073ddcc <+128>:   ldrd    r0, [r2]
   0x0073ddd0 <+132>:   ldrd    r2, [r3]
   0x0073ddd4 <+136>:   cmp     r1, r3
   0x0073ddd8 <+140>:   cmpeq   r0, r2
   0x0073dddc <+144>:   moveq   r0, #1
   0x0073dde0 <+148>:   movne   r0, #0
   0x0073dde4 <+152>:   pop     {r11, pc}
   0x0073dde8 <+156>:   ldrd    r2, [r0, #8]
=> 0x0073ddec <+160>:   vldr    d17, [r2]
   0x0073ddf0 <+164>:   vldr    d16, [r3]
   0x0073ddf4 <+168>:   vcmp.f64        d17, d16
   0x0073ddf8 <+172>:   vmrs    APSR_nzcv, fpscr
   0x0073ddfc <+176>:   moveq   r0, #1
   0x0073de00 <+180>:   movne   r0, #0
   0x0073de04 <+184>:   pop     {r11, pc}
   0x0073de08 <+188>:   ldr     r3, [r3, #12]
   0x0073de0c <+192>:   ldr     r2, [r0, #8]
   0x0073de10 <+196>:   ldr     r1, [r3]
   0x0073de14 <+200>:   ldr     r0, [r2]
   0x0073de18 <+204>:   cmp     r1, #0
   0x0073de1c <+208>:   bne     0x73de2c <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+224>
   0x0073de20 <+212>:   clz     r0, r0
   0x0073de24 <+216>:   lsr     r0, r0, #5
   0x0073de28 <+220>:   pop     {r11, pc}
   0x0073de2c <+224>:   cmp     r0, #0
   0x0073de30 <+228>:   popeq   {r11, pc}
   0x0073de34 <+232>:   bl      0x44c434 <strcmp@plt>
   0x0073de38 <+236>:   b       0x73de20 <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+212>
   0x0073de3c <+240>:   ldr     r3, [r0, #8]
   0x0073de40 <+244>:   ldr     r0, [r3]
   0x0073de44 <+248>:   b       0x73de20 <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+212>
   0x0073de48 <+252>:   ldr     r2, [pc, #8]    ; 0x73de58 <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+268>
   0x0073de4c <+256>:   mov     r1, #0
   0x0073de50 <+260>:   ldr     r0, [pc, #4]    ; 0x73de5c <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+272>
   0x0073de54 <+264>:   bl      0x9e0720 <_Z8V8_FatalPKciS0_z>
   0x0073de58 <+268>:   sbcseq  r7, r0, r8, ror #22
   0x0073de5c <+272>:   ldrheq  r11, [r4], #27
End of assembler dump.

@dansmolik
Copy link
Author

There is output of disassembled core dump.

@dansmolik
Copy link
Author

dansmolik commented Aug 17, 2019

I found another interested thing. If I run node inside gdb I get much interested result. This CPU doesn't have NEON extension.

warning: Can't read data for section '.debug_info' in file '/usr/bin/node'
(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/node 

Program received signal SIGILL, Illegal instruction.
_armv7_neon_probe () at crypto/armv4cpuid.S:126
126     crypto/armv4cpuid.S: No such file or directory.
(gdb) bt
#0  _armv7_neon_probe () at crypto/armv4cpuid.S:126
#1  0xb6d560ac in OPENSSL_cpuid_setup () at crypto/armcap.c:179
#2  0xb6f5244a in _dl_run_array_forward (loadaddr=<optimized out>, 
    size=<optimized out>, array=<optimized out>) at ldso/ldso/dl-array.c:45
#3  _dl_run_init_array (tpnt=tpnt@entry=0xb6f64940) at ldso/ldso/dl-array.c:53
#4  0xb6f55178 in _dl_get_ready_to_run (tpnt=0xb6f64940, 
    tpnt@entry=0xbed4fce8, load_addr=load_addr@entry=3069513728, 
    auxvt=auxvt@entry=0xbed4fc70, envp=envp@entry=0xbed4fe2c, 
    argv=<optimized out>) at ldso/ldso/ldso.c:1425
#5  0xb6f558b6 in _dl_start (args=3201629728) at ldso/ldso/dl-startup.c:358
#6  0xb6f52016 in _start () from /lib/ld-uClibc.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) disassemble
Dump of assembler code for function _armv7_neon_probe:
=> 0xb6d5b6e0 <+0>:     vorr    q0, q0, q0
   0xb6d5b6e4 <+4>:     bx      lr
End of assembler dump.

@bnoordhuis
Copy link
Member

According to SAMA5D3's datasheet it supports VFPv4 so I have no idea why that vldr instruction would generate a SIGILL. I'm pretty sure it's this code:

double float_default() const {
DCHECK(type_ == TYPE_FLOAT);
return *reinterpret_cast<const double*>(defptr_);
}

My guess would be that it's an unaligned access, but AFAIK the Cortex-A5 supports unaligned accesses. You can check with print $r2 in gdb or info registers what the register contains.

That SIGILL in _armv7_neon_probe() is harmless, that's openssl's CPU feature detection code. It catches the signal and then continues. Just hit c<cr> a few times to get past it.

@dansmolik
Copy link
Author

dansmolik commented Aug 18, 2019

There it is:
(gdb) info registers
r0             0xdabea0            14335648
r1             0x0                 0
r2             0xdaddd0            14343632
r3             0xd40f40            13897536
r4             0x5c                92
r5             0xdabea0            14335648
r6             0x18                24
r7             0xdab600            14333440
r8             0x4                 4
r9             0xdc6880            14444672
r10            0x0                 0
r11            0xbefd8b04          3204287236
r12            0xdaaf9c            14331804
sp             0xbefd8b00          0xbefd8b00
lr             0x73e14c            7594316
pc             0x73ddec            0x73ddec <v8::internal::(anonymous namespace)::Flag::IsDefault() const+160>
cpsr           0x80000010          -2147483632
fpscr          0x60000000          1610612736

@bnoordhuis
Copy link
Member

bnoordhuis commented Aug 20, 2019

Assuming it's still crashing at that vldr d17, [r2], then r2 is properly aligned (on a 32 byte boundary.)

Is it possible the SAMA5D3 only supports VFP4-D16 mode? I.e., only the d0-d15 registers, not d16-d31?

Can you try this patch and do a rebuild with rm -rf out/ && ./configure --with-arm-fpu=vfpv4-d16 # whatever else you pass to configure?

diff --git a/configure.py b/configure.py
index beab9ceccc..b24b948926 100755
--- a/configure.py
+++ b/configure.py
@@ -48,7 +48,7 @@ valid_os = ('win', 'mac', 'solaris', 'freebsd', 'openbsd', 'linux',
 valid_arch = ('arm', 'arm64', 'ia32', 'mips', 'mipsel', 'mips64el', 'ppc',
               'ppc64', 'x32','x64', 'x86', 'x86_64', 's390x')
 valid_arm_float_abi = ('soft', 'softfp', 'hard')
-valid_arm_fpu = ('vfp', 'vfpv3', 'vfpv3-d16', 'neon')
+valid_arm_fpu = ('vfp', 'vfpv3', 'vfpv3-d16', 'vfpv4-d16', 'neon')
 valid_mips_arch = ('loongson', 'r1', 'r2', 'r6', 'rx')
 valid_mips_fpu = ('fp32', 'fp64', 'fpxx')
 valid_mips_float_abi = ('soft', 'hard')

@dansmolik
Copy link
Author

Ok I test it. I recompile whole buildroot with vfp4-d16 and apply patch above to nodejs.

@dansmolik
Copy link
Author

dansmolik commented Aug 20, 2019

There is output of /proc/cpuinfo

processor	: 0
model name	: ARMv7 Processor rev 1 (v7l)
BogoMIPS	: 351.43
Features	: half thumb fastmult vfp edsp vfpv3 vfpv3d16 tls vfpv4 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xc05
CPU revision	: 1

Hardware	: Atmel SAMA5
Revision	: 0000
Serial		: 0000000000000000

@bnoordhuis
Copy link
Member

I suppose you could always try vfpv3 or vfpv3-d16 if vfpv4-d16 doesn't work.

Now that I think of it, an explicit --with-arm-fpu=vfpv4 option probably wouldn't hurt either.

@dansmolik
Copy link
Author

I think that vfpv3 and vfpv3-d16 and vfpv4 I tested before. I use buildroot because I haven't success
with self crosscompiling nodejs with toolchain. It looks like that nodejs use some executables to build itself. But executables are build for arm not host amd64 and it doesn't work. Now I build
for vfpv4-d16.

@dansmolik
Copy link
Author

dansmolik commented Aug 20, 2019

Hm still the same.

Dump of assembler code for function _ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv:
   0x0073dd4c <+0>:     push    {r11, lr}
   0x0073dd50 <+4>:     mov     r3, r0
   0x0073dd54 <+8>:     ldr     r2, [r0]
   0x0073dd58 <+12>:    add     r11, sp, #4
   0x0073dd5c <+16>:    cmp     r2, #8
   0x0073dd60 <+20>:    ldrls   pc, [pc, r2, lsl #2]
   0x0073dd64 <+24>:    b       0x73de48 <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+252>
   0x0073dd68 <+28>:    rsbseq  sp, r3, r12, lsl #27
   0x0073dd6c <+32>:    rsbseq  sp, r3, r8, lsr #27
   0x0073dd70 <+36>:    ldrhteq sp, [r3], #-216 ; 0xffffff28
   0x0073dd74 <+40>:    ldrhteq sp, [r3], #-216 ; 0xffffff28
   0x0073dd78 <+44>:    rsbseq  sp, r3, r8, asr #27
   0x0073dd7c <+48>:    rsbseq  sp, r3, r8, ror #27
   0x0073dd80 <+52>:    ldrhteq sp, [r3], #-216 ; 0xffffff28
   0x0073dd84 <+56>:    rsbseq  sp, r3, r8, lsl #28
   0x0073dd88 <+60>:    rsbseq  sp, r3, r12, lsr lr
   0x0073dd8c <+64>:    ldrd    r2, [r0, #8]
   0x0073dd90 <+68>:    ldrb    r0, [r2]
   0x0073dd94 <+72>:    ldrb    r3, [r3]
   0x0073dd98 <+76>:    sub     r0, r0, r3
--Type <RET> for more, q to quit, c to continue without paging--c
   0x0073dd9c <+80>:    clz     r0, r0
   0x0073dda0 <+84>:    lsr     r0, r0, #5
   0x0073dda4 <+88>:    pop     {r11, pc}
   0x0073dda8 <+92>:    ldr     r3, [r0, #8]
   0x0073ddac <+96>:    ldrb    r0, [r3]
   0x0073ddb0 <+100>:   eor     r0, r0, #1
   0x0073ddb4 <+104>:   pop     {r11, pc}
   0x0073ddb8 <+108>:   ldrd    r2, [r0, #8]
   0x0073ddbc <+112>:   ldr     r0, [r2]
   0x0073ddc0 <+116>:   ldr     r3, [r3]
   0x0073ddc4 <+120>:   b       0x73dd98 <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+76>
   0x0073ddc8 <+124>:   ldrd    r2, [r0, #8]
   0x0073ddcc <+128>:   ldrd    r0, [r2]
   0x0073ddd0 <+132>:   ldrd    r2, [r3]
   0x0073ddd4 <+136>:   cmp     r1, r3
   0x0073ddd8 <+140>:   cmpeq   r0, r2
   0x0073dddc <+144>:   moveq   r0, #1
   0x0073dde0 <+148>:   movne   r0, #0
   0x0073dde4 <+152>:   pop     {r11, pc}
   0x0073dde8 <+156>:   ldrd    r2, [r0, #8]
=> 0x0073ddec <+160>:   vldr    d17, [r2]
   0x0073ddf0 <+164>:   vldr    d16, [r3]
   0x0073ddf4 <+168>:   vcmp.f64        d17, d16
   0x0073ddf8 <+172>:   vmrs    APSR_nzcv, fpscr
   0x0073ddfc <+176>:   moveq   r0, #1
   0x0073de00 <+180>:   movne   r0, #0
   0x0073de04 <+184>:   pop     {r11, pc}
   0x0073de08 <+188>:   ldr     r3, [r3, #12]
   0x0073de0c <+192>:   ldr     r2, [r0, #8]
   0x0073de10 <+196>:   ldr     r1, [r3]
   0x0073de14 <+200>:   ldr     r0, [r2]
   0x0073de18 <+204>:   cmp     r1, #0
   0x0073de1c <+208>:   bne     0x73de2c <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+224>
   0x0073de20 <+212>:   clz     r0, r0
   0x0073de24 <+216>:   lsr     r0, r0, #5
   0x0073de28 <+220>:   pop     {r11, pc}
   0x0073de2c <+224>:   cmp     r0, #0
   0x0073de30 <+228>:   popeq   {r11, pc}
   0x0073de34 <+232>:   bl      0x44c434 <strcmp@plt>
   0x0073de38 <+236>:   b       0x73de20 <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+212>
   0x0073de3c <+240>:   ldr     r3, [r0, #8]
   0x0073de40 <+244>:   ldr     r0, [r3]
   0x0073de44 <+248>:   b       0x73de20 <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+212>
   0x0073de48 <+252>:   ldr     r2, [pc, #8]    ; 0x73de58 <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+268>
   0x0073de4c <+256>:   mov     r1, #0
   0x0073de50 <+260>:   ldr     r0, [pc, #4]    ; 0x73de5c <_ZNK2v88internal12_GLOBAL__N_14Flag9IsDefaultEv+272>
   0x0073de54 <+264>:   bl      0x9e0720 <_Z8V8_FatalPKciS0_z>
   0x0073de58 <+268>:   sbcseq  r7, r0, r8, ror #22
   0x0073de5c <+272>:   ldrheq  r11, [r4], #27
End of assembler dump.

@dansmolik
Copy link
Author

But as you say crash in vldr d17 and can be there only register number max 15 ? I look to configure.py and patch is applied. I don't understand this.

@bnoordhuis
Copy link
Member

I've run out of ideas as well. Let me know if you find something.

@dansmolik
Copy link
Author

Hi, victory :-)
I compile nodejs out of buildroot (but use buildroot crosscompiler.
./configure --prefix=../install --dest-cpu=arm --cross-compiling --dest-os=linux --with-arm-float-abi=hard --with-arm-fpu=vfpv3-d16
with this params. And all woks no illegal instruction :-). It looks like that in buidroot is bug when compiling nodejs and isn't propagate correct settings of fpu. I try fill bug report to buidroot.
Thanks for your help.
Regards Dan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arm Issues and PRs related to the ARM platform. build Issues and PRs related to build files or the CI.
Projects
None yet
Development

No branches or pull requests

2 participants