illegal instruction on armv7 (udiv)

[dnr]

I'm trying to use rust to build something on an ARM board (cubox-i2):

$ cat /proc/cpuinfo 
processor       : 0
model name      : ARMv7 Processor rev 10 (v7l)
BogoMIPS        : 3.00
Features        : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x2
CPU part        : 0xc09
CPU revision    : 10

processor       : 1
model name      : ARMv7 Processor rev 10 (v7l)
BogoMIPS        : 3.00
Features        : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x2
CPU part        : 0xc09
CPU revision    : 10

Hardware        : Freescale i.MX6 Quad/DualLite (Device Tree)
Revision        : 0000
Serial          : 0000000000000000

I installed rust 1.25.0 (from the archlinuxarm repo) and get this:

$ usr/bin/rustc -V
rustc 1.25.0
$ usr/bin/rustc -v
Illegal instruction (core dumped)

More detail with gdb:

$ gdb --args usr/bin/rustc -v
GNU gdb (GDB) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "armv7l-unknown-linux-gnueabihf".
Type "show configuration" for configuration details.  
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from usr/bin/rustc...(no debugging symbols found)...done.
(gdb) r
Starting program: /tmp/r125/usr/bin/rustc -v
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x74fff1a0 (LWP 26265)]

Thread 2 "rustc" received signal SIGILL, Illegal instruction.
[Switching to Thread 0x74fff1a0 (LWP 26265)]
0x72bf8fa8 in cashew::IString::set(char const*, bool) ()
   from /tmp/r125/usr/lib/rustlib/armv7-unknown-linux-gnueabihf/codegen-backends/librustc_trans-llvm.so
(gdb) bt
#0  0x72bf8fa8 in cashew::IString::set(char const*, bool) ()
   from /tmp/r125/usr/lib/rustlib/armv7-unknown-linux-gnueabihf/codegen-backends/librustc_trans-llvm.so
#1  0x70d9feec in ?? ()
   from /tmp/r125/usr/lib/rustlib/armv7-unknown-linux-gnueabihf/codegen-backends/librustc_trans-llvm.so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) disassemble
Dump of assembler code for function _ZN6cashew7IString3setEPKcb:
   0x72bf8ef8 <+0>:     push    {r4, r5, r6, r7, r8, r9, r10, r11, lr}
   0x72bf8efc <+4>:     mov     r7, r0
   0x72bf8f00 <+8>:     ldr     r4, [pc, #908]  ; 0x72bf9294 <_ZN6cashew7IString3setEPKcb+924>
   0x72bf8f04 <+12>:    add     r11, sp, #32
   0x72bf8f08 <+16>:    ldr     r0, [pc, #904]  ; 0x72bf9298 <_ZN6cashew7IString3setEPKcb+928>
   0x72bf8f0c <+20>:    sub     sp, sp, #52     ; 0x34
   0x72bf8f10 <+24>:    add     r4, pc, r4
   0x72bf8f14 <+28>:    mov     r8, r2
   0x72bf8f18 <+32>:    str     r1, [r11, #-64] ; 0xffffffc0
   0x72bf8f1c <+36>:    ldr     r2, [r4, r0]
   0x72bf8f20 <+40>:    ldr     r3, [pc, #884]  ; 0x72bf929c <_ZN6cashew7IString3setEPKcb+932>
   0x72bf8f24 <+44>:    str     r2, [r11, #-72] ; 0xffffffb8
   0x72bf8f28 <+48>:    ldr     r2, [r2]
   0x72bf8f2c <+52>:    str     r2, [r11, #-40] ; 0xffffffd8
   0x72bf8f30 <+56>:    ldr     r3, [r4, r3]
   0x72bf8f34 <+60>:    cmp     r3, #0
   0x72bf8f38 <+64>:    str     r3, [r11, #-68] ; 0xffffffbc
   0x72bf8f3c <+68>:    beq     0x72bf8f54 <_ZN6cashew7IString3setEPKcb+92>
   0x72bf8f40 <+72>:    ldr     r3, [pc, #856]  ; 0x72bf92a0 <_ZN6cashew7IString3setEPKcb+936>
   0x72bf8f44 <+76>:    ldr     r0, [r4, r3]
   0x72bf8f48 <+80>:    bl      0x70cda270 <pthread_mutex_lock@plt>
   0x72bf8f4c <+84>:    cmp     r0, #0
   0x72bf8f50 <+88>:    bne     0x72bf923c <_ZN6cashew7IString3setEPKcb+836>
   0x72bf8f54 <+92>:    ldr     r3, [pc, #840]  ; 0x72bf92a4 <_ZN6cashew7IString3setEPKcb+940>
   0x72bf8f58 <+96>:    ldr     r5, [r4, r3]
   0x72bf8f5c <+100>:   ldrb    r3, [r5]
   0x72bf8f60 <+104>:   dmb     ish
   0x72bf8f64 <+108>:   ands    r6, r3, #1
   0x72bf8f68 <+112>:   beq     0x72bf9120 <_ZN6cashew7IString3setEPKcb+552>
   0x72bf8f6c <+116>:   ldr     r3, [pc, #820]  ; 0x72bf92a8 <_ZN6cashew7IString3setEPKcb+944>
   0x72bf8f70 <+120>:   ldr     r5, [r4, r3]
   0x72bf8f74 <+124>:   ldr     r3, [r11, #-64] ; 0xffffffc0
   0x72bf8f78 <+128>:   add     r1, r3, #1
   0x72bf8f7c <+132>:   ldrb    r2, [r3]
   0x72bf8f80 <+136>:   movw    r3, #5381       ; 0x1505
   0x72bf8f84 <+140>:   cmp     r2, #0
   0x72bf8f88 <+144>:   beq     0x72bf8fa0 <_ZN6cashew7IString3setEPKcb+168>
   0x72bf8f8c <+148>:   add     r3, r3, r3, lsl #5
   0x72bf8f90 <+152>:   eor     r3, r3, r2
   0x72bf8f94 <+156>:   ldrb    r2, [r1], #1
   0x72bf8f98 <+160>:   cmp     r2, #0
   0x72bf8f9c <+164>:   bne     0x72bf8f8c <_ZN6cashew7IString3setEPKcb+148>
   0x72bf8fa0 <+168>:   ldr     r1, [r5, #4]
   0x72bf8fa4 <+172>:   sub     r6, r11, #64    ; 0x40
=> 0x72bf8fa8 <+176>:   udiv    r0, r3, r1
   0x72bf8fac <+180>:   mov     r2, r6
   0x72bf8fb0 <+184>:   mls     r1, r1, r0, r3
   0x72bf8fb4 <+188>:   mov     r0, r5
   0x72bf8fb8 <+192>:   bl      0x70ccda90 <_ZNKSt10_HashtableIPKcS1_SaIS1_ENSt8__detail9_IdentityEN6cashew7IString12CStringEqualENS6_11CStringHashENS3_18_Mod_range_hashingENS3_20_Default_ranged_hashENS3_20_Prime_rehash_policyENS3_17_Hashtable_traitsILb1ELb1ELb1EEEE19_M_find_before_nodeEjRKS1_j@plt>
   0x72bf8fbc <+196>:   cmp     r0, #0
   0x72bf8fc0 <+200>:   beq     0x72bf9010 <_ZN6cashew7IString3setEPKcb+280>
   0x72bf8fc4 <+204>:   ldr     r3, [r0]
   0x72bf8fc8 <+208>:   cmp     r3, #0
   0x72bf8fcc <+212>:   beq     0x72bf9010 <_ZN6cashew7IString3setEPKcb+280>
   0x72bf8fd0 <+216>:   ldr     r3, [r3, #4]
   0x72bf8fd4 <+220>:   str     r3, [r11, #-64] ; 0xffffffc0
   0x72bf8fd8 <+224>:   ldr     r2, [r11, #-68] ; 0xffffffbc
   0x72bf8fdc <+228>:   str     r3, [r7]
   0x72bf8fe0 <+232>:   cmp     r2, #0
   0x72bf8fe4 <+236>:   beq     0x72bf8ff4 <_ZN6cashew7IString3setEPKcb+252>
...

Raw instruction:

=> 0x72bf8fa8 <+176>:   13 f1 30 e7     udiv    r0, r3, r1

I tried downgrading version by version to see if I could find one that works, and it turns out 1.22.1 works (and was able to build what I wanted to build) but 1.23.0 does not.

[nikomatsakis]

@dnr well that helps narrows it down! Would it be possible to use

https://github.com/rust-lang-nursery/cargo-bisect-rustc

to try and figure out which the nightly release in which this broke? That would be excellent. Alternatively, do you have any idea of the source?

Most likely, this is some kind of LLVM bug, so cc #50422

[nikomatsakis]

Marking as P-high so that we stay on top of this, but not assigning anyone as I don't have any immediate ideas for next steps. =)

[nagisa]

Support for the SDIV/UDIV instructions is mandatory in ARMv7-M and for the Thumb instruction set in ARMv7-R. It is optional for the ARM instruction set in ARMv7-R. It is optional in ARMv7-A and, if supported, may be in the Thumb instruction set only or in both Thumb and ARM. In ARMv7-A with the Virtualization Extensions, it is mandatory in Thumb and ARM.

The target definition for the armv7-unknown-linux-gnueabi (which is what you probably used) specifies the +thumb2, which is what allows compiler to generate the udiv instruction, as udiv is a required part of the thumb2 instruction set.

Alas, your core is a Cortex-A9 which does not support hardware divide in either Thumb or ARM mode (nor does it support the thumb2 instruction set), so you will have to make a target that does not specify the thumb2 feature and compile the compiler using that custom target. The fact that the older version of the compiler worked for you is a coincidence, as the thumb2 feature is required for this target since at least 2 years ago.

[nagisa]

Just to make myself clear, this is not a bug in the compiler, but rather an issue with how many different variants of ARM cores exist. This particular i.MX6 just happens to not fall within what we consider to be armv7.

[nikomatsakis]

@nagisa is there some way for @dnr to specify a more conservative target definition?

[nagisa]

Since they will want to cross-bootstrap their own compiler, adding a new in-tree target definition would probably be the easiest way (just copying the file I linked above and changing the list of features should suffice).

A PR with such target would likely be accepted but we most likely wouldn’t distribute prebuilt compilers using that target.

[nikomatsakis]

I see. It's not something you can do without doing a local build?

[nagisa]

Well if running rustc itself on i.MX6 is not the goal, then cross-compiling your programs with something like xargo and custom target jsons (documented in the readme of xargo) would be an easier solution.

[parched]

Alas, your core is a Cortex-A9 which does not support hardware divide in either Thumb or ARM mode (nor does it support the thumb2 instruction set), so you will have to make a target that does not specify the thumb2 feature and compile the compiler using that custom target. The fact that the older version of the compiler worked for you is a coincidence, as the thumb2 feature is required for this target since at least 2 years ago.

This isn't quite right, Thumb 2 is mandatory in Armv7-A, but hardware divide isn't mandatory in Thumb 2. The Cortex-A9 is still quite a common core and should definitely be supported by that target.

Is the udiv definitely in rust code not C/C++ (and could just be bad C compiler flags)? I searched for cashew but couldn't find it.

[nagisa]

This isn't quite right, Thumb 2 is mandatory in Armv7-A, but hardware divide isn't mandatory in Thumb 2.

Ah, my bad. I wasn’t wrong about ARM being hard to grok, though.

The instruction indeed appears in the C++ code (cashew is a namespace within binaryen/emscripten), which is something I didn’t notice before. We could definitely verify the target we pass through to compile llvm and other native libraries then.

That’s mostly a T-infra problem in that case.

[dnr]

To clarify, I didn't build rustc myself, I used prebuilt packages from the archlinuxarm project (archived at http://tardis.tiny-vps.com/aarm/ because the official repos only keep the latest version).

I didn't mention this at first, but I also tried the package built for armv6h and it failed in the same way. I don't have a core/disassembly from this right now but I can grab one tonight. Of course that would make sense if the problem is including C++ code from a mismatched target.

(I'm not actively developing this software on arm or otherwise, but if I do I'll definitely look into cross compilation.)

[nagisa]

@dnr can you please try rust installed via rustup?

[dnr]

First:

$ rustup update stable
info: syncing channel updates for 'stable-armv7-unknown-linux-gnueabihf'
info: latest update on 2018-05-10, rust version 1.26.0 (a77568041 2018-05-07)
error: component 'rust-docs' for 'armv7-unknown-linux-gnueabihf' is unavailable for download

I used rustup on my laptop to download nightly for armv7-unknown-linux-gnueabihf, and then copied to the ARM device. I can run rustc -v with no problems.

$ ./rustc -Vv
rustc 1.28.0-nightly (c2d46037f 2018-05-24)
binary: rustc
commit-hash: c2d46037fa6617408155dcba6251d7c4e123b4f5
commit-date: 2018-05-24
host: armv7-unknown-linux-gnueabihf
release: 1.28.0-nightly
LLVM version: 6.0

Disassembling the rustdoc and cargo binaries, I don't see any udiv instructions, so it's probably good.

Does this mean archlinuxarm is building their binaries with wrong flags or other settings, or that it's been fixed in recent versions?

[sanxiyn]

Reading the log, this does seem to be a problem with Arch Linux ARM, not Rust. Closing.