Cannot run bootstrapped build on mipsel-linux

[The-King-of-Toasters]

Zig Version

0.14.0-dev.1078+0f0f543a9

Steps to Reproduce and Observed Behavior

In light of ziglang/zig-bootstrap#12 being closed, I wanted to try if I could run zig on my mipsel router. It is a D-Link DIR-853 A1 with a MediaTek MT7621 CPU. Unfortunately, running zig still gives an Illegal instruction error.

System Info:

root@Bridge:~# uname -a
Linux Bridge 5.15.134 #0 SMP Mon Oct 9 21:45:35 2023 mips GNU/Linux
root@Bridge:~# cat /proc/cpuinfo | awk '{print} NF==0{exit}'
system type             : MediaTek MT7621 ver:1 eco:3
machine                 : D-Link DIR-853 A1
processor               : 0
cpu model               : MIPS 1004Kc V2.15
BogoMIPS                : 586.13
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 32
extra interrupt vector  : yes
hardware watchpoint     : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
isa                     : mips1 mips2 mips32r1 mips32r2
ASEs implemented        : mips16 dsp mt
Options implemented     : tlb 4kex 4k_cache prefetch mcheck ejtag llsc pindexed_dcache userlocal vint perf_cntr_intr_bit cdmm perf
shadow register sets    : 1
kscratch registers      : 0
package                 : 0
core                    : 0
VPE                     : 0
VCED exceptions         : not available
VCEI exceptions         : not available

Steps:

1. Modify zig-bootstrap's build script to not strip symbols for the native zig.
2. Copy the result to some storage and mount it on the device.
3. Run zig on the device.

Backtrace via gdbserver:

Remote debugging using 192.168.20.14:5555
Reading /usr/lib/debug/.build-id/af/102d7c8f53b176d7e21b2c5c558c045281f0fe.debug from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.archlinux.org>
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
0x031c2eb0 in _start ()
Continuing.

Program received signal SIGILL, Illegal instruction.
0x03751fc0 in llvm::cl::opt<double, false, llvm::cl::parser<double> >::opt<char [24], llvm::cl::initializer<double>, llvm::cl::OptionHidden, llvm::cl::desc>(char const (&) [24], llvm::cl::initializer<double> const&, llvm::cl::OptionHidden const&, llvm::cl::desc const&) ()
#0  0x03751fc0 in llvm::cl::opt<double, false, llvm::cl::parser<double> >::opt<char [24], llvm::cl::initializer<double>, llvm::cl::OptionHidden, llvm::cl::desc>(char const (&) [24], llvm::cl::initializer<double> const&, llvm::cl::OptionHidden const&, llvm::cl::desc const&) ()
#1  0x0375c188 in _GLOBAL__sub_I_BlockFrequencyInfoImpl.cpp ()
#2  0x0adeb59c in libc_start_init () at /home/sg/zig-bootstrap/out/host/lib/zig/libc/musl/src/env/__libc_start_main.c:64
#3  0x0adeb540 in libc_start_main_stage2 (main=0x9e8cf78 <start.main>, argc=1, argv=0x7fffcd54) at /home/sg/zig-bootstrap/out/host/lib/zig/libc/musl/src/env/__libc_start_main.c:92
#4  0x0adeb4f4 in __libc_start_main (main=0xb15b848 <llvm::IterativeBFIPrecision>, argc=1, argv=0x7fffcd54, init_dummy=<optimized out>, fini_dummy=0xae109d0 <_fini>, ldso_dummy=0x0) at /home/sg/zig-bootstrap/out/host/lib/zig/libc/musl/src/env/__libc_start_main.c:86
#5  0x031c2f2c in _start_c (p=<optimized out>) at /home/sg/zig-bootstrap/out/host/lib/zig/libc/musl/crt/crt1.c:18
#6  0x031c2ef0 in _start ()
Backtrace stopped: frame did not save the PC
Dump of assembler code for function _ZN4llvm2cl3optIdLb0ENS0_6parserIdEEEC2IJA24_cNS0_11initializerIdEENS0_12OptionHiddenENS0_4descEEEEDpRKT_:
   0x03751e50 <+0>:	3c02079a	lui	v0,0x79a
   0x03751e54 <+4>:	2442b8b0	addiu	v0,v0,-18256
   0x03751e58 <+8>:	27bdffc8	addiu	sp,sp,-56
   0x03751e5c <+12>:	afbf0034	sw	ra,52(sp)
   0x03751e60 <+16>:	afb70030	sw	s7,48(sp)
   0x03751e64 <+20>:	afb6002c	sw	s6,44(sp)
   0x03751e68 <+24>:	afb50028	sw	s5,40(sp)
   0x03751e6c <+28>:	afb40024	sw	s4,36(sp)
   0x03751e70 <+32>:	afb30020	sw	s3,32(sp)
   0x03751e74 <+36>:	afb2001c	sw	s2,28(sp)
   0x03751e78 <+40>:	afb10018	sw	s1,24(sp)
   0x03751e7c <+44>:	afb00014	sw	s0,20(sp)
   0x03751e80 <+48>:	00598021	addu	s0,v0,t9
   0x03751e84 <+52>:	00e09025	move	s2,a3
   0x03751e88 <+56>:	00c09825	move	s3,a2
   0x03751e8c <+60>:	00a0a025	move	s4,a1
   0x03751e90 <+64>:	00808825	move	s1,a0
   0x03751e94 <+68>:	94810006	lhu	at,6(a0)
   0x03751e98 <+72>:	24170001	li	s7,1
   0x03751e9c <+76>:	8e02ad08	lw	v0,-21240(s0)
   0x03751ea0 <+80>:	30218000	andi	at,at,0x8000
   0x03751ea4 <+84>:	24950030	addiu	s5,a0,48
   0x03751ea8 <+88>:	24830048	addiu	v1,a0,72
   0x03751eac <+92>:	24420008	addiu	v0,v0,8
   0x03751eb0 <+96>:	8fb60048	lw	s6,72(sp)
   0x03751eb4 <+100>:	a4800004	sh	zero,4(a0)
   0x03751eb8 <+104>:	ac97003c	sw	s7,60(a0)
   0x03751ebc <+108>:	ac97002c	sw	s7,44(a0)
   0x03751ec0 <+112>:	ac820000	sw	v0,0(a0)
   0x03751ec4 <+116>:	ac830038	sw	v1,56(a0)
   0x03751ec8 <+120>:	ac830034	sw	v1,52(a0)
   0x03751ecc <+124>:	ac950024	sw	s5,36(a0)
   0x03751ed0 <+128>:	a4810006	sh	at,6(a0)
   0x03751ed4 <+132>:	ac80001c	sw	zero,28(a0)
   0x03751ed8 <+136>:	ac800014	sw	zero,20(a0)
   0x03751edc <+140>:	ac80000c	sw	zero,12(a0)
   0x03751ee0 <+144>:	ac800008	sw	zero,8(a0)
   0x03751ee4 <+148>:	ac800010	sw	zero,16(a0)
   0x03751ee8 <+152>:	ac800018	sw	zero,24(a0)
   0x03751eec <+156>:	ac800020	sw	zero,32(a0)
   0x03751ef0 <+160>:	ac800044	sw	zero,68(a0)
   0x03751ef4 <+164>:	ac800040	sw	zero,64(a0)
   0x03751ef8 <+168>:	ac800028	sw	zero,40(a0)
   0x03751efc <+172>:	8e19ad0c	lw	t9,-21236(s0)
   0x03751f00 <+176>:	0320f809	jalr	t9
   0x03751f04 <+180>:	0200e025	move	gp,s0
   0x03751f08 <+184>:	8e21002c	lw	at,44(s1)
   0x03751f0c <+188>:	8e230028	lw	v1,40(s1)
   0x03751f10 <+192>:	24660001	addiu	a2,v1,1
   0x03751f14 <+196>:	0026082b	sltu	at,at,a2
   0x03751f18 <+200>:	14200047	bnez	at,0x3752038 <_ZN4llvm2cl3optIdLb0ENS0_6parserIdEEEC2IJA24_cNS0_11initializerIdEENS0_12OptionHiddenENS0_4descEEEEDpRKT_+488>
   0x03751f1c <+204>:	00000000	nop
   0x03751f20 <+208>:	00030880	sll	at,v1,0x2
   0x03751f24 <+212>:	8e230024	lw	v1,36(s1)
   0x03751f28 <+216>:	00610821	addu	at,v1,at
   0x03751f2c <+220>:	a8220003	swl	v0,3(at)
   0x03751f30 <+224>:	b8220000	swr	v0,0(at)
   0x03751f34 <+228>:	8e011e68	lw	at,7784(s0)
   0x03751f38 <+232>:	8e021e6c	lw	v0,7788(s0)
   0x03751f3c <+236>:	8e031e70	lw	v1,7792(s0)
   0x03751f40 <+240>:	24630008	addiu	v1,v1,8
   0x03751f44 <+244>:	24420008	addiu	v0,v0,8
   0x03751f48 <+248>:	24210008	addiu	at,at,8
   0x03751f4c <+252>:	26240078	addiu	a0,s1,120
   0x03751f50 <+256>:	8e051e74	lw	a1,7796(s0)
   0x03751f54 <+260>:	24a50008	addiu	a1,a1,8
   0x03751f58 <+264>:	ae250058	sw	a1,88(s1)
   0x03751f5c <+268>:	ae240088	sw	a0,136(s1)
   0x03751f60 <+272>:	ae210078	sw	at,120(s1)
   0x03751f64 <+276>:	ae220070	sw	v0,112(s1)
   0x03751f68 <+280>:	ae230000	sw	v1,0(s1)
   0x03751f6c <+284>:	ae200054	sw	zero,84(s1)
   0x03751f70 <+288>:	ae20005c	sw	zero,92(s1)
   0x03751f74 <+292>:	ae200060	sw	zero,96(s1)
   0x03751f78 <+296>:	ae200064	sw	zero,100(s1)
   0x03751f7c <+300>:	ae200068	sw	zero,104(s1)
   0x03751f80 <+304>:	ae20006c	sw	zero,108(s1)
   0x03751f84 <+308>:	8e210028	lw	at,40(s1)
   0x03751f88 <+312>:	24210001	addiu	at,at,1
   0x03751f8c <+316>:	ae210028	sw	at,40(s1)
   0x03751f90 <+320>:	ae200050	sw	zero,80(s1)
   0x03751f94 <+324>:	8e19ab10	lw	t9,-21744(s0)
   0x03751f98 <+328>:	02802025	move	a0,s4
   0x03751f9c <+332>:	0320f809	jalr	t9
   0x03751fa0 <+336>:	0200e025	move	gp,s0
   0x03751fa4 <+340>:	00403025	move	a2,v0
   0x03751fa8 <+344>:	8e19ad28	lw	t9,-21208(s0)
   0x03751fac <+348>:	02202025	move	a0,s1
   0x03751fb0 <+352>:	02802825	move	a1,s4
   0x03751fb4 <+356>:	0320f809	jalr	t9
   0x03751fb8 <+360>:	0200e025	move	gp,s0
   0x03751fbc <+364>:	8e610000	lw	at,0(s3)
=> 0x03751fc0 <+368>:	d4200000	ldc1	$f0,0(at)
   0x03751fc4 <+372>:	a2370068	sb	s7,104(s1)
   0x03751fc8 <+376>:	f6200050	sdc1	$f0,80(s1)
   0x03751fcc <+380>:	f6200060	sdc1	$f0,96(s1)
   0x03751fd0 <+384>:	96210006	lhu	at,6(s1)
   0x03751fd4 <+388>:	3021ff9f	andi	at,at,0xff9f
   0x03751fd8 <+392>:	96420000	lhu	v0,0(s2)
   0x03751fdc <+396>:	00021140	sll	v0,v0,0x5
   0x03751fe0 <+400>:	30420060	andi	v0,v0,0x60
   0x03751fe4 <+404>:	00410825	or	at,v0,at
   0x03751fe8 <+408>:	a6210006	sh	at,6(s1)
   0x03751fec <+412>:	8ec10000	lw	at,0(s6)
   0x03751ff0 <+416>:	8ec20004	lw	v0,4(s6)
   0x03751ff4 <+420>:	ae220018	sw	v0,24(s1)
   0x03751ff8 <+424>:	ae210014	sw	at,20(s1)
   0x03751ffc <+428>:	8e19ad24	lw	t9,-21212(s0)
   0x03752000 <+432>:	02202025	move	a0,s1
   0x03752004 <+436>:	0320f809	jalr	t9
   0x03752008 <+440>:	0200e025	move	gp,s0
   0x0375200c <+444>:	8fb00014	lw	s0,20(sp)
   0x03752010 <+448>:	8fb10018	lw	s1,24(sp)
   0x03752014 <+452>:	8fb2001c	lw	s2,28(sp)
   0x03752018 <+456>:	8fb30020	lw	s3,32(sp)
   0x0375201c <+460>:	8fb40024	lw	s4,36(sp)
   0x03752020 <+464>:	8fb50028	lw	s5,40(sp)
   0x03752024 <+468>:	8fb6002c	lw	s6,44(sp)
   0x03752028 <+472>:	8fb70030	lw	s7,48(sp)
   0x0375202c <+476>:	8fbf0034	lw	ra,52(sp)
   0x03752030 <+480>:	03e00008	jr	ra
   0x03752034 <+484>:	27bd0038	addiu	sp,sp,56
   0x03752038 <+488>:	26240024	addiu	a0,s1,36
   0x0375203c <+492>:	8e19aacc	lw	t9,-21812(s0)
   0x03752040 <+496>:	02a02825	move	a1,s5
   0x03752044 <+500>:	24070004	li	a3,4
   0x03752048 <+504>:	0200e025	move	gp,s0
   0x0375204c <+508>:	0320f809	jalr	t9
   0x03752050 <+512>:	0040a825	move	s5,v0
   0x03752054 <+516>:	8e230028	lw	v1,40(s1)
   0x03752058 <+520>:	1000ffb1	b	0x3751f20 <_ZN4llvm2cl3optIdLb0ENS0_6parserIdEEEC2IJA24_cNS0_11initializerIdEENS0_12OptionHiddenENS0_4descEEEEDpRKT_+208>
   0x0375205c <+524>:	02a01025	move	v0,s5
End of assembler dump.
Continuing.

Program terminated with signal SIGILL, Illegal instruction.
The program no longer exists.

Expected Behavior

For zig to work.

[The-King-of-Toasters]

Ok, I see the problem: The system is compiled for soft-float. The loader for /bin/busybox is /lib/ld-musl-mipsel-sf.so.1.

[alexrp]

Yeah, in general, for targets where we have -gnueabi(hf) and there's just -musl, it means the latter is hard float, and soft float just isn't supported for musl in that case. I don't think there's any particular reason we can't add -musleabi(hf) variants for these - it just hasn't been done yet.

[alexrp]

I guess we can leave this open to track splitting -musl into -musleabi(hf) for these targets:

• mips-linux-musl
• mipsel-linux-musl
• powerpc-linux-musl

[The-King-of-Toasters]

Is there a way to hack around the current build system to force soft float anyway? I'd still like to try in the interim.

[alexrp]

Not really, I don't think. But I'll take a look at this soon.

[alexrp]

@The-King-of-Toasters it's not quite PR-ready yet, but if you want to give it a shot, you can try to pull my musl-hf-sf branch and do a bootstrap with it.

[The-King-of-Toasters]

So I copied over your branch onto zig-bootstrap, but building stage3 native just segfaults. Unfortunately I can't give a backtrack atm since the machine I'm on (cfarm420) doesn't deposit the corefiles.

[alexrp]

@The-King-of-Toasters just force-pushed a bunch more fixes; it now fully passes zig build test -fqemu on my end. Maybe you can give it another try?

[The-King-of-Toasters]

No dice. Stage2 building stage3 native still dies. I ran it in GDB and it says it's in InternPool_Index_unwrap__11099.

[alexrp]

1. Can you get a full backtrace? t a a bt f
2. Is there any way I can get access to a system that's similar enough to repro the crash?

[The-King-of-Toasters]

Actually, it could just be because I didn't clean the build dir 🙃. I'll let you know when it's done.

Update: yep.

[alexrp]

Update: yep.

Does that mean it's making more progress than before?

[The-King-of-Toasters]

Yes, the bootstrap compiled successfully and I'm running Zig on my router. It's not exactly a speed demon since it's taking 4s to print zig zen.

[alexrp]

Given the size of the zig binary, I'm tempted to say that a lot of that time might just literally be the process of loading it. 🤔 Might be interesting to investigate separately.

Thanks for testing; I'll have a PR up soon.

[The-King-of-Toasters]

Yes, it seems that a lot of the time is just mmap'ng and oddly writing to stdout. I can kinda build now:

After a while I get the same error found when building on power10:

error: unable to build zig's multitarget libc: UnableToWriteArchive
error: unable to build compiler_rt: UnableToWriteArchive

[alexrp]

That's an error bubbling up from LLVM's writeArchive() API.

If you're up for it, you can modify the call in src/zig_llvm.cpp to omit the last parameter so that messages go to standard error, rebuild the compiler, and see what's up.

[The-King-of-Toasters]

AFAICT the last parameter is already omitted, at least going by the API reference. I tried removing the nullptr anyway but it did nothing.

[alexrp]

Hmm, I see. I guess you'd have to step into LLVM and see if you can glean anything helpful.