Fix C++ exceptions / threads

[devnoname120]

Throwing an exception on the Vita and catching it crashes the program.

Temporary workaround by @DaveeFTW:

Add this before main():

extern "C"
{
    unsigned int sleep(unsigned int seconds)
    {
        sceKernelDelayThread(seconds*1000*1000);
        return 0;
    }

    int usleep(useconds_t usec)
    {
        sceKernelDelayThread(usec);
        return 0;
    }

    void __sinit(struct _reent *);
}

__attribute__((constructor(101)))
void pthread_setup(void) 
{
    pthread_init();
    __sinit(_REENT);
}

Add the following to target_link_libraries():

-Wl,--whole-archive pthread -Wl,--no-whole-archive

[blastrock]

I ran into a strange issue regarding C++ exceptions. I am leaving this here in case it helps someone...

In a moderately sized app (~7k SLOC), I didn't have any issue with exceptions for quite some time. Then I added a simple code that would throw and catch an exception at the beginning of main(). That throw would make the app crash. Then by commenting out a line in an unrelated file, the app would run fine.

The crash happens in __gnu_Unwind_RaiseException, when a custom personality function is called to unwind the stack:

00102ad6 <__gnu_Unwind_RaiseException+0x26>:
  102ad6:	000f      	movs	r7, r1
  102ad8:	f8cd c000 	str.w	ip, [sp]
  102adc:	e006      	b.n	102aec <__gnu_Unwind_RaiseException+0x3c>
  102ade:	6933      	ldr	r3, [r6, #16]
  102ae0:	466a      	mov	r2, sp
  102ae2:	4631      	mov	r1, r6
  102ae4:	4798      	blx	r3
!!! 		2808      	cmp	r0, #8 !!!
  102ae8:	4604      	mov	r4, r0
  102aea:	d108      	bne.n	102afe <__gnu_Unwind_RaiseException+0x4e>
  102aec:	4630      	mov	r0, r6
  102aee:	9910      	ldr	r1, [sp, #64]	; 0x40
  102af0:	f7ff feca 	bl	102888 <get_eit_entry>
  102af4:	2800      	cmp	r0, #0

The real issue is at the instruction before, the blx, r3 points to an address too far. The unwind table (the .ARM.exidx section) seems broken, though I didn't confirm it. I tried to readelf the velf (the output of vita-elf-create) to dump and diff the unwind tables of the working app and the crashing one, but they are exactly the same. It doesn't seem like vita-make-fself changes the layout of the elf file, so the velf should be quite similar to the eboot.bin.

I found a workaround for the moment, but I can't say for sure that it fixes the issue or just hides it until it reappears in another way. Compiling in Release instead of RelWithDebInfo seems to fix the problem. AFAIK, the only difference there is the absence of the -g flag when compiling.

I am thinking that maybe when the relocation tables are getting too big, the ELF/SCE headers generated by the vita toolchain are broken and the vita fails to relocate some things? The vita does do some dynamic relocation when doing ASLR, right? Is there some code that assumes that there are less than some number of relocations? or that sections are not bigger than some size?

[yne]

This remind me of someone who ported ScummVM (IIRC) and exporting all of it functions, making the exports (or the relocs ? or the symtable ?) unbearable for the ModuleMgr.

Then by commenting out a line in an unrelated file, the app would run fine

Could you double check if this comment is really responsible for the crash/working state ?
If proven, please post the readelf diff between your two velf files.
diff -y <(readelf -hS working.velf) <(readelf -hS buggy.velf)

Maybe the diff will highlight some kind of sections limits that you just hit.

[blastrock]

Could you double check if this comment is really responsible for the crash/working state ?

I've checked it once more to be sure, probably the 5th time ^^

Here's the diff of the velf files:

readelf: Warning: [40]: Info field (39) should index a relocatable section.
readelf: Warning: [40]: Info field (39) should index a relocatable section.
ELF Header:							ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 	  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32			  Class:                             ELF32
  Data:                              2's complement, little e	  Data:                              2's complement, little e
  Version:                           1 (current)		  Version:                           1 (current)
  OS/ABI:                            UNIX - System V		  OS/ABI:                            UNIX - System V
  ABI Version:                       0				  ABI Version:                       0
  Type:                              OS Specific: (fe04)	  Type:                              OS Specific: (fe04)
  Machine:                           ARM			  Machine:                           ARM
  Version:                           0x1			  Version:                           0x1
  Entry point address:               0x1686e0		      |	  Entry point address:               0x1686f0
  Start of program headers:          52 (bytes into file)	  Start of program headers:          52 (bytes into file)
  Start of section headers:          18434194 (bytes into fil |	  Start of section headers:          18436234 (bytes into fil
  Flags:                             0x5000400, Version5 EABI	  Flags:                             0x5000400, Version5 EABI
  Size of this header:               52 (bytes)			  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)			  Size of program headers:           32 (bytes)
  Number of program headers:         3				  Number of program headers:         3
  Size of section headers:           40 (bytes)			  Size of section headers:           40 (bytes)
  Number of section headers:         72				  Number of section headers:         72
  Section header string table index: 63				  Section header string table index: 63

Section Headers:						Section Headers:
  [Nr] Name              Type            Addr     Off    Size	  [Nr] Name              Type            Addr     Off    Size
  [ 0]                   NULL            00000000 000000 0000	  [ 0]                   NULL            00000000 000000 0000
  [ 1] .init             PROGBITS        00008000 008000 0000	  [ 1] .init             PROGBITS        00008000 008000 0000
  [ 2] .text             PROGBITS        00008040 008040 10ee	  [ 2] .text             PROGBITS        00008040 008040 10ee
  [ 3] .rel.text         REL             00000000 ba3050 04a4 |	  [ 3] .rel.text         REL             00000000 ba34b0 04a4
  [ 4] .text.fstubs.SceA PROGBITS        00116e90 116e90 0000	  [ 4] .text.fstubs.SceA PROGBITS        00116e90 116e90 0000
  [ 5] .text.fstubs.SceC PROGBITS        00116ec0 116ec0 0000	  [ 5] .text.fstubs.SceC PROGBITS        00116ec0 116ec0 0000
  [ 6] .text.fstubs.SceC PROGBITS        00116f20 116f20 0000	  [ 6] .text.fstubs.SceC PROGBITS        00116f20 116f20 0000
  [ 7] .text.fstubs.SceD PROGBITS        00116f40 116f40 0000	  [ 7] .text.fstubs.SceD PROGBITS        00116f40 116f40 0000
  [ 8] .text.fstubs.SceD PROGBITS        00116f50 116f50 0000	  [ 8] .text.fstubs.SceD PROGBITS        00116f50 116f50 0000
  [ 9] .text.fstubs.SceG PROGBITS        00116f60 116f60 0003	  [ 9] .text.fstubs.SceG PROGBITS        00116f60 116f60 0003
  [10] .text.fstubs.SceH PROGBITS        001172e0 1172e0 0000	  [10] .text.fstubs.SceH PROGBITS        001172e0 1172e0 0000
  [11] .text.fstubs.SceN PROGBITS        001173c0 1173c0 0000	  [11] .text.fstubs.SceN PROGBITS        001173c0 1173c0 0000
  [12] .text.fstubs.SceN PROGBITS        00117440 117440 0000	  [12] .text.fstubs.SceN PROGBITS        00117440 117440 0000
  [13] .text.fstubs.SceP PROGBITS        00117460 117460 0000	  [13] .text.fstubs.SceP PROGBITS        00117460 117460 0000
  [14] .text.fstubs.SceP PROGBITS        001174f0 1174f0 0000	  [14] .text.fstubs.SceP PROGBITS        001174f0 1174f0 0000
  [15] .text.fstubs.SceP PROGBITS        00117530 117530 0000	  [15] .text.fstubs.SceP PROGBITS        00117530 117530 0000
  [16] .text.fstubs.SceS PROGBITS        00117570 117570 0000	  [16] .text.fstubs.SceS PROGBITS        00117570 117570 0000
  [17] .text.fstubs.SceS PROGBITS        001175a0 1175a0 0000	  [17] .text.fstubs.SceS PROGBITS        001175a0 1175a0 0000
  [18] .text.fstubs.SceS PROGBITS        001175b0 1175b0 0000	  [18] .text.fstubs.SceS PROGBITS        001175b0 1175b0 0000
  [19] .text.fstubs.SceV PROGBITS        00117610 117610 0000	  [19] .text.fstubs.SceV PROGBITS        00117610 117610 0000
  [20] .text.fstubs.SceS PROGBITS        00117620 117620 0001	  [20] .text.fstubs.SceS PROGBITS        00117620 117620 0001
  [21] .text.fstubs.SceR PROGBITS        00117750 117750 0000	  [21] .text.fstubs.SceR PROGBITS        00117750 117750 0000
  [22] .text.fstubs.SceS PROGBITS        00117760 117760 0000	  [22] .text.fstubs.SceS PROGBITS        00117760 117760 0000
  [23] .text.fstubs.SceT PROGBITS        001177a0 1177a0 0000	  [23] .text.fstubs.SceT PROGBITS        001177a0 1177a0 0000
  [24] .text.fstubs.SceT PROGBITS        00117810 117810 0000	  [24] .text.fstubs.SceT PROGBITS        00117810 117810 0000
  [25] .text.fstubs.SceI PROGBITS        00117820 117820 0000	  [25] .text.fstubs.SceI PROGBITS        00117820 117820 0000
  [26] .text.fstubs.SceP PROGBITS        00117880 117880 0000	  [26] .text.fstubs.SceP PROGBITS        00117880 117880 0000
  [27] .text.fstubs.SceL PROGBITS        001178a0 1178a0 0002	  [27] .text.fstubs.SceL PROGBITS        001178a0 1178a0 0002
  [28] .fini             PROGBITS        00117ae0 117ae0 0000	  [28] .fini             PROGBITS        00117ae0 117ae0 0000
  [29] .rodata           PROGBITS        00117af0 117af0 0413	  [29] .rodata           PROGBITS        00117af0 117af0 0413
  [30] .rel.rodata       REL             00000000 bed490 0065 |	  [30] .rel.rodata       REL             00000000 bed900 0065
  [31] .ARM.extab        PROGBITS        00158e24 158e24 00a4 |	  [31] .ARM.extab        PROGBITS        00158e24 158e24 00a4
  [32] .rel.ARM.extab    REL             00000000 bf39a8 0022 |	  [32] .rel.ARM.extab    REL             00000000 bf3e18 0022
  [33] .ARM.exidx        ARM_EXIDX       00163238 163238 0054 |	  [33] .ARM.exidx        ARM_EXIDX       00163250 163250 0054
  [34] .rel.ARM.exidx    REL             00000000 bf5c90 00da |	  [34] .rel.ARM.exidx    REL             00000000 bf6108 00da
  [35] .eh_frame         PROGBITS        00168660 168660 0000 |	  [35] .eh_frame         PROGBITS        00168678 168678 0000
  [36] .rel.eh_frame     REL             00000000 c036c0 0000 |	  [36] .rel.eh_frame     REL             00000000 c03b40 0000
  [37] .init_array       INIT_ARRAY      00180000 170d1c 0000 |	  [37] .init_array       INIT_ARRAY      00180000 170d14 0000
  [38] .rel.init_array   REL             00000000 c036c8 0000 |	  [38] .rel.init_array   REL             00000000 c03b48 0000
  [39] .fini_array       FINI_ARRAY      00180044 170d60 0000 |	  [39] .fini_array       FINI_ARRAY      00180044 170d58 0000
  [40] .rel.fini_array   REL             00000000 c03750 0000 |	  [40] .rel.fini_array   REL             00000000 c03bd0 0000
  [41] .data             PROGBITS        00180048 170d64 0021 |	  [41] .data             PROGBITS        00180048 170d5c 0021
  [42] .rel.data         REL             00000000 c03758 000e |	  [42] .rel.data         REL             00000000 c03bd8 000e
  [43] .tm_clone_table   PROGBITS        001821ec 172f08 0000 |	  [43] .tm_clone_table   PROGBITS        001821ec 172f00 0000
  [44] .bss              NOBITS          001821f0 172f08 14c5 |	  [44] .bss              NOBITS          001821f0 172f00 14c5
  [45] .comment          PROGBITS        00000000 172f08 0000 |	  [45] .comment          PROGBITS        00000000 172f00 0000
  [46] .debug_aranges    PROGBITS        00000000 172f3b 0026 |	  [46] .debug_aranges    PROGBITS        00000000 172f33 0026
  [47] .rel.debug_arange REL             00000000 c04630 0021 |	  [47] .rel.debug_arange REL             00000000 c04ab0 0021
  [48] .debug_info       PROGBITS        00000000 175603 4dd0 |	  [48] .debug_info       PROGBITS        00000000 1755fb 4dd2
  [49] .rel.debug_info   REL             00000000 c06748 2898 |	  [49] .rel.debug_info   REL             00000000 c06bc8 2899
  [50] .debug_abbrev     PROGBITS        00000000 652633 024b |	  [50] .debug_abbrev     PROGBITS        00000000 65285b 024b
  [51] .debug_line       PROGBITS        00000000 6771b8 0764 |	  [51] .debug_line       PROGBITS        00000000 6773e0 0764
  [52] .rel.debug_line   REL             00000000 e8ff70 001e |	  [52] .rel.debug_line   REL             00000000 e90560 001e
  [53] .debug_frame      PROGBITS        00000000 6ed600 033c |	  [53] .debug_frame      PROGBITS        00000000 6ed870 033c
  [54] .rel.debug_frame  REL             00000000 e91d98 017a |	  [54] .rel.debug_frame  REL             00000000 e92388 017a
  [55] .debug_str        PROGBITS        00000000 721254 1749 |	  [55] .debug_str        PROGBITS        00000000 7214c8 1749
  [56] .debug_loc        PROGBITS        00000000 895bb2 1e52 |	  [56] .debug_loc        PROGBITS        00000000 895e26 1e53
  [57] .rel.debug_loc    REL             00000000 ea97e8 1c95 |	  [57] .rel.debug_loc    REL             00000000 ea9dd8 1c96
  [58] .debug_ranges     PROGBITS        00000000 a7adfb 0676 |	  [58] .debug_ranges     PROGBITS        00000000 a7b1d1 0677
  [59] .rel.debug_ranges REL             00000000 1072d90 093 |	  [59] .rel.debug_ranges REL             00000000 10734c8 093
  [60] .ARM.attributes   ARM_ATTRIBUTES  00000000 ae24e3 0000 |	  [60] .ARM.attributes   ARM_ATTRIBUTES  00000000 ae2921 0000
  [61] .symtab           SYMTAB          00000000 ae2514 05f6 |	  [61] .symtab           SYMTAB          00000000 ae2954 05f7
  [62] .strtab           STRTAB          00000000 b41bf4 0614 |	  [62] .strtab           STRTAB          00000000 b42054 0614
  [63] .shstrtab         STRTAB          00000000 1106a00 000 |	  [63] .shstrtab         STRTAB          00000000 11071c8 000
  [64] .sceModuleInfo.ro PROGBITS        001686e0 1686e0 0000 |	  [64] .sceModuleInfo.ro PROGBITS        001686f0 1686f0 0000
  [65] .sceLib.ent       PROGBITS        00168770 168770 0000 |	  [65] .sceLib.ent       PROGBITS        00168780 168780 0000
  [66] .sceExport.rodata PROGBITS        00168790 168790 0000 |	  [66] .sceExport.rodata PROGBITS        001687a0 1687a0 0000
  [67] .sceLib.stubs     PROGBITS        001687a8 1687a8 0004 |	  [67] .sceLib.stubs     PROGBITS        001687b8 1687b8 0004
  [68] .sceImport.rodata PROGBITS        00168c88 168c88 0001 |	  [68] .sceImport.rodata PROGBITS        00168c98 168c98 0001
  [69] .sceFNID.rodata   PROGBITS        00168dc8 168dc8 0003 |	  [69] .sceFNID.rodata   PROGBITS        00168dd8 168dd8 0003
  [70] .sceFStub.rodata  PROGBITS        001690dc 1690dc 0003 |	  [70] .sceFStub.rodata  PROGBITS        001690ec 1690ec 0003
  [71] .sce.rel          LOOS+0          00000000 1106e70 08d |	  [71] .sce.rel          LOOS+0          00000000 1107640 08d
Key to Flags:							Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), 	  W (write), A (alloc), X (execute), M (merge), S (strings), 
  L (link order), O (extra OS processing required), G (group)	  L (link order), O (extra OS processing required), G (group)
  C (compressed), x (unknown), o (OS specific), E (exclude),	  C (compressed), x (unknown), o (OS specific), E (exclude),
  y (purecode), p (processor specific)				  y (purecode), p (processor specific)

Nothing strange about section sizes... or I may have missed something.

Another strange thing is that the exception tables in ARM are supposed to contain relative addresses, so they shouldn't need to be relocated.

If you want to see what it looks like, the following commit is the one that makes the app crash: blastrock/pkgj@724ec4c . As I said, this is unrelated code, the main() function triggers the bug long before it is called. The nop are there to keep the alignment of the rest of the code to be able to diff it. The exception tables however are all messed up.

The commit before should work, it shows "fatal error: test" but the app does not crash.

If you want to compile the code, run ci/ci.sh from the root of the project. The artifacts will be put in ci/build.

[blastrock]

I have dug deeper, but still haven't found a fix...

The issue (in my case at least) is with the entry for __cxa_throw in the unwind table (among others), which means that all exceptions will make the program crash.

Entries in the unwind table look like struct { uint32_t fnoffset; uint32_t content; }

__cxa_throw's entry in the unwind table uses the personality routine 0 (__aeabi_unwind_cpp_pr0) to handle unwinding, and as such content does not contain an address but just some data. Reading the relocation table with objdump shows that there is a relocation for the address of __cxa_throw in fnoffset, but nothing for the content of the exidx entry. Yet, that field is changed when the app is loaded in memory.

I dumped the eboot.bin with an hexadecimal editor, and there's clearly a ab08b180 in the content, which is correct. When I load the app and read that same memory, I get ab08b100, the 80 part was transformed. I tried replacing 80 by ff or 55, then it get replaced by 04.

I have no idea why it gets replaced, and since it's done by the vita's elf loader I have no idea how to investigate further. If anyone has any idea as to why it behaves like that, it would be much appreciated.

[blastrock]

I have spent some more time on this... The faulty relocation is actually there, in the ELF output from ld. I tried removing just that one manually and it fixed my bug. Then I tried to understand why I get that relocation in the first place.

I used readelf on eh_throw.o from libsupc++ in libstdc++, there are two relocations on __cxa_throw's unwind table, but they are not on the content field, so everything is fine there.

I tried writing a custom linker script (from the default one) with this:

  .ARM.exidx.eh : { *eh_throw.o(.ARM.exidx.text.__cxa_throw) }
  .ARM.exidx   : { *(.ARM.exidx* .gnu.linkonce.armexidx.*) }

I've put out the problematic part in its own section. Of course this will never work as the exidx table must be ordered, but it allows me analyzing the final binary and check ld's behavior.

I can see that there are several relocations in .rel.ARM.exidx that change addresses before .ARM.exidx, including in the section I have extracted. That's why I think the bug is in ld.

I tried reading the code of ld, but it's a mass of chaos coming straight from hell, can't do anything in there.

I tried using llvm's lld to link the binary, I had to change the linker script to add some padding for vita-elf-create for the module info structure. I managed to get an eboot, and the problematic relocation is not in there, but it won't run on my psvita giving me a generic error :(

[blastrock]

I think I found the root of my issue and worked around it in a patch on ld.

Reported here: https://sourceware.org/bugzilla/show_bug.cgi?id=23839

Workaround here: https://sourceware.org/bugzilla/show_bug.cgi?id=20595#c1

Here is my fork of vitasdk (a bit old), with the ld patch: https://github.com/blastrock/autobuilds/releases/tag/master-linux-v8

Hope this gets fixed...

[devnoname120]

This is really cool, thanks for investigating and reporting the bug upstream.

[d3m3vilurr]

@blastrock gcc 7.4 & 8.2 was applied that patch except NO_IMPLICIT_EXTERN_C part.
do we need both of binutils patch and that macro patch?

[blastrock]

The NO_IMPLICIT_EXTERN_C was merged here vitasdk/buildscripts@d27627d so you don't need it anymore if you compile master. The backported patch doesn't seem that important actually, but I didn't double check.

You will need however the binutils patch blastrock/autobuilds@b2f8cd5#diff-3b86632c44a047a317357da4d98eaf0a if you have issues with exceptions. The ld bug report didn't get any answer, I guess it won't get any for a long time.

[d3m3vilurr]

okay, I made the patch for that.
if pass in some local test, I'll merge it.

[d3m3vilurr]

I got the report from @xerpi that this issue is still alive.
at the merging time, I might use wrong test steps.
this issue requires testing with pthread call.

std::once_flag flag; 
  
void thread() {
  std::call_once(flag,
    [&] () {
    }
  );
}
  
int main(int argc, char *argv[]) {
  psvDebugScreenInit();
  printf("testing try-catch\n");

  try {
    int i = 1/0;
  } catch(...) {
    printf("catched normal exception\n");
  }
  printf("%d\n", sceKernelGetThreadId());
  std::thread t{thread};
  printf("%d\n", sceKernelGetThreadId());
  
  t.join();
  
  sceKernelDelayThread(10000*1000);
  //sceKernelExitProcess(0);
  return 0;
}

this test code will occur the crash.

[blastrock]

I'm not sure what this snippet should do. Division by zero is not an exception in c++, programs usually crash because of that (depending on the platform). On linux, you get a SIGFPE I believe.
Clang reports a warning:

/tmp/o.cpp:17:14: warning: division by zero is undefined [-Wdivision-by-zero]

Btw, the crash that I worked around was fixed upstream, it seems: https://sourceware.org/bugzilla/show_bug.cgi?id=23839
I didn't test the fix though.

[d3m3vilurr]

I test it with various conditions.

version up binutils or apply the patch have been occurred segfault on vita-elf-create
this would be related with #44

catch exception would be normal on the main thread. but when we make the thread with pthread-embedded that means make the thread with std::thread or std::mutex, it could be crash.

idk main reason, it could be problem of the .ARM.exids section.
imo, vita-elf-create process doesn't have logic for it.

ps. div 0 isn't an important part(the main issue is the crash when we use the pthread on c++).
i just want to test normal catching exceptions, and it would work well.

[blastrock]

Then I don't think this is related to the exidx section. That section is read when the stack is unwound, which happens when you throw an exception. Since there is no throw in that program, that binutil patch shouldn't change anything. That's only my understanding however, I could be wrong.

You should remove that division by 0 for your tests if it isn't important. It is undefined behavior, a crash is a valid result for that.

[d3m3vilurr]

sorry I'm late.
@blastrock you are right. exids is not a reason. built result binary was wrong and I made the patch for it. vitasdk/buildscripts#66

[frangarcj]

@blastrock do you still have an example to reproduce it? Binutils issues have been closed and merged so I wan't to remove the patch

[blastrock]

I haven't worked on psvita in a while, so I can't be sure, but I had a branch for that.

You can try to build blastrock/pkgj@724ec4c . This used to crash pkgj right after it starts with the psvita C-whatever error. The correct behavior is no crash, but a black screen with an error.

Building it might prove a little hard. You need to install a bunch of tools and run ci/ci.sh from the root of the repository. You'll get a vpk in ci/build. This build will be against an old version of the vitasdk that I forked. To test against the last version, you can replace this line: https://github.com/blastrock/pkgj/blob/724ec4c53705b3db20703f08b4bee66ad3c405cf/ci/conan-vitasdk/conanfile.py#L15 with the URL of the last version.