Illegal instruction on core load when compiled with any optimizations

[thieman]

I'm just starting to investigate this but I figured I'd post it now. I'm developing a core which loads and runs fine when compiled at opt-level=0 (debug level default). However, if I compile it with any other level of optimizations, whether speed- or size-based, I get an illegal instruction core dump as soon as I load the core in Retroarch. This is before actually attempting to load any content, this is when I select "Load Core" from the main menu. I'm seeing this both in a Linux->Windows GNU crosscompile, and in a Mac native compile.

If I don't make progress soon I'll try to come up with a minimal repro, but a larger repro is available now by cloning https://github.com/thieman/exa-rs at commit 6035360da4f74d72705d88c234d4328a7547a291.

I'm also wondering if this may be related to #13.

[thieman]

FWIW I seem to be getting the exact same error when compiling Pinky with --release. https://github.com/koute/pinky/blob/master/pinky-libretro

@koute I realize you might not be active on this project any more but if you have a second, I'd be curious if you can still get it to work in release mode however you were doing it before. I'm wondering if this is all related to something with my toolchain rather than with libretro-backend.

[thieman]

Core dump from the Mac side https://gist.github.com/thieman/90f858299fe3c1502ef36300fc2057c0

[thieman]

More findings: if I comment out the body of on_set_environment I survive loading the core, but then fail later on when I try to load some content. I think this is more evidence of the cause being related to #13, though I'm a Rust novice and a lot of what's going on in that issue is over my head. Continuing to tinker with it.

[thieman]

Still digging but I think it might be this issue that was just fixed, I'm going to see if it's in nightly and try there. It seems like I'm getting different results for pointer equality comparisons in debug vs release, which is affecting both setting the environment callback and then later triggering all the assert_ne! asserts against LIBRETRO_INSTANCE. rust-lang/rust#54685

[thieman]

I think I've managed to hackily fix it in my fork https://github.com/thieman/exa-rs/blob/master/src/libretro/mod.rs

I ended up doing what @HeroicKatora mentioned in #13 and changing the callbacks to use Options so that they're never null function pointers. This has seemed to stop the crashes in release mode.

Will wait to get some feedback here before trying to contribute a patch as it seems like this repo might be inactive

[ghost]

I ran into this as well, implementing a core for psxact.

$ retroarch --verbose -L libpsxact_libretro.so ~/Games/sony/playstation/spyro/disc.bin 
[INFO] [Config]: Loading default config.
[INFO] [Config]: Looking for config in: "/home/adam/.config/retroarch/retroarch.cfg".
[INFO] RetroArch 1.9.9 (Git 36888612eb)
[INFO] === Build =======================================
[INFO] CPU Model Name: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
[INFO] Capabilities:  MMX MMXEXT SSE SSE2 SSE3 SSSE3 SSE4 SSE4.2 AES AVX AVX2
[INFO] Built: Sep  6 2021
[INFO] Version: 1.9.9
[INFO] Git: 36888612eb
[INFO] =================================================
[INFO] [Input]: Found input driver: "x".
[INFO] [Core]: Loading dynamic libretro core from: "libpsxact_libretro.so"
[INFO] [Overrides]: No core-specific overrides found at "/home/adam/.config/retroarch/config/psxact/psxact.cfg".
[INFO] [Overrides]: No content-dir-specific overrides found at "/home/adam/.config/retroarch/config/psxact/spyro.cfg".
[INFO] [Overrides]: No game-specific overrides found at "/home/adam/.config/retroarch/config/psxact/disc.cfg".
Illegal instruction (core dumped)

@koute Is this repository being maintained still?
@thieman Is your fork with the fixes pushed to crates.io?

[thieman]

@adam-becker my fork isn't pushed to crates.io but you're welcome to grab it, my changes are under MIT license https://github.com/thieman/exa-rs/blob/main/src/libretro/mod.rs