App works on Xcode release mode, crashes on Testflight.

[dunyakilavuz]

Godot version:
Godot 3.3 Mono

Issue description:
Managed to run my app on both Debug and Release modes on Xcode without any problems. Then I uploaded it to App Store Connect for distribution and tried downloading & running it from Testflight.

The app opens up with a loading scene and when the loading is complete it proceeds to the main scene. But when this loading is complete, app crashes immediately.

Testflight feedback was as follows.

testflight_feedback.zip

I tried detaching most of the scripts I use. The scene without scripts works on Testflight, so I know its not a rendering issue. Also tried loading a test script to see if it was related to Mono, again worked on Testflight. So its not a Mono issue as well?

Couldn't figure out why it would only crash on Testflight though. Everything works perfectly when I run it from Xcode.

I would be very happy If anyone can make up anything from the feedback I uploaded. Thanks for your help!

[naithar]

@dunyakilavuz you should be able to examine crashlog.crash file from feedback archive inside Xcode with your project to turn

14  SolarX_iOS                    	0x00000001030bb03c 0x102dfc000 + 2879548
15  SolarX_iOS                    	0x0000000103dbb1bc 0x102dfc000 + 16511420

into something more readable and can be debugged.

Edit:
You should probably also try to upload a project from non-mono (classic) Godot version. This would narrow the issue down to Mono version only.

[dunyakilavuz]

@naithar Thanks for the suggestion. I opened up the log file in xcode as you suggested and indeed it was more readable.

It seems lines 14 & 15 are the cause of the crash. What's weird is that the app is already started in this state. The loading scene is up and running and the loader script is written in C#. Maybe a script im using in the loaded scene is causing the crash? I wonder why Xcode release won't point out any errors.

[naithar]

Great, that would help, I guess.
Godot should also log the exception that's causing the crash. You should be able to see the logs from Console that you can gain access to from Xcode's Devices and Simulator->Open Console.

I think you can also try setting different optimization levels to see if you can reproduce the issue while running the application from Xcode.
Have you tried editing Xcode scheme to run application in Release mode and turning on/off different scheme settings to check if that would help to reproduce the issue?

[naithar]

I wasn't able to reproduce your exact crash, but I've got crash on closing an application.

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000258
VM Region Info: 0x258 is not in any region.  Bytes before following region: 4339596712

#0	0x0000000105d0a2e4 in mono_debug_domain_unload ()
#1	0x0000000105cad5c0 in mono_domain_free ()
#2	0x0000000105cad5c0 in mono_domain_free ()
#3	0x0000000105c5a5ac in mini_cleanup ()
#4	0x0000000105c3b364 in mono_jit_cleanup ()
#5	(null) in GDMono::~GDMono+ 2884928 () ()
#6	(null) in -[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:] ()
#7	(null) in -[_UISceneLifecycleMultiplexer forceExitWithTransitionContext:scene:] ()
#8	(null) in -[UIApplication workspaceShouldExit:withTransitionContext:] ()
#9	(null) in -[FBSUIApplicationWorkspaceShim workspaceShouldExit:withTransitionContext:] ()
#10	(null) in __63-[FBSWorkspaceScenesClient willTerminateWithTransitionContext:]_block_invoke_2 ()
#11	(null) in -[FBSWorkspace _calloutQueue_executeCalloutFromSource:withBlock:] ()
#12	(null) in __63-[FBSWorkspaceScenesClient willTerminateWithTransitionContext:]_block_invoke ()
#13	(null) in _dispatch_client_callout ()
#14	(null) in _dispatch_block_invoke_direct ()
#15	(null) in __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__ ()
#16	(null) in -[FBSSerialQueue _targetQueue_performNextIfPossible] ()
#17	(null) in -[FBSSerialQueue _performNextFromRunLoopSource] ()
#18	(null) in __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ ()
#19	(null) in __CFRunLoopDoSource0 ()
#20	(null) in __CFRunLoopDoSources0 ()
#21	(null) in __CFRunLoopRun ()
#22	(null) in CFRunLoopRunSpecific ()
#23	(null) in GSEventRunModal ()
#24	(null) in -[UIApplication _run] ()
#25	(null) in UIApplicationMain ()

But I have iPhone XS with iOS 14.5.1, so it might be related.
My friend got the crash on startup running application on iPhone X (iOS 13.7), but TestFlight give me same crash report as for my crash.

[neikeq]

@naithar's crash should be fixed by #49248. I can't repro OP's crash though.

[selfsame]

I am getting the same exception, Godot 3.3 mono. Can't reproduce in Xcode, with debug or release build.

12  Idlematcher                   	0x00000001058df4bc StringName::unref+ 19920060 () + 68
13  libsystem_c.dylib             	0x00000001b50365e4 exit + 28
14  Idlematcher                   	0x00000001048e2158 GDMono::initialize_load_assemblies+ 3154264 () + 0
15  Idlematcher                   	0x00000001052205fc PackedScene::instance+ 12846588 (PackedScene::GenEditState) const + 32

(full symbolized crash report https://gist.github.com/selfsame/303cc85be8c32ce1957fbb601489aad6#file-ios-crash-report)

Eager to help investigate if there's anything I can try

[dunyakilavuz]

Would be very good to finally get rid of this issue, I would investigate too, but don't know where to start.

[naithar]

You should try switching Xcode's Run scheme to the same values as Archive. This should allow you to reproduce the crash in Xcode as well as get some logs that can be useful to fix it.
From current crash reports it seems like the issue is with initialize_load_assemblies. You can also try to build iOS mono template in debug mode - this should give more information about the issue.
You can also try to bisect exact commit/release that introduced this issue.

[Calinou]

You can also try to bisect exact commit/release that introduced this issue.

See Bisecting regressions in the documentation for instructions 🙂

[Difference]

I have the same issue with godot 3.4 beta 2 and 3.
App will start, but some call will make it crash.

[basdp]

I have this same issue. The crash log in the Console shows this Mono exception:

Unhandled Exception:
System.TypeInitializationException: The type initializer for 'System.Random' threw an exception. ---> System.EntryPointNotFoundException: SystemNative_GetNonCryptographicallySecureRandomBytes
  at Interop.GetRandomBytes (System.Byte* buffer, System.Int32 length) [0x00000] in <11afb9645ddb4e8fae4163490ca47a50>:0 
  at System.Random.GenerateGlobalSeed () [0x00000] in <11afb9645ddb4e8fae4163490ca47a50>:0 
  at System.Random..cctor () [0x00000] in <11afb9645ddb4e8fae4163490ca47a50>:0 
   --- End of inner exception stack trace ---
  at Grid..ctor () [0x000a2] in <132935bdaba3435ea6571a1b11c21ad4>:0

Using Godot 3.4.4.

[basdp]

If I allow the game to continue running when an unhandled Mono exception occurs (set unhandled_exception_policy=1 under [mono]), I'm able to start games downloaded from Testflight. No c# code runs though, so the game is just an empty opening scene, nothing happens.

[basdp]

I did some more research, and it looks like it is because I use .NET's System.Random. If I remove the code that uses System.Random, everything works fine. Might be that this issue occurs with more .NET Standard Library stuff, but for now it's only System.Random for me.

@selfsame @dunyakilavuz, could you please verify if you use System.Random or any other .NET specific stuff?

[selfsame]

@basdp I'd confirmed ToString was triggering this, still getting it without ToString. I don't use System.Random but I do use System.Collections.Generic and System.Linq

[basdp]

@selfsame thanks, will try to find out why.

do you know if the app also crashes when released to the App Store?

[selfsame]

do you know if the app also crashes when released to the App Store?

@basdp Yes crash in app store and test flight builds

[basdp]

Ok, that is worrisome.

I have created a very minimal Godot project that reproduces the crash. See:
https://basduprecom-my.sharepoint.com/:u:/g/personal/me_basdupre_com1/Ecc2SPzm1PtLubshqNZ4N4EB5Z_iVw8bY_adRPKqt232bw?e=lT4tel

I am currently still doing some research on this, but I might need some help as I'm not very familiar with Godot's internals.

Also, it looks like this is a duplicate of #38810, which was incorrectly closed as this is still crashing with the fixes mentioned there. I think it was incorrectly flagged as a duplicate of #40757, as it's a totally different scenario (this issue 'only' happens in App Store releases, and the suspect duplicate happened on all real device builds).

Not sure what the best procedure is (re-open the #38810 issue or continue here)

[neikeq]

#38810 was about ad hoc distribution, not Testflight.

The issue here is likely the same (P/Invoke symbols are being stripped), but it only happens on Testflight and App Store releases.

Currently, my focus is 100% on the Godot 4.0 release. After that, I'll come back to this.

[neikeq]

The other issue was fixed via fake references in source code rather than linker flags. If any one is interested in experimenting, you could try using the linker flags the way it's explained here:

I can confirm passing -u to the linker correctly makes it not strip the symbol, fixing the issue. However it has to be done this way:

• The linker flags are of the form: -Wl,-u,_SystemNative_Stat2
• These linker flags have to be added to the Other Linker Flags build setting for the TARGET, not the PROJECT.

If you're getting an EntryPointNotFoundException for the symbol SystemNative_GetNonCryptographicallySecureRandomBytes pass the flag -Wl,-u,_SystemNative_GetNonCryptographicallySecureRandomBytes (notice the _ prefix) and see if that fixes it or if the exception is about a different symbol now.

If that doesn't work either, we may need to reference the symbols in a code path that can actually be executed, e.g.: at startup, if env var is defined, print address to symbol. If this doesn't work either, the issue may be a different one.

[basdp]

@neikeq great insights! I can also confirm that if I set Strip Style under Deployment in the Xcode Build Settings to Non-Global Symbols, it looks like it fixes this. Of course, that's more of a workaround than a fix.

I will try if the linker flags make it a little less shotgun-on-a-fly type of thing, though the 'overhead' is only 200kb in the archive output size, of which some bytes (namely the global exports) are actually desired.

[basdp]

For what it's worth, I worked around this problem for now, by adding this script to my CI pipeline:

cd ~/Library/Application\ Support/Godot/templates/${{ variables.godot_version }}.${{ replace(variables.godot_branch, '_', '.') }}
mkdir godot_ios.xcodeproj
unzip -p iphone.zip godot_ios.xcodeproj/project.pbxproj >godot_ios.xcodeproj/project.pbxproj
gsed -i '/CODE_SIGN_IDENTITY = "$code_sign_identity_release";/a STRIP_STYLE = non-global;' godot_ios.xcodeproj/project.pbxproj
zip -u iphone.zip godot_ios.xcodeproj/project.pbxproj
rm -rf godot_ios.xcodeproj

Please be aware that this is only safe to do if you reset your build environment each build, which happens in my CI as it's running in a container. I'm patching the live export templates during the build. It would be better to create your own export template for iphone.zip with any changes that you might need. Also, if you would go that route, probably you should use Godot's built-in custom template options.

For now, I'm waiting until this problem can be picked up by a more experienced dev. More than happy to help, but my knowledge of the Godot/Mono code structures and patterns is too little to lead a PR on this.

[oknpk]

@neikeq great insights! I can also confirm that if I set Strip Style under Deployment in the Xcode Build Settings to Non-Global Symbols, it looks like it fixes this. Of course, that's more of a workaround than a fix.

I will try if the linker flags make it a little less shotgun-on-a-fly type of thing, though the 'overhead' is only 200kb in the archive output size, of which some bytes (namely the global exports) are actually desired.

The problem was solved this way. If anyone is looking for a solution to this issue, it works. I uploaded it to Testflight and AppStore and it works fine. I use most C# libraries in my project. (such as Random, Thread, TimeZoneInfo, IO, Collections.Generic, Linq). Thanks @basdp