FullAOT test failures/crashes on arm64,armel or armhf

[akoeplinger]

Steps to Reproduce

1. Run runtime tests and various BCL tests on FullAOT config on arm64, armel or amhf
2. Observe test failures and crashes

This started with https://jenkins.mono-project.com/job/test-mono-mainline-linux-fullaot/656/:

• [packaging] Apply F# patch that fixes FSharp build is broken due to new Path.GetFileName() overload #8310 (commit: d0db2e0)
• [interp] remove useless assert in mint_newobj (commit: 8127483)
• Remove unused signal.h include in interp.c (Remove unused signal.h include in interp.c #8334) (commit: cc3666f)
• [mini] fix some warnings with newer GCC ([mini] fix some warnings with newer GCC #8308) (commit: 384c4a4)
• [Tailcall amd64] Use calli/tailcall_reg in method-to-ir for F# FullAOT. (commit: 736683e)
• [System.Net] do not use Thread for SocketTest.TestSelect1 ([System.Net] do not use Thread for SocketTest.TestSelect1 #8364) (commit: 9b47d56)
• [interp] fix warning introduced with 8e0a121 ([interp] fix warning introduced with 8e0a12189a84 #8359) (commit: 8d30d41)
• [watch] Align cinfo->stack_usage to 16 on watchos. ([watch] Align cinfo->stack_usage to 16 on watchos. #8354) (commit: 3106c27)
• [sdks] Match more closely the pre-SDKs configure flags passed to LLVM (commit: e78cae2)
• [runtime] Add a signal handler for sigterm ([runtime] Add a signal handler for sigterm #8240) (commit: 24ef425)
• [ci] Add jit_llvm configuration ([ci] Add jit_llvm configuration #8317) (commit: f9a7746)
• [corlib] Bump allowed diff time in FileTest.LastWriteTimeSubMs (commit: f93b967)

Current Behavior

arm64 runtime test failures:

MonoTests.runtime.tailcall/2.exe

MESSAGE:
tailcall failure FFFFFFFFFFFFFF90

+++++++++++++++++++
STACK TRACE:

armel/armhf runtime test failures:

MonoTests.runtime.runtime-invoke.gen.exe

MESSAGE:
* Assertion at mini-arm.c:2880, condition `ainfo->cinfo->stack_usage % MONO_ARCH_FRAME_ALIGNMENT == 0' not met

--------

MonoTests.runtime.runtime-invoke.exe

MESSAGE:
TEST: System.String ToString() System.String Test()
A
Test
* Assertion at mini-arm.c:2880, condition `ainfo->cinfo->stack_usage % MONO_ARCH_FRAME_ALIGNMENT == 0' not met

armel/armhf BCL test crashes:

corlib:

***** MonoTests.System.GuidTest.TryParse
* Assertion at mini-arm.c:2880, condition `ainfo->cinfo->stack_usage % MONO_ARCH_FRAME_ALIGNMENT == 0' not met

nitLite.Runner.TextUI.RunTests (NUnit.Framework.Api.ITestFilter) [0x0001e] in <f8e3494c86a44d05aacefdf05529e8d3#97F8BA07-4546-E473-D9B6-2BD796223972>:0
  at NUnitLite.Runner.TextUI.Execute (string[]) [0x002c4] in <f8e3494c86a44d05aacefdf05529e8d3#97F8BA07-4546-E473-D9B6-2BD796223972>:0
  at NUnitLite.Console.Main (string[]) [0x00005] in <d2444ca3d8ea4ff2aab762e98017c09d#97F8BA07-4546-E473-D9B6-2BD796223972>:0
  at (wrapper runtime-invoke) object.runtime_invoke_dynamic (intptr,intptr,intptr,intptr) [0x0001e] in <29ae26ecc93e43cc81f0b710ca2b2e92#97F8BA07-4546-E473-D9B6-2BD796223972>:0

Expected Behavior

No test failures / crashes.

On which platforms did you notice this

[ ] macOS
[ X ] Linux
[ ] Windows

Version Used: master

[akoeplinger]

@vargaz This seems related to 3106c27

[jaykrell]

Don't worry too much about tailcall/2.exe.

[jaykrell]

It is a new test. It actually should be failing, until soon. I didn't put in the support for arm64 yet so it should fail. I have the support in PR.

[akoeplinger]

@jaykrell ok thanks. If you expect the PR to take longer than ~48hrs to land then please disable it in the meantime :)

[jaykrell]

@akoeplinger
It is ready to go here:
#8390

been ready since circa Friday, but CI is flaky or broken.

And the test did fail for me locally which is good, and fixed by this PR.
I don't know how/why it didn't fail in CI earlier.

I suggest it just go in now.

[jaykrell]

Also, tailcall/2.exe might be overly sensitive. It maybe should allow like one frame of divergence, like a diff of 8 or 16 but no more. I ran into this in some runs on some architectures. It might be a general problem with the tailcall tests I've been writing.