Tizen armel CI job frequently hangs

[tannergooding]

If you look at the failures for the armel_cross_checked_tizen_prtest, you will see that they frequently fail due to timeout issues.

On one side, it looks like a given job will hang after some tests have hit an 'Unsupported syscall'.

On other jobs, they will succesfully execute all tests and then stall with Perform an action if the job was performed on an Azure VM Agent. is waiting for a checkpoint on dotnet_coreclr » master » armel_cross_checked_tizen_prtest #### (the job will be waiting on another prtest job, from the same queue, and for an unrelated PR to complete before allowing itself to finish).

The timeout on all of these jobs is currently 4 hours which can quickly cause the queue to get backed up or cause PRs to get delayed for long periods of time.

[tannergooding]

I would like to see one or more of the following happen, as I believe they may help:

• Remove whatever is causing the PR jobs to wait for a previous, unrelated PR job to complete
• Reduce the timeout on these jobs to something like 2 hours, rather than 4 (timeouts start from the time the job actually starts executing, so this shouldn't be an issue)
• Have the jobs pre-emptively abort after the first test failure
• Have some monitor that will abort the run if a given test executes for more than some predetermined time

[tannergooding]

Similar issues seem to impact the arm_cross_debug_ubuntu_prtest job.

[jkotas]

@RussKeldorph Have we looked into what can be done about this?

[RussKeldorph]

@hseok-oh @hqueue Is there someone that can look into the armel failures? We're going to have to remove these from PR testing if they can't be made reliable more or less immediately.

[danmoseley]

cc @Anipik

[hseok-oh]

@RussKeldorph You can find failure began after dotnet/coreclr#15878 is merged.

Here is ci's first failure job for push master.
https://ci.dot.net/job/dotnet_coreclr/job/master/job/armel_cross_debug_tizen/2510/

Here is ci's last success job for push master.
https://ci.dot.net/job/dotnet_coreclr/job/master/job/armel_cross_debug_tizen/2509/

[RussKeldorph]

@hseok-oh Thanks. @mikem8361 Can you look into this?

[benaadams]

Might be coincidence but qemu: Unsupported syscall: happens on Tizen then CI jobs start to time out

13:46:04 Skip preparing for GC stress test. Dependent package is not supported on this architecture.
13:46:05 The tests have been prepared
13:46:06 The tests have been prepared
13:46:06 FAILED   - JIT/Directed/coverage/importer/ldelemnullarr2/ldelemnullarr2.sh
13:46:06   BEGIN EXECUTION
13:46:06   /home/coreclr/bin/tests/Windows_NT.x64.Checked/Tests/coreoverlay/corerun ldelemnullarr2.exe
13:46:06   qemu: Unsupported syscall: 389

[mikem8361]

I know nothing about how to debug this on arm. Can we putty to the machine when the tests hang?

Since it is a checked build, can the coreclr logging be enabled (COMPlus_LogEnable env var, etc.)?

My changes really shouldn't have affected Linux/arm, but you never know.

[jkotas]

This should be fixed now

[benaadams]

Looking at recent PRs its still jamming and/or failing with qemu: Unsupported syscall: 389 - 819 Segmentation fault

[jkotas]

The unsupported syscall is tracked by #8614