[mono] Tests System.IO.Hashing.Tests.Crc32Tests crashing on tvos-arm64

[ivanpovazan]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=371961

Affected pipelines:

• runtime-extra-platforms

Build error legs:

• net8.0-tvos-Release-arm64-Mono_Release-OSX.13.Amd64.AppleTV.Open

Tests failing:

• System.IO.Hashing.Tests.Crc32Tests.StaticVerifyOneShotSpanToArray

Crash log: https://gist.github.com/ivanpovazan/d459960870e36af4a23c2fbddce14aeb

Pull request: #90023

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "at System.IO.Hashing.Tests.Crc32Tests",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=371961
Error message validated: at System.IO.Hashing.Tests.Crc32Tests
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 8/14/2023 10:36:45 PM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[ghost]

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger, @kotlarmilos
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=371961

Affected pipelines:

• runtime-extra-platforms

Build error legs:

• net8.0-tvos-Release-arm64-Mono_Release-OSX.13.Amd64.AppleTV.Open

Tests failing:

• System.IO.Hashing.Tests.Crc32Tests.StaticVerifyOneShotSpanToArray

Crash log: https://gist.github.com/ivanpovazan/d459960870e36af4a23c2fbddce14aeb

Pull request: #90023

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "at System.IO.Hashing.Tests.Crc32Tests:StaticVerifyOneShotSpanToArray",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	ivanpovazan
Assignees:	-
Labels:	blocking-clean-ci, os-ios, Known Build Error
Milestone:	8.0.0

[ivanpovazan]

For some reason the Build analysis is not picking up these failures.

Going through history of runtime-extra-platforms it seems this failure was shadowed by tvos-arm64 runs timing out.
However, it can be seen that a log from 3 weeks ago: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-66c0274ea26a40c9a0/System.IO.Hashing.Tests.Attempt.3/1/net.dot.System.IO.Hashing.Tests.log?helixlogtype=result also has the same crashing message.

I tried to reproduce this locally on an ios-arm64 device but with no luck.

@vargaz do you have an idea what could have caused this regression?

[vargaz]

Is this reproducible on CI ?

[ivanpovazan]

Is this reproducible on CI ?

Yes, it is a consistent failure for some time now. It basically fails on every runtime-extra-platforms run, on the tvos-arm64 Release AllSubsets_Mono leg

Here is the crash log: https://gist.github.com/ivanpovazan/d459960870e36af4a23c2fbddce14aeb

[vargaz]

The crash log is not very useful. Maybe it happens only on tvos, or do we have a different build on tvos ?

[ivanpovazan]

Maybe it happens only on tvos

Yes that seems to be the case - at least from analysing the runtime-extra-platforms failures.

On the other hand, System.IO.Hashing tests are not being tested on ios-arm64 on CI, so I tried it locally with the same build settings as tvos-arm64 is using (as the platforms are pretty similar), but could not reproduce.

We do have couple of 'mysterious' tvos-arm64 failures which were excluded before:

runtime/src/libraries/tests.proj

Lines 263 to 268 in 9d53816

	<!--
	Mysterious crashes on tvOS
	-->
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Formats.Cbor\tests\System.Formats.Cbor.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.IO.MemoryMappedFiles\tests\System.IO.MemoryMappedFiles.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Runtime.Numerics\tests\System.Runtime.Numerics.Tests.csproj" />

Maybe this one goes in the same bucket.

[kotlarmilos]

I've noticed that the bundle size for iOS is significantly smaller 232.3 MB, compared to tvOS which is 1009.3 MB. Additionally, the architectures differ, so it might be good testing with tvos-arm64 Helix queue.

[ivanpovazan]

I've noticed that the bundle size for iOS is significantly smaller 232.3 MB, compared to tvOS which is 1009.3 MB. Additionally, the architectures differ, so it might be good testing with tvos-arm64 Helix queue.

Yes on CI, on iOS device we run only the following libraries tests:

• System.IO.MemoryMappedFiles.Tests
• System.Runtime.Numerics.Tests
• System.Runtime.Tests

Additionally, the architectures differ

Sorry, not sure I understand which architectures?

[kotlarmilos]

I've noticed that the bundle size for iOS is significantly smaller 232.3 MB, compared to tvOS which is 1009.3 MB. Additionally, the architectures differ, so it might be good testing with tvos-arm64 Helix queue.

Yes on CI, on iOS device we run only the following libraries tests:

• System.IO.MemoryMappedFiles.Tests
• System.Runtime.Numerics.Tests
• System.Runtime.Tests

When I compared bundle size of the System.IO.Hashing.Tests app built for iOS device locally, it was much smaller compared to the bundle for tvOS device built on the CI.

Additionally, the architectures differ

Sorry, not sure I understand which architectures?

The CI is using the OSX.13.Amd64.AppleTV.Open Helix queue, while locally I built for arm64 iOS device. Have you tried to reproduce the issue on x64 iOS device?

[ivanpovazan]

When I compared bundle size of the System.IO.Hashing.Tests app built for iOS device locally, it was much smaller compared to the bundle for tvOS device built on the CI.

Ah I see. Can we download the bundle somehow from helix or mimic how AOTing is performed on Helix - locally?

The CI is using the OSX.13.Amd64.AppleTV.Open Helix queue, while locally I built for arm64 iOS device. Have you tried to reproduce the issue on x64 iOS device?

The Amd64 in the queue name (afaik) just indicates that the host machine is x64, but it then cross compiles for ios/tvos-arm64 (there are no ios-x64 devices only simulators). But yeah, another difference is that it uses a cross-compiler.

[kotlarmilos]

Ah I see. Can we download the bundle somehow from helix or mimic how AOTing is performed on Helix - locally?

Unfortunatelly we don't publish app bundles as artifacts.

The Amd64 in the queue name (afaik) just indicates that the host machine is x64, but it then cross compiles for ios/tvos-arm64 (there are no ios-x64 devices only simulators). But yeah, another difference is that it uses a cross-compiler.

Correct, sorry for confusion, my error.

[ivanpovazan]

I wonder if we could do the same thing that @steveisok did here: aac729f#diff-efa78f3f0523ef8b61f94772d9c204608051d411527cb9be83ca60266dc1d2f9
Where tests that were failing on tvos-arm64 got excluded, and enabled on ios-arm64 keeping the same coverage.

But tracking the actual fix separately.

[kotlarmilos]

Good idea. If we have capacity, we can do that. I suggest we enable it and track load on Grafana dashboards.