[NAOT][Apple][mobile] Undefined symbols for architecture x86_64: do_vxsort_avx2 causing AppleAppBuilderTask to fail

[matouskozak]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=629964
Build error leg or test failing:

• tvossimulator-x64 Release AllSubsets_NativeAOT_RuntimeTests
• iossimulator-x64 Release AllSubsets_NativeAOT_RuntimeTests

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "/Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: Undefined symbols for architecture x86_64: [/Users/runner/work/1/s/src/tests/build.proj]",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Error log:

normal/x86_64/nativeaot_SmokeTests_UnitTests_dependency_info.dat -o /Users/runner/work/1/s/artifacts/tests/coreclr/obj/iossimulator.x64.Release/Managed/build/iOSApps/nativeaot_SmokeTests_UnitTests/bin-nativeaot_SmokeTests_UnitTests-x64/nativeaot_SmokeTests_UnitTests/Debug-iphonesimulator/nativeaot_SmokeTests_UnitTests.app/nativeaot_SmokeTests_UnitTests [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3624320Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: Undefined symbols for architecture x86_64: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3624960Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:   "do_vxsort_avx2(unsigned char**, unsigned char**, unsigned char*, unsigned char*)", referenced from: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3625650Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::plan_phase(int) in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3626330Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:   "do_vxsort_avx512(unsigned char**, unsigned char**, unsigned char*, unsigned char*)", referenced from: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3627020Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::plan_phase(int) in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3627650Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:   "IsSupportedInstructionSet(InstructionSet)", referenced from: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3628360Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::grow_mark_list() in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3629010Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::plan_phase(int) in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3629730Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::do_post_gc() in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3630350Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:   "InitSupportedInstructionSet(int)", referenced from: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3631020Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::initialize_gc(unsigned long, unsigned long, unsigned long) in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3631690Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: ld: symbol(s) not found for architecture x86_64 [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3632460Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: clang: error: linker command failed with exit code 1 (use -v to see invocation) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3633540Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: note: Run script build phase 'Generate CMakeFiles/ALL_BUILD' will be run during every build because the option to run the script phase "Based on dependency analysis" is unchecked. (in target 'ALL_BUILD' from project 'nativeaot_SmokeTests_UnitTests') [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3634320Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: ** BUILD FAILED ** [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3634840Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: The following build commands failed: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3636140Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: 	Ld /Users/runner/work/1/s/artifacts/tests/coreclr/obj/iossimulator.x64.Release/Managed/build/iOSApps/nativeaot_SmokeTests_UnitTests/bin-nativeaot_SmokeTests_UnitTests-x64/nativeaot_SmokeTests_UnitTests/Debug-iphonesimulator/nativeaot_SmokeTests_UnitTests.app/nativeaot_SmokeTests_UnitTests normal (in target 'nativeaot_SmokeTests_UnitTests' from project 'nativeaot_SmokeTests_UnitTests') [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3637100Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: (1 failure) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3637870Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:    at Utils.RunProcess(TaskLoggingHelper logger, String path, String args, IDictionary`2 envVars, String workingDir, Boolean ignoreErrors, Boolean silent, MessageImportance debugMessageImportance) in /_/src/tasks/Common/Utils.cs:line 123 [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3638790Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:    at Xcode.BuildAppBundle(String xcodePrjPath, Boolean optimized, String devTeamProvisioning) in /_/src/tasks/AppleAppBuilder/Xcode.cs:line 644 [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3639540Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:    at AppleAppBuilderTask.Execute() in /_/src/tasks/AppleAppBuilder/AppleAppBuilder.cs:line 334 [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3640320Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3641250Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3641830Z     0 Warning(s)
2024-04-03T21:32:33.3641990Z     6 Error(s)

Report

Build	Definition	Step Name	Console log	Pull Request
632885	dotnet/runtime	Build Tests	Log	#100389
631365	dotnet/runtime	Build Tests	Log	#100389
630062	dotnet/runtime	Build Tests	Log	#100389
629964	dotnet/runtime	Build Tests	Log

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	4	4

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=629964
Error message validated: [/Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: Undefined symbols for architecture x86_64: [/Users/runner/work/1/s/src/tests/build.proj]]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/4/2024 7:53:18 PM UTC

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger, @kotlarmilos
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to 'os-tvos': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[matouskozak]

Could it be caused by #98712 @cshung @janvorli?

[cshung]

Yes, it is my problem.
For NativeAOT compile on x64 platform, we need to link one more library.
Who would be a good person to talk to about AppleAppBuilderTask?

[am11]

Unlike other smoke tests, seems like mono library-mode smoke tests are using aotsdk from package while the aot runtime build took place with live bits, hence the conflict. It should be using live build for both. @ivanpovazan any idea how to bring library-mode tests to the same plan as rest of the platforms (use live build of nativeaot)?

[ivanpovazan]

Unlike other smoke tests, seems like mono library-mode smoke tests are using aotsdk from package while the aot runtime build took place with live bits, hence the conflict. It should be using live build for both. @ivanpovazan any idea how to bring library-mode tests to the same plan as rest of the platforms (use live build of nativeaot)?

I think that all NativeAOT runtime tests (tests under: https://github.com/dotnet/runtime/tree/main/src/tests) are using aotsdk based on:

runtime/src/tests/Directory.Build.targets

Line 563 in ffb2578

<IlcSdkPath>$(CoreCLRAotSdkDir)</IlcSdkPath>

so I don't think that is the problem.

The problem seems to be specific to inclusion of libRuntime.Vxsort*.a libraries and looking at the binlogs from the failed test build, it seems that when building runtime tests for iossimulator (most probably all ios platforms) _targetArchitecture is not being set, which in turn prevents the required library to be included in NativeLibrary item group and further passed to the native linker when bundling iOS app bundle.

@matouskozak @kotlarmilos do you remember why we had to explicitly set these properties here:

runtime/src/tests/Directory.Build.props

Lines 213 to 216 in e610649

	<PropertyGroup Condition="'$(RuntimeFlavor)' == 'coreclr' and '$(TargetsAppleMobile)' == 'true'">
	<_targetOS>$(TargetOS)</_targetOS>
	<UseNativeAOTRuntime>true</UseNativeAOTRuntime>
	<NativeLib>static</NativeLib>

I would assume we were hitting similar problems for _targetOS, as we are now having with _targetArchitecture not being resolved at the adequate time during build.

[kotlarmilos]

@matouskozak @kotlarmilos do you remember why we had to explicitly set these properties here:

They are used in https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets.

[ivanpovazan]

Yes, but _targetOS (and _targetArchitecture) should be resolved in https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/BuildIntegration/Microsoft.DotNet.ILCompiler.SingleEntry.targets
so I was asking if you remember why did we have to set it manually

[kotlarmilos]

Can you check from the binlog if the condition in Microsoft.DotNet.ILCompiler.SingleEntry.targets is true?
<PropertyGroup Condition="'$(RuntimeIdentifier)' != '' and '$(NETCoreSdkPortableRuntimeIdentifier)' != ''">

[am11]

I think that all NativeAOT runtime tests (tests under: main/src/tests) are using aotsdk based on:

Yes, that is resolved to the live build of runtime. I was suspecting that iOS platforms end up using BuildIntegration targets from the SDK (which is still at preview 1 per global.json), rather than the clr artifacts directory (live build) which causes the missing libvxsort in the logs and lead to the missing symbol during the link. With current situation, any time we will add a new dependency or remove one, it will run into the same error.

Thanks for looking into it everyone. 👍

[cshung]

Ideally, the test should use the target files in the repo, if we could do that, that would be the best.

[ivanpovazan]

@cshung the problem is most probably related to how we configure runtime tests for NativeAOT on iOS, and not with the build integration targets file.

---

@kotlarmilos NETCoreSdkPortableRuntimeIdentifier is not set.

• As a short-term solution, we could probably just add <_targetArchitecture>$(TargetArchitecture)</_targetArchitecture> in the place I linked above.
• As a long-term solution, we might want to investigate how we can configure NativeAOT runtime tests to include Microsoft.NET.Sdk.props and with it Microsoft.NETCoreSdk.BundledVersions.props which defines NETCoreSdkPortableRuntimeIdentifier. This way we would avoid any similar future problems with changes introduced to NativeAOT build (or its dependencies).