Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove duplicated ILLink PackageReference and update target version of the SDK to 6.0 #48462

Merged
merged 14 commits into from
Feb 25, 2021

Conversation

ViktorHofer
Copy link
Member

@ViktorHofer ViktorHofer commented Feb 18, 2021

The Arcade.SDK already package refrences the ILLink package. The
duplicate reference in illink.targets caused SDK errors as the Arcade
reference has IsImplicitlyDefined set which doesn't allow an additional
reference with the same identity.

Also, as the ILLink package already exposes the path to the assembly via
its props file, using that instead of manually constructing the path to
the assembly.

The SDK target version update is required as the sequencing of the
ILLink.props file was wrong and is required for this change. This isn't
considered a breaking change, as the SDK's minimum required version
isn't changed.

Fixes #48409
Fixes #44578
Fixes #46960

cc @missymessa

@ViktorHofer ViktorHofer added this to the 6.0.0 milestone Feb 18, 2021
@ViktorHofer ViktorHofer self-assigned this Feb 18, 2021
@ghost
Copy link

ghost commented Feb 18, 2021

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

The Arcade.SDK already package refrences the ILLink package. The
duplicate reference in illink.targets caused SDK errors as the Arcade
reference has IsImplicitlyDefined set which doesn't allow an additional
reference with the same identity.

Also, as the ILLink package already exposes the path to the assembly via
its props file, using that instead of manually constructing the path to
the assembly.

The SDK target version update is required as the sequencing of the
ILLink.props file was wrong and is required for this change. This isn't
considered a breaking change, as the SDK's minimum required version
isn't changed.

Fixes https://github.com/dotnet/runtime

cc @missymessa

Author: ViktorHofer
Assignees: ViktorHofer
Labels:

area-Infrastructure

Milestone: 6.0.0

global.json Show resolved Hide resolved
@ViktorHofer
Copy link
Member Author

Waiting for dotnet/linker#1842 to be merged so that I can consume the newer package.

@ViktorHofer
Copy link
Member Author

@jkoritzinsky @hoyosjs can you please take a look at the coreclr error? Seems like the runtime hardcodes the 5.0.0 shared framework somewhere?

@hoyosjs
Copy link
Member

hoyosjs commented Feb 19, 2021

@ViktorHofer Yes, there are projects that run on 5.0. Namely:

References to `NetCoreAppToolCurrent`
Directory.Build.props
42:    <NetCoreAppToolCurrentVersion>5.0</NetCoreAppToolCurrentVersion>
47:    <NetCoreAppToolCurrent>net$(NetCoreAppToolCurrentVersion)</NetCoreAppToolCurrent>
48:    <NetCoreAppCurrentToolTargetFrameworkMoniker>$(NetCoreAppCurrentIdentifier),Version=v$(NetCoreAppToolCurrentVersion)</NetCoreAppCurrentToolTargetFrameworkMoniker>
67:    <AppleAppBuilderDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'AppleAppBuilder', 'Debug', '$(NetCoreAppToolCurrent)'))</AppleAppBuilderDir>
68:    <AndroidAppBuilderDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'AndroidAppBuilder', 'Debug', '$(NetCoreAppToolCurrent)', 'publish'))</AndroidAppBuilderDir>
69:    <WasmAppBuilderDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'WasmAppBuilder', 'Debug', '$(NetCoreAppToolCurrent)', 'publish'))</WasmAppBuilderDir>
70:    <WasmBuildTasksDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'WasmBuildTasks', 'Debug', '$(NetCoreAppToolCurrent)', 'publish'))</WasmBuildTasksDir>
71:    <MonoAOTCompilerDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'MonoAOTCompiler', 'Debug', '$(NetCoreAppToolCurrent)'))</MonoAOTCompilerDir>

src/coreclr/crossgen-corelib.proj
4:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/runtime.proj
4:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/ToolBox/SOS/DacTableGen/DacTableGen.csproj
4:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/ToolBox/SOS/DIALib/DIALib.ilproj
3:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/tools/aot/crossgen2/crossgen2.csproj
6:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj
7:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj
5:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/tools/aot/ILCompiler.TypeSystem.ReadyToRun.Tests/ILCompiler.TypeSystem.ReadyToRun.Tests.csproj
4:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkGenerator.csproj
5:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj
6:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/tools/r2rdump/R2RDump.csproj
10:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/tools/r2rtest/R2RTest.csproj
6:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/coreclr/tools/runincontext/runincontext.csproj
4:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/installer/tests/Directory.Build.props
12:    <TestInfraTargetFramework>$(NetCoreAppToolCurrent)</TestInfraTargetFramework>

src/mono/llvm/llvm-init.proj
3:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/mono/monoaotcross.proj
3:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/mono/nuget/Microsoft.NET.Runtime.Android.Sample.Mono/Microsoft.NET.Runtime.Android.Sample.Mono.pkgproj
14:    <PackageFile Include="@(_AndroidSampleFiles)" TargetPath="tools\$(NetCoreAppToolCurrent)\" />

src/mono/nuget/Microsoft.NET.Runtime.iOS.Sample.Mono/Microsoft.NET.Runtime.iOS.Sample.Mono.pkgproj
17:    <PackageFile Include="@(_iOSSampleFiles)" TargetPath="tools\$(NetCoreAppToolCurrent)\" />

src/mono/nuget/Microsoft.NET.Runtime.wasm.Sample.Mono/Microsoft.NET.Runtime.wasm.Sample.Mono.pkgproj
14:    <PackageFile Include="@(_wasmSampleFiles)" TargetPath="tools\$(NetCoreAppToolCurrent)\" />

src/mono/sample/Android/AndroidSampleApp.csproj
5:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/mono/sample/HelloWorld/HelloWorld.csproj
4:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/mono/sample/iOS/Program.csproj
5:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/mono/sample/mbr/console/ConsoleDelta.csproj
4:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/mono/sample/mbr/DeltaHelper/DeltaHelper.csproj
3:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/mono/wasm/build/WasmApp.InTree.props
8:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/mono/wasm/build/WasmApp.LocalBuild.props
23:    <_NetCoreAppToolCurrent>net5.0</_NetCoreAppToolCurrent>
32:    <WasmAppBuilderDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'WasmAppBuilder', 'Debug', '$(_NetCoreAppToolCurrent)', 'publish'))</WasmAppBuilderDir>
33:    <WasmBuildTasksDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'WasmBuildTasks', 'Debug', '$(_NetCoreAppToolCurrent)', 'publish'))</WasmBuildTasksDir>
34:    <MonoAOTCompilerDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'MonoAOTCompiler', 'Debug', '$(_NetCoreAppToolCurrent)'))</MonoAOTCompilerDir>

src/mono/wasm/wasm.proj
4:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/tasks/AndroidAppBuilder/AndroidAppBuilder.csproj
3:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/tasks/AotCompilerTask/MonoAOTCompiler.csproj
3:    <TargetFrameworks>$(NetCoreAppToolCurrent)</TargetFrameworks>
29:      <FilesToPackage Include="$(OutputPath)$(NetCoreAppToolCurrent)\$(AssemblyName).dll" TargetPath="tasks" />

src/tasks/AppleAppBuilder/AppleAppBuilder.csproj
3:    <TargetFrameworks>$(NetCoreAppToolCurrent)</TargetFrameworks>

src/tasks/WasmAppBuilder/WasmAppBuilder.csproj
3:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/tasks/WasmBuildTasks/WasmBuildTasks.csproj
3:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/tests/Common/Coreclr.TestWrapper/Coreclr.TestWrapper.csproj
6:    <NuGetTargetMonikerShort>$(NetCoreAppToolCurrent)</NuGetTargetMonikerShort>

src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj
8:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/tests/Common/dir.sdkbuild.props
9:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/tests/Common/Directory.Build.targets
137:          Include="$(ArtifactsBinDir)\WasmAppBuilder\Debug\$(NetCoreAppToolCurrent)\publish\**"

src/tests/Common/external/external.csproj
15:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/tests/Common/test_dependencies/test_dependencies.csproj
5:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/tests/Common/test_dependencies_fs/test_dependencies.fsproj
5:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

src/tests/Directory.Build.props
133:    <NuGetTargetMonikerShort>$(NetCoreAppToolCurrent)</NuGetTargetMonikerShort>

src/tests/Directory.Build.targets
359:          Include="$(ArtifactsBinDir)\WasmAppBuilder\Debug\$(NetCoreAppToolCurrent)\publish\**"
364:          Include="$(ArtifactsBinDir)\MonoAOTCompiler\Debug\$(NetCoreAppToolCurrent)\**"

src/tests/JIT/Directed/tailcall/mutual_recursion.fsproj
10:    <TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>

Then just a few questions, as I don't know if it's safe to just remap NetCoreAppToolCurrent to net6.0:

  1. Do we want to build and run tools we use for the product against a 6.0 version?
  2. I don't know about a lot of the tools that reference the net5.0 TFM. A bunch of them are mono tasks (AppleAppBuilder, WasmAppBuilder) and samles (like AndroidSampleApp). @directhex are those safe to move to net6.0?
  3. @trylek @davidwrighton @AntonLapounov is it safe to move the crossgen2 and dotnet-pgo projects to net6.0?

@ViktorHofer
Copy link
Member Author

Yes, I realized that after posting the comment... The tasks are definitely fine to be upgraded. I think crossgen2 should be fine as well.

The Arcade.SDK already package refrences the ILLink package. The
duplicate reference in illink.targets caused SDK errors as the Arcade
reference has IsImplicitlyDefined set which doesn't allow an additional
reference with the same identity.

Also, as the ILLink package already exposes the path to the assembly via
its props file, using that instead of manually constructing the path to
the assembly.

The SDK target version update is required as the sequencing of the
ILLink.props file was wrong and is required for this change. This isn't
considered a breaking change, as the SDK's minimum required version
isn't changed.
@ViktorHofer
Copy link
Member Author

Apparently, they way that coreclr uses the pkgproj infrastructure causes these failures. Likely because the props and targets files are imported directly there:

.packages/microsoft.dotnet.build.tasks.packaging/6.0.0-beta.21120.1/build/Packaging.targets(747,5): error MSB4018: (NETCORE_ENGINEERING_TELEMETRY=Build) The "GenerateRuntimeDependencies" task failed unexpectedly.
System.MissingMethodException: Method not found: 'Void NuGet.RuntimeModel.JsonObjectWriter..ctor(Newtonsoft.Json.JsonWriter)'.
   at Microsoft.DotNet.Build.Tasks.Packaging.NuGetUtility.WriteRuntimeGraph(String filePath, RuntimeGraph runtimeGraph)
   at Microsoft.DotNet.Build.Tasks.Packaging.GenerateRuntimeDependencies.Execute() in /_/src/Microsoft.DotNet.Build.Tasks.Packaging/src/GenerateRuntimeDependencies.cs:line 144
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask)

@ericstj can you please help out here? I also updated the NuGet dependencies in Arcade and they already flowed into this PR. The binlog shows that the right assembly with the NuGet breaking change is used. It's unclear to me why there's a MissingMethodException.

@ViktorHofer
Copy link
Member Author

OK I got the fix ready in Arcade, and will update this PR tomorrow.

@jkoritzinsky
Copy link
Member

The runtime tests themselves run on the built shared framework, but the xunit wrappers around the tests run on the CLI we pull down. Otherwise when something breaks that xunit uses, we lose all test coverage, which makes the problem harder to track down.

@ViktorHofer
Copy link
Member Author

Thanks Jeremy. I would really love to hear more about what the xunit wrappers are for and why libraries and installer don't need them. Will probably ask about that at some point offline, as this isn't relevant for this PR.

@jkoritzinsky
Copy link
Member

The xunit wrappers execute the test and assert the return code matches the expected value. Without them, we wouldn’t have a good way to execute our tests and get an xunit test results file to upload to AzDO. We could use a different runner or create a new one (which we do for onboarding new targets), but the xunit system works well for now.

@ViktorHofer
Copy link
Member Author

@safern @akoeplinger I think you guys touched the code that handles the xunit asset publishing with xharness. Can you please take a look at the error?

/__w/1/s/.dotnet/sdk/6.0.100-preview.2.21118.7/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.ConflictResolution.targets(108,5): error NETSDK1148: Found multiple publish output files with the same relative path: /__w/1/s/artifacts/bin/Microsoft.XmlSerializer.Generator.Tests/net6.0-Release/browser-wasm/Microsoft.XmlSerializer.Generator.Tests.deps.json, /__w/1/s/artifacts/obj/Microsoft.XmlSerializer.Generator.Tests/net6.0-Release/browser-wasm/Microsoft.XmlSerializer.Generator.Tests.deps.json, /__w/1/s/.packages/xunit.abstractions/2.0.3/lib/netstandard2.0/xunit.abstractions.dll, /__w/1/s/artifacts/bin/WasmTestRunner/net6.0-Release/xunit.abstractions.dll, /__w/1/s/.packages/xunit.extensibility.core/2.4.1/lib/netstandard1.1/xunit.core.dll, /__w/1/s/artifacts/bin/WasmTestRunner/net6.0-Release/xunit.core.dll, /__w/1/s/.packages/xunit.extensibility.execution/2.4.1/lib/netstandard1.1/xunit.execution.dotnet.dll, /__w/1/s/artifacts/bin/WasmTestRunner/net6.0-Release/xunit.execution.dotnet.dll. [/__w/1/s/src/libraries/Microsoft.XmlSerializer.Generator/tests/Microsoft.XmlSerializer.Generator.Tests.csproj]

@ViktorHofer
Copy link
Member Author

The remaining issue are duplicated xunit artifacts in mobile builds that @safern is currently looking at. I expect (or better to say hope) that we get this in by EOD.

Copy link
Member

@safern safern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me.

Copy link
Member

@eerhardt eerhardt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

I still don't understand why we need the empty TestProjects.targets file, but if you think it is valuable, then I'm OK with it.

@ViktorHofer ViktorHofer changed the title Remove duplicated ILLink PackageReference Remove duplicated ILLink PackageReference and update target version of the SDK to 6.0 Feb 24, 2021
@ghost
Copy link

ghost commented Feb 25, 2021

Hello @ViktorHofer!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

@ViktorHofer
Copy link
Member Author

DNS failures are #48751

@ViktorHofer ViktorHofer merged commit 653b310 into dotnet:master Feb 25, 2021
@ViktorHofer ViktorHofer deleted the ILLinkFixPkgReferences branch February 25, 2021 10:23
@sdmaclea
Copy link
Contributor

I thought this was supposed to wait until the March infras rollout. Or am I missing something?

@ViktorHofer
Copy link
Member Author

I thought this was supposed to wait until the March infras rollout. Or am I missing something?

The march rollout updates the SDK's minimum required version (the sdk entry in the repo's global.json file). msbuild checks that version when its being invoked (including VS). The target version that was being updated as part of this PR is used for bootstrapping the repo local SDK.

@ghost ghost locked as resolved and limited conversation to collaborators Mar 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
7 participants