Update to arcade 6.0 #6143

benvillalobos · 2021-02-10T01:03:44Z

Fixes #6119

Context

This is to get MSBuild on arcade powered source-build.

Changes Made

Ran darc update-dependencies --source-repo arcade --channel ".NET 5 Eng - Latest"

3/3: Ran darc update-dependencies --source-repo arcade --channel ".NET Eng - Latest"

Testing

CI is enough.

Notes

I expect this PR to be in flight as I chip away at it over time. Or it will magically work first time through.

For reviewers: The first commit is the biggest diff and was entirely generated by running the darc command above, so you may want to start at commit 2 when it comes up.

benvillalobos · 2021-02-10T02:22:40Z

PR looking good so far.

Should look into this before merging. May just need to update a path to look for 3.1.

Forgind · 2021-02-10T15:42:50Z

I thought the whole reason we didn't go straight to 6.0 was that we didn't want preview bits in release MSBuild?

benvillalobos · 2021-02-10T18:14:11Z

@Forgind that's at odds with what source-build needs, which is for us to be on arcade 6. @marcpopMSFT thoughts? arcade 6 is LTS FYI.

benvillalobos · 2021-02-10T18:53:31Z

VS PR: https://dev.azure.com/devdiv/DevDiv/_git/VS/pullrequest/303587

benvillalobos · 2021-02-10T22:58:23Z

The VS PR failed almost identically to https://dev.azure.com/devdiv/DevDiv/_git/VS/pullrequest/303587, which suggests that merging arcade 6 won't cause any issues (famous last words).

We should wait until master is unblocked before merging this.

benvillalobos · 2021-03-04T02:22:54Z

@epinanth, is there a channel for "I want to be on and stay on arcade 6" yet?

benvillalobos · 2021-03-04T20:05:54Z

@epananth friendly ping (mispelled your username in my previous post). Seeing a lot of strange errors running darc update-dependencies --source-repo arcade --channel ".NET Eng - Latest". Should we not be using this channel if we want to be on arcade 6?

epananth · 2021-03-04T22:27:00Z

Going to look at this next week, I'm on FR next week

benvillalobos · 2021-03-09T20:52:10Z

@epananth friendly ping 🙂

epananth · 2021-03-09T20:55:18Z

Looking into this one today, I will post an update soon.

benvillalobos · 2021-03-23T01:17:41Z

Notes to self: I was wondering why this succeeded on my local machine but was seeing "sdk not found" issues for CI builds. Turns out I had the net5 sdk installed under the .dotnet folder in the repo. Time to start updating projects to net6.0

benvillalobos · 2021-03-23T18:09:43Z

We're hitting this breaking change by updating to net6.0: dotnet/docs#19625

Thread.ResetAbort() is obsolete: https://source.dot.net/#System.Private.CoreLib/Thread.cs,47d38cd18e02048c

Looks like we can just ifdef around this.

benvillalobos · 2021-03-31T22:40:48Z

I can't seem to repro the (no bootstrap) build. Even after running git clean -xdf and removing my 6.0.100 preview sdk.

Got it to repro on linux and seeing the same "could not find nuget.frameworks.dll" error. Not sure what the root cause is.

Found this when digging through the assembly

// NuGet.Frameworks, Version=5.10.0.7148, Culture=neutral, PublicKeyToken=31bf3856ad364e35
	// Assembly reference loading information:
	// There were some problems during assembly reference load, see below for more information!
	// Error: Could not find reference: NuGet.Frameworks, Version=5.10.0.7148, Culture=neutral, PublicKeyToken=31bf3856ad364e35

ProcessFrameworkReferences is an sdk task, any ideas @dsplaisted?

Forgind · 2021-03-31T22:54:39Z

Try merging in master?

benvillalobos · 2021-03-31T23:41:18Z

@Forgind Not sure what you mean? latest/master into this branch?

It was updated to latest main recently.

Forgind · 2021-04-01T00:09:49Z

The PR I was thinking might have fixed it went in this morning. Doesn't look like you've picked it up yet. I've seen some issues with NuGet.Frameworks that looked very similar to that from the change to assembly loading.

benvillalobos · 2021-04-01T18:28:12Z

It's worth a shot!

benvillalobos · 2021-04-01T23:18:54Z

/azp run

azure-pipelines · 2021-04-01T23:19:04Z

Azure Pipelines successfully started running 1 pipeline(s).

benvillalobos · 2021-04-28T23:50:28Z

Tried building from a VM with a clean VS install and running eng\cibuild_bootstrapped_msbuild.cmd and no repro.

rainersigwald · 2021-05-03T22:27:58Z

eng/Version.Details.xml

@@ -14,4 +14,10 @@
      <Sha>e9fd4dc7d74932c0d4b042251bc5a88bb5b3c437</Sha>
    </Dependency>
  </ToolsetDependencies>
+  <ProductDependencies>
+    <Dependency Name="Microsoft.Extensions.DependencyModel" Version="6.0.0-preview.2.21154.6">


Self, is this actually a product dependency?

rainersigwald · 2021-05-03T22:28:27Z

eng/Versions.props

    <MicrosoftNetCompilersToolsetVersion>3.9.0-2.20574.26</MicrosoftNetCompilersToolsetVersion>
-    <NuGetBuildTasksVersion>5.9.1-rc.8</NuGetBuildTasksVersion>
+    <NuGetBuildTasksVersion>5.10.0-preview.1.7148</NuGetBuildTasksVersion>


Shouldn't darc have been doing this for us? Are we missing a subscription?

You're right: #6327 looks to be the last time this version updated automatically.

I'm thinking it was either the master->main rename or the switch from nuget.client.trusted to nuget.client that broke this.

Just going to log my thought process here.

Likely not due to the switch from nuget.client.trusted since we got a few PR's since then. It's more likely that it's darc.

Looks like there's no sub for it.
darc get-subscriptions --channel "VS 16.10"

https://github.com/dotnet/fsharp (VS 16.10) ==> 'https://github.com/dotnet/sdk' ('main') - Id: 1b3ed604-a240-43c5-f3e6-08d8e97c775d - Update Frequency: EveryDay - Enabled: True - Batchable: False - Merge Policies: [] - Last Build: 20210427.5 (b6a352ab0a11a0584348dd77811af7eecaf9d072) https://github.com/dotnet/fsharp (VS 16.10) ==> 'https://github.com/dotnet/sdk' ('release/5.0.3xx') - Id: 8d76063a-5130-4369-8278-08d8e9754760 - Update Frequency: EveryDay - Enabled: True - Batchable: False - Merge Policies: Standard - Last Build: 20210430.7 (e411df7595bbe379f07dd2bcbb6e4046eca9eb31) https://github.com/dotnet/fsharp (VS 16.10) ==> 'https://github.com/dotnet/sdk' ('release/6.0.1xx-preview4') - Id: fec82357-f505-450c-c78c-08d8ff796cf5 - Update Frequency: EveryBuild - Enabled: True - Batchable: False - Merge Policies: Standard - Last Build: 20210430.7 (e411df7595bbe379f07dd2bcbb6e4046eca9eb31) https://github.com/dotnet/msbuild (VS 16.10) ==> 'https://github.com/dotnet/sdk' ('main') - Id: 9b9c2fc1-dee5-447e-827b-08d8e9754760 - Update Frequency: EveryDay - Enabled: True - Batchable: False - Merge Policies: Standard - Last Build: 20210428.2 (5b9216a75e98e19eba84e04a5f30bd35a68f317a) https://github.com/dotnet/msbuild (VS 16.10) ==> 'https://github.com/dotnet/sdk' ('release/5.0.3xx') - Id: 08d8569c-a921-4400-1fc7-08d8c8766580 - Update Frequency: EveryDay - Enabled: True - Batchable: False - Merge Policies: Standard - Last Build: 20210428.2 (5b9216a75e98e19eba84e04a5f30bd35a68f317a) https://github.com/dotnet/msbuild (VS 16.10) ==> 'https://github.com/dotnet/sdk' ('release/6.0.1xx-preview4') - Id: f3da650c-3ec0-4a87-5a84-08d8ff79a3bc - Update Frequency: EveryBuild - Enabled: True - Batchable: False - Merge Policies: Standard - Last Build: 20210503.2 (fa96a2a81e0fb8c028057fa204bbf386bfb36aec) https://github.com/dotnet/roslyn (VS 16.10) ==> 'https://github.com/dotnet/arcade' ('main') - Id: 71f51b42-1554-444f-7918-08d8e262a02c - Update Frequency: EveryWeek - Enabled: True - Batchable: True - Merge Policies: Standard - Last Build: 20210501.8 (4c32f5e4e9c0828a94fd4d1c9c0994082c85aaf3) https://github.com/microsoft/vstest (VS 16.10) ==> 'https://github.com/dotnet/sdk' ('release/5.0.3xx') - Id: b4754456-bab5-48e6-093c-08d8ff7930ba - Update Frequency: EveryBuild - Enabled: True - Batchable: False - Merge Policies: [] - Last Build: 20210429-01 (1050e72d7a92a5fb5737aa6a464e8e43e7dc52f4) https://github.com/nuget/nuget.client (VS 16.10) ==> 'https://github.com/dotnet/sdk' ('main') - Id: fcb199f6-ff33-44a0-f3ef-08d8e97c775d - Update Frequency: EveryBuild - Enabled: True - Batchable: False - Merge Policies: [] - Last Build: 5.10.0.7228 (d2bb0de35242b04802c1a7856ebb7718f674565c) https://github.com/nuget/nuget.client (VS 16.10) ==> 'https://github.com/dotnet/sdk' ('release/5.0.3xx') - Id: 61b07feb-68a2-4b3c-64a4-08d8e9757c68 - Update Frequency: EveryBuild - Enabled: True - Batchable: False - Merge Policies: [] - Last Build: 5.10.0.7228 (d2bb0de35242b04802c1a7856ebb7718f674565c) https://github.com/nuget/nuget.client (VS 16.10) ==> 'https://github.com/dotnet/sdk' ('release/6.0.1xx-preview4') - Id: 5184e9dc-a7d8-43a0-c78a-08d8ff796cf5 - Update Frequency: EveryBuild - Enabled: True - Batchable: False - Merge Policies: Standard - Last Build: 5.10.0.7240 (dca1d060f38e1e02f6bfca41e25f081f19fd534b)

Correctomundo! darc add-subscription --channel "VS 16.10" --source-repo "https://github.com/nuget/nuget.client" --target-repo "https://github.com/dotnet/msbuild" --target-branch main --update-frequency "everyWeek" --trigger

See #6405

rainersigwald · 2021-05-03T22:29:08Z

ref/Microsoft.Build/Microsoft.Build.cs

@@ -0,0 +1,2158 @@
+// Copyright (c) Microsoft. All rights reserved.


Something's up here; this shouldn't be a new file.

rainersigwald · 2021-05-03T22:29:46Z

src/Directory.BeforeCommon.targets

@@ -122,9 +122,10 @@
    <DefineConstants>$(DefineConstants);FEATURE_RUNTIMEINFORMATION</DefineConstants>
    <DefineConstants>$(DefineConstants);USE_MSBUILD_DLL_EXTN</DefineConstants>
    <DefineConstants>$(DefineConstants);WORKAROUND_COREFX_19110</DefineConstants>
+    <DefineConstants Condition="$([MSBuild]::IsTargetFrameworkCompatible('$(TargetFramework)', 'net6.0'))">$(DefineConstants);NET6_PLUS</DefineConstants>


IIRC the SDK defines something much like this. Can we use that instead?

running rg -Fi "<defineconstants" -tmsbuild in the sdk doesn't look promising. Sounds like something we can merge into the SDK to allow compiler directives in code for breaking changes between sdk versions. Something like:

<DefineConstants Condition="$([MSBuild]::IsTargetFrameworkCompatible('$(TargetFramework)', 'netcoreapp3.1'))">$(DefineConstants);NET31_PLUS</DefineConstants> <DefineConstants Condition="$([MSBuild]::IsTargetFrameworkCompatible('$(TargetFramework)', 'net5.0'))">$(DefineConstants);NET5_PLUS</DefineConstants> <DefineConstants Condition="$([MSBuild]::IsTargetFrameworkCompatible('$(TargetFramework)', 'net6.0'))">$(DefineConstants);NET6_PLUS</DefineConstants>

@dsplaisted does this sound reasonable?

Questions answered with some greppin'

Where is DefineConstants even consumed?

Looks like it's passed as an argument to the compiler. https://github.com/dotnet/msbuild/blob/main/src/Tasks/Microsoft.CSharp.CurrentVersion.targets#L249

dotnet/sdk#14798 added it; in this case it's NET6_0_OR_GREATER.

* darc update-dependencies --source-repo arcade --channel '.NET Eng - Latest' * update DotNetCliVersion to match global.json

Because we use binaries from the initial .NET SDK to build up our bootstrap directory, we're tied to the runtime version of that SDK. Since we will target .NET 6 for 17.0, this is a reasonable move now. * Keep ref file for .NET 6 in netstandard * This isn't strictly correct but is where we have been keeping ".NET Core" API surface. * Use NET6_0_OR_GREATER preprocessor symbol * This API is no longer available in .NET 6. * Update bootstrap build scripts to look for net6.0 binaries * Update Microsoft.Extensions.DependencyModel * This needs to roughly match the SDK that we're assembling in the bootstrap folder.

This should fix CI builds running on older VS images.

rainersigwald

LGTM pending #6408.

Forgind · 2021-05-10T17:29:15Z

eng/Version.Details.xml

    </Dependency>
    <Dependency Name="NuGet.Build.Tasks" Version="5.10.0-rc.7240">
      <Uri>https://github.com/nuget/nuget.client</Uri>
      <Sha>dca1d060f38e1e02f6bfca41e25f081f19fd534b</Sha>
    </Dependency>
+    <Dependency Name="Microsoft.Extensions.DependencyModel" Version="6.0.0-preview.2.21154.6">


Do we own this/should we be specifying its version? Do we have a plan for keeping it up-to-date with the version specified in Versions.props?

yes: darc subscription. Can't set it up until this merges though.

MichaelSimons · 2021-05-21T15:48:33Z

Can this get merged now? The one issue I saw mentioned as blocking appears to have been merged - #6408. This is blocking #6387. Thanks

rainersigwald · 2021-05-21T15:58:22Z

@MichaelSimons Not yet but Soon™. We have finally gotten unblocked on checking in changes to VS and are working through backlog. This is high on the priority list.

benvillalobos force-pushed the arcade-6.0-update branch from 0948169 to 9dee02a Compare February 10, 2021 18:04

benvillalobos added the merge-when-branch-open PRs that are approved, except that there is a problem that means we are not merging stuff right now. label Feb 10, 2021

benvillalobos removed the merge-when-branch-open PRs that are approved, except that there is a problem that means we are not merging stuff right now. label Mar 4, 2021

Base automatically changed from master to main March 15, 2021 20:09

benvillalobos force-pushed the arcade-6.0-update branch from 461dc86 to 198c784 Compare April 1, 2021 21:47

rainersigwald force-pushed the arcade-6.0-update branch from e139569 to f9a923d Compare April 16, 2021 21:14

benvillalobos mentioned this pull request Apr 20, 2021

Move to V3 publishing #6349

Merged

1 task

MichaelSimons mentioned this pull request Apr 29, 2021

Onboarding to ArPow (arcade-powered source-build) #6387

Merged

rainersigwald reviewed May 3, 2021

View reviewed changes

rainersigwald force-pushed the arcade-6.0-update branch from a8bb518 to a3dd38b Compare May 4, 2021 22:04

benvillalobos and others added 4 commits May 4, 2021 17:06

Update to arcade 6.0

8e34de8

* darc update-dependencies --source-repo arcade --channel '.NET Eng - Latest' * update DotNetCliVersion to match global.json

Update build scripts to match Arcade netcoreapp3.1

72be073

Opt into prerel SDKs when in VS/MSBuild.exe-driven builds

29e17e9

This should fix CI builds running on older VS images.

rainersigwald force-pushed the arcade-6.0-update branch from a3dd38b to 29e17e9 Compare May 4, 2021 22:06

rainersigwald approved these changes May 4, 2021

View reviewed changes

Forgind reviewed May 10, 2021

View reviewed changes

Forgind approved these changes May 10, 2021

View reviewed changes

Forgind added the merge-when-branch-open PRs that are approved, except that there is a problem that means we are not merging stuff right now. label May 14, 2021

Forgind merged commit 97ba42a into dotnet:main May 24, 2021

		@@ -0,0 +1,2158 @@
		// Copyright (c) Microsoft. All rights reserved.

Update to arcade 6.0 #6143

Update to arcade 6.0 #6143

Conversation

benvillalobos commented Feb 10, 2021 • edited Loading

Context

Changes Made

Testing

Notes

benvillalobos commented Feb 10, 2021

Forgind commented Feb 10, 2021

benvillalobos commented Feb 10, 2021

benvillalobos commented Feb 10, 2021

benvillalobos commented Feb 10, 2021

benvillalobos commented Mar 4, 2021

benvillalobos commented Mar 4, 2021

epananth commented Mar 4, 2021

benvillalobos commented Mar 9, 2021

epananth commented Mar 9, 2021

benvillalobos commented Mar 23, 2021

benvillalobos commented Mar 23, 2021

benvillalobos commented Mar 31, 2021

Forgind commented Mar 31, 2021

benvillalobos commented Mar 31, 2021

Forgind commented Apr 1, 2021

benvillalobos commented Apr 1, 2021

benvillalobos commented Apr 1, 2021

azure-pipelines bot commented Apr 1, 2021

benvillalobos commented Apr 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rainersigwald left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelSimons commented May 21, 2021

rainersigwald commented May 21, 2021

benvillalobos commented Feb 10, 2021 •

edited

Loading