Update to 5.0 GA #1845

dseefeld · 2020-10-28T22:27:30Z

No description provided.

eng/Version.Details.xml

patches/installer/0008-Update-blob-versions-to-use-ref-pkg-versions-to-trac.patch

repos/linker.proj

repos/runtime.proj

Note: Removed aspnetcore version number patch because it is set to a stable 5.0 version. Note: Removed roslyn Humanizer removal patch because there is now more usage of Humanizer in the code base. May need to redo this patch or add Humanizer. (See dotnet#1767)

Include patch to remove import of restore/harvestPackages.targets in runtime libraries build to fix issue with portable build

dseefeld · 2020-11-03T16:07:06Z

@dagood @crummel Any more comments on this? I think it will be ready to merge as soon as it goes green.

dagood · 2020-11-03T16:11:51Z

Filed an issue for my remaining comment from earlier. Going through it again now.

dagood · 2020-11-03T16:12:57Z

Directory.Build.props

@@ -20,7 +20,7 @@
    <Platform Condition="'$(Platform)' == '' AND '$(BuildArchitecture)' == 'arm64'">$(BuildArchitecture)</Platform>
    <Platform Condition="'$(Platform)' == ''">x64</Platform>

-    <UseStableVersions Condition="'$(UseStableVersions)' == ''">false</UseStableVersions>
+    <UseStableVersions Condition="'$(UseStableVersions)' == ''">true</UseStableVersions>


Can we delete this line? #1627

No. Without it, Roslyn wasn't building stable versions.

Roslyn wasn't building stable versions.

Why is that a goal? The Microsoft Roslyn build didn't produce stable versions.
https://maestro-prod.westus2.cloudapp.azure.com/1297/https:%2F%2Fgithub.com%2Fdotnet%2Froslyn/68310/graph

And in Version.Details.xml, <Dependency Name="Microsoft.NETCore.Compilers" Version="3.8.0-5.20519.18" ...

vs.

[14:25:41.87] Building 'roslyn'...done New NuGet package(s) after building roslyn: -> Microsoft.CodeAnalysis.Build.Tasks 3.8.0 -> Microsoft.CodeAnalysis.Common 3.8.0

Hmm... A later repo was looking for a stable version and wasn't finding it. I was mistaken in my fix for that.

I think it's fine to leave #1627 tracking it for followup but wanted to dig into any issues to make it easier to tackle later.

Without the patch, the production build produces Microsoft.NETCore.Compilers.3.8.0-dev.20519.18.nupkg. The "dev" version doesn't seem right.

That sounds like a missing -ci or OfficialBuildId or something along those lines.

I didn't change roslyn.proj. It has OfficialBuild=true here:

source-build/repos/roslyn.proj

Line 12 in 92d2aef

<BuildCommandArgs>$(BuildCommandArgs) /p:OfficialBuild=true</BuildCommandArgs>

Adding --ci to roslyn.proj caused it to generate the desired version. Microsoft.Net.Compilers.Toolset.3.8.0-5.20519.18.nupkg I'll fix this under #1627 once this PR is merged

tools-local/prebuilt-baseline-offline.xml

crummel

LGTM, just one question about the ASP.NET version.

patches/aspnetcore/0005-Fix-version-number.patch

… 5.0GA

tmds · 2020-11-13T14:44:11Z

patches/runtime/0026-Pin-System.Security.Cryptography.Cng-to-single-versi.patch

+results in an error:
+   An item with the same key has already been added.
+   Key: (System.Security.Cryptography.Cng, 5.0.0)
+---


@dseefeld @dagood does this mean something doesn't allow us to consume System.Security.Cryptography.Cng 5.0.0 to produce System.Security.Cryptography.Cng 5.0.0?

Yes, that's what I was seeing. Here's the error I was seeing without this patch:

3000 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : An item with the same key has already been added. Key: (System.Security.Cryptography.Cng, 5.0.0) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3001 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3002 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at System.Linq.Enumerable.ToDictionary[TSource,TKey](List`1 source, Func`2 keySelector, IEqualityComparer`1 comparer) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3003 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at NuGet.Commands.LockFileBuilder.CreateLockFile(LockFile previousLockFile, PackageSpec project, IEnumerable`1 targetGraphs, IReadOnlyList`1 localRepositories, RemoteWalkContext context) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3004 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at NuGet.Commands.RestoreCommand.ExecuteAsync(CancellationToken token) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3005 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at NuGet.Commands.RestoreRunner.ExecuteAsync(RestoreSummaryRequest summaryRequest, CancellationToken token) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3006 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at NuGet.Commands.RestoreRunner.ExecuteAndCommitAsync(RestoreSummaryRequest summaryRequest, CancellationToken token) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3007 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at NuGet.Commands.RestoreRunner.CompleteTaskAsync(List`1 restoreTasks) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3008 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at NuGet.Commands.RestoreRunner.RunAsync(IEnumerable`1 restoreRequests, RestoreArgs restoreContext, CancellationToken token) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3009 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at NuGet.Commands.RestoreRunner.RunAsync(RestoreArgs restoreContext, CancellationToken token) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3010 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at NuGet.Build.Tasks.BuildTasksUtility.RestoreAsync(DependencyGraphSpec dependencyGraphSpec, Boolean interactive, Boolean recursive, Boolean noCache, Boolean ignoreFailedSources, Boolean disableParallel, Boolean force, Boolean forceEvaluate, Boolean hideWarningsAndErrors, Boolean restorePC, Boolean cleanupAssetsForUnsupportedProjects, ILogger log, CancellationToken cancellationToken) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj] 3011 /src/tb-5.0GA/.dotnet/sdk/5.0.100-rc.1.20452.10/NuGet.RestoreEx.targets(10,5): error : at NuGet.Build.Tasks.Console.MSBuildStaticGraphRestore.RestoreAsync(String entryProjectFilePath, IDictionary`2 globalProperties, IReadOnlyDictionary`2 options) [/src/tb-5.0GA/src/runtime.cf258a14b70ad9069470a108f13765e0e5988f51/Build.proj]

Doesn't this happen for a lot of packages since we're consuming 5.0.0 prebuilt packages to build 5.0.0 source-built packages? Or is issue somehow specific to the Cng package?

I don't know why this issue popped up for this specific package. We could certainly try to remove the patch and see if the issue repros. It may have been a "point in time" issue.

Here's some more info from our old chat about it. dotnet/runtime seems to be doing something specific that aggravates some NuGet bug. It won't be a point in time issue, I doubt the NuGet issue will be fixed in our timeline. I was able to put together a simplified repro, but we dropped it because we probably don't actually need a fix here. (And we always want to cut work because time is not something we have much of.)

dagood: [in response to a binlog] Yeah, good old RestoreTaskEx... another issue we have with static restore is #1825 FWIW, but unrelated.

...

Seems like we might be running into NuGet/Home#6795 ? But not sure why we would but the Microsoft build wouldn't...

[10/30 9:51 AM] Davis Goodin
Hmm, noticed a few things earlier in those RestoreTaskEx logs:

Could not parse version for LatestPackage System.Security.Cryptography.Cng, will use latest stable. ... Installing System.Security.Cryptography.Cng 5.0.0.

(And actually the lines do seem scrambled around that installing line, so not sure which project it's coming from.)

[10/30 9:52 AM] Davis Goodin
I guess some other project is referring to the Cng package. Not totally sure that's important though.

[10/30 9:55 AM] Davis Goodin
Hmm. Wonder if it's choking on this, where in the prod build we don't have a PVP value but in the tarball build we're overwriting it?
https://github.com/dotnet/runtime/blob/7ef6d50b312217d2f7c17b9697891fa8ab98a19d/src/libraries/System.Security.Cryptography.Pkcs/src/System.Security.Cryptography.Pkcs.csproj#L639-L642

<ItemGroup Condition="$(TargetFramework.StartsWith('netstandard2.0'))"> <PackageReference Include="System.Buffers" Version="$(SystemBuffersVersion)" /> <PackageReference Include="System.Memory" Version="$(SystemMemoryVersion)" /> <PackageReference Include="System.Security.Cryptography.Cng" Version="$(SystemSecurityCryptographyCngVersion)" />

[10/30 9:57 AM] Davis Goodin
For a sanity check... does the PVP being passed in in your local build have SystemSecurityCryptographyCngVersion defined?

[10/30 9:58 AM] Dan Seefeldt
Yes.
[10/30 9:59 AM] Dan Seefeldt
The difference between production and tarball is that production build has it defined as 4.7.0 while the tarball build is 5.0.0.

[10/30 10:40 AM] Davis Goodin
Got it, looks like TargetFrameworks is part of the key for NuGet to mash them together and fail: https://github.com/dagood/repro/tree/repro-static-restore-ref-mashup

Oh huh. Static restore is not necessary to repro.

(I was wondering if we could also fix it by disabling static restore.)

After mulling over what Tom's suggesting (and Wes suggested in the past) I think the real long term fix is to stop building all TFMs except net5.0... (so that we don't have this PackageReference to begin with) [referring to #1763]

dseefeld: BTW, forcing System.Security.Cryptography.Cng to 4.7.0 fixed the issue.

4.7.0 isn't a problem because we should be able to build it in SBRP, so we should be able to leave it at that.

This info probably should have made it to the patch, or an issue linked to from the patch. (But we're rushing.)

@dagood since you have a small reproducer, it seems worth filing a bug in https://github.com/NuGet/Home (even if it won't be fixed for our 5.0).

Agreed, but I don't feel I have time to make it make enough sense for discussion. Filed #1888 to track it though.

FYI, I've filed the issue for this: NuGet/Home#10368.

dseefeld self-assigned this Oct 28, 2020

dagood reviewed Oct 29, 2020

View reviewed changes

eng/Version.Details.xml Outdated Show resolved Hide resolved

patches/installer/0008-Update-blob-versions-to-use-ref-pkg-versions-to-trac.patch Show resolved Hide resolved

repos/linker.proj Show resolved Hide resolved

repos/runtime.proj Outdated Show resolved Hide resolved

dagood mentioned this pull request Oct 30, 2020

Consider adding Humanizer to source-build #1767

Closed

dseefeld added 18 commits October 30, 2020 19:30

Update to 5.0.100-rtm.20521.5, BAR Build ID 68494

cde2f44

Update previously source-built to N-1

754ba87

Reconcile patches

9d1cbb7

Note: Removed aspnetcore version number patch because it is set to a stable 5.0 version. Note: Removed roslyn Humanizer removal patch because there is now more usage of Humanizer in the code base. May need to redo this patch or add Humanizer. (See dotnet#1767)

Update to 5.0.100-rtm.20522.4, BAR Build ID 68583

0a95e42

Update commits in Version.Details.xml and reconcile patches

ce1386d

Remove temporary RC1 fix in aspnetcore

0a3ce07

Install built runtime for usage by aspnetcore

f434b4e

Update roslyn to build CodeStyle packages

f46b98f

Update / add installer patches

317e0ae

Fix broken patch

084a2a7

Update prevSourceBuilt to GA version

5160b7c

Pin System.Security.Cryptography.Cng version in runtime build

adae8b2

Update path to sdk installation

9a9a56f

Update smoke test nuget.config entries

e99ca58

Update online prebuilt baseline

3fcb32f

Update package name for aspnetcore reference

909a8cc

Add runtime feed to smoke-testNuGet.Config

50bfc0d

Update runtime config entry

82e9958

dseefeld force-pushed the 5.0GA branch from 16f3390 to 82e9958 Compare October 30, 2020 20:19

dseefeld added 4 commits October 30, 2020 22:06

Update to 20201026.5, BAR Build ID 68880

059cd96

Set UseStableVersions flag

ecdb22a

Update online baseline

45e1c26

Update offline prebuilt baseline

a77ec25

crummel self-requested a review November 2, 2020 21:06

runfoapp bot mentioned this pull request Nov 2, 2020

OSX deprovision jaredpar/runfo#41

Closed

Include patch from dotnet#1850

943c33a

Include patch to remove import of restore/harvestPackages.targets in runtime libraries build to fix issue with portable build

dagood mentioned this pull request Nov 3, 2020

[release/5.0] Remove Mono.Cecil/0.11.2 prebuilt #1804

Closed

Merge branch 'master' into 5.0GA

a1ad7a4

dagood mentioned this pull request Nov 3, 2020

Fix installation of runtime into the tooling SDK dir during the build #1854

Closed

dagood approved these changes Nov 3, 2020

View reviewed changes

crummel approved these changes Nov 3, 2020

View reviewed changes

patches/aspnetcore/0005-Fix-version-number.patch Show resolved Hide resolved

dseefeld added 2 commits November 3, 2020 18:24

Re-add portable build only prebuilts

9e05d3e

Merge branch '5.0GA' of https://github.com/dseefeld/source-build into…

5fa1a69

… 5.0GA

dseefeld merged commit dd02c2f into dotnet:master Nov 3, 2020

dseefeld deleted the 5.0GA branch November 3, 2020 22:34

tmds reviewed Nov 13, 2020

View reviewed changes

dagood mentioned this pull request Nov 16, 2020

Follow up on NuGet restore bug with project and package reference at same version on different TFMs: "An item with the same key has already been added." #1888

Closed

dagood mentioned this pull request Dec 8, 2020

When a ProjectReference and PackageReference in different TFMs have the same identity/version, Restore throws "An item with the same key has already been added. [...]" NuGet/Home#10368

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to 5.0 GA #1845

Update to 5.0 GA #1845

dseefeld commented Oct 28, 2020

dseefeld commented Nov 3, 2020

dagood commented Nov 3, 2020

dagood Nov 3, 2020

dseefeld Nov 3, 2020 •

edited

Loading

dagood Nov 3, 2020

dseefeld Nov 3, 2020

dagood Nov 3, 2020

dseefeld Nov 3, 2020

dagood Nov 3, 2020

dseefeld Nov 3, 2020

dseefeld Nov 3, 2020 •

edited

Loading

crummel left a comment

tmds Nov 13, 2020

dseefeld Nov 13, 2020 •

edited

Loading

tmds Nov 13, 2020

dseefeld Nov 13, 2020

dagood Nov 13, 2020 •

edited

Loading

tmds Nov 16, 2020

dagood Nov 16, 2020

dagood Dec 8, 2020

Update to 5.0 GA #1845

Update to 5.0 GA #1845

Conversation

dseefeld commented Oct 28, 2020

dseefeld commented Nov 3, 2020

dagood commented Nov 3, 2020

Choose a reason for hiding this comment

dseefeld Nov 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dseefeld Nov 3, 2020 • edited Loading

Choose a reason for hiding this comment

crummel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dseefeld Nov 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dagood Nov 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dseefeld Nov 3, 2020 •

edited

Loading

dseefeld Nov 3, 2020 •

edited

Loading

dseefeld Nov 13, 2020 •

edited

Loading

dagood Nov 13, 2020 •

edited

Loading