Consume new strict CPDs in dotnet/installer for 5.0 source-build

[dagood]

Tracking #1495 (comment):

Had a meeting to discuss this with mmitche and others, and thinking now we'll be able to rely on the CPD strictification of dotnet/arcade#5195 and storing a complete coherent build graph in the dotnet/installer repo. That seems like it will remove the need for special darc clone features (until we want to do speculative builds).

@mmitche plans to get some updated CPD code to me quickly to start validating this.

[dagood]

After getting a WIP copy of darc, I realized the new direction for CPD means source-build is basically using what I already validated to work fine in my prototype (#1466): Version.Details.xml files that give us the graph with --depth 0. So, risk already seems minimal here.

@mmitche is working on the strict coherency feature and applying strict CPDs from the bottom up in 5.0, and gave me a very rough estimate of about a week to have it through to dotnet/installer. We should verify that it ends up working in source-build but, as mentioned, it doesn't seem risky to me and waiting for that seems fine.

---

While thinking about this, I realized the new strict CPDs are being applied to 3.1 as well, so I think we need to update the source-build eng/Version.Details.xml to work with it. Filed #1569 for that.

[dagood]

Issue tracking propagating strict CPD through the 5.0 branches: dotnet/arcade#5685.

And a few notes after reading this again in the future: the strict CPD feature doesn't require that there's a full CPD graph in dotnet/installer. We may have to add that ourselves while moving our infra to dotnet/installer (which is fine: we have our own CPD graph in dotnet/source-build that the dotnet/installer owners should fix up if needed). Maintaining the graph also doesn't seem to require the strict CPD feature (I may be forgetting a reason), but it makes the process simpler/more reliable.

[dagood]

Issue tracking propagating strict CPD through the 5.0 branches: dotnet/arcade#5685.

We don't need to wait for this: we can try adding the source-build graph into the dotnet/installer release/5.0.1xx Version.Details.xml with CPD attributes and use update-dependencies --coherency-only --strict-coherency to see how it works. Some repos have already been added, but there are some missing like NuGet, MSBuild, Roslyn, SourceLink, and Linker.

(dotnet/extensions shouldn't be necessary anymore at all because it isn't used in the .NET SDK. (Because ASP.NET Core sharedfx and packs no longer use it.))

Then, we can merge the additions, because we will use them in the future for arcade-powered source-build. This gets a significant chunk of uncertainty out of the way.

[dagood]

Issues uncovered while moving the dependencies over:

• ✔️ Strict CPD on dotnet/sourcelink via dotnet/arcade causes darc failure (due to empty <Sha/>?) Strict CPD on dotnet/sourcelink via dotnet/arcade causes darc failure (due to empty <Sha/>?) arcade#6064
  • ✔️ Waiting on rollout to dotnet/installer release/5.0.1xx
• ✔️ Why is the XDT dependency pinned to 3.1.0 in release/5.0.1xx? Why is the XDT dependency pinned to 3.1.0 in release/5.0.1xx? sdk#13214
  • VS coordination and stability. We can work around by treating it as a library that isn't maintained by us.
• ✔️ dotnet/aspnetcore-tooling dependency isn't used anywhere?
  • This is already known, I was just migrating based on outdated data.

Non-blocker: strict CPD update not working yet in dotnet/installer master, only release/5.0.1xx. mmitche's working through this.

[dagood]

Merged in dotnet/installer#8377. If there are missing CPDs or repos, I expect we'll add them when adding source-build to the repo, or during prebuilt reduction.