Create full PlatformManifest.txt in "live-live" dotnet/runtime builds

[dagood]

The platform manifest is a text file that describes the files in every platform's shared framework. The SDK uses it to filter FDE/FDD publish outputs. The manifest ends up in the targeting pack and runtime packs, at data\PlatformManifest.txt. A little example of the contents:

mscorlib.dll|Microsoft.NETCore.App.Ref|4.0.0.0|4.700.19.46214
System.IO.Compression.Native.a|Microsoft.NETCore.App.Ref||0.0.0.0
System.IO.Compression.Native.so|Microsoft.NETCore.App.Ref||0.0.0.0
...
libcoreclr.so|Microsoft.NETCore.App.Ref||0.0.0.0
...
libcoreclr.dylib|Microsoft.NETCore.App.Ref||0.0.0.0
...
api-ms-win-crt-utility-l1-1-0.dll|Microsoft.NETCore.App.Ref||10.0.17134.12
...
coreclr.dll|Microsoft.NETCore.App.Ref||4.700.19.46205
...

Note that coreclr.dll/libcoreclr.so/libcoreclr.dylib are all listed: each platform's native binary. This list and data is harvested from the actual files, so multiple machines are necessarily involved.

The consolidation effort (dotnet/runtime) is going to be building "live-live" builds: builds that take place on a single machine all the way through coreclr, corefx, and core-setup. It won't rely on cached files built by an earlier run of the repo. This means we can't use the current approach, because it needs multiple machines to produce all the native artifacts.

---

I think we need to stop using harvesting to populate this file, and instead calculate the contents. We probably need to keep a platform manifest template (list of files) in the source to act as a template.

This would normally be a maintenance hazard, but we can prevent that:

1. Each platform uses the template to generate an expected platform manifest, which is used when building targeting packs, runtime packs, installers, etc.
2. When running a platform-specific job, validate:
   • All sharedfx files generated on that platform are in the manifest template.
   • There are no files in the template that are expected to be built on the current platform but weren't.
3. By the time any CI/PR validation build ends, all platforms have run the validation step. This means that if all jobs succeed, we know the platform manifest is valid. Even if some platforms silently produced bad platform manifests, the build would fail overall.
4. Since official builds stop if some jobs failed, there is no risk of releasing an outdated platform manifest.

This does impact dev flows. Someone may add a new file to the shared framework and not know how to properly update the manifest template. To mitigate this, when the validation step fails, it should emit the corrected list of files and tell devs where they need to put it to fix up their PR.

This also fixes source-build, where there are even more constraints than "live-live" builds: https://github.com/dotnet/core-setup/issues/5297. (The platform manifest is already broken for source-build even without consolidation.)

Here are some other options I considered that I don't think are viable (especially short-term) and don't fix source-build. (Click me)

1. Have a "fan in" stage in the dotnet/runtime official build.

   • The new stage would download the pipeline artifacts from all the platform builds and harvest the platform manifest. It would then produce the artifacts the manifest is included in. This stage would have its own matrix of platforms to fan back out, because the platform manifest is included in some installers, and installers can only be built on the target platform.
   • This is not easily reproducible on dev machines, and pinches the build down to a choke point which could prevent build time improvements.

2. Defer the harvest to downstream.

   • We can skip producing the platform manifest in dotnet/runtime, and instead make it in a downstream repo that isn't in the "live live" build. The downstream repo needs to download dotnet/runtime artifacts anyway, so there's no problem with harvesting from multiple platforms.
   • Lots of downstream effects depending on how exactly this is split out.

3. Eliminate the need for a platform manifest file.

   • I don't see how to do this, but if we can, it's an option. The platform manifest can be thought of as a performance improvement: instead of downloading every single runtime pack to check what files they contain, this text file in targeting pack is used.
   • A major caveat for some ways of doing this is that the targeting pack version matches the runtime pack version. We can't simply have the SDK generate and contain a platform manifest, because the SDK needs to be able to target downlevel runtimes.

I think I should work on this, and in the meantime I think it's reasonable to disable "full" platform manifest generation (set BuildFullPlatformManifest to false) to unblock dotnet/runtime "live live" builds.

/cc @MichaelSimons @dleeapho @jkoritzinsky @jashook @dseefeld @crummel @NikolaMilosavljevic @adaggarwal

[dagood]

Turns out this is blocking 5.0 uptake by various upstream repos: #1129.

[dagood]

I looked deeper at the templating approach, and it is fairly complex. A decent implementation requires Arcade changes, and there are versions present that we can't trivially calculate to replace in a set of templates:

• Redistributed files (UCRT, DiaSymReader) for specific platforms.
• CoreFX/Libraries projects that don't use the standard assembly/file versions (Microsoft.VisualBasic.Core).
• 0.0.0.0-versioned files if a version isn't found after scanning all platforms.

---

Something has changed that opens up more possibilities:

The consolidation effort (dotnet/runtime) is going to be building "live-live" builds: builds that take place on a single machine all the way through coreclr, corefx, and core-setup.

This is no longer true: each subset runs on its own set of machines. That means we can have every installer job depend on all outputs from the earlier subsets' jobs, just like the pre-consolidation approach with Core-Setup.

Changing our approach means won't be fixing source-build's bad platform manifest, but that's not a new problem (not a regression).

I think there will still be a decent amount of work because we no longer use NuGet transport packages. However, keeping the old approach is at least more straightforward and I believe this will unblock runtime build uptake much more quickly. At the moment I think this will be done EOW (Jan 10) but that's a very early estimate.

[dagood]

@wtgodbe, re dotnet/aspnetcore#18250, I think we should standardize on the templating approach above. Also, this thread may be useful for context on the way things are done for the NETCoreApp platform manifests.

[dagood]

Build running with the fix (using harvesting): https://dev.azure.com/dnceng/internal/_build/results?buildId=479360&view=results

Once we get it through, I'm going to close this issue and open a new one tracking solving this for source-build.

[dagood]

Build completed, manifest looks as expected. Closing as resolved. I have #1622 open to track fixing source-build's incomplete manifest.