Publish dotnet/runtime using standard Arcade Publish tooling #111934

jkoritzinsky · 2025-01-28T22:33:53Z

Publish the assets from dotnet/runtime using the standard Arcade publish tooling. This tooling doesn't allow any duplicate packages, so change our publishing logic to the following:

In a given build, publish all packages that were produced in the current build that contain that job's target RID.
For some target RIDs, also publish an additional subset of packages (specified in Publishing.props).
For some jobs (specified in runtime-official), publish all packages produced.
In the VMR, mark all "packages not to be published" as "Vertical" visibility so they're still available upstack.
Outside the VMR, skip publishing all "packages not to be published" packages.

~~Blocked on MicroBuild fixes for Linux and Mac: dotnet/source-build#4793~~ As per #111934 (comment), we are no longer blocked on this.

Verified in https://dev.azure.com/dnceng/internal/_build/results?buildId=2629649&view=results that all of the same assets that we expect to produce in a runtime official build are produced on this branch.

Fixes #102672
Fixes #1719

…n. Don't copy workloads packages as they're already in the right format now.

…ic jobs publish specific packages

…specific packages

…es. Use that when possible for unique names and produce our own names when necessary.

…manifests.

…st packages

…o the Artifacts items in Signing.props

…-publish

eng/pipelines/runtime-official.yml

…lob paths to make that work

…e signed coming out of runtime and otherwise we'd miss signing them

…g/internal/_git/dotnet-runtime into standard-arcade-publish-with-join

jkoritzinsky · 2025-03-06T23:48:24Z

I've updated this PR to provide a mechanism for us to merge it today and get rid of the long-running patch.

Here's my basic approach:

Instead of publishing to AzDO, each runtime vertical will instead publish to disk with a layout like the artifacts directory.
Runtime verticals won't sign in the runtime official build
We bring back the final join job. This job downloads all of the artifacts from each runtime vertical into its artifacts directory and then does a standard arcade sign/publish with the same signing rules as exist in main.
We still get rid of IntermediateArtifacts and instead use separate pipeline artifacts for each vertical.

I have a passing official build from this PR here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2657037&view=results

jkoritzinsky · 2025-03-07T00:31:45Z

New official build for a314fc3 at https://dev.azure.com/dnceng/internal/_build/results?buildId=2657219&view=results

ViktorHofer · 2025-03-07T14:15:17Z

eng/DotNetBuild.props

+  <PropertyGroup Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">
+    <!-- Source-build always needs all artifacts to be published. -->
+    <EnableDefaultRidSpecificArtifacts Condition="'$(DotNetBuildSourceOnly)' != ''">false</EnableDefaultRidSpecificArtifacts>
+    <!-- Short-stack builds should always only publish RID-specific artifacts. -->
+    <EnableDefaultRidSpecificArtifacts Condition="'$(ShortStack)' == 'true'">true</EnableDefaultRidSpecificArtifacts>
+    <!-- If no override is specified, don't use RID-specific publishing. Instead, publish everything. -->
+    <EnableDefaultRidSpecificArtifacts Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">false</EnableDefaultRidSpecificArtifacts>
+  </PropertyGroup>


These settings make sense to me but wouldn't they better fit into the VMR orchestrator as they aren't specific to runtime?

Yes, they do fit better in the VMR. However, we need to put them here first so they don't break the VMR until I change the VMR to pass these options down (also in PR)

Once the VMR controls are in, I'll come back and remove these.

Consider filing a tracking issue and linking to it. Also, so that others can follow along and understand the potential for improvement in these code paths.

eng/Signing.props

Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>

jkoritzinsky · 2025-03-07T22:54:44Z

/ba-g timeouts unrelated

jkoritzinsky added 23 commits January 21, 2025 15:25

Remove prepare-signed-artifacts and publish from each leg directly.

0364ee2

Publish in a separate stage to make dependency tracking easy

7e284a4

Set _SignType so we can pass -sign

d2c9cf8

Remove old validation target

3338d4d

!drop remove -sign so we can figure out publishing first

01bca60

Remove InitialTargets

88ee36c

Only generate product version files once. Fix asset manifest extensio…

d2fa244

…n. Don't copy workloads packages as they're already in the right format now.

Completely restructure how we publish to be more explicit that specif…

f93b7e4

…ic jobs publish specific packages

Move "non-RID-specific" special packages logic into Publishing.props

8ee9ac1

Make sure we build the list of skipped packages after we add the RID-…

4455301

…specific packages

Arcade now handles TargetOS/TargetArchitecture for asset manifest nam…

e36e87e

…es. Use that when possible for unique names and produce our own names when necessary.

Extract handling hostOS and hostArch into another props file

747bffe

Fix import and adjust condition for SB

fbe3515

Fix asset manifest name uniqueness to ensue we don't overwrite asset …

6bbbc4c

…manifests.

Set AssetManifestFilePath again

c1068eb

Fix PublishAllPackages condition and make sure to publish the VS redi…

9e61e42

…st packages

Explicitly look up packages as we add zips and other blob artifacts t…

05fe23b

…o the Artifacts items in Signing.props

Search in the right folder.

d8b032a

Look up zips in VSSetup for workloads

2ac88a1

Add missing semicolon

552bb1d

Include VSSetup artifacts separately

9247027

Re-enable signing (depends on Microbuild signing on Linux/Mac)

1eb6a85

Merge branch 'main' of github.com:dotnet/runtime into standard-arcade…

72af6e0

…-publish

jkoritzinsky added blocked Issue/PR is blocked on something - see comments area-Infrastructure labels Jan 28, 2025

jkoritzinsky requested review from akoeplinger and ViktorHofer January 28, 2025 22:33

dotnet-policy-service bot assigned jkoritzinsky Jan 28, 2025

build-analysis bot mentioned this pull request Jan 29, 2025

[wasm] Out of memory exception in System.Xml.XmlWriterApiTests.TCFullEndElement tests #110090

Closed

ViktorHofer reviewed Jan 29, 2025

View reviewed changes

eng/pipelines/runtime-official.yml Outdated Show resolved Hide resolved

jkoritzinsky added 14 commits March 4, 2025 10:54

Put the VSSetup blobs in the VSSetup folder and adjust the relative b…

3702f3c

…lob paths to make that work

Fix crossdac and workloads downloads

5fe309f

inputs happen before clone. Don't download into the sources directory

2343ebe

Fix paths

b08df1e

Skip extracting packages that don't match so we can see the whole list.

f37e6a5

Add arm32 downloads for crossdac

d3b9cfd

Publish crossos dac packages

2c90296

Don't use RID-specific artifacts for the cross-dac build

a0e9fe6

Fix source-build and deps

53c44db

Fix join-job signing

0e820f4

Add back split macos job to the official build. These files need to b…

fbf49fc

…e signed coming out of runtime and otherwise we'd miss signing them

Merge branch 'standard-arcade-publish' of https://dev.azure.com/dncen…

e91c434

…g/internal/_git/dotnet-runtime into standard-arcade-publish-with-join

Remove duplicate jobs

9557e70

Adjust commands for macos split job

745a8a1

jkoritzinsky removed the blocked Issue/PR is blocked on something - see comments label Mar 6, 2025

Use the right property for the signing items base directory

a314fc3

jkoritzinsky mentioned this pull request Mar 7, 2025

Remove primary vertical concept from JoinVerticals dotnet/sdk#47338

Open

mmitche approved these changes Mar 7, 2025

View reviewed changes

ViktorHofer reviewed Mar 7, 2025

View reviewed changes

eng/Signing.props Outdated Show resolved Hide resolved

ViktorHofer approved these changes Mar 7, 2025

View reviewed changes

Update Signing.props

5312597

Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>

build-analysis bot mentioned this pull request Mar 7, 2025

restarted. Azure DevOps can't recover from restarts. dotnet/dnceng#3879

Open

3 tasks

Remove duplicate signing inputs

f93980b

build-analysis bot mentioned this pull request Mar 7, 2025

Brew command passes (now invalid) --no-lock parameter #113276

Closed

jkoritzinsky merged commit 891ff91 into dotnet:main Mar 7, 2025
138 of 148 checks passed

jkoritzinsky deleted the standard-arcade-publish branch March 7, 2025 22:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish dotnet/runtime using standard Arcade Publish tooling #111934

Publish dotnet/runtime using standard Arcade Publish tooling #111934

jkoritzinsky commented Jan 28, 2025 •

edited

Loading

jkoritzinsky commented Mar 6, 2025

jkoritzinsky commented Mar 7, 2025

ViktorHofer Mar 7, 2025

jkoritzinsky Mar 7, 2025

jkoritzinsky Mar 7, 2025

ViktorHofer Mar 7, 2025

jkoritzinsky commented Mar 7, 2025

Publish dotnet/runtime using standard Arcade Publish tooling #111934

Publish dotnet/runtime using standard Arcade Publish tooling #111934

Conversation

jkoritzinsky commented Jan 28, 2025 • edited Loading

jkoritzinsky commented Mar 6, 2025

jkoritzinsky commented Mar 7, 2025

ViktorHofer Mar 7, 2025

Choose a reason for hiding this comment

jkoritzinsky Mar 7, 2025

Choose a reason for hiding this comment

jkoritzinsky Mar 7, 2025

Choose a reason for hiding this comment

ViktorHofer Mar 7, 2025

Choose a reason for hiding this comment

jkoritzinsky commented Mar 7, 2025

jkoritzinsky commented Jan 28, 2025 •

edited

Loading