Undo revert of Arcade update to ingest Workload Manifest task

[wtgodbe]

Basically, undo #71110 (but don't undo the SDK update), and fix up the publish task as described below (copy/pasted from the PR):

Runtime will have to take an arcade update, a llvm-project update, and Jacques' change next month, plus a fix to the publish stage invoked here: https://github.com/dotnet/runtime/blob/cd19e8bf35d18cef6f30d896313cbeb1be2a06dd/eng/pipelines/official/jobs/prepare-signed-artifacts.yml#L48-L57. The problem seems to be that you're trying to upload all blobs to `blobArtifacts`, but there are now .msi's with duplicate names. From @joeloff:

> [7:52 PM] Jacques Eloff
it organizes builds by feature band for VS insertions, so MSIs can be duplicated in multiple folders. Copying them the same location won't work.

> because in staging, each sets of files for each feature band is published to a separate VSDROP container and there can be overlap, but not all blobbies have the same files.

Wanted to make sure this was on your radar. The failure happens during the AzDO artifact upload task that happens at the end of that step:

> Async Command Start: Upload Artifact
> ...
> Fail to upload 'D:\a\_work\1\s\artifacts\PackageDownload\IntermediateArtifacts\workloads-vs\Insertion\6.0.300\Microsoft.AOT.win-x64.Cross.android-arm.6.0.7.msi' due to 'Blob is incomplete (missing block). Blob: af95b7c0d1f1ec11b656501ac504b8de, Expected Offset: 0, Actual Offset: 4194304'.

The root cause of this was the change to the workload manifest tasks in Arcade: dotnet/arcade#8645

This requires reaction in runtime's usage of those tasks:

runtime/src/workloads/workloads.csproj

Lines 113 to 133 in 6212eab

	<GenerateManifestMsi
	IntermediateBaseOutputPath="$(IntermediateOutputPath)"
	OutputPath="$(OutputPath)"
	MsiVersion="$(MsiVersion)"
	WixToolsetPath="$(WixToolsetPath)"
	WorkloadManifestPackage="%(ManifestPackages.Identity)"
	ShortNames="@(ShortNames)" >
	<Output TaskParameter="Msis" ItemName="ManifestMsis" />
	</GenerateManifestMsi>
	<GenerateVisualStudioWorkload IntermediateBaseOutputPath="$(WorkloadIntermediateOutputPath)"
	WixToolsetPath="$(WixToolsetPath)"
	GenerateMsis="true"
	ComponentVersions="@(ComponentVersions)"
	OutputPath="$(WorkloadOutputPath)"
	PackagesPath="$(PackageSource)"
	ShortNames="@(ShortNames)"
	WorkloadPackages="@(ManifestPackages)">
	<Output TaskParameter="SwixProjects" ItemName="SwixProjects" />
	<Output TaskParameter="Msis" ItemName="Msis" />
	</GenerateVisualStudioWorkload>

We tried to react with #71046, but that caused us to try to publish multiple .msi's with the same name to the same place. Basically we need to publish the output of the new task & preserve its output folder structure, rather than flattening.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Basically, undo #71110 (but don't undo the SDK update), and fix up the publish task as described below (copy/pasted from the PR):

Runtime will have to take an arcade update, a llvm-project update, and Jacques' change next month, plus a fix to the publish stage invoked here:

runtime/eng/pipelines/official/jobs/prepare-signed-artifacts.yml

Lines 48 to 57 in cd19e8b

	- script: >-
	build.cmd -ci
	-subset publish
	-configuration Release
	/p:PublishRidAgnosticPackagesFromPlatform=${{ parameters.PublishRidAgnosticPackagesFromPlatform }}
	/p:OfficialBuildId=$(Build.BuildNumber)
	/p:SignType=$(SignType)
	/p:DotNetSignType=$(SignType)
	/bl:$(Build.SourcesDirectory)\prepare-artifacts.binlog
	displayName: Prepare artifacts and upload to build

. The problem seems to be that you're trying to upload all blobs to blobArtifacts, but there are now .msi's with duplicate names. From @joeloff:

[7:52 PM] Jacques Eloff
it organizes builds by feature band for VS insertions, so MSIs can be duplicated in multiple folders. Copying them the same location won't work.

because in staging, each sets of files for each feature band is published to a separate VSDROP container and there can be overlap, but not all blobbies have the same files.

Wanted to make sure this was on your radar. The failure happens during the AzDO artifact upload task that happens at the end of that step:

Async Command Start: Upload Artifact
...
Fail to upload 'D:\a_work\1\s\artifacts\PackageDownload\IntermediateArtifacts\workloads-vs\Insertion\6.0.300\Microsoft.AOT.win-x64.Cross.android-arm.6.0.7.msi' due to 'Blob is incomplete (missing block). Blob: af95b7c0d1f1ec11b656501ac504b8de, Expected Offset: 0, Actual Offset: 4194304'.

Author:	wtgodbe
Assignees:	-
Labels:	area-Infrastructure
Milestone:	6.0.8

[ghost]

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Basically, undo #71110 (but don't undo the SDK update), and fix up the publish task as described below (copy/pasted from the PR):

Runtime will have to take an arcade update, a llvm-project update, and Jacques' change next month, plus a fix to the publish stage invoked here: https://github.com/dotnet/runtime/blob/cd19e8bf35d18cef6f30d896313cbeb1be2a06dd/eng/pipelines/official/jobs/prepare-signed-artifacts.yml#L48-L57. The problem seems to be that you're trying to upload all blobs to `blobArtifacts`, but there are now .msi's with duplicate names. From @joeloff:

> [7:52 PM] Jacques Eloff
it organizes builds by feature band for VS insertions, so MSIs can be duplicated in multiple folders. Copying them the same location won't work.

> because in staging, each sets of files for each feature band is published to a separate VSDROP container and there can be overlap, but not all blobbies have the same files.

Wanted to make sure this was on your radar. The failure happens during the AzDO artifact upload task that happens at the end of that step:

> Async Command Start: Upload Artifact
> ...
> Fail to upload 'D:\a\_work\1\s\artifacts\PackageDownload\IntermediateArtifacts\workloads-vs\Insertion\6.0.300\Microsoft.AOT.win-x64.Cross.android-arm.6.0.7.msi' due to 'Blob is incomplete (missing block). Blob: af95b7c0d1f1ec11b656501ac504b8de, Expected Offset: 0, Actual Offset: 4194304'.

The root cause of this was the change to the workload manifest tasks in Arcade: dotnet/arcade#8645

This requires reaction in runtime's usage of those tasks:

runtime/src/workloads/workloads.csproj

Lines 113 to 133 in 6212eab

	<GenerateManifestMsi
	IntermediateBaseOutputPath="$(IntermediateOutputPath)"
	OutputPath="$(OutputPath)"
	MsiVersion="$(MsiVersion)"
	WixToolsetPath="$(WixToolsetPath)"
	WorkloadManifestPackage="%(ManifestPackages.Identity)"
	ShortNames="@(ShortNames)" >
	<Output TaskParameter="Msis" ItemName="ManifestMsis" />
	</GenerateManifestMsi>
	<GenerateVisualStudioWorkload IntermediateBaseOutputPath="$(WorkloadIntermediateOutputPath)"
	WixToolsetPath="$(WixToolsetPath)"
	GenerateMsis="true"
	ComponentVersions="@(ComponentVersions)"
	OutputPath="$(WorkloadOutputPath)"
	PackagesPath="$(PackageSource)"
	ShortNames="@(ShortNames)"
	WorkloadPackages="@(ManifestPackages)">
	<Output TaskParameter="SwixProjects" ItemName="SwixProjects" />
	<Output TaskParameter="Msis" ItemName="Msis" />
	</GenerateVisualStudioWorkload>

We tried to react with #71046, but that caused us to try to publish multiple .msi's with the same name to the same place. Basically we need to publish the output of the new task & preserve its output folder structure, rather than flattening.

Author:	wtgodbe
Assignees:	-
Labels:	area-Infrastructure-mono
Milestone:	6.0.8

[ericstj]

@directhex @steveisok can you prioritize a change to workloads.csproj to fix this and help get Arcade flowing again in 6.0?

[joeloff]

Some details. The workload outputs will be organized by feature band to facilitate publishing the specific content for the VS versions being targeted. There's some automation tasks that we have in staging that we haven't been able to use.

artifacts\VSSetup\Debug\Insertion\6.0.100
artifacts\VSSetup\Debug\Insertion\6.0.200
artifacts\VSSetup\Debug\Insertion\6.0.300

Because workload packs can be the same, it's likely there will be overlap in these folders and it seems that publishing was pushing the content to the same blob container

[directhex]

Hi @wtgodbe, do logs still exist for one of the builds which failed due to the Arcade bump?

[wtgodbe]

https://dev.azure.com/dnceng/internal/_build/results?buildId=1837162&view=logs&j=138e5e17-b2cd-527a-2da4-df51693fdead&t=7c441ab8-f67c-56a1-1eef-407c16f44b55&s=fbeceea2-8819-56cc-d4aa-12918796648b

[directhex]

OK, that binlog has revealed a lot

The publish subset runs though src/installer/prepare-artifacts.proj. One thing happening in that file is here;

runtime/src/installer/prepare-artifacts.proj

Lines 121 to 124 in 9aec9ad

	<ItemsToPush Include="@(WorkloadsVSInsertionFile)">
	<RelativeBlobPath>$(InstallersRelativePath)workloads/$(SdkBandVersion)/%(Filename)%(Extension)</RelativeBlobPath>
	<PublishFlatContainer>true</PublishFlatContainer>
	</ItemsToPush>

@(WorkloadsVSInsertionFile) contains the paths of all files generated by the workload generation task which we want to keep, but because SdkBandVersion is static (from Versions.props), that's the specific place where all three versions of the workload files get an identical target RelativeBlobPath of 6.0.300. With any luck, fixing this up to derive from the source path will be enough (maybe setting PublishFlatContainer to false is also needed, will see)

[directhex]

This is increasingly looking like an annoying problem to resolve.

PushToAzureDevOpsArtifacts, part of Arcade, is apparently where things are going wrong. If the Items have a PublishFlatContainer=false property attached, then directory structure is not flattened... but the task will fail for non-nupkg files (e.g. json files). If PublishFlatContainer=true then we run into the observed behaviour with System.InvalidOperationException: Blob is incomplete (missing block). and non-zero offsets.

So I need to pull Arcade apart to try and fix it (and really need to set up a minimal repro case so I'm not waiting hours for every attempt)

[directhex]

I can confirm my analysis from a simple repro case. https://github.com/directhex/arcade-repro runs on https://dev.azure.com/joshield/testing/_build?definitionId=1&_a=summary and behaves in exactly the same way as runtime.git: PublishFlatContainer=false only supports NuPkg files, and RelativeBlobPath is thrown away during the upload. PublishToAzureDevOpsArtifacts just doesn't support folders in the general case.

For small files (smaller than the upload chunk size), the last uploaded file simply overwrites older files of the same name. For files larger than the chunk size, you get the Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Bad Request which is largely typified by Blob is incomplete (missing block).

Should I open an issue against Arcade for this?

[joeloff]

Yeah,. might be worth to pull them in and see if there's a good solution. Right now we have duplicated packs, but the idea was always that workloads for different feature bands can diverge and we want to upload clean drops for VS for each feature band.

[directhex]

@joeloff what's the status of the issue here? Are we still producing files with duplicate names in workloads? Or have other discussions surrounding multi-targeting of workloads affected the underlying intent behind this PR?

[joeloff]

No, we should close this. Let me link the PRs for this.

[joeloff]

Fixed with #73981