Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Join assets from verticals in the final join point #43627

Merged
merged 23 commits into from
Oct 17, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
0270b04
merge vertical manifests
MilenaHristova Sep 23, 2024
260f356
exclude verticals from join
MilenaHristova Sep 25, 2024
c5c14ac
add comment
MilenaHristova Sep 25, 2024
0aa6762
list duplicate assets
MilenaHristova Sep 27, 2024
d1784b7
parallelism
MilenaHristova Sep 27, 2024
745f552
download build artifact
MilenaHristova Sep 27, 2024
6f70d33
fix path
MilenaHristova Sep 27, 2024
82c18d0
add comment
MilenaHristova Sep 27, 2024
ed711dc
add extra parallelism
MilenaHristova Oct 7, 2024
c34e878
Update src/SourceBuild/content/eng/tools/tasks/Microsoft.DotNet.Unifi…
MilenaHristova Oct 7, 2024
717bfcb
Merge branch 'main' into mhristova/join-verticals
MilenaHristova Oct 7, 2024
a08dc29
add comment
MilenaHristova Oct 9, 2024
0a9cb47
Merge branch 'mhristova/join-verticals' of https://github.com/MilenaH…
MilenaHristova Oct 9, 2024
afd565d
Merge branch 'main' into mhristova/join-verticals
mmitche Oct 10, 2024
1093ff3
Merge branch 'main' into mhristova/join-verticals
mmitche Oct 10, 2024
ea8dd95
Update src/SourceBuild/content/eng/join-verticals.proj
MilenaHristova Oct 15, 2024
90400d7
Update src/SourceBuild/content/eng/pipelines/ci.yml
MilenaHristova Oct 15, 2024
2e256ba
Update src/SourceBuild/content/eng/pipelines/pr.yml
MilenaHristova Oct 15, 2024
0d501d1
Remove the unused target
ViktorHofer Oct 16, 2024
5d36eba
Update src/SourceBuild/content/eng/pipelines/ci.yml
ViktorHofer Oct 16, 2024
d33abcc
Update src/SourceBuild/content/eng/pipelines/pr.yml
ViktorHofer Oct 16, 2024
37d25d5
Merge branch 'main' into mhristova/join-verticals
ViktorHofer Oct 17, 2024
875c991
Update vmr-build.yml
ViktorHofer Oct 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 34 additions & 2 deletions eng/pipelines/templates/jobs/vmr-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,13 @@ jobs:
displayName: Publish Artifacts
sbomEnabled: true

# Using build artifacts to enable publishing the vertical manifests to a single artifact from different jobs
- output: buildArtifacts
PathtoPublish: $(Build.ArtifactStagingDirectory)/manifests/$(Agent.JobName).xml
ArtifactName: VerticalManifests
displayName: Publish Vertical Manifest
sbomEnabled: false

- ${{ if not(parameters.isBuiltFromVmr) }}:
- output: pipelineArtifact
displayName: Upload failed patches
Expand Down Expand Up @@ -272,7 +279,7 @@ jobs:
- script: |
set extraBuildProperties=
if not [${{ parameters.buildPass }}]==[] set extraBuildProperties=%extraBuildProperties% /p:DotNetBuildPass=${{ parameters.buildPass }}
call build.cmd -ci -cleanWhileBuilding -prepareMachine %devArgument% /p:TargetOS=${{ parameters.targetOS }} /p:TargetArchitecture=${{ parameters.targetArchitecture }} %extraBuildProperties% ${{ parameters.extraProperties }}
call build.cmd -ci -cleanWhileBuilding -prepareMachine %devArgument% /p:TargetOS=${{ parameters.targetOS }} /p:TargetArchitecture=${{ parameters.targetArchitecture }} /p:VerticalName=$(Agent.JobName) %extraBuildProperties% ${{ parameters.extraProperties }}
displayName: Build
workingDirectory: ${{ variables.sourcesPath }}
env:
Expand All @@ -283,7 +290,7 @@ jobs:

- ${{ if eq(parameters.runTests, 'True') }}:
- script: |
call build.cmd -ci -prepareMachine -test -excludeCIBinarylog /bl:artifacts/log/Release/Test.binlog /p:TargetOS=${{ parameters.targetOS }} /p:TargetArchitecture=${{ parameters.targetArchitecture }} ${{ parameters.extraProperties }}
call build.cmd -ci -prepareMachine -test -excludeCIBinarylog /bl:artifacts/log/Release/Test.binlog /p:TargetOS=${{ parameters.targetOS }} /p:TargetArchitecture=${{ parameters.targetArchitecture }} /p:VerticalName=$(Agent.JobName) ${{ parameters.extraProperties }}
displayName: Run Tests
workingDirectory: ${{ variables.sourcesPath }}
timeoutInMinutes: ${{ variables.runTestsTimeout }}
Expand Down Expand Up @@ -381,6 +388,8 @@ jobs:
extraBuildProperties="$extraBuildProperties ${{ parameters.extraProperties }}"
fi

extraBuildProperties="$extraBuildProperties /p:VerticalName=$(Agent.JobName)"

buildArgs="$(additionalBuildArgs) $customBuildArgs $extraBuildProperties"

# Only use Docker when a container is specified
Expand Down Expand Up @@ -456,6 +465,8 @@ jobs:
customBuildArgs="$customBuildArgs --target-rid ${{ parameters.targetRid }}"
fi

extraBuildProperties="$extraBuildProperties /p:VerticalName=$(Agent.JobName)"

if [[ -n "${{ parameters.extraProperties }}" ]]; then
extraBuildProperties="$extraBuildProperties ${{ parameters.extraProperties }}"
fi
Expand Down Expand Up @@ -597,6 +608,19 @@ jobs:
TargetFolder: $(Build.ArtifactStagingDirectory)/publishing
displayName: Copy artifacts to Artifact Staging Directory

- ${{ if eq(parameters.targetOS, 'windows') }}:
mmitche marked this conversation as resolved.
Show resolved Hide resolved
- powershell: |
$sourcePath = "$(sourcesPath)/artifacts/VerticalManifest.xml"
$targetPath = "$(Build.ArtifactStagingDirectory)/manifests/$(Agent.JobName).xml"
New-Item -ItemType Directory -Path "$(Build.ArtifactStagingDirectory)/manifests" -Force | Out-Null
Copy-Item $sourcePath -Destination $targetPath -Force
displayName: Copy vertical manifest to Artifact Staging Directory
- ${{ else }}:
- script: |
mkdir -p "$(Build.ArtifactStagingDirectory)/manifests"
cp "$(sourcesPath)/artifacts/VerticalManifest.xml" "$(Build.ArtifactStagingDirectory)/manifests/$(Agent.JobName).xml"
displayName: Copy vertical manifest to Artifact Staging Directory

# When building from source, the Private.SourceBuilt.Artifacts archive already contains the nuget packages
- ${{ if ne(parameters.buildSourceOnly, 'true') }}:
- task: CopyFiles@2
Expand All @@ -610,3 +634,11 @@ jobs:
artifact: $(Agent.JobName)_Artifacts$(artifactsSuffix)
displayName: Publish Artifacts
continueOnError: true

# Using build artifacts to enable publishing the vertical manifests to a single artifact from different jobs
- task: PublishBuildArtifacts@1
MilenaHristova marked this conversation as resolved.
Show resolved Hide resolved
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)/manifests/$(Agent.JobName).xml
ArtifactName: VerticalManifests
displayName: Publish Vertical Manifest
condition: succeededOrFailed()
52 changes: 52 additions & 0 deletions eng/pipelines/templates/stages/vmr-final-join.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
parameters:
# Branch of the VMR to use (to push to for internal builds)
- name: vmrBranch
type: string
default: $(Build.SourceBranch)

- name: pool_Windows
type: object
default:
name: $(defaultPoolName)
image: $(poolImage_Windows)
demands: ImageOverride -equals $(poolImage_Windows)
os: windows

stages:
- stage: VMR_Final_Join
displayName: VMR Final Join
dependsOn: VMR_Vertical_Build
condition: succeededOrFailed()
variables:
- template: ../variables/vmr-build.yml
parameters:
vmrBranch: ${{ parameters.vmrBranch }}

jobs:
- job: FinalJoin
displayName: Final Build Pass
pool: ${{ parameters.pool_Windows }}
timeoutInMinutes: 240
templateContext:
outputs:
- output: buildArtifacts
PathtoPublish: $(Build.ArtifactStagingDirectory)/artifacts/MergedManifest.xml
ArtifactName: AssetManifests
displayName: Publish Merged Manifest
sbomEnabled: false
- output: buildArtifacts
PathtoPublish: $(Build.ArtifactStagingDirectory)/artifacts/assets
ArtifactName: BlobArtifacts
displayName: Publish Blob Artifacts
sbomEnabled: false
- output: buildArtifacts
PathtoPublish: $(Build.ArtifactStagingDirectory)/artifacts/packages
ArtifactName: PackageArtifacts
displayName: Publish Package Artifacts
sbomEnabled: false
steps:
- template: ../steps/vmr-join-verticals.yml
parameters:
dotNetBuildPass: final
primaryDependentJob: Windows_x64
outputFolder: $(Build.ArtifactStagingDirectory)/artifacts
37 changes: 37 additions & 0 deletions eng/pipelines/templates/steps/vmr-join-verticals.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
parameters:
- name: dotNetBuildPass
type: string
default: final

- name: primaryDependentJob
type: string
default: Windows_x64

- name: outputFolder
type: string
default: $(Build.ArtifactStagingDirectory)/artifacts

steps:
- task: DownloadBuildArtifacts@1
inputs:
artifactName: 'VerticalManifests'
downloadPath: $(Build.ArtifactStagingDirectory)
checkDownloadedFiles: true

- task: DownloadPipelineArtifact@2
inputs:
artifactName: ${{ parameters.primaryDependentJob }}_Artifacts
targetPath: $(Build.ArtifactStagingDirectory)/${{ parameters.primaryDependentJob }}_Artifacts
checkDownloadedFiles: true

- powershell: eng/join-verticals.ps1
/p:VerticalManifestsPath=$(Build.ArtifactStagingDirectory)/VerticalManifests
/p:MainVertical=${{ parameters.primaryDependentJob }}
/p:DotNetBuildPass=${{ parameters.dotNetBuildPass }}
/p:BuildId=$(Build.BuildId)
/p:AzureDevOpsToken=$(System.AccessToken)
/p:AzureDevOpsBaseUri=$(System.CollectionUri)
/p:AzureDevOpsProject=$(System.TeamProject)
/p:MainVerticalArtifactsFolder=$(Build.ArtifactStagingDirectory)/${{ parameters.primaryDependentJob }}_Artifacts
/p:OutputFolder=${{ parameters.outputFolder }}
displayName: Join Verticals
3 changes: 3 additions & 0 deletions src/SourceBuild/content/build.proj
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,12 @@
<PropertyGroup>
<BuildModeInfoText Condition="'$(DotNetBuildSourceOnly)' == 'true'">source-build</BuildModeInfoText>
<BuildModeInfoText Condition="'$(DotNetBuildSourceOnly)' != 'true'">non-source-build</BuildModeInfoText>
<BuildPassInfoText Condition="'$(DotNetBuildPass)' == ''">1</BuildPassInfoText>
<BuildPassInfoText Condition="'$(DotNetBuildPass)' != ''">$(DotNetBuildPass)</BuildPassInfoText>
</PropertyGroup>

<Message Text="Build Mode: $(BuildModeInfoText)" Importance="high" />
<Message Text="Build Pass: $(BuildPassInfoText)" Importance="high" />
<Message Text="Build Environment: $(TargetArchitecture) $(Configuration) $(TargetOS) $(TargetRid)" Importance="high" />
<Message Condition="'$(EnablePoison)' == 'true'" Text="Crossgen is disabled due to incompatibility with poison checks." Importance="high" />
</Target>
Expand Down
44 changes: 44 additions & 0 deletions src/SourceBuild/content/eng/join-verticals.proj
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
<Project Sdk="Microsoft.Build.Traversal">

<PropertyGroup>
<TargetFramework>$(NetCurrent)</TargetFramework>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="$(TasksDir)Microsoft.DotNet.UnifiedBuild.Tasks\Microsoft.DotNet.UnifiedBuild.Tasks.csproj" BuildInParallel="true" />
</ItemGroup>

<Target Name="BuildTasksProject">
<MSBuild Projects="$(TasksDir)Microsoft.DotNet.UnifiedBuild.Tasks\Microsoft.DotNet.UnifiedBuild.Tasks.csproj" Targets="Build" />
</Target>
ViktorHofer marked this conversation as resolved.
Show resolved Hide resolved

<UsingTask TaskName="Microsoft.DotNet.UnifiedBuild.Tasks.JoinVerticals" AssemblyFile="$(MicrosoftDotNetUnifiedBuildTasksAssembly)" TaskFactory="TaskHostFactory" />

<Target Name="JoinVerticals" DependsOnTargets="BuildTasksProject" AfterTargets="BuildTasksProject" >
MilenaHristova marked this conversation as resolved.
Show resolved Hide resolved
<Error Condition="'$(MainVertical)' == ''" Text="MainVertical is not set." />
<Error Condition="'$(VerticalManifestsPath)' == ''" Text="VerticalManifestsPath is not set." />
<Error Condition="'$(BuildId)' == ''" Text="BuildId is not set." />
<Error Condition="'$(AzureDevOpsBaseUri)' == ''" Text="AzureDevOpsBaseUri is not set." />
<Error Condition="'$(AzureDevOpsProject)' == ''" Text="AzureDevOpsProject is not set." />
<Error Condition="'$(MainVerticalArtifactsFolder)' == ''" Text="MainVerticalArtifactsFolder is not set." />
<Error Condition="'$(OutputFolder)' == ''" Text="OutputFolder is not set." />

<ItemGroup>
<VerticalManifest
Include="$(VerticalManifestsPath)\*.xml"
Exclude="$(VerticalManifestsPath)\*Mono*.xml;$(VerticalManifestsPath)\*Pgo*.xml;$(VerticalManifestsPath)\*Shortstack*.xml;$(VerticalManifestsPath)\*DevVersions*.xml" />
</ItemGroup>

<!-- AzureDevOpsToken shouldn't be set when running in dnceng-public -->
<Microsoft.DotNet.UnifiedBuild.Tasks.JoinVerticals
VerticalManifest="@(VerticalManifest)"
MainVertical="$(MainVertical)"
BuildId="$(BuildId)"
AzureDevOpsToken="$(AzureDevOpsToken)"
AzureDevOpsBaseUri="$(AzureDevOpsBaseUri)"
AzureDevOpsProject="$(AzureDevOpsProject)"
MainVerticalArtifactsFolder="$(MainVerticalArtifactsFolder)"
OutputFolder="$(OutputFolder)" />
</Target>

</Project>
33 changes: 33 additions & 0 deletions src/SourceBuild/content/eng/join-verticals.ps1
ViktorHofer marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
[CmdletBinding(PositionalBinding=$false)]
Param(
[string][Alias('v')]$verbosity = "minimal",
[Parameter(ValueFromRemainingArguments=$true)][String[]]$properties
)

$useGlobalNuGetCache=$false
$ci = $true

. $PSScriptRoot\common\tools.ps1

$project = Join-Path $EngRoot "join-verticals.proj"
$arguments = @()
$targets = "/t:JoinVerticals"

try {
$bl = '/bl:' + (Join-Path $LogDir 'JoinVerticals.binlog')

MSBuild -restore `
$project `
$bl `
$targets `
/p:Configuration=Release `
@properties `
@arguments
}
catch {
Write-Host $_.ScriptStackTrace
Write-PipelineTelemetryError -Category 'Build' -Message $_
ExitWithExitCode 1
}

ExitWithExitCode 0
3 changes: 2 additions & 1 deletion src/SourceBuild/content/eng/merge-asset-manifests.proj
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@
<Microsoft.DotNet.UnifiedBuild.Tasks.MergeAssetManifests
AssetManifest="@(RepoAssetManifest)"
MergedAssetManifestOutputPath="$(MergedAssetManifestOutputPath)"
VmrBuildNumber="$(BUILD_BUILDNUMBER)" />
VmrBuildNumber="$(BUILD_BUILDNUMBER)"
VerticalName="$(VerticalName)" />
</Target>

</Project>
3 changes: 3 additions & 0 deletions src/SourceBuild/content/eng/pipelines/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,3 +95,6 @@ extends:
scope: lite
${{ else }}:
scope: full

- ${{ if eq(variables['isSourceOnlyBuild'], 'false') }}:
MilenaHristova marked this conversation as resolved.
Show resolved Hide resolved
- template: /src/sdk/eng/pipelines/templates/stages/vmr-final-join.yml@self
3 changes: 3 additions & 0 deletions src/SourceBuild/content/eng/pipelines/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,6 @@ stages:
scope: lite
${{ else }}:
scope: full

- ${{ if eq(variables['isSourceOnlyBuild'], 'false') }}:
ViktorHofer marked this conversation as resolved.
Show resolved Hide resolved
MilenaHristova marked this conversation as resolved.
Show resolved Hide resolved
- template: /src/sdk/eng/pipelines/templates/stages/vmr-post-build.yml@self
ViktorHofer marked this conversation as resolved.
Show resolved Hide resolved
Loading
Loading