Add VMR unified build pipelines #18409

akoeplinger · 2024-01-26T13:10:06Z

Contributes to dotnet/source-build#3823

Removes and consolidates vmr-cross-build.yml/ci-cross-build.yml POC into vmr-build.yml/ci.yml
Add stage for building vertical/unified build legs into vmr-build.yml
Adds Windows build and handling for Windows/Unix differences

MichaelSimons

Can you please link to sample runs.

src/SourceBuild/content/eng/pipelines/templates/stages/vmr-scan.yml

src/SourceBuild/content/eng/pipelines/ci.yml

eng/pipelines/templates/jobs/vmr-build.yml

src/SourceBuild/content/eng/pipelines/ci.yml

eng/pipelines/templates/stages/vmr-build.yml

src/SourceBuild/content/eng/pipelines/ci.yml

eng/pipelines/templates/variables/vmr-stage.yml

akoeplinger · 2024-02-01T14:20:05Z

I've incorporated the feedback and here's the suggested pipeline setup that we discussed on Teams (let me know if I transcribed that correctly into the .yml):

ultralite = 1 leg for smoke tests
lite = several legs e.g. win/linux/mac for basic testing
full = everything

## Public pipelines

installer-source-build (vmr-build.yml)
 - PR (release/* and main, ultralite build)

installer-unified-build (vmr-build.yml)
 - PR (release/* and main, lite build)

dotnet-source-build (ci.yml)
 - PR (ultralite build)
 - CI (release/* only, every batched commit, full build)
 - Schedule (main only, full build)

dotnet-unified-build (ci.yml)
 - PR (lite build)
 - CI (release/* only, every batched commit, full build)
 - Schedule (main only, full build)


## Internal pipelines

dotnet-source-build (ci.yml)
 - PR (ultralite build)
 - CI (release/* and internal/release/* only, every batched commit, full build)
 - CI (main only, every batched commit, lite build)
 - Schedule (main only, full build)

dotnet-source-build-lite (ci-lite.yml)
 - PR (release/* and main, lite build)
 - CI (main only, every batched commit, lite build)

dotnet-unified-build (ci.yml)
 - PR (lite build)
 - CI (release/*, internal/release/* and main, every batched commit, full build)

I removed vmr-build-internal.yml since it is only used by the dotnet-installer-source-build-internal pipeline and I think that can be 8.0-only ?

ViktorHofer

This looks great 💯

eng/vmr-sync.ps1

ViktorHofer · 2024-02-05T12:26:56Z

/azp run installer-unified-build

azure-pipelines · 2024-02-05T12:27:08Z

Azure Pipelines successfully started running 1 pipeline(s).

akoeplinger · 2024-02-05T17:28:29Z

/azp run installer-unified-build

azure-pipelines · 2024-02-05T17:28:40Z

Azure Pipelines successfully started running 1 pipeline(s).

akoeplinger · 2024-02-05T17:38:12Z

Triggered internal builds with these changes:

MichaelSimons · 2024-02-05T18:04:20Z

Triggered internal builds with these changes:

Can you trigger a full source-build run?

akoeplinger · 2024-02-05T18:52:45Z

Hmm I triggered https://dev.azure.com/dnceng/internal/_build/results?buildId=2370500&view=results (manual run of the vmr-vertical-pipeline branch) but it didn't trigger the full scope even though it should based on how I see the if/elseif/else. Will take a look.

akoeplinger · 2024-02-05T19:25:30Z

Ok, found the reason: variables are strings and casting to bool uses empty/non-empty as false/true so you need to explicitly compare to "true"

Here's the new source-build "full" run: https://dev.azure.com/dnceng/internal/_build/results?buildId=2370535&view=results

ViktorHofer · 2024-02-05T21:34:37Z

The full VMR build is green and doesn't show any new regressions compared to the last scheduled build. @MichaelSimons any more feedback? Otherwise I would like to get this in so that we can test some other PRs with the new pipeline.

MichaelSimons · 2024-02-05T23:45:22Z

The full VMR build is green and doesn't show any new regressions compared to the last scheduled build. @MichaelSimons any more feedback? Otherwise I would like to get this in so that we can test some other PRs with the new pipeline.

Nope - the should build test runs passed. I will sign off.

MichaelSimons

Thanks for all your work on this @akoeplinger!

Follow-up from #18409

… log (#18501) Noticed in dotnet/installer#18409 that errors aren't propagated which resulted in builds not to fail. The fix here is to use `OnError` in the correct target (but only when using the minimal console log feature) and not set `IgnoreStandardErrorWarningFormat=true` on the Exec task that invokes the repo build script. I also cleaned targets up as those introduced unnecessary complexity and logged in cases when they shouldn't (i.e. in dotnet.proj or package-source-build.proj). This affected runtime which errors after 30s of building because of RuntimeOS and BaseOS being passed in in-correctly. Regressed with dotnet/installer@a265140#diff-86602308e6bb519266bc2f224ea65e39589d273804d40ad0f9c6e0eea2a263dc. Fixed that in runtime.proj. Kudos to Alexander who made the fix. I just copied it in form his CI PR.

… log (#18501) Noticed in dotnet/installer#18409 that errors aren't propagated which resulted in builds not to fail. The fix here is to use `OnError` in the correct target (but only when using the minimal console log feature) and not set `IgnoreStandardErrorWarningFormat=true` on the Exec task that invokes the repo build script. I also cleaned targets up as those introduced unnecessary complexity and logged in cases when they shouldn't (i.e. in dotnet.proj or package-source-build.proj). This affected runtime which errors after 30s of building because of RuntimeOS and BaseOS being passed in in-correctly. Regressed with dotnet/installer@37ddec7#diff-86602308e6bb519266bc2f224ea65e39589d273804d40ad0f9c6e0eea2a263dc. Fixed that in runtime.proj. Kudos to Alexander who made the fix. I just copied it in form his CI PR.

akoeplinger force-pushed the vmr-vertical-pipeline branch 7 times, most recently from 5791eeb to cb1d852 Compare January 26, 2024 13:32

Add VMR unified build pipelines

2df04fe

akoeplinger force-pushed the vmr-vertical-pipeline branch from cb1d852 to 2df04fe Compare January 26, 2024 13:34

dotnet deleted a comment from azure-pipelines bot Jan 26, 2024

akoeplinger marked this pull request as ready for review January 26, 2024 14:44

akoeplinger requested a review from a team as a code owner January 26, 2024 14:44

MichaelSimons reviewed Jan 26, 2024

View reviewed changes

MichaelSimons requested review from ellahathaway and mthalman January 26, 2024 15:28

This comment was marked as outdated.

Sign in to view

ellahathaway reviewed Jan 26, 2024

View reviewed changes

src/SourceBuild/content/eng/pipelines/ci.yml Outdated Show resolved Hide resolved

src/SourceBuild/content/eng/pipelines/ci.yml Outdated Show resolved Hide resolved

src/SourceBuild/content/eng/pipelines/ci.yml Outdated Show resolved Hide resolved

premun reviewed Jan 30, 2024

View reviewed changes

eng/pipelines/templates/variables/vmr-stage.yml Show resolved Hide resolved

eng/pipelines/templates/variables/vmr-stage.yml Show resolved Hide resolved

akoeplinger added 9 commits January 31, 2024 15:51

PR feedback: don't condition vmr-scan.yml

1a64f7e

PR feedback: simplify docker and build args

261e76d

PR feedback: remove _Online_MsftSdk suffix

fcadc9c

PR feedback: combine targetRid cases

d458b91

PR feedback: always run vmr-scan

42a4f8d

PR feedback: vmrBranch parameter

dffb44f

Only pass --source-only flag for source only builds

450f818

Rework pipeline triggers

7079a8f

Fix yml

1002956

akoeplinger added 5 commits February 3, 2024 15:47

PR feedback: condition

583fb52

PR feedback: only use docker in Prep the Build

238d067

PR feedback: combine conditions and cleanup

3d63407

PR feedback: condition on runTests

27c3d48

Merge remote-tracking branch 'upstream/main' into vmr-vertical-pipeline

ac7e1cd

ViktorHofer approved these changes Feb 5, 2024

View reviewed changes

premun reviewed Feb 5, 2024

View reviewed changes

eng/vmr-sync.ps1 Show resolved Hide resolved

ViktorHofer mentioned this pull request Feb 5, 2024

Support for short stack builds. #18463

Merged

Remove triggers from ci.yml and simplify ci-lite.yml

4822608

Fix variable cast to boolean

2f5232f

MichaelSimons approved these changes Feb 5, 2024

View reviewed changes

akoeplinger merged commit 28012e7 into main Feb 6, 2024

akoeplinger deleted the vmr-vertical-pipeline branch February 6, 2024 08:57

akoeplinger added a commit that referenced this pull request Feb 6, 2024

Don't run vmr-scan.yml for PRs in ci.yml

b83a384

Follow-up from #18409

akoeplinger mentioned this pull request Feb 6, 2024

Don't run vmr-scan.yml for PRs in ci.yml and treat main-ub as a main branch #18515

Merged

ViktorHofer mentioned this pull request Feb 6, 2024

VMR pipelines dotnet/source-build#4074

Closed

3 tasks

This was referenced Feb 14, 2024

Implement Vertical Build for MacOS x64 dotnet/source-build#3875

Closed

Implement Vertical Build for Windows x64 dotnet/source-build#3890

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add VMR unified build pipelines #18409

Add VMR unified build pipelines #18409

akoeplinger commented Jan 26, 2024 •

edited

Loading

MichaelSimons left a comment

This comment was marked as outdated.

This comment was marked as outdated.

akoeplinger commented Feb 1, 2024 •

edited

Loading

ViktorHofer left a comment

ViktorHofer commented Feb 5, 2024

azure-pipelines bot commented Feb 5, 2024

akoeplinger commented Feb 5, 2024

azure-pipelines bot commented Feb 5, 2024

akoeplinger commented Feb 5, 2024 •

edited

Loading

MichaelSimons commented Feb 5, 2024

akoeplinger commented Feb 5, 2024

akoeplinger commented Feb 5, 2024

ViktorHofer commented Feb 5, 2024 •

edited

Loading

MichaelSimons commented Feb 5, 2024

MichaelSimons left a comment

Add VMR unified build pipelines #18409

Add VMR unified build pipelines #18409

Conversation

akoeplinger commented Jan 26, 2024 • edited Loading

MichaelSimons left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

akoeplinger commented Feb 1, 2024 • edited Loading

ViktorHofer left a comment

Choose a reason for hiding this comment

ViktorHofer commented Feb 5, 2024

azure-pipelines bot commented Feb 5, 2024

akoeplinger commented Feb 5, 2024

azure-pipelines bot commented Feb 5, 2024

akoeplinger commented Feb 5, 2024 • edited Loading

MichaelSimons commented Feb 5, 2024

akoeplinger commented Feb 5, 2024

akoeplinger commented Feb 5, 2024

ViktorHofer commented Feb 5, 2024 • edited Loading

MichaelSimons commented Feb 5, 2024

MichaelSimons left a comment

Choose a reason for hiding this comment

akoeplinger commented Jan 26, 2024 •

edited

Loading

akoeplinger commented Feb 1, 2024 •

edited

Loading

akoeplinger commented Feb 5, 2024 •

edited

Loading

ViktorHofer commented Feb 5, 2024 •

edited

Loading