Add source-index build job to pipeline #37028

dougbu · 2021-09-27T19:05:33Z

Add source index build step #33534
use 1ES agents because default VS is incompatible with our .NET SDK version
- windows-latest might work but prefer to protect binary log
wait to source-index for other jobs
use binary log from Linux_x64 build
- we're waiting for that build anyhow
only source index the 'main' branch
- anything else would be redundant

nits:

remove unnecessary backslashes in the Linux_x64_build job's scripts
- instead separate commands with a semicolon
confirm test build success before BAR updates in manual builds

.azure/pipelines/ci.yml

wtgodbe · 2021-09-27T19:09:37Z

.azure/pipelines/ci.yml

This list is a little spooky, it seems likely people won't know to update it if/when our set of jobs changes

Been this way forever unfortunately. AzDO schema doesn't provide a way to automagically wait for almost every other job.

Main mitigation is @dotnet/aspnet-build are automagically reviewers whenever a .azure/ file is changed.

wtgodbe

LGTM other than the comment

dougbu · 2021-09-27T19:23:01Z

Fortunately, my typo was in the last commit and didn't invalidate internal build 20210927.7

dougbu · 2021-09-27T20:38:06Z

New internal (real test) build is 20210927.10

.azure/pipelines/ci.yml

dougbu · 2021-09-28T03:54:39Z

Sorry to those playing along for squashing all prior commits on you.

New test build is 20210927.27

davidfowl · 2021-09-28T16:53:53Z

This link is currently broken on http://source.dot.net

dougbu · 2021-09-28T20:45:27Z

The approach of using a binary log from another job isn't going to work at all. The processing seems to require generated files.

Unhandled exception. System.IO.DirectoryNotFoundException: Could not find a part of the path 'D:\a\_work\1\s\artifacts\obj\Microsoft.Extensions.Caching.SqlServer\Release\net7.0\.NETCoreApp,Version=v7.0.AssemblyAttributes.cs'.

My main idea is to build and process the binary log in one job then do the "Upload stage1 artifacts to source index" part using a new artifact from the first job (after the rest of the build completes successfully). The main downside will be not keeping us up-to-date w/ eng/common/templates/job/source-index-stage1.yml. Ideal would be a step template or two from Arcade that installs and runs BinLogToSln (no need for UseDotNet@2 or UploadIndexStage1 there) and another job or step template containing the UploadIndexStage1 part. But, I could copy the bits and pieces over for now.

Thoughts @alexperovich @davidfowl @wtgodbe❔ Other suggestions❔

alexperovich · 2021-09-28T21:37:31Z

would this just be downloading a new artifact from the first build? Would the preSteps parameter not work for this?

dougbu · 2021-09-28T21:44:19Z

would this just be downloading a new artifact from the first build? Would the preSteps parameter not work for this?

Since there's no reason not to proactively run BinLogToSln, my thought was to do that in the first job (together w/ a regular build) then only run UploadIndexStage1 later, after everything else succeeds.

Just using preSteps would require a new artifact containing at least all artifacts/**/*.cs. That wouldn't be too gigantic an artifact but I'm not sure how many other types of files we'd need nor whether we would need generated files from elsewhere in the sources directory.

dougbu · 2021-09-29T17:07:53Z

Have the C# upload / download working now, I hope. Latest test build is https://dev.azure.com/dnceng/internal/_build/results?buildId=1393133&view=results

This time, the exact internal build commit is present here on GitHub too. I just added the main-branch-only commit here. That should help your checks @davidfowl (assuming the test build works).

.azure/pipelines/ci.yml

dougbu · 2021-09-29T17:16:58Z

.azure/pipelines/jobs/default-build.yml

The key word here is "runtime" i.e. the value of artifact.path for the Windows_arm_csharp artifact is $(Build.ArtifactStagingDirectory), not e.g. D:\a\_work\1\a.

.azure/pipelines/ci.yml

alexperovich · 2021-09-29T20:58:18Z

I don't have a problem with running BinLogToSln in a separate job, I can update the code in arcade once we get a design that works for aspnetcore.

- #33534 - start source indexing after other jobs - do not update https://source.dot.net/ from failed builds - use 1ES agents - `windows-latest` might work but prefer to protect binary log - use Windows_arm_build binary log - align binary log with source-indexing agent i.e. always use Windows - much faster than Windows x64/x86 build - provide another binary log, even in internal builds nits: - remove unnecessary backslashes in the Linux_x64_build job's `script`s - instead separate commands with a semicolon - confirm test build success before BAR updates in manual builds - merge a couple of `in(...)` expressions

- required for source-indexing from the binary log

- this approach is taking us down a rabbit hole This reverts commit 033ba5696bbc545538bcf9172825a70e5e7bff49.

- this approach is taking us down a rabbit hole This reverts commit 6426a6195972dc778de4552eb70a978f1ed57adf.

- current template not well suited to reuse a binary log from another job

dougbu · 2021-10-02T04:41:00Z

Very different than previous iterations but working 🎉 Build 20211001.17 was successful and uploaded what looks like a complete set of sources.

Due to how I organized the template expressions, the commits here are identical. https://source.dot.net/ should have no broken links. But, right now, it's using the now-lost commit from my first (prototype) successful run.

dougbu · 2021-10-02T04:48:25Z

.azure/pipelines/ci.yml

+- name: testSourceIndexing
+  default: false
+  displayName: Test source indexing? !Danger! see comments in YAML.
+  type: boolean


This parameter is visible in the AzDO UI but, as mentioned in the comments, ignored in public builds.

.azure/pipelines/ci.yml

dougbu · 2021-10-04T02:29:10Z

@alexperovich, @wtgodbe is OOF now. I'd appreciate you double-checking this PR before I merge it. Thanks…

dougbu added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Sep 27, 2021

dougbu requested review from a team, KirillOsenkov, alexperovich and davidfowl September 27, 2021 19:05

dougbu commented Sep 27, 2021

View reviewed changes

.azure/pipelines/ci.yml Outdated Show resolved Hide resolved

wtgodbe reviewed Sep 27, 2021

View reviewed changes

wtgodbe approved these changes Sep 27, 2021

View reviewed changes

alexperovich approved these changes Sep 27, 2021

View reviewed changes

dougbu force-pushed the dougbu/source.indexing.33534 branch from 7838407 to 90a72d6 Compare September 27, 2021 20:37

dougbu commented Sep 27, 2021

View reviewed changes

.azure/pipelines/ci.yml Outdated Show resolved Hide resolved

dougbu mentioned this pull request Sep 27, 2021

Add source index build step #33534

Closed

dougbu force-pushed the dougbu/source.indexing.33534 branch from 90a72d6 to 6e94598 Compare September 28, 2021 03:46

dougbu commented Sep 28, 2021

View reviewed changes

.azure/pipelines/ci.yml Outdated Show resolved Hide resolved

.azure/pipelines/ci.yml Outdated Show resolved Hide resolved

dougbu force-pushed the dougbu/source.indexing.33534 branch from 6e94598 to a138fbb Compare September 28, 2021 05:32

dougbu force-pushed the dougbu/source.indexing.33534 branch from a138fbb to 7023165 Compare September 29, 2021 17:04

dougbu commented Sep 29, 2021

View reviewed changes

.azure/pipelines/ci.yml Outdated Show resolved Hide resolved

dougbu force-pushed the dougbu/source.indexing.33534 branch from 7023165 to ebbcd42 Compare September 29, 2021 22:26

dougbu mentioned this pull request Oct 1, 2021

Skip Helix tests when skipping tests #37171

Merged

dougbu added 6 commits October 1, 2021 17:45

Save and reuse generated files from Windows ARM

c60f33e

- required for source-indexing from the binary log

Copy some assemblies around too

8d6db59

Revert "Copy some assemblies around too"

e7e8971

- this approach is taking us down a rabbit hole This reverts commit 033ba5696bbc545538bcf9172825a70e5e7bff49.

Revert "Save and reuse generated files from Windows ARM"

a5419b4

- this approach is taking us down a rabbit hole This reverts commit 6426a6195972dc778de4552eb70a978f1ed57adf.

Copy source-index-stage1 template into ci.yml

0c8e33c

- current template not well suited to reuse a binary log from another job

Add testSourceIndexing parameter

9721e92

dougbu force-pushed the dougbu/source.indexing.33534 branch from ebbcd42 to 9721e92 Compare October 2, 2021 04:23

dougbu commented Oct 2, 2021

View reviewed changes

alexperovich approved these changes Oct 4, 2021

View reviewed changes

dougbu merged commit fc286da into main Oct 4, 2021

dougbu deleted the dougbu/source.indexing.33534 branch October 4, 2021 20:54

ghost added this to the 7.0-preview1 milestone Oct 4, 2021

Add source-index build job to pipeline #37028

Add source-index build job to pipeline #37028

Uh oh!

Conversation

dougbu commented Sep 27, 2021

Uh oh!

Uh oh!

wtgodbe Sep 27, 2021

Choose a reason for hiding this comment

Uh oh!

dougbu Sep 27, 2021

Choose a reason for hiding this comment

Uh oh!

wtgodbe left a comment

Choose a reason for hiding this comment

Uh oh!

dougbu commented Sep 27, 2021

Uh oh!

dougbu commented Sep 27, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dougbu commented Sep 28, 2021

Uh oh!

davidfowl commented Sep 28, 2021

Uh oh!

dougbu commented Sep 28, 2021

Uh oh!

alexperovich commented Sep 28, 2021

Uh oh!

dougbu commented Sep 28, 2021

Uh oh!

dougbu commented Sep 29, 2021

Uh oh!

Uh oh!

Uh oh!

dougbu Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexperovich commented Sep 29, 2021

Uh oh!

dougbu commented Oct 2, 2021

Uh oh!

dougbu Oct 2, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dougbu commented Oct 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants