Add source index build step

[davidfowl]

See dotnet/runtime#46823 for more details.

cc @alexperovich
cc @dougbu

[dougbu]

I thought source indexing was something covered automatically (along w/ SDL checks) in our release pipeline❔

Separately, I'm curious how source indexing relates to rich navigation. Please explain.

@jepetty set up rich navigation for us in https://github.com/dotnet/aspnetcore/blob/main/.azure/pipelines/richnav.yml but I can't find the actual pipeline (@jepetty❔).

/cc @dotnet/aspnet-build

[davidfowl]

This is for https://source.dot.net/ I think but I dunno how the 2 are related.

[dougbu]

Thanks @davidfowl for this and the mention in the internal conversation. Makes much more sense now❕

I added this to the dashboard we (the build v-team) use to track work. Seems pretty simple…

/fyi @Pilchie

[alexperovich]

To simplify this change, if the repo uses eng/common/templates/jobs/jobs.yml, you can just set the "enableSourceIndex" parameter there. If the code to be indexed gets built by default with eng/common/build.ps1, then you can stop there. Otherwise the sourceIndexBuildCommand value in the sourceIndexParams parameter will need to be set to the command that will produce a binlog containing builds of all the projects to be indexed.

[dougbu]

@alexperovich can we use enableSourceIndex at a lower level e.g. in a single job that already produces a source index❔

Separately (kinda) we build multiple times to handle native assets and avoid desktop msbuild otherwise. Our builds therefore produce a minimum of three binary logs where binlogs are enabled (and we don't enable them often -- too big). Does the source index need just the binary log for building our C# code❔ Is it possible to tell your stuff which binlog to use❔

[alexperovich]

it only needs to be in one job, and that job can only build C#, or whatever you need. The only requirement is that it produce a binlog in the normal place that contains all the csc commands for the code that should show up on source.dot.net

[alexperovich]

it will actually read all the binlogs, so if it produces multiple that is fine, but one is faster.

[KirillOsenkov]

How do we get this in? I'm getting complaints that these assemblies aren't on https://source.dot.net

[dougbu]

I'm getting complaints that these assemblies aren't on https://source.dot.net

We (a) missed the urgency in this issue -- everything was quiet since June and (b) only just learned the simple approach.

---

The next question is which Eng channels contain Arcade bit containing enableSourceIndex. The corresponding dotnet/aspnetcore branches are where we'd fix things in the next few days.

Separately, is this also an issue for dotnet/efcore code❔

[alexperovich]

Its just a part of the arcade sdk, the source index bits are delivered with the eng/common folder and the sdk.

[dougbu]

Its just a part of the arcade sdk, the source index bits are delivered with the eng/common folder and the sdk.

@alexperovich I understand that part. What I'm not sure is whether, say, the '.NET 3 Eng' bits include the feature.

[alexperovich]

I doubt they do, I added the code to arcade/main, and there is only one source-index so it should be done on main. Release branches don't need anything.