Replace KoreBuild with Arcade #11122

natemcmaster · 2019-06-12T00:28:18Z

Part of #7280

High level changes

Update build.sh/ps1 to call into Arcade instead of KoreBuild. This completely removes our usage of KoreBuild
Move all contents in build/ (the KoreBuild convention) into their matching version in eng/
Remove dead code and other features which were KoreBuild only
- Remove usage of NuGetPackageVerifier
- Remove signcheck
- Remove AddAllProjectRefsToSolution.ps1 (Resolves AddAllProjectRefsToSolution fails for Middleware solution #7740 )

TODO (before this PR is ready to merge)

Get tests running
Ensure we get the same build output
Ensure code gen steps still work
Consume a new update of Arcade which has necessary fixes
- Cannot specify dotnet/x86 runtime and still build on *nix arcade#3006
- Support detecting VS BuildTools arcade#3025

TODO (will be done in a separate PR)
* This removed support for generating Maestro manifests during build. We'll need to re-enable this somehow. handled in this PR by keeping existing Maestro manifest generation

Use the Arcade YAML templates for CI definition
Fix site extensions build SiteExtensions do not build with Arcade #11186
Run sign check SignValidationExclusionList does not work arcade#2888

…rSolutionBuild.targets

build.ps1

* capture test results in xunit form * attempt to fix code check * restore before linux build * remove duplicate signinfos

natemcmaster · 2019-06-12T21:36:26Z

Hmm, SiteExtensions are super broken by this. I'm looking at the targets which implement the site extension build and big chunks will probably need to be rewritten to work with Arcade.

@JunTaoLuo would you be okay temporarily disabling the site extension projects and doing that separately?

JunTaoLuo · 2019-06-12T21:38:08Z

Sure let's separate that work out of this PR. Let's discuss what the issues are.

* exclude node_modules from unique project check * fixup signing props

* Remove unused NoWarns * Skip building site extension * Suppress xunit color in console output * Install x86 runtime

JunTaoLuo · 2019-06-18T16:52:43Z

@aspnet-hello triage https://dev.azure.com/dnceng/public/_build/results?buildId=228611&view=logs.

aspnet-hello · 2019-06-18T16:57:11Z

This comment was made automatically. If there is a problem contact aspnetcore-build@microsoft.com.

I've triaged the above build. I've created/commented on the following issue(s)
https://github.com/aspnet/AspNetCore-Internal/issues/2623
https://github.com/aspnet/AspNetCore-Internal/issues/2410
https://github.com/aspnet/AspNetCore-Internal/issues/2415

JunTaoLuo · 2019-06-18T18:07:29Z

Tests are passing! Just need to verify build output before merging.

Directory.Build.props

eng/AfterSolutionBuild.targets

src/ProjectTemplates/Web.Spa.ProjectTemplates/Microsoft.DotNet.Web.Spa.ProjectTemplates.csproj

JunTaoLuo · 2019-06-19T00:10:16Z

Just missing a signoff from @dougbu and another run of the tests to see if there are any flakiness.

natemcmaster · 2019-06-19T03:44:34Z

Nice work! Thanks for following through on this John!

dougbu

Generally looks good (though I'm not sure I really got everything in this huge PR)

dougbu · 2019-06-19T02:20:31Z

.azure/pipelines/ci.yml

            -arch x86
-            -NoRestore


Why is this gone? Same for a few other -NoRestore and --no-restore removals.

Hmm I might want to investigate this separately. Specifically because I know there were some changes with lifetimes of targets and interleaving of steps introduced with converting to arcade so these changes may be deliberate. Since I wouldn't want to hold up the PR and this is more of an optimization, I would do this as a follow up.

.azure/pipelines/helix-test.yml

dougbu · 2019-06-19T04:40:40Z

...uilder/test/Microsoft.AspNetCore.FunctionalTests/Microsoft.AspNetCore.FunctionalTests.csproj

+    <ProjectReference Include="$(RepoRoot)src\DefaultBuilder\testassets\DependencyInjectionApp\DependencyInjectionApp.csproj" ReferenceOutputAssemblies="false" />
+    <ProjectReference Include="$(RepoRoot)src\DefaultBuilder\testassets\StartRequestDelegateUrlApp\StartRequestDelegateUrlApp.csproj" ReferenceOutputAssemblies="false" />
+    <ProjectReference Include="$(RepoRoot)src\DefaultBuilder\testassets\StartRouteBuilderUrlApp\StartRouteBuilderUrlApp.csproj" ReferenceOutputAssemblies="false" />
+    <ProjectReference Include="$(RepoRoot)src\DefaultBuilder\testassets\StartWithIApplicationBuilderUrlApp\StartWithIApplicationBuilderUrlApp.csproj" ReferenceOutputAssemblies="false" />


Are these new references to projects not mentioned elsewhere showing up due to merges from 'master'?

I added these references to ensure build order between test assets and functional tests.

dougbu · 2019-06-19T04:41:44Z

src/Hosting/Hosting/test/WebHostBuilderTests.cs

@@ -173,7 +173,8 @@ public void ConfigureDefaultServiceProvider(IWebHostBuilder builder)
                    options.ValidateScopes = true;
                });

-            Assert.Throws<InvalidOperationException>(() => hostBuilder.Build().Start());
+            using var host = hostBuilder.Build();
+            Assert.Throws<InvalidOperationException>(() => host.Start());


Lots 'o C# changes from here on also look odd. Are these also due to merges from 'master'?

No this is to prevent a hang. Not disposing generic host builders caused this so I had to add using in several places. @Tratcher is this the issue tacking that bug dotnet/extensions#1363?

JunTaoLuo · 2019-06-19T05:01:47Z

If the next round of tests succeed, I'll conclude that these changes have not introduced any additional flakiness (a total of 3 successes) in our builds and will merge. I'll also file follow up issues.

JunTaoLuo · 2019-06-19T05:47:31Z

Followup items:

jkotalik · 2019-06-19T15:07:03Z

@JunTaoLuo I'll also make a follow-up PR to fix potential native dll copying issues. Let's get this in!

Helix completed, not sure why the PR check isn't showing green/red.

dougbu · 2019-06-19T15:49:49Z

@aspnet-hello triage https://dev.azure.com/dnceng/public/_build/results?buildId=230108

aspnet-hello · 2019-06-19T15:53:23Z

This comment was made automatically. If there is a problem contact aspnetcore-build@microsoft.com.

I've triaged the above build. I've created/commented on the following issue(s)
https://github.com/aspnet/AspNetCore-Internal/issues/2489

dougbu · 2019-06-19T15:54:54Z

Helix completed, not sure why the PR check isn't showing green/red.

We're still seeing the occasional missed messages between AzDO and GitHub. Since the Helix failure is of a known-flaky test, I suggest we get this in. @JunTaoLuo please do the honours.

This was referenced Jun 12, 2019

Support detecting VS BuildTools dotnet/arcade#3025

Merged

Converge "KoreBuild" and Arcade dotnet/arcade#88

Closed

natemcmaster added 15 commits June 11, 2019 20:48

Replace usages of $(CI) with $(ContinuousIntegrationBuild)

15a5a2c

Move RepoTasks to eng/tools/

d0671e6

Remove dead code

9707264

Update build.sh/ps1 to invoke Arcade instead of KoreBuild

8b3cec5

build/repo.props => eng/Build.props

d419e8f

build/CodeSign.{props,targets} => eng/Signing.props

18a8744

build/repo.targets => eng/Build.props, eng/CodeGen.proj, and eng/Afte…

90b3b8e

…rSolutionBuild.targets

Remove AddAllProjectRefsToSolution.ps1

bb0e21b

Put selenium process tracking into artifacts/tmp/

4b5151d

build/docker => eng/docker

67204a7

Move content from korebuild.json to global.json

db3e3c6

Remove cibuild.{sh,cmd} and update ci.yml

48ac512

Ensure vswhere detects BuildTools installations

963d192

Remove sign check exclusions (Arcade does not support)

45c02fd

list VS required components in global.json

9618ad7

Eilon added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Jun 12, 2019

dougbu reviewed Jun 12, 2019

View reviewed changes

build.ps1 Show resolved Hide resolved

JunTaoLuo self-requested a review June 12, 2019 17:22

Update CodeCheck.ps1

31563be

natemcmaster force-pushed the namc/arcade branch from af61c9d to 31563be Compare June 12, 2019 18:59

natemcmaster mentioned this pull request Jun 12, 2019

Consider updating Kestrel error message about missing dev cert #11145

Closed

John Luo and others added 2 commits June 12, 2019 13:46

Update arcade (#11143)

f4bf319

Fixups to broken build

4903f13

* capture test results in xunit form * attempt to fix code check * restore before linux build * remove duplicate signinfos

natemcmaster added 3 commits June 12, 2019 22:48

More build fixes

015a94f

* exclude node_modules from unique project check * fixup signing props

Merge branch 'master' into namc/arcade

a6641e3

More build fixes

4ed04d2

* Remove unused NoWarns * Skip building site extension * Suppress xunit color in console output * Install x86 runtime

John Luo added 5 commits June 17, 2019 17:54

Disable signalr npm tests

7a963cf

Merge branch 'master' into namc/arcade

dad5e7a

Debug template test

36f5457

Merge branch 'master' into namc/arcade

3f7f6b9

Try to fix templates again and make steps more reliable

84cb2cd

JunTaoLuo approved these changes Jun 18, 2019

View reviewed changes

jkotalik approved these changes Jun 18, 2019

View reviewed changes

Directory.Build.props Show resolved Hide resolved

JunTaoLuo reviewed Jun 19, 2019

View reviewed changes

eng/AfterSolutionBuild.targets Show resolved Hide resolved

JunTaoLuo reviewed Jun 19, 2019

View reviewed changes

src/ProjectTemplates/Web.Spa.ProjectTemplates/Microsoft.DotNet.Web.Spa.ProjectTemplates.csproj Outdated Show resolved Hide resolved

John Luo added 2 commits June 18, 2019 17:08

Cleanup

f2490db

Merge branch 'master' into namc/arcade

449cc0e

dougbu approved these changes Jun 19, 2019

View reviewed changes

Feedback

9a42266

This was referenced Jun 19, 2019

Re-enable SignalR npm functional tests in Arcade. #11360

Closed

Investigate no restore options for build steps on AzDO #11361

Closed

JunTaoLuo merged commit 4fde84a into master Jun 19, 2019

ghost deleted the namc/arcade branch June 19, 2019 17:03

JunTaoLuo mentioned this pull request Jul 5, 2019

Convert to arcade YAML templates #11924

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace KoreBuild with Arcade #11122

Replace KoreBuild with Arcade #11122

natemcmaster commented Jun 12, 2019 •

edited by JunTaoLuo

Loading

natemcmaster commented Jun 12, 2019

JunTaoLuo commented Jun 12, 2019

JunTaoLuo commented Jun 18, 2019

aspnet-hello commented Jun 18, 2019

JunTaoLuo commented Jun 18, 2019

JunTaoLuo commented Jun 19, 2019

natemcmaster commented Jun 19, 2019

dougbu left a comment

dougbu Jun 19, 2019

JunTaoLuo Jun 19, 2019

dougbu Jun 19, 2019

JunTaoLuo Jun 19, 2019

dougbu Jun 19, 2019

JunTaoLuo Jun 19, 2019

Tratcher Jun 19, 2019

JunTaoLuo commented Jun 19, 2019

JunTaoLuo commented Jun 19, 2019

jkotalik commented Jun 19, 2019 •

edited

Loading

dougbu commented Jun 19, 2019

aspnet-hello commented Jun 19, 2019

dougbu commented Jun 19, 2019

Replace KoreBuild with Arcade #11122

Replace KoreBuild with Arcade #11122

Conversation

natemcmaster commented Jun 12, 2019 • edited by JunTaoLuo Loading

natemcmaster commented Jun 12, 2019

JunTaoLuo commented Jun 12, 2019

JunTaoLuo commented Jun 18, 2019

aspnet-hello commented Jun 18, 2019

JunTaoLuo commented Jun 18, 2019

JunTaoLuo commented Jun 19, 2019

natemcmaster commented Jun 19, 2019

dougbu left a comment

Choose a reason for hiding this comment

dougbu Jun 19, 2019

Choose a reason for hiding this comment

JunTaoLuo Jun 19, 2019

Choose a reason for hiding this comment

dougbu Jun 19, 2019

Choose a reason for hiding this comment

JunTaoLuo Jun 19, 2019

Choose a reason for hiding this comment

dougbu Jun 19, 2019

Choose a reason for hiding this comment

JunTaoLuo Jun 19, 2019

Choose a reason for hiding this comment

Tratcher Jun 19, 2019

Choose a reason for hiding this comment

JunTaoLuo commented Jun 19, 2019

JunTaoLuo commented Jun 19, 2019

jkotalik commented Jun 19, 2019 • edited Loading

dougbu commented Jun 19, 2019

aspnet-hello commented Jun 19, 2019

dougbu commented Jun 19, 2019

natemcmaster commented Jun 12, 2019 •

edited by JunTaoLuo

Loading

jkotalik commented Jun 19, 2019 •

edited

Loading