Selenium tests no longer run locally in VS

[SteveSandersonMS]

Maybe I missed a memo about something we're meant to do differently now, but I think this problem was introduced here: f76be64#diff-d25609f62dd6bb91ad82d484d1f32bfc

Since that commit, local builds in VS no longer write the necessary assembly metadata to enable the Selenium tests, so you can't run and debug them from the IDE. We definitely need to be able to do that.

@JunTaoLuo Is there a new and intended way to run those tests from VS? Or am I just doing something wrong?

[JunTaoLuo]

The motivation for the original issue we were trying to solve (#10504) is that building the repo doesn't require Java and should not need to specify -NoBuildJava. I can look into potential implementations where builds don't check for Java installations but tests will still fail without Java. In the meantime, the expected workflow we had in mind is that you would set BuildJava as an environment variable or passed in via command line arguments. In the meantime, you should be able to resolve the issue by launching via a shell using startvs.cmd with the environment variable set.

[JunTaoLuo]

cc @mkArtakMSFT

[SteveSandersonMS]

I don’t think it’s ok to have the requirement that people reverse-engineer our build system to figure out what magic environment variables to pass to VS just so they can run tests.

It’s not just about engineers at Microsoft - this is meant to be an open source project, which implies external developers should be able to make sense of how to contribute.

[JunTaoLuo]

The alternative is that ./build.cmd will fail for everyone working in this repo, even for people working in areas that don't run Seleium tests.

Again, I'm not proposing a permanent workflow change, but to unblock, see if setting the environment variable would resolve your issue. I agree that we shouldn't require external developers reverse engineering our builds, but we'll need to investigate possible improvements.

Some alternatives that come to mind include add a warning that selenium tests are not running and instructions on how to enable them, or run tests automatically if Java is detected but skip them if Java is not found.

[mkArtakMSFT]

@JunTaoLuo I thought we agreed earlier to default the flag to a value, which would result simple ./build.cmd to succeed. Are you saying that's the goal, but it's not straight forward, so for now we'll need to have a workaround?

[JunTaoLuo]

I realized that the solution we were considering probably won't be sufficient. Modifying https://github.com/aspnet/AspNetCore/blob/master/src/Components/build.cmd isn't enough because the actual issue here is test not build. We can run tests in two ways: command line and in VS. If we edited build.cmd and tell devs to test via ./build.cmd -Test, that would fix the command line test workflow but I don't think that's what most people are doing. Also, changing the behaviour of build.cmd will not affect how VS runs tests which is also a requirement here. Hence, I suggested the workaround of setting the environment variable for now while we investigate if there's a better way to implement this. I have made two proposals but open to any other ideas.

[rynowak]

Hence, I suggested the workaround of setting the environment variable for now while we investigate if there's a better way to implement this.

I'm still blocked on this. Setting the environment variable is not enough to make this work.

System.ArgumentNullException: Value cannot be null.
Parameter name: path2
   at System.IO.Path.Combine(String path1, String path2)
   at Microsoft.AspNetCore.Blazor.DevServer.Server.Startup.ResolveApplicationAssemblyFullPath(IWebHostEnvironment environment) in C:\git\aspnet\AspNetCore\src\Components\Blazor\DevServer\src\Server\Startup.cs:line 59
   at Microsoft.AspNetCore.Blazor.DevServer.Server.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment environment, IConfiguration configuration) in C:\git\aspnet\AspNetCore\src\Components\Blazor\DevServer\src\Server\Startup.cs:line 38
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.MethodInfoExtensions.InvokeWithoutWrappingExceptions(MethodInfo methodInfo, Object obj, Object[] parameters) in C:\git\aspnet\AspNetCore\src\Hosting\Hosting\src\Internal\MethodInfoExtensions.cs:line 17
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder) in C:\git\aspnet\AspNetCore\src\Hosting\Hosting\src\Internal\ConfigureBuilder.cs:line 56
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder) in C:\git\aspnet\AspNetCore\src\Hosting\Hosting\src\Internal\ConfigureBuilder.cs:line 20
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.Configure(IApplicationBuilder app) in C:\git\aspnet\AspNetCore\src\Hosting\Hosting\src\Startup\ConventionBasedStartup.cs:line 25
   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app) in C:\git\aspnet\AspNetCore\src\DefaultBuilder\src\HostFilteringStartupFilter.cs:line 17
   at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication() in C:\git\aspnet\AspNetCore\src\Hosting\Hosting\src\Internal\WebHost.cs:line 228

[mkArtakMSFT]

@JunTaoLuo, @dougbu this is still blocking the team. I'm marking this as P0 so it gets enough attention.