Update performance tests to build against netcoreapp

[jkotas]

Performance tests are building against .NET Standard 1.4: https://github.com/dotnet/coreclr/blob/master/tests/src/JIT/config/benchmark/benchmark.csproj#L5

This is a problem for testing netcoreapp-specific APIs. We have hit with MemoryMarshal.CreateSpan API that was changed to netcoreapp-specific. The test for this API was disable via #if false and link to this issue: dotnet/coreclr#16091

category:correctness
theme:testing
skill-level:expert
cost:small

[RussKeldorph]

Would this supercede #8991? There was some debate about what is appropriate there. Do we all agree this issue is the path forward? @DrewScoggins @jorive @AndyAyersMS

[AndyAyersMS]

In the past we've wanted to keep perf tests building against a lowest common denominator so we could run the exact same tests on older releases to assess release to release perf changes.

I don't believe we need to support running current tests against 1.1 anymore, but it would be nice to be able to run them vs 2.0. So maybe we need to multi-target this test or just let the 2.0 version diverge.

Moving up to at least netstandard2.0 would let us clean up some clunky code that searches for related files (#9281).

For whatever reason updating the test dependencies have proven difficult.

[jorive]

@jkotas Why not create a tests/src/JIT/config/benchmark+span/benchmark.csproj where it targets a different netstandard? I do not see a reason to move all benchmarks for this reason.
Or even better, we could create different configs/projects like this: tests/src/JIT/config/benchmark+netstandard[1.1|1.2|1.3|1.4|1.5|1.6|2.0] as needed :)

[jkotas]

Sounds fine to me.

[jkotas]

I do think you really need the tests/src/JIT/config/benchmark+netstandard* .csproj files. They have a lot of cruft in them, e.g. the 4.4.0-beta-24913-0 package versions do not look right. I think you should be just able to reference netstandard1.1/.../2.0 TFM in the .csproj file.

[jorive]

I do not know what was the initial reasoning behind these config/csproj, but I do agree with each test being able to specify its own target.

[AndyAyersMS]

I'll start working on this.

[AndyAyersMS]

Because we produce a unified environment for all the CoreCLR tests, all the projects involved have to agree on which version of a package we'll use, both for building and running. We have complex web of dependencies here which makes updating all this challenging (at least for us jit developers). I know previous attempts have all hit roadblocks.

I'm not sure if it is worth the trouble to keep individual projects referencing the minimum level of standard they require, since this is something we'd need to discover test by test. It seems a lot simpler to baseline all test projects at the same version, say netstandard 2.0, and then make exceptions for the tests that need 2.1. I realize this means we can't back port the perf test changes to the 1.1 branch, but I think that's ok.

[AndyAyersMS]

The dependencies were updated via dotnet/coreclr#16378, so the disabled tests can now be re-enabled.

@ahsonkhan is doing this as part of dotnet/coreclr#16521. 😕

[AndyAyersMS]

Had to revert the changes in dotnet/coreclr#16378 via dotnet/coreclr#16647.

[AndyAyersMS]

Will get to this after 2.1 is out the door.

[AndyAyersMS]

No longer relevant.