Update runtime metrics tests to avoid failures on some platforms

[stevejgordon]

This updates the jit metrics assertions to only assertions for greater than zero measurements when the JIT appears to have run.

I've further limited the GcCollectionsCount test to allow zero measurements, which is best in cases where a forced GC may not have run.

This should fix #105202 and #105203.

What do you think of this approach @tarekgh? We could also more stringently skip platforms with an IsNotMobile check, but I think it's worth having these run if possible while allowing zero values for measurements when those occur. We mostly want to test that the metrics return a measurement for the expected metric name, and the value is less important and less deterministic.

[matouskozak]

/azp run runtime-ioslikesimulator,runtime-maccatalyst

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[stevejgordon]

@matouskozak I'm taking a look at the latest failures.

[stevejgordon]

@matouskozak I'm hoping the next run is good!

[matouskozak]

/azp run runtime-ioslikesimulator,runtime-maccatalyst

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[tarekgh]

@stevejgordon thanks for helping with that. Is it possible to add the fix for #105203 in this PR too?

[stevejgordon]

I was hoping this would solve those too.

[matouskozak]

/azp run runtime-androidemulator

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tarekgh]

@stevejgordon looks the tests still failing. Most likely we need to skip running the cpu time test on iOS and tvos I guess.

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-105300-merge-aedfb0c0c29d4254ad/System.Diagnostics.DiagnosticSource.Tests/1/console.c59c4be1.log?helixlogtype=result

https://dev.azure.com/dnceng-public/public/_build/results?buildId=752200&view=logs&j=cd385208-11d3-5d70-fa38-1c21a65d17f1&t=e9ee4177-53d1-5e6c-e1b1-ff3d9929ea47&l=55

[13:36:59] dbug: 2024-07-23 13:36:59.630777-0400 System.Diagnostics.DiagnosticSource.Tests[19264:232555] Failed tests:
[13:36:59] dbug: 2024-07-23 13:36:59.631174-0400 System.Diagnostics.DiagnosticSource.Tests[19264:232555] 1) 	[FAIL] System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.CpuTime   Test name: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.CpuTime
[13:36:59] dbug: 2024-07-23 13:36:59.631497-0400 System.Diagnostics.DiagnosticSource.Tests[19264:232555]    Assembly:  [System.Diagnostics.DiagnosticSource.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
[13:36:59] dbug: Exception messages: Expected to find a measurement for 'user' CPU mode.   Exception stack traces:    at System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.CpuTime()
[13:36:59] dbug: 2024-07-23 13:36:59.631689-0400 System.Diagnostics.DiagnosticSource.Tests[19264:232555]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
[13:36:59] dbug: at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
[13:36:59] dbug: Execution time: 0.1062381
[13:36:59] dbug: 2024-07-23 13:36:59.632417-0400 System.Diagnostics.DiagnosticSource.Tests[19264:232555] 2) 	[FAIL] System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements   Test name: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements<Int64>(metricName: "dotnet.process.memory.working_set", valueAssertion: Func`2 { Method = System.ValueTuple`2[System.Boolean,System.String] <.cctor>b__23_1(Int64), Target = <>c { } }, beforeRecord: null)   Test case: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements
[13:36:59] dbug: 2024-07-23 13:36:59.632794-0400 System.Diagnostics.DiagnosticSource.Tests[19264:232555]    Assembly:  [System.Diagnostics.DiagnosticSource.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
[13:36:59] dbug: 2024-07-23 13:36:59.633169-0400 System.Diagnostics.DiagnosticSource.Tests[19264:232555]    Exception messages: Expected value to be greater than zero. Actual value was: 0.   Exception stack traces:    at System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateSingleMeasurement[Int64](String metricName, Func`2 valueAssertion, Func`1 beforeRecord)
[13:36:59] dbug: at System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements[Int64](String metricName, Func`2 valueAssertion, Func`1 beforeRecord)
[13:36:59] dbug: 2024-07-23 13:36:59.633534-0400 System.Diagnostics.DiagnosticSource.Tests[19264:232555]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)

[matouskozak]

/azp run runtime-ioslikesimulator,runtime-maccatalyst,runtime-androidemulator

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).

[tarekgh]

@stevejgordon looks we need more tweaks; it is failing in tvos.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=753129&view=logs&j=cd385208-11d3-5d70-fa38-1c21a65d17f1&t=e9ee4177-53d1-5e6c-e1b1-ff3d9929ea47&l=55

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-105300-merge-ef15edb3920b4b6280/System.Diagnostics.DiagnosticSource.Tests/1/console.acc52e6c.log?helixlogtype=result

[02:58:43] dbug: 2024-07-24 02:58:43.682251-0400 System.Diagnostics.DiagnosticSource.Tests[54344:3234808] 1) 	[FAIL] System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements   Test name: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements<Int64>(metricName: "dotnet.process.memory.working_set", valueAssertion: Func`2 { Method = System.ValueTuple`2[System.Boolean,System.String] <.cctor>b__23_1(Int64), Target = <>c { } }, beforeRecord: null)   Test case: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements
[02:58:43] dbug: 2024-07-24 02:58:43.682362-0400 System.Diagnostics.DiagnosticSource.Tests[54344:3234808]    Assembly:  [System.Diagnostics.DiagnosticSource.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
[02:58:43] dbug: 2024-07-24 02:58:43.682476-0400 System.Diagnostics.DiagnosticSource.Tests[54344:3234808]    Exception messages: Expected value to be greater than zero. Actual value was: 0.   Exception stack traces:    at System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateSingleMeasurement[Int64](String metricName, Func`2 valueAssertion, Func`1 beforeRecord)
[02:58:43] dbug: at System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements[Int64](String metricName, Func`2 valueAssertion, Func`1 beforeRecord)
[02:58:43] dbug: 2024-07-24 02:58:43.682575-0400 System.Diagnostics.DiagnosticSource.Tests[54344:3234808]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)

[stevejgordon]

@tarekgh, sorry for the delay. I've been knocked out with a virus. I'll hopefully be back to work next week and can tweak this then if that's not too late.

[tarekgh]

@stevejgordon I have added a small change to the PR hopefully this will make the test pass. I have rebased with the main to ensure we have the latest from the main.

[tarekgh]

/azp run runtime-ioslikesimulator,runtime-maccatalyst,runtime-androidemulator

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).

[tarekgh]

/ba-g the failing test didn't run at all and not producing any logs. https://dev.azure.com/dnceng-public/public/_build/results?buildId=760893&view=ms.vss-test-web.build-test-results-tab&runId=19324756&resultId=100028&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

[matouskozak]

/ba-g the failing test didn't run at all and not producing any logs. https://dev.azure.com/dnceng-public/public/_build/results?buildId=760893&view=ms.vss-test-web.build-test-results-tab&runId=19324756&resultId=100028&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

Was it intended to disable the tests on mobile by 7109a74 ? Based on the PR description, I thought the idea of this PR was to modify the tests so that they are able to run on mobile as well.

From tvos-sim log:

2024-07-30 15:16:33.209 Df System.Diagnostics.DiagnosticSource.Tests[61224:1d8725] 	[IGNORED] System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements

[tarekgh]

Look at my comment #105300 (comment). The test was failing on the tvos too. I don't think we can claim these metrics are supported on such platforms without doing more work I guess.