Regressions in System.IO.Tests.BinaryReaderTests

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	851c274cc50544bf11a59e1011dbc594ca7010b0
Compare	5233eea73e60915d849b945dccec150bc57827c6
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Tests.BinaryReaderTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ReadHalf - Duration of single invocation 📈 - ADX Test Multi Config Graph	5.29 ns	6.29 ns	1.19	0.08	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.BinaryReaderTests*'

Payloads

Baseline
Compare

System.IO.Tests.BinaryReaderTests.ReadHalf

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/13/2023 7:46:10 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -15.773360937890658 (T) = (0 -6.424371170856452) / Math.Sqrt((0.0010989257886529443 / (11)) + (0.03524672407763912 / (7))) is less than -2.1199052992212764 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (7) - 2, .025) and -0.21350717532034122 = (5.29405289190857 - 6.424371170856452) / 5.29405289190857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[EgorBo]

Also dotnet/perf-autofiling-issues#23338

Regressed by #80331 cc @teo-tsirpanis

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	851c274cc50544bf11a59e1011dbc594ca7010b0
Compare	5233eea73e60915d849b945dccec150bc57827c6
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Tests.BinaryReaderTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ReadHalf - Duration of single invocation 📈 - ADX Test Multi Config Graph	5.29 ns	6.29 ns	1.19	0.08	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.BinaryReaderTests*'

Payloads

Baseline
Compare

System.IO.Tests.BinaryReaderTests.ReadHalf

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/13/2023 7:46:10 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -15.773360937890658 (T) = (0 -6.424371170856452) / Math.Sqrt((0.0010989257886529443 / (11)) + (0.03524672407763912 / (7))) is less than -2.1199052992212764 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (7) - 2, .025) and -0.21350717532034122 = (5.29405289190857 - 6.424371170856452) / 5.29405289190857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	arch-arm64, area-System.IO, os-linux, tenet-performance, tenet-performance-benchmarks, untriaged, runtime-coreclr
Milestone:	-

[teo-tsirpanis]

Weird, on Linux it improved. 🤔

[adamsitnik]

Weird, on Linux it improved.

It has improved for all OSes on x64, but regressed on arm64.

[adamsitnik]

I think it's OK to accept this tradeoff.