Not possible to compute the (Ratio, RatioSD)

[sastru]

I get the warning

// * Warnings *
BaselineCustomAnalyzer
Summary -> A question mark '?' symbol indicates that it was not possible to compute the (Ratio, RatioSD) column(s) because the baseline value is too close to zero.

I suspect it is because somehow all my benchmarks are separated by a line, why? how can I fix that?

Program.cs

BenchmarkRunner.Run<Benchmarks>(DefaultConfig.Instance
    .AddJob(Job.InProcess
        .WithStrategy(RunStrategy.Monitoring)
        .DontEnforcePowerPlan()
        .WithToolchain(new InProcessEmitToolchain(true))
    ), args);

Benchmarks.cs

...
[Benchmark(Baseline = true)]
[Arguments(1028, 1028, false, OutsideQueryVariation.Default, false, null, 30)]
public async Task Baseline(int limit, int totalCount, bool includeOutsidePoints, OutsideQueryVariation useOutsideQueryVariation, bool useMaxOutsideInterval, int? status, int timeSpanInMin) { .. }

[Benchmark]
[Arguments(1028,  1028,  false, OutsideQueryVariation.Default,  false, 192,  30)]
[Arguments(1028,  1030,  true,  OutsideQueryVariation.Default,  false, null, 30)]
public async Task Benchmark(int limit, int totalCount, bool includeOutsidePoints, OutsideQueryVariation useOutsideQueryVariation, bool useMaxOutsideInterval, int? status, int timeSpanInMin) { .. }

[AndreyAkinshin]

@sastru thanks for the report, looks like a bug. I filed an issue: #2591

[timcassell]

Are we sure it's a bug? To me, it looks like benchmarks are only comparable if their inputs are the same, and this is working as intended. Perhaps we just need a new feature to force logical groupings.

[AndreyAkinshin]

The current behaviour (summary + warning) is confusing and misleading, so we definitely have a room for improvement.

[timcassell]

The warning is confusing, because it doesn't actually have a baseline to compare to. It seems there's 2 separate concerns here. The warning message being incorrect, and the expected logical groupings.

[timcassell]

This seems closely related to #881 (though it's for arguments instead of params).

[timcassell]

If you refactor like this, you should get the expected groupings (though you will lose all the arguments in the summary table).

[Benchmark(Baseline = true)]
public Task Baseline() => Impl(1028, 1028, false, OutsideQueryVariation.Default, false, null, 30);
[Benchmark]
public Task Alt1() => Impl(1028,  1028,  false, OutsideQueryVariation.Default,  false, 192,  30);
[Benchmark]
public Task Alt2() => Impl(1028,  1030,  true,  OutsideQueryVariation.Default,  false, null, 30);

private async Task Impl(int limit, int totalCount, bool includeOutsidePoints, OutsideQueryVariation useOutsideQueryVariation, bool useMaxOutsideInterval, int? status, int timeSpanInMin)
{ ... }

[sastru]

I have like 40 permutations, missing the arguments in the summary table would be painful when presenting the benchmarks to decision makers. 😓