Rewrite complexity_test to use (hardcoded) manual time #1757
Conversation
LebedevRI
force-pushed
the
complexity
branch
2 times, most recently
from
b2496c8 to
0806db7
Compare
|
what happened? this seemed like a good idea... |
|
I don't understand what the heck is going on with automatic complexity fitting, |
This test is fundamentally flaky, because it tried to read tea leafs, and is inherently misbehaving in CI environments, since there are unmitigated sources of noise. That being said, the computed Big-O also depends on the `--benchmark_min_time=` Fixes google#272
LebedevRI
force-pushed
the
complexity
branch
from
0806db7 to
4aa8882
Compare
|
OOOOH this is a fun one! |
LebedevRI
force-pushed
the
complexity
branch
9 times, most recently
from
bd48182 to
6107f69
Compare
LebedevRI
force-pushed
the
complexity
branch
from
6107f69 to
13e8159
Compare
|
@dmah42 ok, this is more like it. |
LebedevRI
force-pushed
the
complexity
branch
from
2781dd7 to
6a4f5b9
Compare
Looks like on windows, some of these tests still fail, i guess clock precision is too small.
LebedevRI
force-pushed
the
complexity
branch
from
6a4f5b9 to
8f64ed2
Compare
| @@ -96,6 +96,7 @@ BenchmarkReporter::Run CreateRunReport( | |||
| } else { | |||
| report.real_accumulated_time = results.real_time_used; | |||
| } | |||
| report.use_real_time_for_initial_big_o = b.use_manual_time(); | |||
There was a problem hiding this comment.
to check this logic:
if we're using manual time, we'll report manual time as "real" but then use "real" time for bigO.
if we're not using manual time, then we'll report real time as "real" but use "cpu" time for bigO.
maybe this could be a comment somewhere just to track the interaction between manual, real, and cpu times for bigO.
| @@ -186,8 +186,19 @@ std::vector<BenchmarkReporter::Run> ComputeBigO( | |||
| result_cpu = MinimalLeastSq(n, cpu_time, reports[0].complexity_lambda); | |||
| result_real = MinimalLeastSq(n, real_time, reports[0].complexity_lambda); | |||
| } else { | |||
| result_cpu = MinimalLeastSq(n, cpu_time, reports[0].complexity); | |||
| result_real = MinimalLeastSq(n, real_time, result_cpu.complexity); | |||
| const BigO* InitialBigO = &reports[0].complexity; | |||
There was a problem hiding this comment.
this section needs some comments. maybe there's a way to simplify the logic?
| BENCHMARK(BM_Complexity_O1) | ||
| ->Range(1, 1 << 18) | ||
| ->UseManualTime() |
There was a problem hiding this comment.
does this mean complexity only works for manual time? do we have any tests where we use complexity with real time?
|
@dmah42 we always compute complexity for both cpu and real time, |
|
@dmah42 thank you. Looks like there's some residual flakiness in CI left, so i'll see if these nits can be revisited later... |
This test is fundamentally flaky, because it tried to read tea leafs, and is inherently misbehaving in CI environments,
since there are unmitigated sources of noise.
Fixes #272
Fixes #1758