Implementation of random interleaving. #1105

haihuang-ml · 2021-03-28T19:34:39Z

See http://github.com/google/benchmark/issues/1051 for the feature requests.

Committer: Hai Huang (http://github.com/haih-g)

On branch fr-1051
Changes to be committed:
modified: include/benchmark/benchmark.h
modified: src/benchmark.cc
new file: src/benchmark_adjust_repetitions.cc
new file: src/benchmark_adjust_repetitions.h
modified: src/benchmark_api_internal.cc
modified: src/benchmark_api_internal.h
modified: src/benchmark_register.cc
modified: src/benchmark_runner.cc
modified: src/benchmark_runner.h
modified: test/CMakeLists.txt
new file: test/benchmark_random_interleaving_gtest.cc

http://github.com/google/benchmark/issues/1051 for the feature requests. Committer: Hai Huang (http://github.com/haih-g) On branch fr-1051 Changes to be committed: modified: include/benchmark/benchmark.h modified: src/benchmark.cc new file: src/benchmark_adjust_repetitions.cc new file: src/benchmark_adjust_repetitions.h modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: src/benchmark_register.cc modified: src/benchmark_runner.cc modified: src/benchmark_runner.h modified: test/CMakeLists.txt new file: test/benchmark_random_interleaving_gtest.cc

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc modified: src/benchmark_runner.cc modified: test/benchmark_random_interleaving_gtest.cc

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: src/benchmark_runner.cc

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_runner.cc

…-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: test/benchmark_random_interleaving_gtest.cc

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: test/benchmark_random_interleaving_gtest.cc

dmah42

there's a lot here but i've left some initial comments.

it would also be good to see the results of a run of something like the basic tests with and without interleaving enabled.

src/benchmark.cc

haihuang-ml · 2021-03-30T05:16:01Z

Without Random Interleaving

$ build/test/benchmark_test
2021-03-30T01:12:21-04:00
Running build/test/benchmark_test
Run on (8 X 2200 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x4)
  L1 Instruction 32 KiB (x4)
  L2 Unified 256 KiB (x4)
  L3 Unified 56320 KiB (x1)
Load Average: 0.83, 1.31, 0.91
-----------------------------------------------------------------------------------
Benchmark                                         Time             CPU   Iterations
-----------------------------------------------------------------------------------
BM_Factorial                                   3963 ns         2186 ns            1 40320
BM_Factorial/real_time                         1859 ns         1714 ns            1 40320
BM_CalculatePiRange/1                          1633 ns         1581 ns            1 0
BM_CalculatePiRange/8                          1739 ns         1639 ns            1 3.28374
BM_CalculatePiRange/64                         1848 ns         1770 ns            1 3.15746
BM_CalculatePiRange/512                        3225 ns         3138 ns            1 3.14355
BM_CalculatePiRange/4096                      15139 ns        15004 ns            1 3.14184
BM_CalculatePiRange/32768                    120244 ns       120441 ns            1 3.14162
BM_CalculatePiRange/262144                   833346 ns       833730 ns            1 3.1416
BM_CalculatePiRange/1048576                 3568729 ns      3569552 ns            1 3.14159
BM_CalculatePi/threads:8                        596 ns         4678 ns            8
BM_CalculatePi/threads:1                       5256 ns         4864 ns            1
BM_CalculatePi/threads:2                       6815 ns        13596 ns            2
BM_CalculatePi/threads:4                       1287 ns         5105 ns            4
BM_CalculatePi/threads:8                        607 ns         4807 ns            8
BM_CalculatePi/threads:16                       338 ns         4898 ns           16
BM_CalculatePi/threads:32                       127 ns         3893 ns           32
BM_CalculatePi/threads:8                        354 ns         2662 ns            8
BM_SetInsert/1024/128                         32948 ns        32498 ns            1 bytes_per_second=15.025M/s items_per_second=3.9387M/s
BM_SetInsert/4096/128                         28031 ns        27960 ns            1 bytes_per_second=17.4636M/s items_per_second=4.57797M/s
BM_SetInsert/8192/128                         33194 ns        33106 ns            1 bytes_per_second=14.749M/s items_per_second=3.86637M/s
BM_SetInsert/1024/512                         80817 ns        80508 ns            1 bytes_per_second=24.26M/s items_per_second=6.35962M/s
BM_SetInsert/4096/512                         81093 ns        81047 ns            1 bytes_per_second=24.0987M/s items_per_second=6.31732M/s
BM_SetInsert/8192/512                        116500 ns       116322 ns            1 bytes_per_second=16.7907M/s items_per_second=4.40157M/s
BM_Sequential<std::vector<int>,int>/1          2272 ns         1854 ns            1 bytes_per_second=2.05755M/s items_per_second=539.374k/s
BM_Sequential<std::vector<int>,int>/8          1380 ns         1370 ns            1 bytes_per_second=22.2756M/s items_per_second=5.83942M/s
BM_Sequential<std::vector<int>,int>/64         5267 ns         5200 ns            1 bytes_per_second=46.9501M/s items_per_second=12.3077M/s
BM_Sequential<std::vector<int>,int>/512       81292 ns        81352 ns            1 bytes_per_second=24.0083M/s items_per_second=6.29364M/s
BM_Sequential<std::vector<int>,int>/1024       2508 ns         2502 ns            1 bytes_per_second=1.52466G/s items_per_second=409.273M/s
BM_Sequential<std::list<int>>/1                1000 ns          986 ns            1 bytes_per_second=3.86886M/s items_per_second=1014.2k/s
BM_Sequential<std::list<int>>/8                3538 ns         3534 ns            1 bytes_per_second=8.63542M/s items_per_second=2.26372M/s
BM_Sequential<std::list<int>>/64               4909 ns         4904 ns            1 bytes_per_second=49.784M/s items_per_second=13.0506M/s
BM_Sequential<std::list<int>>/512             29050 ns        29078 ns            1 bytes_per_second=67.1685M/s items_per_second=17.6078M/s
BM_Sequential<std::list<int>>/1024            50999 ns        51065 ns            1 bytes_per_second=76.4956M/s items_per_second=20.0529M/s
BM_Sequential<std::vector<int>, int>/512      11810 ns        11855 ns            1 bytes_per_second=164.751M/s items_per_second=43.1885M/s
BM_StringCompare/1                             1092 ns         1071 ns            1
BM_StringCompare/8                             1041 ns         1028 ns            1
BM_StringCompare/64                            1010 ns         1003 ns            1
BM_StringCompare/512                           1049 ns         1041 ns            1
BM_StringCompare/4096                          1079 ns         1079 ns            1
BM_StringCompare/32768                         2068 ns         2072 ns            1
BM_StringCompare/262144                       16743 ns        16529 ns            1
BM_StringCompare/1048576                      82413 ns        82304 ns            1
BM_SetupTeardown/threads:8                     1567 ns         9832 ns            8
BM_LongTest/65536                            235058 ns       234905 ns            1
BM_LongTest/262144                           885519 ns       885945 ns            1
BM_LongTest/2097152                         7299766 ns      7300481 ns            1
BM_LongTest/16777216                       59865919 ns     59531136 ns            1
BM_LongTest/134217728                     459461403 ns    457979182 ns            1
BM_LongTest/268435456                     918575224 ns    917545465 ns            1
BM_ParallelMemset/10485760/threads:1        4054349 ns      4054427 ns            1
BM_ParallelMemset/10485760/threads:2        1037273 ns      2059556 ns            2
BM_ParallelMemset/10485760/threads:4         283396 ns      1098383 ns            4
BM_ManualTiming/1/real_time                   82348 ns        18913 ns            1 items_per_second=12.1436k/s
BM_ManualTiming/8/real_time                   77933 ns        10890 ns            1 items_per_second=102.652k/s
BM_ManualTiming/64/real_time                 134085 ns        10297 ns            1 items_per_second=477.309k/s
BM_ManualTiming/512/real_time                583666 ns         8756 ns            1 items_per_second=877.214k/s
BM_ManualTiming/4096/real_time              4187032 ns        17825 ns            1 items_per_second=978.259k/s
BM_ManualTiming/16384/real_time            16472653 ns        19631 ns            1 items_per_second=994.618k/s
BM_ManualTiming/1/manual_time                 74830 ns        11672 ns            1 items_per_second=13.3636k/s
BM_ManualTiming/8/manual_time                 76543 ns        10155 ns            1 items_per_second=104.516k/s
BM_ManualTiming/64/manual_time               132944 ns         9969 ns            1 items_per_second=481.406k/s
BM_ManualTiming/512/manual_time              582629 ns        10053 ns            1 items_per_second=878.775k/s
BM_ManualTiming/4096/manual_time            4226672 ns        19723 ns            1 items_per_second=969.084k/s
BM_ManualTiming/16384/manual_time          16484715 ns        23253 ns            1 items_per_second=993.89k/s
BM_with_args/int_test                          1120 ns         1051 ns            1
BM_with_args/string_and_pair_test               991 ns          979 ns            1
BM_non_template_args/basic_test                1009 ns         1026 ns            1
BM_DenseThreadRanges/1/threads:1                977 ns          982 ns            1
BM_DenseThreadRanges/1/threads:2               1203 ns         1764 ns            2
BM_DenseThreadRanges/1/threads:3               2013 ns         1542 ns            3
BM_DenseThreadRanges/2/threads:1               1239 ns         1062 ns            1
BM_DenseThreadRanges/2/threads:3                395 ns         1063 ns            3
BM_DenseThreadRanges/2/threads:4                335 ns         1147 ns            4
BM_DenseThreadRanges/3/threads:5                253 ns         1128 ns            5
BM_DenseThreadRanges/3/threads:8                188 ns         1259 ns            8
BM_DenseThreadRanges/3/threads:11               120 ns         1255 ns           11
BM_DenseThreadRanges/3/threads:14              92.2 ns         1129 ns           14

With Random Interleaving

$ build/test/benchmark_test --benchmark_enable_random_interleaving
2021-03-30T01:14:26-04:00
Running build/test/benchmark_test
Run on (8 X 2200 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x4)
  L1 Instruction 32 KiB (x4)
  L2 Unified 256 KiB (x4)
  L3 Unified 56320 KiB (x1)
Load Average: 1.17, 1.42, 1.01
------------------------------------------------------------------------------------------
Benchmark                                                Time             CPU   Iterations
------------------------------------------------------------------------------------------
BM_Factorial                                          1885 ns         1673 ns            1 40320
BM_Factorial                                          1104 ns         1097 ns            1 40320
BM_Factorial                                          1204 ns         1204 ns            1 40320
BM_Factorial                                          1308 ns         1275 ns            1 40320
BM_Factorial_mean                                     1375 ns         1312 ns            4 40320
BM_Factorial_median                                   1256 ns         1239 ns            4 40320
BM_Factorial_stddev                                    350 ns          251 ns            4 40320
BM_Factorial/real_time                                1948 ns         1809 ns            1 40320
BM_Factorial/real_time                                2025 ns         1698 ns            1 40320
BM_Factorial/real_time                                1212 ns         1196 ns            1 40320
BM_Factorial/real_time_mean                           1728 ns         1568 ns            3 40320
BM_Factorial/real_time_median                         1948 ns         1698 ns            3 40320
BM_Factorial/real_time_stddev                          449 ns          327 ns            3 40320
BM_CalculatePiRange/1                                 1232 ns         1200 ns            1 0
BM_CalculatePiRange/1                                 1962 ns         1763 ns            1 0
BM_CalculatePiRange/1_mean                            1597 ns         1482 ns            2 0
BM_CalculatePiRange/1_median                          1597 ns         1482 ns            2 0
BM_CalculatePiRange/1_stddev                           516 ns          398 ns            2 0
BM_CalculatePiRange/8                                 1083 ns         1078 ns            1 3.28374
BM_CalculatePiRange/8                                 1200 ns         1250 ns            1 3.28374
BM_CalculatePiRange/8_mean                            1142 ns         1164 ns            2 3.28374
BM_CalculatePiRange/8_median                          1142 ns         1164 ns            2 3.28374
BM_CalculatePiRange/8_stddev                          82.7 ns          122 ns            2 3.28374
BM_CalculatePiRange/64                                3817 ns         1968 ns            1 3.15746
BM_CalculatePiRange/64                                2242 ns         2146 ns            1 3.15746
BM_CalculatePiRange/64                                1210 ns         1188 ns            1 3.15746
BM_CalculatePiRange/64                                2456 ns         2531 ns            1 3.15746
BM_CalculatePiRange/64_mean                           2431 ns         1958 ns            4 3.15746
BM_CalculatePiRange/64_median                         2349 ns         2057 ns            4 3.15746
BM_CalculatePiRange/64_stddev                         1072 ns          565 ns            4 3.15746
BM_CalculatePiRange/512                               4154 ns         4027 ns            1 3.14355
BM_CalculatePiRange/512                               1969 ns         1928 ns            1 3.14355
BM_CalculatePiRange/512                               1968 ns         1940 ns            1 3.14355
BM_CalculatePiRange/512_mean                          2697 ns         2632 ns            3 3.14355
BM_CalculatePiRange/512_median                        1969 ns         1940 ns            3 3.14355
BM_CalculatePiRange/512_stddev                        1262 ns         1208 ns            3 3.14355
BM_CalculatePiRange/4096                             10847 ns        10771 ns            1 3.14184
BM_CalculatePiRange/4096                             11247 ns        11207 ns            1 3.14184
BM_CalculatePiRange/4096                              7307 ns         7294 ns            1 3.14184
BM_CalculatePiRange/4096                              7299 ns         7271 ns            1 3.14184
BM_CalculatePiRange/4096_mean                         9175 ns         9136 ns            4 3.14184
BM_CalculatePiRange/4096_median                       9077 ns         9033 ns            4 3.14184
BM_CalculatePiRange/4096_stddev                       2168 ns         2147 ns            4 3.14184
BM_CalculatePiRange/32768                            48069 ns        48044 ns            1 3.14162
BM_CalculatePiRange/32768                            49780 ns        49775 ns            1 3.14162
BM_CalculatePiRange/32768                            48087 ns        48058 ns            1 3.14162
BM_CalculatePiRange/32768                            47848 ns        47888 ns            1 3.14162
BM_CalculatePiRange/32768                            67074 ns        67133 ns            1 3.14162
BM_CalculatePiRange/32768                            47990 ns        48029 ns            1 3.14162
BM_CalculatePiRange/32768                            49615 ns        49647 ns            1 3.14162
BM_CalculatePiRange/32768                            48030 ns        47994 ns            1 3.14162
BM_CalculatePiRange/32768                            49817 ns        49817 ns            1 3.14162
BM_CalculatePiRange/32768                            75234 ns        75282 ns            1 3.14162
BM_CalculatePiRange/32768                           103238 ns       102896 ns            1 3.14162
BM_CalculatePiRange/32768_mean                       57707 ns        57688 ns           11 3.14162
BM_CalculatePiRange/32768_median                     49615 ns        49647 ns           11 3.14162
BM_CalculatePiRange/32768_stddev                     17690 ns        17609 ns           11 3.14162
BM_CalculatePiRange/262144                          462995 ns       463002 ns            1 3.1416
BM_CalculatePiRange/262144                          453519 ns       454600 ns            1 3.1416
BM_CalculatePiRange/262144                          609242 ns       609906 ns            1 3.1416
BM_CalculatePiRange/262144                          401616 ns       401813 ns            1 3.1416
BM_CalculatePiRange/262144                          505822 ns       506699 ns            1 3.1416
BM_CalculatePiRange/262144                          398796 ns       399123 ns            1 3.1416
BM_CalculatePiRange/262144                          622734 ns       624134 ns            1 3.1416
BM_CalculatePiRange/262144                          444521 ns       420865 ns            1 3.1416
BM_CalculatePiRange/262144                          395735 ns       395783 ns            1 3.1416
BM_CalculatePiRange/262144                          429722 ns       429985 ns            1 3.1416
BM_CalculatePiRange/262144                          815558 ns       815797 ns            1 3.1416
BM_CalculatePiRange/262144                          894011 ns       865769 ns            1 3.1416
BM_CalculatePiRange/262144_mean                     536189 ns       532290 ns           12 3.1416
BM_CalculatePiRange/262144_median                   458257 ns       458801 ns           12 3.1416
BM_CalculatePiRange/262144_stddev                   167461 ns       163432 ns           12 3.1416
BM_CalculatePiRange/1048576                        3202652 ns      3205087 ns            1 3.14159
BM_CalculatePiRange/1048576                        1694699 ns      1695075 ns            1 3.14159
BM_CalculatePiRange/1048576                        1613708 ns      1613774 ns            1 3.14159
BM_CalculatePiRange/1048576                        1636065 ns      1636364 ns            1 3.14159
BM_CalculatePiRange/1048576                        1669860 ns      1670143 ns            1 3.14159
BM_CalculatePiRange/1048576                        1721356 ns      1652747 ns            1 3.14159
BM_CalculatePiRange/1048576                        1572047 ns      1572094 ns            1 3.14159
BM_CalculatePiRange/1048576                        1737978 ns      1738402 ns            1 3.14159
BM_CalculatePiRange/1048576                        1745102 ns      1718382 ns            1 3.14159
BM_CalculatePiRange/1048576                        3653504 ns      3360496 ns            1 3.14159
BM_CalculatePiRange/1048576                        1684214 ns      1684546 ns            1 3.14159
BM_CalculatePiRange/1048576                        1602507 ns      1602738 ns            1 3.14159
BM_CalculatePiRange/1048576_mean                   1961141 ns      1929154 ns           12 3.14159
BM_CalculatePiRange/1048576_median                 1689456 ns      1677345 ns           12 3.14159
BM_CalculatePiRange/1048576_stddev                  694022 ns       634959 ns           12 3.14159
BM_CalculatePi/threads:8                               530 ns         3950 ns            8
BM_CalculatePi/threads:1                              5360 ns         4530 ns            1
BM_CalculatePi/threads:1                              2630 ns         2604 ns            1
BM_CalculatePi/threads:1                              4112 ns         3930 ns            1
BM_CalculatePi/threads:1                              2767 ns         2677 ns            1
BM_CalculatePi/threads:1                              4078 ns         3876 ns            1
BM_CalculatePi/threads:1                              2578 ns         2513 ns            1
BM_CalculatePi/threads:1                              2442 ns         2399 ns            1
BM_CalculatePi/threads:1                              2872 ns         2755 ns            1
BM_CalculatePi/threads:1                              2621 ns         2582 ns            1
BM_CalculatePi/threads:1                              4899 ns         4771 ns            1
BM_CalculatePi/threads:1_mean                         3436 ns         3264 ns           10
BM_CalculatePi/threads:1_median                       2820 ns         2716 ns           10
BM_CalculatePi/threads:1_stddev                       1081 ns          913 ns           10
BM_CalculatePi/threads:2                              2356 ns         4053 ns            2
BM_CalculatePi/threads:4                               852 ns         2956 ns            4
BM_CalculatePi/threads:8                               618 ns         4730 ns            8
BM_CalculatePi/threads:16                              212 ns         3191 ns           16
BM_CalculatePi/threads:32                              109 ns         3649 ns           32
BM_CalculatePi/threads:8                               410 ns         3079 ns            8
BM_SetInsert/1024/128                                31285 ns        31004 ns            1 bytes_per_second=15.749M/s items_per_second=4.1285M/s
BM_SetInsert/1024/128                                25183 ns        24985 ns            1 bytes_per_second=19.543M/s items_per_second=5.12307M/s
BM_SetInsert/1024/128                                23597 ns        23374 ns            1 bytes_per_second=20.8899M/s items_per_second=5.47617M/s
BM_SetInsert/1024/128                                25602 ns        25504 ns            1 bytes_per_second=19.1453M/s items_per_second=5.01882M/s
BM_SetInsert/1024/128_mean                           26417 ns        26217 ns            4 bytes_per_second=18.8318M/s items_per_second=4.93664M/s
BM_SetInsert/1024/128_median                         25393 ns        25244 ns            4 bytes_per_second=19.3441M/s items_per_second=5.07095M/s
BM_SetInsert/1024/128_stddev                          3358 ns         3318 ns            4 bytes_per_second=2.18661M/s items_per_second=573.206k/s
BM_SetInsert/4096/128                                33817 ns        33659 ns            1 bytes_per_second=14.5067M/s items_per_second=3.80285M/s
BM_SetInsert/4096/128                                38652 ns        38209 ns            1 bytes_per_second=12.7792M/s items_per_second=3.35M/s
BM_SetInsert/4096/128_mean                           36235 ns        35934 ns            2 bytes_per_second=13.643M/s items_per_second=3.57642M/s
BM_SetInsert/4096/128_median                         36235 ns        35934 ns            2 bytes_per_second=13.643M/s items_per_second=3.57642M/s
BM_SetInsert/4096/128_stddev                          3419 ns         3217 ns            2 bytes_per_second=1.22152M/s items_per_second=320.213k/s
BM_SetInsert/8192/128                                39662 ns        38391 ns            1 bytes_per_second=12.7186M/s items_per_second=3.33411M/s
BM_SetInsert/8192/128                                25337 ns        25160 ns            1 bytes_per_second=19.407M/s items_per_second=5.08744M/s
BM_SetInsert/8192/128_mean                           32499 ns        31775 ns            2 bytes_per_second=16.0628M/s items_per_second=4.21078M/s
BM_SetInsert/8192/128_median                         32499 ns        31775 ns            2 bytes_per_second=16.0628M/s items_per_second=4.21078M/s
BM_SetInsert/8192/128_stddev                         10129 ns         9356 ns            2 bytes_per_second=4.72942M/s items_per_second=1.23979M/s
BM_SetInsert/1024/512                               123033 ns       122866 ns            1 bytes_per_second=15.8964M/s items_per_second=4.16714M/s
BM_SetInsert/1024/512                               123495 ns       123659 ns            1 bytes_per_second=15.7944M/s items_per_second=4.14042M/s
BM_SetInsert/1024/512                               120988 ns       120985 ns            1 bytes_per_second=16.1435M/s items_per_second=4.23193M/s
BM_SetInsert/1024/512                                91419 ns        91345 ns            1 bytes_per_second=21.3818M/s items_per_second=5.60512M/s
BM_SetInsert/1024/512                                93305 ns        93241 ns            1 bytes_per_second=20.9471M/s items_per_second=5.49115M/s
BM_SetInsert/1024/512                                90849 ns        90605 ns            1 bytes_per_second=21.5565M/s items_per_second=5.6509M/s
BM_SetInsert/1024/512                                93365 ns        93325 ns            1 bytes_per_second=20.9282M/s items_per_second=5.4862M/s
BM_SetInsert/1024/512                               151504 ns       151832 ns            1 bytes_per_second=12.8637M/s items_per_second=3.37215M/s
BM_SetInsert/1024/512_mean                          110995 ns       110982 ns            8 bytes_per_second=18.189M/s items_per_second=4.76813M/s
BM_SetInsert/1024/512_median                        107177 ns       107155 ns            8 bytes_per_second=18.5359M/s items_per_second=4.85907M/s
BM_SetInsert/1024/512_stddev                         22216 ns        22354 ns            8 bytes_per_second=3.38435M/s items_per_second=887.188k/s
BM_SetInsert/4096/512                               174728 ns       173342 ns            1 bytes_per_second=11.2675M/s items_per_second=2.9537M/s
BM_SetInsert/4096/512                               104087 ns       103717 ns            1 bytes_per_second=18.8313M/s items_per_second=4.93651M/s
BM_SetInsert/4096/512                               146116 ns       102860 ns            1 bytes_per_second=18.9882M/s items_per_second=4.97764M/s
BM_SetInsert/4096/512                                90335 ns        90225 ns            1 bytes_per_second=21.6473M/s items_per_second=5.6747M/s
BM_SetInsert/4096/512_mean                          128816 ns       117536 ns            4 bytes_per_second=17.6836M/s items_per_second=4.63564M/s
BM_SetInsert/4096/512_median                        125101 ns       103288 ns            4 bytes_per_second=18.9097M/s items_per_second=4.95707M/s
BM_SetInsert/4096/512_stddev                         38728 ns        37712 ns            4 bytes_per_second=4.46828M/s items_per_second=1.17133M/s
BM_SetInsert/8192/512                               152587 ns       151796 ns            1 bytes_per_second=12.8668M/s items_per_second=3.37295M/s
BM_SetInsert/8192/512                               112077 ns       111913 ns            1 bytes_per_second=17.4522M/s items_per_second=4.57498M/s
BM_SetInsert/8192/512                               110262 ns       110196 ns            1 bytes_per_second=17.7241M/s items_per_second=4.64627M/s
BM_SetInsert/8192/512_mean                          124975 ns       124635 ns            3 bytes_per_second=16.0143M/s items_per_second=4.19807M/s
BM_SetInsert/8192/512_median                        112077 ns       111913 ns            3 bytes_per_second=17.4522M/s items_per_second=4.57498M/s
BM_SetInsert/8192/512_stddev                         23930 ns        23538 ns            3 bytes_per_second=2.72927M/s items_per_second=715.461k/s
BM_Sequential<std::vector<int>,int>/1                 1930 ns         1667 ns            1 bytes_per_second=2.28836M/s items_per_second=599.88k/s
BM_Sequential<std::vector<int>,int>/1                 1312 ns         1195 ns            1 bytes_per_second=3.19222M/s items_per_second=836.82k/s
BM_Sequential<std::vector<int>,int>/1                 1620 ns         1479 ns            1 bytes_per_second=2.57924M/s items_per_second=676.133k/s
BM_Sequential<std::vector<int>,int>/1_mean            1621 ns         1447 ns            3 bytes_per_second=2.68661M/s items_per_second=704.278k/s
BM_Sequential<std::vector<int>,int>/1_median          1620 ns         1479 ns            3 bytes_per_second=2.57924M/s items_per_second=676.133k/s
BM_Sequential<std::vector<int>,int>/1_stddev           309 ns          238 ns            3 bytes_per_second=472.467k/s items_per_second=120.951k/s
BM_Sequential<std::vector<int>,int>/8                 2319 ns         1890 ns            1 bytes_per_second=16.1469M/s items_per_second=4.2328M/s
BM_Sequential<std::vector<int>,int>/8                 2447 ns         2332 ns            1 bytes_per_second=13.0864M/s items_per_second=3.43053M/s
BM_Sequential<std::vector<int>,int>/8                 1317 ns         1307 ns            1 bytes_per_second=23.3493M/s items_per_second=6.12089M/s
BM_Sequential<std::vector<int>,int>/8                 1680 ns         1567 ns            1 bytes_per_second=19.4752M/s items_per_second=5.1053M/s
BM_Sequential<std::vector<int>,int>/8                 1420 ns         1409 ns            1 bytes_per_second=21.659M/s items_per_second=5.67779M/s
BM_Sequential<std::vector<int>,int>/8                 1365 ns         1355 ns            1 bytes_per_second=22.5222M/s items_per_second=5.90406M/s
BM_Sequential<std::vector<int>,int>/8_mean            1758 ns         1643 ns            6 bytes_per_second=19.3732M/s items_per_second=5.07856M/s
BM_Sequential<std::vector<int>,int>/8_median          1550 ns         1488 ns            6 bytes_per_second=20.5671M/s items_per_second=5.39154M/s
BM_Sequential<std::vector<int>,int>/8_stddev           502 ns          398 ns            6 bytes_per_second=4.02228M/s items_per_second=1054.42k/s
BM_Sequential<std::vector<int>,int>/64                2176 ns         2174 ns            1 bytes_per_second=112.3M/s items_per_second=29.4388M/s
BM_Sequential<std::vector<int>,int>/64                2033 ns         1988 ns            1 bytes_per_second=122.807M/s items_per_second=32.1932M/s
BM_Sequential<std::vector<int>,int>/64                2093 ns         1944 ns            1 bytes_per_second=125.587M/s items_per_second=32.9218M/s
BM_Sequential<std::vector<int>,int>/64                1576 ns         1499 ns            1 bytes_per_second=162.869M/s items_per_second=42.6951M/s
BM_Sequential<std::vector<int>,int>/64                1698 ns         1648 ns            1 bytes_per_second=148.144M/s items_per_second=38.835M/s
BM_Sequential<std::vector<int>,int>/64                2526 ns         2496 ns            1 bytes_per_second=97.8128M/s items_per_second=25.641M/s
BM_Sequential<std::vector<int>,int>/64_mean           2017 ns         1958 ns            6 bytes_per_second=128.253M/s items_per_second=33.6208M/s
BM_Sequential<std::vector<int>,int>/64_median         2063 ns         1966 ns            6 bytes_per_second=124.197M/s items_per_second=32.5575M/s
BM_Sequential<std::vector<int>,int>/64_stddev          343 ns          359 ns            6 bytes_per_second=23.714M/s items_per_second=6.21648M/s
BM_Sequential<std::vector<int>,int>/512               3719 ns         3605 ns            1 bytes_per_second=541.782M/s items_per_second=142.025M/s
BM_Sequential<std::vector<int>,int>/512               3725 ns         3277 ns            1 bytes_per_second=596.01M/s items_per_second=156.24M/s
BM_Sequential<std::vector<int>,int>/512               3309 ns         3301 ns            1 bytes_per_second=591.677M/s items_per_second=155.105M/s
BM_Sequential<std::vector<int>,int>/512               3162 ns         3162 ns            1 bytes_per_second=617.687M/s items_per_second=161.923M/s
BM_Sequential<std::vector<int>,int>/512               2852 ns         2805 ns            1 bytes_per_second=696.301M/s items_per_second=182.531M/s
BM_Sequential<std::vector<int>,int>/512               2254 ns         2232 ns            1 bytes_per_second=875.056M/s items_per_second=229.391M/s
BM_Sequential<std::vector<int>,int>/512               3110 ns         3005 ns            1 bytes_per_second=649.958M/s items_per_second=170.383M/s
BM_Sequential<std::vector<int>,int>/512_mean          3162 ns         3055 ns            7 bytes_per_second=652.639M/s items_per_second=171.085M/s
BM_Sequential<std::vector<int>,int>/512_median        3162 ns         3162 ns            7 bytes_per_second=617.687M/s items_per_second=161.923M/s
BM_Sequential<std::vector<int>,int>/512_stddev         512 ns          441 ns            7 bytes_per_second=109.405M/s items_per_second=28.6799M/s
BM_Sequential<std::vector<int>,int>/1024              4336 ns         4323 ns            1 bytes_per_second=903.597M/s items_per_second=236.873M/s
BM_Sequential<std::vector<int>,int>/1024              3112 ns         3115 ns            1 bytes_per_second=1.22462G/s items_per_second=328.732M/s
BM_Sequential<std::vector<int>,int>/1024              3839 ns         3832 ns            1 bytes_per_second=1019.38M/s items_per_second=267.223M/s
BM_Sequential<std::vector<int>,int>/1024              4276 ns         4260 ns            1 bytes_per_second=916.96M/s items_per_second=240.376M/s
BM_Sequential<std::vector<int>,int>/1024              3788 ns         3780 ns            1 bytes_per_second=1033.4M/s items_per_second=270.899M/s
BM_Sequential<std::vector<int>,int>/1024              4049 ns         4041 ns            1 bytes_per_second=966.654M/s items_per_second=253.403M/s
BM_Sequential<std::vector<int>,int>/1024              4578 ns         4560 ns            1 bytes_per_second=856.634M/s items_per_second=224.561M/s
BM_Sequential<std::vector<int>,int>/1024_mean         3997 ns         3987 ns            7 bytes_per_second=992.948M/s items_per_second=260.295M/s
BM_Sequential<std::vector<int>,int>/1024_median       4049 ns         4041 ns            7 bytes_per_second=966.654M/s items_per_second=253.403M/s
BM_Sequential<std::vector<int>,int>/1024_stddev        480 ns          474 ns            7 bytes_per_second=131.352M/s items_per_second=34.4332M/s
BM_Sequential<std::list<int>>/1                       1864 ns         1668 ns            1 bytes_per_second=2.28699M/s items_per_second=599.52k/s
BM_Sequential<std::list<int>>/1                       1256 ns         1203 ns            1 bytes_per_second=3.17099M/s items_per_second=831.255k/s
BM_Sequential<std::list<int>>/1                       1286 ns         1213 ns            1 bytes_per_second=3.14485M/s items_per_second=824.402k/s
BM_Sequential<std::list<int>>/1                       1133 ns         1105 ns            1 bytes_per_second=3.45221M/s items_per_second=904.977k/s
BM_Sequential<std::list<int>>/1_mean                  1385 ns         1297 ns            4 bytes_per_second=3.01376M/s items_per_second=790.039k/s
BM_Sequential<std::list<int>>/1_median                1271 ns         1208 ns            4 bytes_per_second=3.15792M/s items_per_second=827.829k/s
BM_Sequential<std::list<int>>/1_stddev                 326 ns          252 ns            4 bytes_per_second=516.196k/s items_per_second=132.146k/s
BM_Sequential<std::list<int>>/8                       4727 ns         4429 ns            1 bytes_per_second=6.8904M/s items_per_second=1.80628M/s
BM_Sequential<std::list<int>>/8                       1614 ns         1611 ns            1 bytes_per_second=18.9433M/s items_per_second=4.96586M/s
BM_Sequential<std::list<int>>/8                       1541 ns         1518 ns            1 bytes_per_second=20.1038M/s items_per_second=5.27009M/s
BM_Sequential<std::list<int>>/8                       5515 ns         4804 ns            1 bytes_per_second=6.35253M/s items_per_second=1.66528M/s
BM_Sequential<std::list<int>>/8                       2822 ns         2631 ns            1 bytes_per_second=11.5992M/s items_per_second=3.04067M/s
BM_Sequential<std::list<int>>/8                       4375 ns         4424 ns            1 bytes_per_second=6.89819M/s items_per_second=1.80832M/s
BM_Sequential<std::list<int>>/8_mean                  3432 ns         3236 ns            6 bytes_per_second=11.7979M/s items_per_second=3.09275M/s
BM_Sequential<std::list<int>>/8_median                3599 ns         3527 ns            6 bytes_per_second=9.24871M/s items_per_second=2.42449M/s
BM_Sequential<std::list<int>>/8_stddev                1683 ns         1500 ns            6 bytes_per_second=6.29009M/s items_per_second=1.64891M/s
BM_Sequential<std::list<int>>/64                      5707 ns         5594 ns            1 bytes_per_second=43.6433M/s items_per_second=11.4408M/s
BM_Sequential<std::list<int>>/64                      7344 ns         7424 ns            1 bytes_per_second=32.8853M/s items_per_second=8.62069M/s
BM_Sequential<std::list<int>>/64                     44780 ns        44978 ns            1 bytes_per_second=5.428M/s items_per_second=1.42292M/s
BM_Sequential<std::list<int>>/64                      5023 ns         4997 ns            1 bytes_per_second=48.8574M/s items_per_second=12.8077M/s
BM_Sequential<std::list<int>>/64                      9690 ns         9207 ns            1 bytes_per_second=26.5168M/s items_per_second=6.95123M/s
BM_Sequential<std::list<int>>/64                      6066 ns         6022 ns            1 bytes_per_second=40.5415M/s items_per_second=10.6277M/s
BM_Sequential<std::list<int>>/64                      8251 ns         8223 ns            1 bytes_per_second=29.69M/s items_per_second=7.78305M/s
BM_Sequential<std::list<int>>/64                      7154 ns         7124 ns            1 bytes_per_second=34.2702M/s items_per_second=8.98372M/s
BM_Sequential<std::list<int>>/64                     11966 ns        11836 ns            1 bytes_per_second=20.627M/s items_per_second=5.40723M/s
BM_Sequential<std::list<int>>/64_mean                11776 ns        11712 ns            9 bytes_per_second=31.3844M/s items_per_second=8.22723M/s
BM_Sequential<std::list<int>>/64_median               7344 ns         7424 ns            9 bytes_per_second=32.8853M/s items_per_second=8.62069M/s
BM_Sequential<std::list<int>>/64_stddev              12561 ns        12646 ns            9 bytes_per_second=13.0673M/s items_per_second=3.42552M/s
BM_Sequential<std::list<int>>/512                    48939 ns        48912 ns            1 bytes_per_second=39.9314M/s items_per_second=10.4678M/s
BM_Sequential<std::list<int>>/512                    39802 ns        39759 ns            1 bytes_per_second=49.1241M/s items_per_second=12.8776M/s
BM_Sequential<std::list<int>>/512                    48071 ns        47833 ns            1 bytes_per_second=40.8322M/s items_per_second=10.7039M/s
BM_Sequential<std::list<int>>/512                    39919 ns        39981 ns            1 bytes_per_second=48.8513M/s items_per_second=12.8061M/s
BM_Sequential<std::list<int>>/512                    59215 ns        59299 ns            1 bytes_per_second=32.9369M/s items_per_second=8.63421M/s
BM_Sequential<std::list<int>>/512                    40938 ns        40919 ns            1 bytes_per_second=47.7315M/s items_per_second=12.5125M/s
BM_Sequential<std::list<int>>/512                    38634 ns        38661 ns            1 bytes_per_second=50.5193M/s items_per_second=13.2433M/s
BM_Sequential<std::list<int>>/512                    40319 ns        40311 ns            1 bytes_per_second=48.4514M/s items_per_second=12.7012M/s
BM_Sequential<std::list<int>>/512                    60054 ns        60256 ns            1 bytes_per_second=32.4138M/s items_per_second=8.49708M/s
BM_Sequential<std::list<int>>/512                    31461 ns        31489 ns            1 bytes_per_second=62.0256M/s items_per_second=16.2596M/s
BM_Sequential<std::list<int>>/512                    61844 ns        61930 ns            1 bytes_per_second=31.5376M/s items_per_second=8.2674M/s
BM_Sequential<std::list<int>>/512                    39184 ns        39204 ns            1 bytes_per_second=49.8195M/s items_per_second=13.0599M/s
BM_Sequential<std::list<int>>/512_mean               45698 ns        45713 ns           12 bytes_per_second=44.5146M/s items_per_second=11.6692M/s
BM_Sequential<std::list<int>>/512_median             40628 ns        40615 ns           12 bytes_per_second=48.0915M/s items_per_second=12.6069M/s
BM_Sequential<std::list<int>>/512_stddev              9911 ns         9948 ns           12 bytes_per_second=9.14617M/s items_per_second=2.39761M/s
BM_Sequential<std::list<int>>/1024                  107028 ns       107378 ns            1 bytes_per_second=36.3785M/s items_per_second=9.5364M/s
BM_Sequential<std::list<int>>/1024                   70385 ns        70416 ns            1 bytes_per_second=55.4739M/s items_per_second=14.5421M/s
BM_Sequential<std::list<int>>/1024                  162524 ns       162703 ns            1 bytes_per_second=24.0085M/s items_per_second=6.29368M/s
BM_Sequential<std::list<int>>/1024                   71453 ns        71548 ns            1 bytes_per_second=54.5962M/s items_per_second=14.3121M/s
BM_Sequential<std::list<int>>/1024                  104422 ns       104413 ns            1 bytes_per_second=37.4115M/s items_per_second=9.80721M/s
BM_Sequential<std::list<int>>/1024                   69853 ns        69900 ns            1 bytes_per_second=55.8834M/s items_per_second=14.6495M/s
BM_Sequential<std::list<int>>/1024                   93644 ns        93585 ns            1 bytes_per_second=41.7401M/s items_per_second=10.9419M/s
BM_Sequential<std::list<int>>/1024                   76643 ns        76719 ns            1 bytes_per_second=50.9163M/s items_per_second=13.3474M/s
BM_Sequential<std::list<int>>/1024                  103287 ns       103517 ns            1 bytes_per_second=37.7353M/s items_per_second=9.8921M/s
BM_Sequential<std::list<int>>/1024                   77370 ns        77417 ns            1 bytes_per_second=50.4573M/s items_per_second=13.2271M/s
BM_Sequential<std::list<int>>/1024                  120593 ns       120485 ns            1 bytes_per_second=32.421M/s items_per_second=8.49898M/s
BM_Sequential<std::list<int>>/1024                  125043 ns       125032 ns            1 bytes_per_second=31.242M/s items_per_second=8.1899M/s
BM_Sequential<std::list<int>>/1024_mean              98520 ns        98593 ns           12 bytes_per_second=42.3553M/s items_per_second=11.1032M/s
BM_Sequential<std::list<int>>/1024_median            98466 ns        98551 ns           12 bytes_per_second=39.7377M/s items_per_second=10.417M/s
BM_Sequential<std::list<int>>/1024_stddev            28144 ns        28162 ns           12 bytes_per_second=10.8064M/s items_per_second=2.83284M/s
BM_Sequential<std::vector<int>, int>/512            150685 ns       150532 ns            1 bytes_per_second=12.9748M/s items_per_second=3.40127M/s
BM_Sequential<std::vector<int>, int>/512              3506 ns         3453 ns            1 bytes_per_second=565.631M/s items_per_second=148.277M/s
BM_Sequential<std::vector<int>, int>/512              3526 ns         3383 ns            1 bytes_per_second=577.335M/s items_per_second=151.345M/s
BM_Sequential<std::vector<int>, int>/512              2569 ns         2560 ns            1 bytes_per_second=762.939M/s items_per_second=200M/s
BM_Sequential<std::vector<int>, int>/512              4035 ns         3670 ns            1 bytes_per_second=532.187M/s items_per_second=139.51M/s
BM_Sequential<std::vector<int>, int>/512              3048 ns         3045 ns            1 bytes_per_second=641.42M/s items_per_second=168.144M/s
BM_Sequential<std::vector<int>, int>/512              3199 ns         3143 ns            1 bytes_per_second=621.421M/s items_per_second=162.902M/s
BM_Sequential<std::vector<int>, int>/512              3170 ns         3120 ns            1 bytes_per_second=626.002M/s items_per_second=164.103M/s
BM_Sequential<std::vector<int>, int>/512              3021 ns         2990 ns            1 bytes_per_second=653.219M/s items_per_second=171.237M/s
BM_Sequential<std::vector<int>, int>/512              2835 ns         2775 ns            1 bytes_per_second=703.829M/s items_per_second=184.505M/s
BM_Sequential<std::vector<int>, int>/512              5625 ns         5313 ns            1 bytes_per_second=367.612M/s items_per_second=96.3674M/s
BM_Sequential<std::vector<int>, int>/512              2905 ns         2889 ns            1 bytes_per_second=676.056M/s items_per_second=177.224M/s
BM_Sequential<std::vector<int>, int>/512_mean        15677 ns        15573 ns           12 bytes_per_second=561.719M/s items_per_second=147.251M/s
BM_Sequential<std::vector<int>, int>/512_median       3184 ns         3132 ns           12 bytes_per_second=623.711M/s items_per_second=163.502M/s
BM_Sequential<std::vector<int>, int>/512_stddev      42524 ns        42507 ns           12 bytes_per_second=199.113M/s items_per_second=52.1964M/s
BM_StringCompare/1                                    1884 ns         1824 ns            1
BM_StringCompare/1                                    1201 ns         1138 ns            1
BM_StringCompare/1                                    1310 ns         1227 ns            1
BM_StringCompare/1                                    1253 ns         1194 ns            1
BM_StringCompare/1                                    1419 ns         1371 ns            1
BM_StringCompare/1                                    1131 ns         1097 ns            1
BM_StringCompare/1_mean                               1366 ns         1308 ns            6
BM_StringCompare/1_median                             1282 ns         1211 ns            6
BM_StringCompare/1_stddev                              272 ns          270 ns            6
BM_StringCompare/8                                    2223 ns         1755 ns            1
BM_StringCompare/8                                    1212 ns         1128 ns            1
BM_StringCompare/8                                    1979 ns         1748 ns            1
BM_StringCompare/8                                    1056 ns         1044 ns            1
BM_StringCompare/8                                    1204 ns         1135 ns            1
BM_StringCompare/8_mean                               1535 ns         1362 ns            5
BM_StringCompare/8_median                             1212 ns         1135 ns            5
BM_StringCompare/8_stddev                              528 ns          357 ns            5
BM_StringCompare/64                                   1124 ns         1117 ns            1
BM_StringCompare/64                                   1188 ns         1154 ns            1
BM_StringCompare/64                                   1118 ns         1099 ns            1
BM_StringCompare/64                                   1126 ns         1100 ns            1
BM_StringCompare/64                                   1044 ns         1040 ns            1
BM_StringCompare/64_mean                              1120 ns         1102 ns            5
BM_StringCompare/64_median                            1124 ns         1100 ns            5
BM_StringCompare/64_stddev                            51.1 ns         41.2 ns            5
BM_StringCompare/512                                  2974 ns         2782 ns            1
BM_StringCompare/512                                  1773 ns         1706 ns            1
BM_StringCompare/512                                  1208 ns         1187 ns            1
BM_StringCompare/512                                  1224 ns         1178 ns            1
BM_StringCompare/512                                  1136 ns         1128 ns            1
BM_StringCompare/512                                  1256 ns         1257 ns            1
BM_StringCompare/512_mean                             1595 ns         1540 ns            6
BM_StringCompare/512_median                           1240 ns         1222 ns            6
BM_StringCompare/512_stddev                            714 ns          644 ns            6
BM_StringCompare/4096                                 2144 ns         2147 ns            1
BM_StringCompare/4096                                 1217 ns         1208 ns            1
BM_StringCompare/4096                                 1492 ns         1459 ns            1
BM_StringCompare/4096                                 1344 ns         1309 ns            1
BM_StringCompare/4096                                 3349 ns         2771 ns            1
BM_StringCompare/4096                                 1351 ns         1303 ns            1
BM_StringCompare/4096_mean                            1816 ns         1699 ns            6
BM_StringCompare/4096_median                          1421 ns         1384 ns            6
BM_StringCompare/4096_stddev                           820 ns          626 ns            6
BM_StringCompare/32768                                7726 ns         6433 ns            1
BM_StringCompare/32768                                2495 ns         2461 ns            1
BM_StringCompare/32768_mean                           5111 ns         4447 ns            2
BM_StringCompare/32768_median                         5111 ns         4447 ns            2
BM_StringCompare/32768_stddev                         3699 ns         2809 ns            2
BM_StringCompare/262144                              16920 ns        16882 ns            1
BM_StringCompare/262144                              18461 ns        18425 ns            1
BM_StringCompare/262144                              48215 ns        48417 ns            1
BM_StringCompare/262144                              47766 ns        47920 ns            1
BM_StringCompare/262144_mean                         32841 ns        32911 ns            4
BM_StringCompare/262144_median                       33114 ns        33173 ns            4
BM_StringCompare/262144_stddev                       17506 ns        17630 ns            4
BM_StringCompare/1048576                             89681 ns        88891 ns            1
BM_StringCompare/1048576                             70396 ns        70537 ns            1
BM_StringCompare/1048576                             83098 ns        83218 ns            1
BM_StringCompare/1048576                             83268 ns        83202 ns            1
BM_StringCompare/1048576_mean                        81611 ns        81462 ns            4
BM_StringCompare/1048576_median                      83183 ns        83210 ns            4
BM_StringCompare/1048576_stddev                       8080 ns         7760 ns            4
BM_SetupTeardown/threads:8                             359 ns         2688 ns            8
BM_LongTest/65536                                   211783 ns       211772 ns            1
BM_LongTest/65536                                   219591 ns       219561 ns            1
BM_LongTest/65536                                   221906 ns       222272 ns            1
BM_LongTest/65536                                   224461 ns       224516 ns            1
BM_LongTest/65536                                   237296 ns       236840 ns            1
BM_LongTest/65536                                   219566 ns       219610 ns            1
BM_LongTest/65536                                   212149 ns       212503 ns            1
BM_LongTest/65536                                   219824 ns       219746 ns            1
BM_LongTest/65536                                   212688 ns       212989 ns            1
BM_LongTest/65536                                   211809 ns       211826 ns            1
BM_LongTest/65536                                   272909 ns       272949 ns            1
BM_LongTest/65536                                   223705 ns       223655 ns            1
BM_LongTest/65536_mean                              223974 ns       224020 ns           12
BM_LongTest/65536_median                            219707 ns       219678 ns           12
BM_LongTest/65536_stddev                             17046 ns        16983 ns           12
BM_LongTest/262144                                 1038499 ns      1038810 ns            1
BM_LongTest/262144                                  983091 ns       983675 ns            1
BM_LongTest/262144                                  892904 ns       893870 ns            1
BM_LongTest/262144                                  879511 ns       879496 ns            1
BM_LongTest/262144                                  879460 ns       882030 ns            1
BM_LongTest/262144                                  948754 ns       949967 ns            1
BM_LongTest/262144                                  882652 ns       883508 ns            1
BM_LongTest/262144                                  879481 ns       879629 ns            1
BM_LongTest/262144                                 1019827 ns      1020161 ns            1
BM_LongTest/262144                                 1032175 ns      1034947 ns            1
BM_LongTest/262144                                 1078228 ns      1078693 ns            1
BM_LongTest/262144                                  905595 ns       905760 ns            1
BM_LongTest/262144_mean                             951681 ns       952546 ns           12
BM_LongTest/262144_median                           927175 ns       927863 ns           12
BM_LongTest/262144_stddev                            74864 ns        74903 ns           12
BM_LongTest/2097152                                7377364 ns      7164346 ns            1
BM_LongTest/2097152                                7023535 ns      7024117 ns            1
BM_LongTest/2097152                                7231007 ns      7232109 ns            1
BM_LongTest/2097152                                7206032 ns      7206377 ns            1
BM_LongTest/2097152                                7153763 ns      7125356 ns            1
BM_LongTest/2097152                                7134342 ns      7134431 ns            1
BM_LongTest/2097152                                7542607 ns      7226429 ns            1
BM_LongTest/2097152                                7161925 ns      7081105 ns            1
BM_LongTest/2097152                                7089616 ns      7089439 ns            1
BM_LongTest/2097152                                8556446 ns      8439017 ns            1
BM_LongTest/2097152                               11364642 ns      8272463 ns            1
BM_LongTest/2097152                                7840612 ns      7754882 ns            1
BM_LongTest/2097152_mean                           7723491 ns      7395839 ns           12
BM_LongTest/2097152_median                         7218520 ns      7185362 ns           12
BM_LongTest/2097152_stddev                         1225009 ns       486472 ns           12
BM_LongTest/16777216                              57487585 ns     56901575 ns            1
BM_LongTest/16777216                              57251285 ns     56853699 ns            1
BM_LongTest/16777216                              57448280 ns     57282926 ns            1
BM_LongTest/16777216                              57247391 ns     57075617 ns            1
BM_LongTest/16777216                              57811875 ns     57588828 ns            1
BM_LongTest/16777216                              57042879 ns     56723367 ns            1
BM_LongTest/16777216                              57434748 ns     56948351 ns            1
BM_LongTest/16777216                              57173880 ns     56946850 ns            1
BM_LongTest/16777216                              57982355 ns     56705055 ns            1
BM_LongTest/16777216                              74062482 ns     65624444 ns            1
BM_LongTest/16777216                              69571040 ns     65939027 ns            1
BM_LongTest/16777216                              58084688 ns     57666492 ns            1
BM_LongTest/16777216_mean                         59883207 ns     58521353 ns           12
BM_LongTest/16777216_median                       57467932 ns     57011984 ns           12
BM_LongTest/16777216_stddev                        5664800 ns      3405742 ns           12
BM_LongTest/134217728                            458897695 ns    455916471 ns            1
BM_LongTest/134217728                            464546643 ns    462919553 ns            1
BM_LongTest/134217728_mean                       461722169 ns    459418012 ns            2
BM_LongTest/134217728_median                     461722169 ns    459418012 ns            2
BM_LongTest/134217728_stddev                       3994410 ns      4951927 ns            2
BM_LongTest/268435456                            918742385 ns    912407071 ns            1
BM_ParallelMemset/10485760/threads:1               4040424 ns      4011123 ns            1
BM_ParallelMemset/10485760/threads:1               4245056 ns      4246564 ns            1
BM_ParallelMemset/10485760/threads:1               4034060 ns      4034309 ns            1
BM_ParallelMemset/10485760/threads:1               4218489 ns      4219840 ns            1
BM_ParallelMemset/10485760/threads:1               4496682 ns      4275055 ns            1
BM_ParallelMemset/10485760/threads:1               4026107 ns      4026839 ns            1
BM_ParallelMemset/10485760/threads:1               4193886 ns      4148575 ns            1
BM_ParallelMemset/10485760/threads:1               4314166 ns      4314503 ns            1
BM_ParallelMemset/10485760/threads:1               4134814 ns      4136226 ns            1
BM_ParallelMemset/10485760/threads:1               5507927 ns      4902491 ns            1
BM_ParallelMemset/10485760/threads:1_mean          4321161 ns      4231552 ns           10
BM_ParallelMemset/10485760/threads:1_median        4206187 ns      4184208 ns           10
BM_ParallelMemset/10485760/threads:1_stddev         441566 ns       259201 ns           10
BM_ParallelMemset/10485760/threads:2               1869736 ns      3682901 ns            2
BM_ParallelMemset/10485760/threads:2               1866413 ns      3658981 ns            2
BM_ParallelMemset/10485760/threads:2               1025590 ns      2052459 ns            2
BM_ParallelMemset/10485760/threads:2               1053835 ns      2109148 ns            2
BM_ParallelMemset/10485760/threads:2               1143083 ns      2288285 ns            2
BM_ParallelMemset/10485760/threads:2_mean          1391731 ns      2758355 ns            5
BM_ParallelMemset/10485760/threads:2_median        1143083 ns      2288285 ns            5
BM_ParallelMemset/10485760/threads:2_stddev         436998 ns       837651 ns            5
BM_ParallelMemset/10485760/threads:4                424600 ns      1377139 ns            4
BM_ParallelMemset/10485760/threads:4                268341 ns      1046685 ns            4
BM_ParallelMemset/10485760/threads:4                382408 ns      1517478 ns            4
BM_ParallelMemset/10485760/threads:4                291343 ns      1166604 ns            4
BM_ParallelMemset/10485760/threads:4                371640 ns      1474776 ns            4
BM_ParallelMemset/10485760/threads:4                367050 ns      1443126 ns            4
BM_ParallelMemset/10485760/threads:4_mean           350897 ns      1337635 ns            6
BM_ParallelMemset/10485760/threads:4_median         369345 ns      1410133 ns            6
BM_ParallelMemset/10485760/threads:4_stddev          59124 ns       188544 ns            6
BM_ManualTiming/1/real_time                         111601 ns        12463 ns            1 items_per_second=8.96049k/s
BM_ManualTiming/1/real_time                          80693 ns        15065 ns            1 items_per_second=12.3926k/s
BM_ManualTiming/1/real_time                          85799 ns        18636 ns            1 items_per_second=11.6551k/s
BM_ManualTiming/1/real_time                          84242 ns        17658 ns            1 items_per_second=11.8706k/s
BM_ManualTiming/1/real_time                          86254 ns        10463 ns            1 items_per_second=11.5937k/s
BM_ManualTiming/1/real_time                          61832 ns         8571 ns            1 items_per_second=16.1729k/s
BM_ManualTiming/1/real_time                          81593 ns         9485 ns            1 items_per_second=12.2559k/s
BM_ManualTiming/1/real_time                          85505 ns         9645 ns            1 items_per_second=11.6952k/s
BM_ManualTiming/1/real_time                          92569 ns        23348 ns            1 items_per_second=10.8028k/s
BM_ManualTiming/1/real_time                         924121 ns        16676 ns            1 items_per_second=1082.11/s
BM_ManualTiming/1/real_time                          82858 ns        29344 ns            1 items_per_second=12.0688k/s
BM_ManualTiming/1/real_time                         157745 ns        23507 ns            1 items_per_second=6.33934k/s
BM_ManualTiming/1/real_time_mean                    161234 ns        16238 ns           12 items_per_second=10.5741k/s
BM_ManualTiming/1/real_time_median                   85652 ns        15870 ns           12 items_per_second=11.6752k/s
BM_ManualTiming/1/real_time_stddev                  241400 ns         6605 ns           12 items_per_second=3.76043k/s
BM_ManualTiming/8/real_time                          84916 ns        13251 ns            1 items_per_second=94.2107k/s
BM_ManualTiming/8/real_time                          85476 ns        12895 ns            1 items_per_second=93.5934k/s
BM_ManualTiming/8/real_time                          85561 ns        13416 ns            1 items_per_second=93.5005k/s
BM_ManualTiming/8/real_time                          84832 ns        13484 ns            1 items_per_second=94.3039k/s
BM_ManualTiming/8/real_time                          83691 ns        12720 ns            1 items_per_second=95.5898k/s
BM_ManualTiming/8/real_time                          95253 ns        11771 ns            1 items_per_second=83.9869k/s
BM_ManualTiming/8/real_time                          85847 ns        14245 ns            1 items_per_second=93.1891k/s
BM_ManualTiming/8/real_time                          79336 ns         9869 ns            1 items_per_second=100.837k/s
BM_ManualTiming/8/real_time                        6058527 ns        19281 ns            1 items_per_second=1.32045k/s
BM_ManualTiming/8/real_time                         120763 ns        10400 ns            1 items_per_second=66.2454k/s
BM_ManualTiming/8/real_time                          88499 ns         7940 ns            1 items_per_second=90.3965k/s
BM_ManualTiming/8/real_time                         166507 ns        15671 ns            1 items_per_second=48.046k/s
BM_ManualTiming/8/real_time_mean                    593267 ns        12912 ns           12 items_per_second=79.6016k/s
BM_ManualTiming/8/real_time_median                   85704 ns        13073 ns           12 items_per_second=93.3448k/s
BM_ManualTiming/8/real_time_stddev                 1721286 ns         2896 ns           12 items_per_second=28.8374k/s
BM_ManualTiming/64/real_time                        141027 ns        11838 ns            1 items_per_second=453.814k/s
BM_ManualTiming/64/real_time                        145412 ns        15272 ns            1 items_per_second=440.129k/s
BM_ManualTiming/64/real_time                        142433 ns        14425 ns            1 items_per_second=449.334k/s
BM_ManualTiming/64/real_time                        137184 ns        12230 ns            1 items_per_second=466.527k/s
BM_ManualTiming/64/real_time                        142177 ns        13240 ns            1 items_per_second=450.143k/s
BM_ManualTiming/64/real_time                        141297 ns         9949 ns            1 items_per_second=452.947k/s
BM_ManualTiming/64/real_time                        135795 ns        10805 ns            1 items_per_second=471.299k/s
BM_ManualTiming/64/real_time                        168247 ns         9812 ns            1 items_per_second=380.393k/s
BM_ManualTiming/64/real_time                        135880 ns         9359 ns            1 items_per_second=471.004k/s
BM_ManualTiming/64/real_time                        212685 ns        15260 ns            1 items_per_second=300.914k/s
BM_ManualTiming/64/real_time                        137955 ns        10889 ns            1 items_per_second=463.92k/s
BM_ManualTiming/64/real_time                        138433 ns        10175 ns            1 items_per_second=462.318k/s
BM_ManualTiming/64/real_time_mean                   148210 ns        11938 ns           12 items_per_second=438.562k/s
BM_ManualTiming/64/real_time_median                 141162 ns        11363 ns           12 items_per_second=453.38k/s
BM_ManualTiming/64/real_time_stddev                  22082 ns         2148 ns           12 items_per_second=49.6681k/s
BM_ManualTiming/512/real_time                       625368 ns        30841 ns            1 items_per_second=818.718k/s
BM_ManualTiming/512/real_time                       592016 ns        13870 ns            1 items_per_second=864.841k/s
BM_ManualTiming/512/real_time                       590173 ns        11774 ns            1 items_per_second=867.542k/s
BM_ManualTiming/512/real_time                       597940 ns        20205 ns            1 items_per_second=856.273k/s
BM_ManualTiming/512/real_time                       592504 ns        15669 ns            1 items_per_second=864.129k/s
BM_ManualTiming/512/real_time                       615374 ns        15630 ns            1 items_per_second=832.014k/s
BM_ManualTiming/512/real_time                       596129 ns        15531 ns            1 items_per_second=858.875k/s
BM_ManualTiming/512/real_time                       589970 ns        11854 ns            1 items_per_second=867.841k/s
BM_ManualTiming/512/real_time                       608691 ns        12896 ns            1 items_per_second=841.149k/s
BM_ManualTiming/512/real_time                       585681 ns        20633 ns            1 items_per_second=874.196k/s
BM_ManualTiming/512/real_time                       583120 ns        17105 ns            1 items_per_second=878.036k/s
BM_ManualTiming/512/real_time                       597717 ns        12934 ns            1 items_per_second=856.593k/s
BM_ManualTiming/512/real_time_mean                  597890 ns        16578 ns           12 items_per_second=856.684k/s
BM_ManualTiming/512/real_time_median                594316 ns        15580 ns           12 items_per_second=861.502k/s
BM_ManualTiming/512/real_time_stddev                 12554 ns         5357 ns           12 items_per_second=17.6409k/s
BM_ManualTiming/4096/real_time                     4200757 ns        22151 ns            1 items_per_second=975.062k/s
BM_ManualTiming/4096/real_time                     4207627 ns        23145 ns            1 items_per_second=973.47k/s
BM_ManualTiming/4096/real_time                     4210919 ns        33320 ns            1 items_per_second=972.709k/s
BM_ManualTiming/4096/real_time                     4181943 ns        16244 ns            1 items_per_second=979.449k/s
BM_ManualTiming/4096/real_time                     4176456 ns        14147 ns            1 items_per_second=980.736k/s
BM_ManualTiming/4096/real_time                     4209909 ns        29085 ns            1 items_per_second=972.943k/s
BM_ManualTiming/4096/real_time                     4202937 ns         8624 ns            1 items_per_second=974.557k/s
BM_ManualTiming/4096/real_time                     4192441 ns        20192 ns            1 items_per_second=976.996k/s
BM_ManualTiming/4096/real_time                     5752538 ns        13749 ns            1 items_per_second=712.034k/s
BM_ManualTiming/4096/real_time                     4178382 ns        25178 ns            1 items_per_second=980.284k/s
BM_ManualTiming/4096/real_time                     4266499 ns        12915 ns            1 items_per_second=960.038k/s
BM_ManualTiming/4096/real_time                    15058227 ns        17291 ns            1 items_per_second=272.011k/s
BM_ManualTiming/4096/real_time_mean                5236553 ns        19670 ns           12 items_per_second=894.191k/s
BM_ManualTiming/4096/real_time_median              4205282 ns        18741 ns           12 items_per_second=974.013k/s
BM_ManualTiming/4096/real_time_stddev              3125035 ns         7226 ns           12 items_per_second=210.044k/s
BM_ManualTiming/16384/real_time                   16506807 ns       130034 ns            1 items_per_second=992.56k/s
BM_ManualTiming/16384/real_time                   16684015 ns        12875 ns            1 items_per_second=982.018k/s
BM_ManualTiming/16384/real_time                   16472079 ns        19319 ns            1 items_per_second=994.653k/s
BM_ManualTiming/16384/real_time                   16520532 ns        41588 ns            1 items_per_second=991.736k/s
BM_ManualTiming/16384/real_time                   16496373 ns        27277 ns            1 items_per_second=993.188k/s
BM_ManualTiming/16384/real_time                   16468731 ns        17459 ns            1 items_per_second=994.855k/s
BM_ManualTiming/16384/real_time                   16530009 ns        15143 ns            1 items_per_second=991.167k/s
BM_ManualTiming/16384/real_time                   16461764 ns        17317 ns            1 items_per_second=995.276k/s
BM_ManualTiming/16384/real_time                   16485025 ns        22278 ns            1 items_per_second=993.872k/s
BM_ManualTiming/16384/real_time                   16462144 ns        22076 ns            1 items_per_second=995.253k/s
BM_ManualTiming/16384/real_time                   16548421 ns        24401 ns            1 items_per_second=990.064k/s
BM_ManualTiming/16384/real_time                   16473779 ns        16715 ns            1 items_per_second=994.55k/s
BM_ManualTiming/16384/real_time_mean              16509140 ns        30540 ns           12 items_per_second=992.433k/s
BM_ManualTiming/16384/real_time_median            16490699 ns        20697 ns           12 items_per_second=993.53k/s
BM_ManualTiming/16384/real_time_stddev               61901 ns        32223 ns           12 items_per_second=3.69453k/s
BM_ManualTiming/1/manual_time                       199470 ns        19633 ns            1 items_per_second=5.01329k/s
BM_ManualTiming/1/manual_time                        85346 ns        16819 ns            1 items_per_second=11.717k/s
BM_ManualTiming/1/manual_time                        83699 ns        18521 ns            1 items_per_second=11.9476k/s
BM_ManualTiming/1/manual_time                       102631 ns        25319 ns            1 items_per_second=9.74364k/s
BM_ManualTiming/1/manual_time                       105427 ns        28606 ns            1 items_per_second=9.48524k/s
BM_ManualTiming/1/manual_time                        74606 ns         6557 ns            1 items_per_second=13.4037k/s
BM_ManualTiming/1/manual_time                        73788 ns        11785 ns            1 items_per_second=13.5523k/s
BM_ManualTiming/1/manual_time                        88991 ns        12859 ns            1 items_per_second=11.2371k/s
BM_ManualTiming/1/manual_time                        78804 ns        13584 ns            1 items_per_second=12.6897k/s
BM_ManualTiming/1/manual_time                        96478 ns        27789 ns            1 items_per_second=10.3651k/s
BM_ManualTiming/1/manual_time                        64114 ns        11482 ns            1 items_per_second=15.5972k/s
BM_ManualTiming/1/manual_time                       117316 ns        15925 ns            1 items_per_second=8.52399k/s
BM_ManualTiming/1/manual_time_mean                   97556 ns        17407 ns           12 items_per_second=11.1063k/s
BM_ManualTiming/1/manual_time_median                 87168 ns        16372 ns           12 items_per_second=11.4771k/s
BM_ManualTiming/1/manual_time_stddev                 35473 ns         6904 ns           12 items_per_second=2.76113k/s
BM_ManualTiming/8/manual_time                       104062 ns        17034 ns            1 items_per_second=76.8772k/s
BM_ManualTiming/8/manual_time                        85433 ns        17071 ns            1 items_per_second=93.6406k/s
BM_ManualTiming/8/manual_time                        89342 ns        16251 ns            1 items_per_second=89.5436k/s
BM_ManualTiming/8/manual_time                        82813 ns        13112 ns            1 items_per_second=96.6032k/s
BM_ManualTiming/8/manual_time                       102901 ns        30285 ns            1 items_per_second=77.7446k/s
BM_ManualTiming/8/manual_time                       116615 ns        12938 ns            1 items_per_second=68.6018k/s
BM_ManualTiming/8/manual_time                        84661 ns        14864 ns            1 items_per_second=94.4945k/s
BM_ManualTiming/8/manual_time                       101697 ns        10546 ns            1 items_per_second=78.6651k/s
BM_ManualTiming/8/manual_time                        96837 ns        19850 ns            1 items_per_second=82.6131k/s
BM_ManualTiming/8/manual_time                        75736 ns        17106 ns            1 items_per_second=105.63k/s
BM_ManualTiming/8/manual_time                        73868 ns        14710 ns            1 items_per_second=108.301k/s
BM_ManualTiming/8/manual_time                        81199 ns        11272 ns            1 items_per_second=98.5234k/s
BM_ManualTiming/8/manual_time_mean                   91264 ns        16253 ns           12 items_per_second=89.2699k/s
BM_ManualTiming/8/manual_time_median                 87388 ns        15557 ns           12 items_per_second=91.5921k/s
BM_ManualTiming/8/manual_time_stddev                 13072 ns         5173 ns           12 items_per_second=12.3806k/s
BM_ManualTiming/64/manual_time                      172693 ns        12934 ns            1 items_per_second=370.6k/s
BM_ManualTiming/64/manual_time                      137859 ns        11754 ns            1 items_per_second=464.242k/s
BM_ManualTiming/64/manual_time                      139304 ns        10970 ns            1 items_per_second=459.427k/s
BM_ManualTiming/64/manual_time                      139857 ns        13783 ns            1 items_per_second=457.61k/s
BM_ManualTiming/64/manual_time                      135655 ns        11138 ns            1 items_per_second=471.785k/s
BM_ManualTiming/64/manual_time                      184438 ns        16058 ns            1 items_per_second=347k/s
BM_ManualTiming/64/manual_time                      155938 ns        17575 ns            1 items_per_second=410.42k/s
BM_ManualTiming/64/manual_time                      144073 ns        15221 ns            1 items_per_second=444.219k/s
BM_ManualTiming/64/manual_time                      139007 ns        12778 ns            1 items_per_second=460.408k/s
BM_ManualTiming/64/manual_time                      199202 ns         7275 ns            1 items_per_second=321.282k/s
BM_ManualTiming/64/manual_time                      133904 ns        19771 ns            1 items_per_second=477.954k/s
BM_ManualTiming/64/manual_time                      134930 ns        11810 ns            1 items_per_second=474.32k/s
BM_ManualTiming/64/manual_time_mean                 151405 ns        13422 ns           12 items_per_second=429.939k/s
BM_ManualTiming/64/manual_time_median               139580 ns        12856 ns           12 items_per_second=458.519k/s
BM_ManualTiming/64/manual_time_stddev                22045 ns         3343 ns           12 items_per_second=54.4041k/s
BM_ManualTiming/512/manual_time                   15469204 ns        22368 ns            1 items_per_second=33.098k/s
BM_ManualTiming/512/manual_time                     587687 ns        13374 ns            1 items_per_second=871.212k/s
BM_ManualTiming/512/manual_time                     589809 ns        14363 ns            1 items_per_second=868.078k/s
BM_ManualTiming/512/manual_time                     621871 ns        28679 ns            1 items_per_second=823.322k/s
BM_ManualTiming/512/manual_time                     589193 ns        12422 ns            1 items_per_second=868.985k/s
BM_ManualTiming/512/manual_time                     599085 ns        15603 ns            1 items_per_second=854.637k/s
BM_ManualTiming/512/manual_time                     599787 ns        18362 ns            1 items_per_second=853.636k/s
BM_ManualTiming/512/manual_time                     592806 ns        17689 ns            1 items_per_second=863.689k/s
BM_ManualTiming/512/manual_time                     581505 ns         9199 ns            1 items_per_second=880.474k/s
BM_ManualTiming/512/manual_time                     614143 ns        26472 ns            1 items_per_second=833.682k/s
BM_ManualTiming/512/manual_time                     581808 ns        17050 ns            1 items_per_second=880.015k/s
BM_ManualTiming/512/manual_time                    2665960 ns        15338 ns            1 items_per_second=192.051k/s
BM_ManualTiming/512/manual_time_mean               2007738 ns        17577 ns           12 items_per_second=735.24k/s
BM_ManualTiming/512/manual_time_median              595946 ns        16326 ns           12 items_per_second=859.163k/s
BM_ManualTiming/512/manual_time_stddev             4280850 ns         5720 ns           12 items_per_second=293.312k/s
BM_ManualTiming/4096/manual_time                   4182052 ns        19954 ns            1 items_per_second=979.423k/s
BM_ManualTiming/4096/manual_time                   4240516 ns        21447 ns            1 items_per_second=965.92k/s
BM_ManualTiming/4096/manual_time                   4193118 ns        16281 ns            1 items_per_second=976.839k/s
BM_ManualTiming/4096/manual_time                   4192942 ns        21364 ns            1 items_per_second=976.88k/s
BM_ManualTiming/4096/manual_time                   4270906 ns        27436 ns            1 items_per_second=959.047k/s
BM_ManualTiming/4096/manual_time                   4172457 ns        15496 ns            1 items_per_second=981.676k/s
BM_ManualTiming/4096/manual_time                   4184650 ns        18769 ns            1 items_per_second=978.815k/s
BM_ManualTiming/4096/manual_time                   4178909 ns        18081 ns            1 items_per_second=980.16k/s
BM_ManualTiming/4096/manual_time                   4194173 ns        29297 ns            1 items_per_second=976.593k/s
BM_ManualTiming/4096/manual_time                   4171065 ns        21064 ns            1 items_per_second=982.003k/s
BM_ManualTiming/4096/manual_time                   4205952 ns        68864 ns            1 items_per_second=973.858k/s
BM_ManualTiming/4096/manual_time                   4180798 ns        17059 ns            1 items_per_second=979.717k/s
BM_ManualTiming/4096/manual_time_mean              4197295 ns        24593 ns           12 items_per_second=975.911k/s
BM_ManualTiming/4096/manual_time_median            4188796 ns        20509 ns           12 items_per_second=977.848k/s
BM_ManualTiming/4096/manual_time_stddev              29719 ns        14552 ns           12 items_per_second=6.83808k/s
BM_ManualTiming/16384/manual_time                 16513779 ns        17707 ns            1 items_per_second=992.141k/s
BM_ManualTiming/16384/manual_time                 16471564 ns        20509 ns            1 items_per_second=994.684k/s
BM_ManualTiming/16384/manual_time                 16482964 ns        23208 ns            1 items_per_second=993.996k/s
BM_ManualTiming/16384/manual_time                 16494558 ns        28488 ns            1 items_per_second=993.297k/s
BM_ManualTiming/16384/manual_time                 16473792 ns        20545 ns            1 items_per_second=994.549k/s
BM_ManualTiming/16384/manual_time                 16482720 ns        23462 ns            1 items_per_second=994.011k/s
BM_ManualTiming/16384/manual_time                 16494458 ns        22581 ns            1 items_per_second=993.303k/s
BM_ManualTiming/16384/manual_time                 16489719 ns        23352 ns            1 items_per_second=993.589k/s
BM_ManualTiming/16384/manual_time                 16613390 ns        31377 ns            1 items_per_second=986.192k/s
BM_ManualTiming/16384/manual_time                 16463222 ns        25442 ns            1 items_per_second=995.188k/s
BM_ManualTiming/16384/manual_time                 16460168 ns        15652 ns            1 items_per_second=995.373k/s
BM_ManualTiming/16384/manual_time                 20386042 ns        21038 ns            1 items_per_second=803.687k/s
BM_ManualTiming/16384/manual_time_mean            16818865 ns        22780 ns           12 items_per_second=977.501k/s
BM_ManualTiming/16384/manual_time_median          16486342 ns        22895 ns           12 items_per_second=993.792k/s
BM_ManualTiming/16384/manual_time_stddev           1124094 ns         4310 ns           12 items_per_second=54.7905k/s
BM_with_args/int_test                                 1113 ns         1075 ns            1
BM_with_args/int_test                                 1041 ns         1027 ns            1
BM_with_args/int_test                                 1648 ns         1513 ns            1
BM_with_args/int_test                                 1105 ns         1085 ns            1
BM_with_args/int_test                                 1042 ns         1028 ns            1
BM_with_args/int_test                                 1188 ns         1087 ns            1
BM_with_args/int_test                                 1308 ns         1176 ns            1
BM_with_args/int_test_mean                            1206 ns         1142 ns            7
BM_with_args/int_test_median                          1113 ns         1085 ns            7
BM_with_args/int_test_stddev                           216 ns          171 ns            7
BM_with_args/string_and_pair_test                     1795 ns         1639 ns            1
BM_with_args/string_and_pair_test                     1404 ns         1288 ns            1
BM_with_args/string_and_pair_test                     1120 ns         1106 ns            1
BM_with_args/string_and_pair_test                     1046 ns         1043 ns            1
BM_with_args/string_and_pair_test                     2199 ns         1918 ns            1
BM_with_args/string_and_pair_test                     1123 ns         1101 ns            1
BM_with_args/string_and_pair_test_mean                1448 ns         1349 ns            6
BM_with_args/string_and_pair_test_median              1263 ns         1197 ns            6
BM_with_args/string_and_pair_test_stddev               461 ns          354 ns            6
BM_non_template_args/basic_test                       1132 ns         1066 ns            1
BM_non_template_args/basic_test                       1079 ns         1062 ns            1
BM_non_template_args/basic_test                       1104 ns         1075 ns            1
BM_non_template_args/basic_test                       1126 ns         1115 ns            1
BM_non_template_args/basic_test                       1210 ns         1190 ns            1
BM_non_template_args/basic_test                       1116 ns         1100 ns            1
BM_non_template_args/basic_test_mean                  1128 ns         1101 ns            6
BM_non_template_args/basic_test_median                1121 ns         1088 ns            6
BM_non_template_args/basic_test_stddev                44.4 ns         48.0 ns            6
BM_DenseThreadRanges/1/threads:1                      1472 ns         1295 ns            1
BM_DenseThreadRanges/1/threads:1                      1028 ns          994 ns            1
BM_DenseThreadRanges/1/threads:1                      1144 ns         1129 ns            1
BM_DenseThreadRanges/1/threads:1                      1069 ns         1048 ns            1
BM_DenseThreadRanges/1/threads:1                      1141 ns         1120 ns            1
BM_DenseThreadRanges/1/threads:1                      1083 ns         1061 ns            1
BM_DenseThreadRanges/1/threads:1_mean                 1156 ns         1108 ns            6
BM_DenseThreadRanges/1/threads:1_median               1112 ns         1090 ns            6
BM_DenseThreadRanges/1/threads:1_stddev                161 ns          104 ns            6
BM_DenseThreadRanges/1/threads:2                      1210 ns         2030 ns            2
BM_DenseThreadRanges/1/threads:3                       533 ns         1270 ns            3
BM_DenseThreadRanges/2/threads:1                      1094 ns          998 ns            1
BM_DenseThreadRanges/2/threads:1                      1035 ns         1030 ns            1
BM_DenseThreadRanges/2/threads:1                      1237 ns         1140 ns            1
BM_DenseThreadRanges/2/threads:1                      1072 ns         1058 ns            1
BM_DenseThreadRanges/2/threads:1                      1086 ns         1073 ns            1
BM_DenseThreadRanges/2/threads:1                      1113 ns         1074 ns            1
BM_DenseThreadRanges/2/threads:1_mean                 1106 ns         1062 ns            6
BM_DenseThreadRanges/2/threads:1_median               1090 ns         1066 ns            6
BM_DenseThreadRanges/2/threads:1_stddev               69.2 ns         47.9 ns            6
BM_DenseThreadRanges/2/threads:3                       678 ns         2181 ns            3
BM_DenseThreadRanges/2/threads:4                       484 ns         1718 ns            4
BM_DenseThreadRanges/3/threads:5                       773 ns         3540 ns            5
BM_DenseThreadRanges/3/threads:8                       203 ns         1392 ns            8
BM_DenseThreadRanges/3/threads:11                      149 ns         1415 ns           11
BM_DenseThreadRanges/3/threads:14                      199 ns         2623 ns           14

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: test/benchmark_random_interleaving_gtest.cc

dmah42 · 2021-03-30T12:03:05Z

tests are showing a lot of 0 CPU result (which is then causing reporter output tests to fail because 'inf' is not a number). i don't see anything obvious in how this has been triggered by this change, but it's worth taking a look.

windows only too, which is also odd.

…fr-1051. Also change sentinel of random_interleaving_repetitions to -1. Hopefully it fixes the failures on Windows. Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_runner.cc

dmah42 · 2021-04-06T16:16:25Z

i think this looks good, but i'm biased because i've seen it before. i'd like @LebedevRI to also take a look.

src/benchmark_adjust_repetitions.cc

src/benchmark_adjust_repetitions.h

LebedevRI · 2021-04-06T17:08:04Z

Missing documentation changes.
Is there any way to make patch more reviewable?
Is there any chunks that could be split off into a preparatory PR?

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_adjust_repetitions.cc modified: src/benchmark_adjust_repetitions.h

haihuang-ml · 2021-04-13T22:58:24Z

Missing documentation changes.
Is there any way to make patch more reviewable?
Is there any chunks that could be split off into a preparatory PR?

This PR introduces a new paradigm to run benchmarks and it has to touch multiple files at the same time. The only piece I can split off is the benchmark_adjust_repetitions.h|cc. I can have a dummy ComputeRandomInterleavingRepetitions() in the first PR and then have another PR to add the real ComputeRandomInterleavingRepetitions(). But I don't think it will make the PR more reviewable. The .h|cc is already isolated from other files.

matta · 2021-04-18T15:11:48Z

src/benchmark.cc

+    std::vector<RunResults> run_results_vector(benchmarks.size());
+    for (size_t i = 0; i < outer_repetitions; i++) {
+      if (FLAGS_benchmark_enable_random_interleaving) {
+        std::random_shuffle(benchmark_indices.begin(), benchmark_indices.end());


std::random_shuffle uses std::rand and was deprecated in C++11 and removed in C++17. std::shuffle is its replacement. See https://en.cppreference.com/w/cpp/algorithm/random_shuffle

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc

LebedevRI · 2021-04-28T09:05:48Z

src/benchmark_adjust_repetitions.cc

+
+namespace {
+
+constexpr double kNanosecondInSecond = 0.000000001;


LebedevRI

I still believe this can be split up at least into 3 parts:

BenchmarkInstance rework (rewrite to getters)
Native interleaving just of the repetitions as they are defined now
Introduction of automatic repetitions for interleaving

As is it's a bit too big to review. I guess it looks fine.

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_adjust_repetitions.cc

dmah42 · 2021-05-17T08:49:14Z

CI failure is real:

/__w/benchmark/benchmark/src/benchmark_api_internal.cc: In member function 'void benchmark::internal::BenchmarkInstance::InitRandomInterleavingRepetitions(int) const':
/__w/benchmark/benchmark/src/benchmark_api_internal.cc:108:22: error: declaration of 'repetitions' shadows a member of 'this' [-Werror=shadow]
     int repetitions) const {

you need to tweak the name to something like repetitionsCount or reps to avoid the error.

haihuang-ml · 2021-05-18T20:40:16Z

CI failure is real:

/__w/benchmark/benchmark/src/benchmark_api_internal.cc: In member function 'void benchmark::internal::BenchmarkInstance::InitRandomInterleavingRepetitions(int) const':
/__w/benchmark/benchmark/src/benchmark_api_internal.cc:108:22: error: declaration of 'repetitions' shadows a member of 'this' [-Werror=shadow]
     int repetitions) const {

you need to tweak the name to something like repetitionsCount or reps to avoid the error.

Done

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: src/benchmark_runner.cc

dmah42 · 2021-05-19T09:24:26Z

One last thing :)

Can you add a section to the docs and link to it from the README.md please that explains the use and benefit of the flag?

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: README.md new file: docs/random_interleaving.md

haihuang-ml · 2021-05-20T05:06:41Z

One last thing :)

Can you add a section to the docs and link to it from the README.md please that explains the use and benefit of the flag?

Done

dmah42 · 2021-05-20T16:09:37Z

thank you!

LebedevRI · 2021-05-30T20:23:48Z

Ok. Now that i have actually tried this locally, i have the very same thought i voiced originally:
unless this is split up, this is too big to review :)

Because this was too big, two pretty minor bugs slipped through:

if i asked for 1000 repetitions, i'd like to have 1000 repetitions.
if i asked for the repetition to take 1 s, the original, full repetition should take 1 sec, not the fractional repetition.

If i was at the helm, this is where i'd back this out, and ask for a complete redesign,
and splitting this into two parts, like i originally suggested.

I am talking about redesign, because we can't just divide min time by the desired number of fractions,
because single iteration may take longer than min time, and presumably we shouldn't just run 12 repetitions?

:/

This reverts commit a6a738c.

dmah42 · 2021-06-01T15:06:46Z

I've reverted this in #1161.

@haih-g, if we want to try this again then I think following the approach outlined by @LebedevRI is worthwhile. We need to make sure that expectations of interactions between different flags are honoured.

haihuang-ml · 2021-06-01T18:25:45Z

Ok. Now that i have actually tried this locally, i have the very same thought i voiced originally:
unless this is split up, this is too big to review :)

Because this was too big, two pretty minor bugs slipped through:

if i asked for 1000 repetitions, i'd like to have 1000 repetitions.

if i asked for the repetition to take 1 s, the original, full repetition should take 1 sec, not the fractional repetition.

If i was at the helm, this is where i'd back this out, and ask for a complete redesign,
and splitting this into two parts, like i originally suggested.

I am talking about redesign, because we can't just divide min time by the desired number of fractions,
because single iteration may take longer than min time, and presumably we shouldn't just run 12 repetitions?

:/

To help me better understand the bug, could you please provide a few run examples?

From what you described, I think this is actually the "repetition auto-adjustment" feature. Random interleaving will introduce some overhead and the overhead is controlled by the flag --benchmark_random_interleaving_max_overhead, which by default is 0.4 and will automatically reduce # of repetitions if the overhead is too large. I'm wondering if we could just set --benchmark_random_interleaving_max_overhead=inf by default to resolve this bug.

One thing we learned from the Google internal implementation is that benchmarks may timeout simply because the overhead of random interleaving may be too much. So we have to have "repetition auto-adjustment" because we cannot afford breaking anyone in a mono-repo environment. But maybe this is not needed in the world of OSS, as everyone is doing version control by themselves.

So if "repetition auto-adjustment" is causing troubles, we can either:

Do not offer it in the OSS version
Still offer it but set --benchmark_random_interleaving_max_overhead=inf by default.
Still offer it and still keep --benchmark_random_interleaving_max_overhead=0.4 by default, which is a proven default value from Google internal implementation. But we improve the help/doc of the flags so users are aware of this feature.

Thoughts?

…le#1051) Based on the original implementation by Hai Huang @haih-g) from google#1105.

Based on original implementation by Hai Huang @haih-g in #1105

…le#1051) Based on the original implementation by Hai Huang @haih-g) from google#1105.

…le#1051) Inspired by the original implementation by Hai Huang @haih-g from google#1105. The original implementation had design deficiencies that weren't really addressable without redesign, so it was reverted. In essence, the original implementation consisted of two separateable parts: * reducing the amount time each repetition is run for, and symmetrically increasing repetition count * running the repetitions in random order While it worked fine for the usual case, it broke down when user would specify repetitions (it would completely ignore that request), or specified per-repetition min time (while it would still adjust the repetition count, it would not adjust the per-repetition time, leading to much greater run times) Here, like i was originally suggesting in the original review, i'm separating the features, and only dealing with a single one - running repetitions in random order. Now that the runs/repetitions are no longer in-order, the tooling may wish to sort the output, and indeed `compare.py` has been updated to do that: google#1168.

… (#1163) Inspired by the original implementation by Hai Huang @haih-g from #1105. The original implementation had design deficiencies that weren't really addressable without redesign, so it was reverted. In essence, the original implementation consisted of two separateable parts: * reducing the amount time each repetition is run for, and symmetrically increasing repetition count * running the repetitions in random order While it worked fine for the usual case, it broke down when user would specify repetitions (it would completely ignore that request), or specified per-repetition min time (while it would still adjust the repetition count, it would not adjust the per-repetition time, leading to much greater run times) Here, like i was originally suggesting in the original review, i'm separating the features, and only dealing with a single one - running repetitions in random order. Now that the runs/repetitions are no longer in-order, the tooling may wish to sort the output, and indeed `compare.py` has been updated to do that: #1168.

* Refactor BenchmarkInstance (precursor to google#1105) * fix bazel (debug) build * clang-format on header * fix build error on g++-4.8

* Implementation of random interleaving. See http://github.com/google/benchmark/issues/1051 for the feature requests. Committer: Hai Huang (http://github.com/haih-g) On branch fr-1051 Changes to be committed: modified: include/benchmark/benchmark.h modified: src/benchmark.cc new file: src/benchmark_adjust_repetitions.cc new file: src/benchmark_adjust_repetitions.h modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: src/benchmark_register.cc modified: src/benchmark_runner.cc modified: src/benchmark_runner.h modified: test/CMakeLists.txt new file: test/benchmark_random_interleaving_gtest.cc * Fix benchmark_random_interleaving_gtest.cc for fr-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc modified: src/benchmark_runner.cc modified: test/benchmark_random_interleaving_gtest.cc * Fix macos build for fr-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: src/benchmark_runner.cc * Fix macos and windows build for fr-1051. Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_runner.cc * Fix benchmark_random_interleaving_test.cc for macos and windows in fr-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: test/benchmark_random_interleaving_gtest.cc * Fix int type benchmark_random_interleaving_gtest for macos in fr-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: test/benchmark_random_interleaving_gtest.cc * Address dominichamon's comments 03/29 for fr-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: test/benchmark_random_interleaving_gtest.cc * Address dominichamon's comment on default min_time / repetitions for fr-1051. Also change sentinel of random_interleaving_repetitions to -1. Hopefully it fixes the failures on Windows. Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h * Fix windows test failures for fr-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_runner.cc * Add license blurb for fr-1051. Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_adjust_repetitions.cc modified: src/benchmark_adjust_repetitions.h * Switch to std::shuffle() for fr-1105. Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc * Change to 1e-9 in fr-1105 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_adjust_repetitions.cc * Fix broken build caused by bad merge for fr-1105. Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_runner.cc * Fix build breakage for fr-1051. Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: src/benchmark_register.cc modified: src/benchmark_runner.cc * Print out reports as they come in if random interleaving is disabled (fr-1051) Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc * size_t, int64_t --> int in benchmark_runner for fr-1051. Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_runner.cc modified: src/benchmark_runner.h * Address comments from dominichamon for fr-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc modified: src/benchmark_adjust_repetitions.cc modified: src/benchmark_adjust_repetitions.h modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: test/benchmark_random_interleaving_gtest.cc * benchmar_indices --> size_t to make CI pass: fr-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc * Fix min_time not initialized issue for fr-1051. Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h * min_time --> MinTime in fr-1051. Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: src/benchmark_runner.cc * Add doc for random interleaving for fr-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: README.md new file: docs/random_interleaving.md Co-authored-by: Dominic Hamon <dominichamon@users.noreply.github.com>

…#1161) This reverts commit a6a738c.

Based on original implementation by Hai Huang @haih-g in google#1105

…le#1051) (google#1163) Inspired by the original implementation by Hai Huang @haih-g from google#1105. The original implementation had design deficiencies that weren't really addressable without redesign, so it was reverted. In essence, the original implementation consisted of two separateable parts: * reducing the amount time each repetition is run for, and symmetrically increasing repetition count * running the repetitions in random order While it worked fine for the usual case, it broke down when user would specify repetitions (it would completely ignore that request), or specified per-repetition min time (while it would still adjust the repetition count, it would not adjust the per-repetition time, leading to much greater run times) Here, like i was originally suggesting in the original review, i'm separating the features, and only dealing with a single one - running repetitions in random order. Now that the runs/repetitions are no longer in-order, the tooling may wish to sort the output, and indeed `compare.py` has been updated to do that: google#1168.

google-cla bot added the cla: yes label Mar 28, 2021

haihuang-ml mentioned this pull request Mar 28, 2021

[FR] Using Random Interleaving to reduce run-to-run variance. #1051

Closed

haihuang-ml added 5 commits March 28, 2021 17:35

Fix macos build for fr-1051

58f3a39

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: src/benchmark_runner.cc

Fix macos and windows build for fr-1051.

89d3a78

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_runner.cc

Fix benchmark_random_interleaving_test.cc for macos and windows in fr…

0aadade

…-1051 Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: test/benchmark_random_interleaving_gtest.cc

Fix int type benchmark_random_interleaving_gtest for macos in fr-1051

f42e87c

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: test/benchmark_random_interleaving_gtest.cc

dmah42 requested changes Mar 29, 2021

View reviewed changes

haihuang-ml added 2 commits March 31, 2021 01:59

Fix windows test failures for fr-1051

1a2b6df

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_runner.cc

dmah42 assigned LebedevRI Apr 6, 2021

LebedevRI reviewed Apr 6, 2021

View reviewed changes

src/benchmark_adjust_repetitions.cc Show resolved Hide resolved

LebedevRI reviewed Apr 6, 2021

View reviewed changes

src/benchmark_adjust_repetitions.h Show resolved Hide resolved

Add license blurb for fr-1051.

81c9ab0

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_adjust_repetitions.cc modified: src/benchmark_adjust_repetitions.h

matta reviewed Apr 18, 2021

View reviewed changes

Switch to std::shuffle() for fr-1105.

70fe24a

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark.cc

dmah42 added enhancement next-release PRs or Issues that should be included in the next release labels Apr 26, 2021

dmah42 approved these changes Apr 28, 2021

View reviewed changes

LebedevRI reviewed Apr 28, 2021

View reviewed changes

haihuang-ml added 2 commits April 29, 2021 02:48

Change to 1e-9 in fr-1105

4966e90

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_adjust_repetitions.cc

Merge branch 'master' into fr-1051

688eaff

haihuang-ml and others added 2 commits May 18, 2021 16:59

min_time --> MinTime in fr-1051.

ee70382

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: src/benchmark_api_internal.cc modified: src/benchmark_api_internal.h modified: src/benchmark_runner.cc

Merge branch 'master' into fr-1051

4bdafa9

haihuang-ml added 2 commits May 20, 2021 01:03

Add doc for random interleaving for fr-1051

736875b

Committer: Hai Huang <haih@google.com> On branch fr-1051 Your branch is up to date with 'origin/fr-1051'. Changes to be committed: modified: README.md new file: docs/random_interleaving.md

Merge branch 'fr-1051' of github.com:haih-g/benchmark into fr-1051

d340447

dmah42 removed the incomplete work needed label May 20, 2021

dmah42 merged commit a6a738c into google:master May 20, 2021

dmah42 added a commit that referenced this pull request Jun 1, 2021

Revert "Implementation of random interleaving. (#1105)"

9a3bf70

This reverts commit a6a738c.

dmah42 pushed a commit that referenced this pull request Jun 1, 2021

Revert "Implementation of random interleaving. (#1105)" (#1161)

e025dd5

This reverts commit a6a738c.

LebedevRI mentioned this pull request Jun 1, 2021

Revert "Implementation of random interleaving. (#1105)" #1161

Merged

LebedevRI added a commit to LebedevRI/benchmark that referenced this pull request Jun 1, 2021

Random interleaving of benchmark repetitions - the sequel (fixes goog…

09b2b9e

…le#1051) Based on the original implementation by Hai Huang @haih-g) from google#1105.

LebedevRI mentioned this pull request Jun 1, 2021

Random interleaving of benchmark repetitions - the sequel (fixes #1051) #1163

Merged

LebedevRI added a commit that referenced this pull request Jun 3, 2021

[NFCI] RunBenchmarks(): extract FlushStreams()/Report() functions

520573f

Based on original implementation by Hai Huang @haih-g in #1105

dmah42 pushed a commit that referenced this pull request Jun 3, 2021

[NFCI] RunBenchmarks(): extract FlushStreams()/Report() functions

01fa773

Based on original implementation by Hai Huang @haih-g in #1105

LebedevRI added a commit to LebedevRI/benchmark that referenced this pull request Jun 3, 2021

Random interleaving of benchmark repetitions - the sequel (fixes goog…

f293839

…le#1051) Based on the original implementation by Hai Huang @haih-g) from google#1105.

vincenzopalazzo pushed a commit to vincenzopalazzo/benchmark that referenced this pull request Feb 8, 2022

Refactor BenchmarkInstance (google#1148)

511f8e9

* Refactor BenchmarkInstance (precursor to google#1105) * fix bazel (debug) build * clang-format on header * fix build error on g++-4.8

vincenzopalazzo pushed a commit to vincenzopalazzo/benchmark that referenced this pull request Feb 8, 2022

Revert "Implementation of random interleaving. (google#1105)" (google…

091bf9f

…#1161) This reverts commit a6a738c.

vincenzopalazzo pushed a commit to vincenzopalazzo/benchmark that referenced this pull request Feb 8, 2022

[NFCI] RunBenchmarks(): extract FlushStreams()/Report() functions

1dae8ce

Based on original implementation by Hai Huang @haih-g in google#1105

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of random interleaving. #1105

Implementation of random interleaving. #1105

haihuang-ml commented Mar 28, 2021

dmah42 left a comment

haihuang-ml commented Mar 30, 2021

dmah42 commented Mar 30, 2021

dmah42 commented Apr 6, 2021

LebedevRI commented Apr 6, 2021

haihuang-ml commented Apr 13, 2021

matta Apr 18, 2021

haihuang-ml Apr 19, 2021

LebedevRI Apr 28, 2021

haihuang-ml Apr 29, 2021

LebedevRI left a comment

dmah42 commented May 17, 2021

haihuang-ml commented May 18, 2021

dmah42 commented May 19, 2021

haihuang-ml commented May 20, 2021

dmah42 commented May 20, 2021

LebedevRI commented May 30, 2021 •

edited

Loading

dmah42 commented Jun 1, 2021

haihuang-ml commented Jun 1, 2021


		namespace {

		constexpr double kNanosecondInSecond = 0.000000001;

Implementation of random interleaving. #1105

Implementation of random interleaving. #1105

Conversation

haihuang-ml commented Mar 28, 2021

dmah42 left a comment

Choose a reason for hiding this comment

haihuang-ml commented Mar 30, 2021

Without Random Interleaving

With Random Interleaving

dmah42 commented Mar 30, 2021

dmah42 commented Apr 6, 2021

LebedevRI commented Apr 6, 2021

haihuang-ml commented Apr 13, 2021

matta Apr 18, 2021

Choose a reason for hiding this comment

haihuang-ml Apr 19, 2021

Choose a reason for hiding this comment

LebedevRI Apr 28, 2021

Choose a reason for hiding this comment

haihuang-ml Apr 29, 2021

Choose a reason for hiding this comment

LebedevRI left a comment

Choose a reason for hiding this comment

dmah42 commented May 17, 2021

haihuang-ml commented May 18, 2021

dmah42 commented May 19, 2021

haihuang-ml commented May 20, 2021

dmah42 commented May 20, 2021

LebedevRI commented May 30, 2021 • edited Loading

dmah42 commented Jun 1, 2021

haihuang-ml commented Jun 1, 2021

LebedevRI commented May 30, 2021 •

edited

Loading