diff --git a/.gitignore b/.gitignore index a293dc33..ba2200aa 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ doc docs /test/tmp save.benchee +/.elixir_ls/ diff --git a/README.md b/README.md index c72c130f..eaa31c7a 100644 --- a/README.md +++ b/README.md @@ -212,6 +212,7 @@ The available options are the following (also documented in [hexdocs](https://he * `print` - a map or keyword list from atoms to `true` or `false` to configure if the output identified by the atom will be printed during the standard benchee benchmarking process. All options are enabled by default (true). Options are: * `:benchmarking` - print when Benchee starts benchmarking a new job (`Benchmarking name ...`) * `:configuration` - a summary of configured benchmarking options including estimated total run time is printed before benchmarking starts + * `:system` - a description of a system benchmark is being run on * `:fast_warning` - warnings are displayed if functions are executed too fast leading to inaccurate measures * `:unit_scaling` - the strategy for choosing a unit for durations, counts & memory measurements. May or may not be implemented by a given formatter (The console formatter implements it). diff --git a/example/project_benchmark.exs b/example/project_benchmark.exs new file mode 100644 index 00000000..f7a848fa --- /dev/null +++ b/example/project_benchmark.exs @@ -0,0 +1,87 @@ +defmodule Clickhouse.Benchmark.Performance do + use Benchee.Benchmark + + # Since it's a module, any other constructs may be used, + # such as defs, defmacro, etc. + def map_fun(i) do + [i] + end + + # Global benchmark setup. Optional. + # + # Called before any benchmark in the module. + # + # If defined, must return {:ok, state} for benchmarks to run + before_all do + {:ok, nil} + end + + # Global benchmark teardown. Optional. + # + # Called after all benchmarks have finished + # (and their possible local teardowns had been called). + # + # Can return anything. + after_all do + :anything + end + + # Benchmarks. Module may have many of them. + benchmark "Flattening list from map", # Name. Mandatory. + warmup: 0, time: 1 # Opts (see Benchee.run). Optional. + do # Do block. Mandatory. + + # Benchmark setup. Optional + # + # If global setup is defined, + # implicit variable "state" is bound to it's result + # + # If defined, must return {:ok, state} for benchmarks to run + before_benchmark do + {:ok, fn x -> [x] end} + end + + # Benchmark teardown. Optional + # + # Implicit variable: state, as it is returned from local setup + # (or from global setup, if no local setup is defined) + # + # Can return anything + after_benchmark do + :anything + end + + # Inputs: the same as passing inputs via :input option + # + # Accepts either an expression or a do block as a 2nd argument + input "Small", Enum.to_list(1..100) + input "Medium" do + n = 10_000 + Enum.to_list(1..n) + end + input "Bigger", Enum.to_list(1..100_000) + + + # Benchmark scenarios + # + # Each scenario has an implicit variables: + # - state: state returned from local or global setup + # And if any input is passed (either with option or as input directive): + # - input: data for benchmark + scenario "Enum.flat_map", # Name. Mandatory + before_scenario: + fn i -> # Scenario options, e.g. local hooks. Optional. + IO.inspect(length(i), label: "Input length"); + i + end + do # Do block. Mandatory + map_fun = state + Enum.flat_map(input, map_fun) + end + + scenario "Enum.map |> List.flatten" do + map_fun = state + input |> Enum.map(map_fun) |> List.flatten() + end + end +end diff --git a/lib/benchee/benchmark.ex b/lib/benchee/benchmark.ex index 3cf41699..7ddf36e7 100644 --- a/lib/benchee/benchmark.ex +++ b/lib/benchee/benchmark.ex @@ -98,9 +98,22 @@ defmodule Benchee.Benchmark do printer \\ Printer, runner \\ Runner ) do + printer.system_information(suite) printer.configuration_information(suite) scenario_context = %ScenarioContext{config: config, printer: printer} scenarios = runner.run_scenarios(scenarios, scenario_context) %Suite{suite | scenarios: scenarios} end + + # delegate use to Benchee.Project.Benchmark.__using__/1 + defmacro __using__(opts) do + require Benchee.Project.Benchmark + + quoted = + quote do + use Benchee.Project.Benchmark, unquote(opts) + end + + Macro.expand(quoted, __ENV__) + end end diff --git a/lib/benchee/configuration.ex b/lib/benchee/configuration.ex index d743186e..71441209 100644 --- a/lib/benchee/configuration.ex +++ b/lib/benchee/configuration.ex @@ -22,8 +22,9 @@ defmodule Benchee.Configuration do percentiles: [50, 99], print: %{ benchmarking: true, + fast_warning: true, configuration: true, - fast_warning: true + system: true }, inputs: nil, save: false, @@ -178,7 +179,8 @@ defmodule Benchee.Configuration do print: %{ benchmarking: true, fast_warning: true, - configuration: true + configuration: true, + system: true }, percentiles: [50, 99], unit_scaling: :best, @@ -206,7 +208,8 @@ defmodule Benchee.Configuration do print: %{ benchmarking: true, fast_warning: true, - configuration: true + configuration: true, + system: true }, percentiles: [50, 99], unit_scaling: :best, @@ -234,7 +237,8 @@ defmodule Benchee.Configuration do print: %{ benchmarking: true, fast_warning: true, - configuration: true + configuration: true, + system: true }, percentiles: [50, 99], unit_scaling: :best, @@ -269,7 +273,8 @@ defmodule Benchee.Configuration do print: %{ benchmarking: true, fast_warning: false, - configuration: true + configuration: true, + system: true }, percentiles: [50, 99], unit_scaling: :smallest, diff --git a/lib/benchee/output/benchmark_printer.ex b/lib/benchee/output/benchmark_printer.ex index 1293efc9..bed65233 100644 --- a/lib/benchee/output/benchmark_printer.ex +++ b/lib/benchee/output/benchmark_printer.ex @@ -15,19 +15,17 @@ defmodule Benchee.Output.BenchmarkPrinter do end @doc """ - Prints general information such as system information and estimated - benchmarking time. + Prints general information about the system such as operating system. """ - def configuration_information(%{configuration: %{print: %{configuration: false}}}) do + def system_information(%{configuration: %{print: %{system: false}}}) do nil end - def configuration_information(%{scenarios: scenarios, system: sys, configuration: config}) do - system_information(sys) - suite_information(scenarios, config) + def system_information(%{system: sys}) do + print_system_information(sys) end - defp system_information(%{ + defp print_system_information(%{ erlang: erlang_version, elixir: elixir_version, os: os, @@ -45,7 +43,18 @@ defmodule Benchee.Output.BenchmarkPrinter do """) end - defp suite_information(scenarios, %{ + @doc """ + Prints general benchmark information such as estimated benchmarking time. + """ + def configuration_information(%{configuration: %{print: %{configuration: false}}}) do + nil + end + + def configuration_information(%{scenarios: scenarios, configuration: config}) do + print_suite_information(scenarios, config) + end + + defp print_suite_information(scenarios, %{ parallel: parallel, time: time, warmup: warmup, diff --git a/lib/benchee/project/benchmark.ex b/lib/benchee/project/benchmark.ex new file mode 100644 index 00000000..2679cb95 --- /dev/null +++ b/lib/benchee/project/benchmark.ex @@ -0,0 +1,190 @@ +defmodule Benchee.Project.Benchmark do + @moduledoc """ + Module with macro for defining project-wide benchmarks. + """ + require Logger + + def benchee_scenario_attr, do: :benchee_benchmarks + + @doc """ + Module-scope setup. Optional. + + Must return {:ok, state} for actual benchmark to start. + """ + defmacro before_all(_opts \\ [], do: doblock) do + quote do + def before_all do + unquote(doblock) + end + end + end + + @doc """ + Module-scope teardown. Optional + + Can return anything. + """ + defmacro after_all(_opts \\ [], do: doblock) do + quote do + def after_all(state) do + unquote(doblock) + end + end + end + + # credo:disable-for-next-line Credo.Check.Refactor.CyclomaticComplexity + defmacro benchmark(name, opts \\ [], do: {:__block__, [], exprs}) do + state_var = atom_to_var(:state) + + run_fun_name = make_fun_name(name) + + inputs = make_inputs(exprs, opts) + scenarios = make_scenarios(exprs, inputs) + + before_benchmark = make_before_benchmark(exprs, state_var) + after_benchmark = make_after_benchmark(exprs, state_var) + + opts = + case inputs do + [] -> opts + [_ | _] -> Keyword.put(opts, :inputs, inputs) + end + + quote generated: true do + @unquote(benchee_scenario_attr())(unquote(run_fun_name)) + def unquote(run_fun_name)(unquote(state_var), opts \\ []) do + opts = DeepMerge.deep_merge(opts, unquote(opts)) + + case unquote(before_benchmark).(unquote(state_var)) do + {:ok, unquote(state_var)} -> + Benchee.run(unquote({:%{}, [], scenarios}), opts) + unquote(after_benchmark).(unquote(state_var)) + + other -> + :ok = + Logger.error( + "Local setup of \"#{unquote(name)}\" in " <> + "#{inspect(__MODULE__)} returned #{inspect(other)}" + ) + end + end + end + end + + defp make_fun_name(name) do + ["run", :erlang.unique_integer() |> Integer.to_string(), name] + |> Enum.join("_") + |> String.to_atom() + end + + defp make_inputs(exprs, opts) do + inputs_dsl = + for {:input, _, [name, expr]} <- exprs do + {name, + quote do + unquote(block_to_fun(expr)).() + end} + end + Keyword.get(opts, :inputs, []) ++ inputs_dsl + end + + defp make_scenarios(exprs, inputs) do + case_args = + case length(inputs) do + 0 -> [] + _ -> [:input] + end + + for {:scenario, _, args} <- exprs do + case args do + [name, block] -> + {name, block_to_fun(block, case_args)} + + [name, opts, block] -> + {name, {block_to_fun(block, case_args), opts}} + end + end + end + + defp make_before_benchmark(exprs, state_var) do + [before_doblock | rest] = + (exprs + |> Enum.filter(fn t -> elem(t, 0) == :before_benchmark end) + |> Enum.map(fn {:before_benchmark, _, [[do: doblock]]} -> doblock end) + ) ++ + [quote generated: true do {:ok, unquote(state_var)} end] + + before_benchmark = block_to_fun(before_doblock, [:state]) + + if length(rest) > 1 do + raise "Multiple before_benchmark found: only one is required" + end + + before_benchmark + end + + defp make_after_benchmark(exprs, state_var) do + [after_doblock | rest] = + (exprs + |> Enum.filter(fn t -> elem(t, 0) == :after_benchmark end) + |> Enum.map(fn {:after_benchmark, _, [[do: doblock]]} -> doblock end) + ) ++ + [quote do {:ok, unquote(state_var)} end] + + after_benchmark = block_to_fun(after_doblock, [:state]) + + if length(rest) > 1 do + raise "Multiple after_benchmark found: only one is required" + end + + after_benchmark + end + + defmacro __using__(_) do + quote do + require Logger + + Module.register_attribute(__MODULE__, unquote(benchee_scenario_attr()), + persist: true, + accumulate: true + ) + + import Benchee.Project.Benchmark, + only: [ + before_all: 1, + after_all: 1, + benchmark: 2, benchmark: 3 + ] + end + end + + # Helpers + defp quote_put_attrs({name, base_attrs, args}, attrs) do + {name, Keyword.merge(base_attrs, attrs), args} + end + + defp atom_to_var(atom, module \\ nil) do + atom + |> Macro.var(module) + |> quote_put_attrs(generated: true) + end + + defp block_to_fun(doblock, args \\ []) + + defp block_to_fun([do: doblock], args), + do: block_to_fun(doblock, args) + + defp block_to_fun(doblock, args) do + case Enum.map(args, &atom_to_var/1) do + [] -> + quote do + fn -> unquote(doblock) end + end + + argvars -> + quote do + fn unquote_splicing(argvars) -> unquote(doblock) end + end + end + end +end diff --git a/lib/mix/tasks/benchmark.ex b/lib/mix/tasks/benchmark.ex new file mode 100644 index 00000000..dd13b61d --- /dev/null +++ b/lib/mix/tasks/benchmark.ex @@ -0,0 +1,201 @@ +defmodule Mix.Tasks.Benchmark.Helper do + @moduledoc false + + defmacro compile_file(file) do + case compare_versions(System.version(), "1.7.0") do + :gt -> + quote do + Code.compile_file(unquote(file)) + end + + _ -> + quote do + Code.load_file(unquote(file)) + end + end + end + + defp compare_versions(v1, v2) do + Version.compare(Version.parse!(v1), Version.parse!(v2)) + end +end + +defmodule Mix.Tasks.Benchmark do + use Mix.Task + require Logger + alias Mix.Tasks.Benchmark.Helper + require Mix.Tasks.Benchmark.Helper + + @moduledoc """ + Runs project benchmarks + """ + @dialyzer [no_match: [find_benchmark_files: 2]] + + defp default_benchmark_path, do: "./benchmarks" + + @switches [] + + @shortdoc "Runs project benchmarks" + @recursive true + @preferred_cli_env :test + @impl Mix.Task + def run(args) do + {opts, files} = OptionParser.parse!(args, strict: @switches) + + with :ok <- compile_project(args, opts), + {:ok, modules} <- compile_benchmarks(files, opts), + :ok <- run_benchmarks(modules, opts) + do + :ok + else + {:error, reason} = e -> + :ok = Logger.error("Benchmark task failed with reason: #{inspect reason}") + e + end + end + + defp compile_project(args, _opts) do + :ok = Logger.debug("Compiling project") + Mix.Task.run("loadpaths", args) + Mix.Project.compile(args) + :ok + end + + defp compile_benchmarks(files, opts) do + with {:ok, files} <- find_benchmark_files(files, opts), + {:ok, modules} <- compile_benchmark_files(files, opts) + do + {:ok, modules} + else + {:error, _} = e -> e + end + end + + defp find_benchmark_files(files, _opts) do + :ok = Logger.debug("Locating benchmarks") + + files = + case files do + [] -> + [default_benchmark_path()] + + already_defined -> + already_defined + end + + files = + Enum.flat_map(files, fn file -> + if File.dir?(file) do + [file, "*.exs"] + |> Path.join() + |> Path.wildcard() + else + [file] + end + end) + + case files do + [] -> + :ok = Logger.debug("No benchmark files found") + {:error, :no_benchmark_files} + + files -> + {:ok, files} + end + end + + defp compile_benchmark_files(files, _opts) do + :ok = Logger.debug("Compiling benchmarks") + + modules = + files + |> Enum.flat_map(fn f -> + f + |> Helper.compile_file() + |> Enum.map(fn {mod, _bin} -> mod end) + end) + + case modules do + [] -> + :ok = Logger.debug("No benchmarks found") + {:error, :no_benchmarks} + + modules -> + {:ok, modules} + end + end + + defp run_benchmarks(modules, _opts) do + :ok = Logger.debug("Running benchmarks") + + modules + |> get_benchmarks() + |> run_all_benchmarks() + end + + defp get_benchmarks(modules) do + case_attr = Benchee.Project.Benchmark.benchee_scenario_attr() + + modules + |> Enum.filter(fn mod -> + Keyword.has_key?(mod.module_info(:attributes), case_attr) + end) + |> Enum.map(fn mod -> + attrs = mod.module_info(:attributes) + run_funs = Keyword.get(attrs, case_attr) + {mod, run_funs} + end) + end + + defp run_all_benchmarks(benchmarks, print_system? \\ true) + defp run_all_benchmarks([], _print?), do: :ok + + defp run_all_benchmarks([{mod, run_funs} | rest], print_system?) do + opts = + case print_system? do + true -> [] + false -> [print: [system: false]] + end + + case apply_if_exists(mod, :before_benchmark, [], {:ok, nil}) do + {:ok, state} -> + state = + Enum.reduce(run_funs, state, fn funname, state -> + apply(mod, funname, [state, opts]) + end) + + try_apply_if_exists(mod, :after_benchmark, [state]) + + other -> + :ok = + Logger.error( + "Global setup of \"#{inspect(mod)}\" " <> + "returned #{inspect(other)}" + ) + end + + run_all_benchmarks(rest, false) + end + + defp try_apply_if_exists(mod, name, args) do + case apply_if_exists(mod, name, args) do + _ -> :ok + end + end + + defp apply_if_exists(mod, name, args, default) do + case apply_if_exists(mod, name, args) do + {:ok, result} -> result + {:error, :not_exists} -> default + end + end + + defp apply_if_exists(mod, fun, args) do + arity = length(args) + + case function_exported?(mod, fun, arity) do + true -> {:ok, apply(mod, fun, args)} + false -> {:error, :not_exists} + end + end +end diff --git a/mix.exs b/mix.exs index 2a80b2c3..9250ed61 100644 --- a/mix.exs +++ b/mix.exs @@ -29,7 +29,8 @@ defmodule Benchee.Mixfile do "safe_coveralls.travis": :test ], dialyzer: [ - flags: [:unmatched_returns, :error_handling, :race_conditions, :underspecs] + flags: [:unmatched_returns, :error_handling, :race_conditions, :underspecs], + plt_add_apps: [:mix] ], name: "Benchee", source_url: "https://github.com/PragTob/benchee", diff --git a/test/benchee/benchmark_test.exs b/test/benchee/benchmark_test.exs index 996dc1bb..de16c4ff 100644 --- a/test/benchee/benchmark_test.exs +++ b/test/benchee/benchmark_test.exs @@ -98,6 +98,7 @@ defmodule Benchee.BenchmarkTest do test "prints the configuration information" do Benchmark.collect(%Suite{}, TestPrinter, TestRunner) + assert_receive :system_information assert_receive :configuration_information end diff --git a/test/benchee/configuration_test.exs b/test/benchee/configuration_test.exs index 09ba2674..568b3e9a 100644 --- a/test/benchee/configuration_test.exs +++ b/test/benchee/configuration_test.exs @@ -65,7 +65,8 @@ defmodule Benchee.ConfigurationTest do print: %{ configuration: false, fast_warning: false, - benchmarking: true + benchmarking: true, + system: true } } diff --git a/test/benchee/output/benchmark_printer_test.exs b/test/benchee/output/benchmark_printer_test.exs index 468cc8d4..b34cbf87 100644 --- a/test/benchee/output/benchmark_printer_test.exs +++ b/test/benchee/output/benchmark_printer_test.exs @@ -25,6 +25,30 @@ defmodule Benchee.Output.BenchmarkPrintertest do assert output =~ "Something" end + describe ".system_information" do + output = + capture_io(fn -> + %{ + configuration: %Configuration{ + parallel: 2, + time: 10_000, + warmup: 0, + inputs: nil + }, + # scenarios: [%Scenario{job_name: "one"}, %Scenario{job_name: "two"}], + system: @system_info + } + |> system_information + end) + + assert output =~ "Erlang 19.2" + assert output =~ "Elixir 1.4" + assert output =~ "Intel" + assert output =~ "Cores: 4" + assert output =~ "macOS" + assert output =~ "8568392814" + end + describe ".configuration_information" do test "sys information" do output = @@ -37,12 +61,6 @@ defmodule Benchee.Output.BenchmarkPrintertest do |> configuration_information end) - assert output =~ "Erlang 19.2" - assert output =~ "Elixir 1.4" - assert output =~ "Intel" - assert output =~ "Cores: 4" - assert output =~ "macOS" - assert output =~ "8568392814" assert output =~ ~r/following configuration/i assert output =~ "warmup: 0 ns" assert output =~ "time: 10 μs" @@ -113,6 +131,14 @@ defmodule Benchee.Output.BenchmarkPrintertest do end) assert output == "" + + output = + capture_io(fn -> + %{configuration: %{print: %{system: false}}} + |> system_information + end) + + assert output == "" end end diff --git a/test/support/fake_benchmark_printer.ex b/test/support/fake_benchmark_printer.ex index 1f95d8ce..c0ddcf75 100644 --- a/test/support/fake_benchmark_printer.ex +++ b/test/support/fake_benchmark_printer.ex @@ -5,6 +5,10 @@ defmodule Benchee.Test.FakeBenchmarkPrinter do send(self(), {:duplicate, name}) end + def system_information(_) do + send(self(), :system_information) + end + def configuration_information(_) do send(self(), :configuration_information) end