Support user-defined benchmark suites.

[ericsnowcurrently]

(I realize this is a big PR. Sorry! The changes to .py files is a smaller change. If necessary I can split it up a little, but it might not be worth it. 🙂 Just let me know.)

Currently pyperformance is two things coupled together: a tool to run a Python benchmark suite and a curated suite of Python benchmarks. This PR splits those apart, with the existing suite used as the default. This allows users to run their own set of benchmarks, perhaps specific to their Python implementation or their PyPI library, e.g. https://github.com/ericsnowcurrently/pyston-macrobenchmarks/tree/pyperformance.

Key changes:

• introduce a new filesystem structure for suites and individual benchmarks
• add the --manifest CLI option to specify the custom suite to use
• the default suite has been changed to the new format and moved to pyperformance/_benchmarks (only a data dir now)
• sometimes run some benchmarks in separate venvs
• do not fail all benchmarks if the dependencies for one cannot be installed

Most notably, this change should not affect benchmark results.

Specifying a Benchmark Suite

A benchmark suite is defined through a manifest file. For example, see the file for the default suite in pyperformance/_benchmarks/MANIFEST. A manifest file has a [benchmarks] section (and zero or more [group NAME] sections). See pyperformance/benchmarks/_manifest.py.

The [benchmarks] section is a TSV (tab-separated-values) table with 4 columns:

• name - the name of the benchmark
• version - (optional) the version of that benchmark to use
• origin - (optional) the URI where the benchmark is found (e.g. PyPI, GitHub, local filesystem)
• metafile - the location of the metadata file for the benchmark (or <local> to find "bm_" relative to the manifest file)

The New Structure of a Benchmark

Each benchmark is in its own directory and looks like this:

<benchmark>/
    pyproject.toml
    bm_<name>.toml    # alternate benchmarks based on this one
    requirements.txt  # if has external dependencies; pins versions
    run_benchmark.py  # essentially the same benchmark script as before

pyproject.toml is the normal PEP 621 format for Python projects. These are the important fields in the [project] section:

• "name" - (may be inferred from the suite manifest)
• "version" - (may be inferred from the suite manifest or base metadata)
• "dependencies" - (if has external dependencies; the top-level list of dependencies)

We add a [tool.pyperformance] section, as supported by PEP 518. These are the important fields:

• "name" - (inferred from the [project] section)
• "metabase" - if any, the pyproject.toml to inherit from (".." expands to "../base.toml"; possibly inferred)
• "extra_opts" - if any, additional CLI args to use when running the benchmark script
• "runscript" - defaults to "./run_benchmark.py"

See pyperformance/benchmark/_metadata.py.

[ericsnowcurrently]

@pablogsal, if possible could you give this a quick look. I don't necessarily need a full review of the code. More than anything I want to be sure the overall approach is acceptable. I'd be glad to hop into a call if that would help. Thanks!

[ericsnowcurrently]

FYI, after a quick chat with @pablogsal, I plan on merging this tomorrow.