Docker Container

The Docker container can be pulled from the Docker Hub or built locally. It contains all benchmarks and dependencies and exposes the benchmark server via port 50051.

We give an exemplary usage of the Docker container in the bencherclient repository.

pwd # /path/to/bencher
docker build -t bencher .
# always keep the container running, can be stopped with docker stop <container-id>
docker run -p 50051:50051 --restart always -d bencher:latest

or

docker pull gaunab/bencher:latest
# always keep the container running, can be stopped with docker stop <container-id>
docker run -p 50051:50051 --restart always -d gaunab/bencher:latest

Apptainer / Singularity Container

You can build an Apptainer container from the Docker image:

Bootstrap: docker
From: gaunab/bencher:latest
Namespace:
Stage: build

%environment
    export LANG=C.UTF-8
    export PATH="/root/.local/bin:$PATH"

%post
    cd /opt
    git clone your-repo
    cd your-repo
    pip install bencherscaffold # you'll need bencherscaffold to call bencher
    pip install your-dependencies

%startscript
    bash -c "/docker-entrypoint.sh"

%runscript
    bash -c "your-command-to-run-your-app"

This will create an Apptainer container with the Docker image gaunab/bencher:latest and the repository your-repo with the dependencies your-dependencies installed.

Usage

Starting the instance

apptainer build container.sif your-apptainer-file

Start the Apptainer instance

This starts all the benchmarks in the container (as defined in the startscript of the Apptainer file).

apptainer instance start container.sif your-instance-name

Run your command that depends on the benchmarks

This runs your command in the instance your-instance-name as defined in the runscript of the Apptainer file.

apptainer run instance://your-instance-name

Evaluating a benchmark

We show how to run all benchmarks in the bencherclient repository. You don't need to use this repository, it is mainly used to test the benchmarks. The general setup to evaluate a benchmark is as follows. First, install the bencherscaffold package:

pip install git+https://github.com/LeoIV/BencherScaffold

Then, you can use the following code to evaluate a benchmark:

from bencherscaffold.client import BencherClient
from bencherscaffold.protoclasses.bencher_pb2 import Value, ValueType

# Create a client to communicate with the Bencher server
# By default, it connects to 127.0.0.1:50051
client = BencherClient()

# Create a list of values to evaluate
values = [Value(type=ValueType.CONTINUOUS, value=0.5) for _ in range(180)]
# The benchmark name is the name of the benchmark you want to evaluate
benchmark_name = 'lasso-dna'

# Evaluate the benchmark with the given values
# This will send the values to the server and return the result
# If the server is not running, it will raise an error
result = client.evaluate_point(
    benchmark_name=benchmark_name,
    point=values,
)
print(f"Result: {result}")

Available Benchmarks

The urban mobility benchmarks (1ramp_*, 2corridor_*, etc.) follow a templated naming convention: BASE-NAME_DATE_HOUR_EVAL-TYPE ¹.

• BASE-NAME: Defines the traffic scenario (1ramp, 2corridor, 3junction, 4smallRegion, 5fullRegion).
• DATE: A date in yymmdd format, from 221008 to 221021.
• HOUR: The time of day (06-07, 08-09, or 17-18).
• EVAL-TYPE: The evaluation metric (count or speed).

For example, a valid benchmark name is 1ramp_221008_08-09_count.

The following benchmarks are available:

Benchmark Name	# Dimensions	Type	Source(s)	Noisy
lasso-dna	180	continuous	2,3	☒
lasso-simple	60	continuous	2	☒
lasso-medium	100	continuous	2	☒
lasso-high	300	continuous	2,3	☒
lasso-hard	1000	continuous	2,3	☒
lasso-leukemia	7129	continuous	2	☒
lasso-rcv1	47236	continuous	2,4	☒
lasso-diabetes	8	continuous	2	☒
lasso-breastcancer	10	continuous	2	☒
mopta08	124	continuous	5,3	☒
maxsat60	60	binary	6,7	☒
maxsat125	125	binary	7	☒
robotpushing	14	continuous	8	☑
lunarlander	12	continuous	8	☑
rover	60	continuous	8	☒
mujoco-ant	888	continuous	9,3	☑
mujoco-hopper	33	continuous	9,3	☑
mujoco-walker	102	continuous	9,3	☑
mujoco-halfcheetah	102	continuous	9,3	☑
mujoco-swimmer	16	continuous	9,3	☑
mujoco-humanoid	6392	continuous	9,3	☑
svm	388	continuous	5,3,10	☒
svmmixed	53	mixed	6,7	☒
1ramp_*	3	integer	1	☒
2corridor_*	21	integer	1	☒
3junction_*	44	integer	1	☒
4smallRegion_*	151	integer	1	☒
5fullRegion_*	10100	integer	1	☒
pestcontrol	25	categorical	11,12	☒
bbob-sphere	any	continuous	13,14	☒
bbob-ellipsoid	any	continuous	13,14	☒
bbob-rastrigin	any	continuous	13,14	☒
bbob-buecherastrigin	any	continuous	13,14	☒
bbob-linearslope	any	continuous	13,14	☒
bbob-attractivesector	any	continuous	13,14	☒
bbob-stepellipsoid	any	continuous	13,14	☒
bbob-rosenbrock	any	continuous	13,14	☒
bbob-rosenbrockrotated	any	continuous	13,14	☒
bbob-ellipsoidrotated	any	continuous	13,14	☒
bbob-discus	any	continuous	13,14	☒
bbob-bentcigar	any	continuous	13,14	☒
bbob-sharpridge	any	continuous	13,14	☒
bbob-differentpowers	any	continuous	13,14	☒
bbob-rastriginrotated	any	continuous	13,14	☒
bbob-weierstrass	any	continuous	13,14	☒
bbob-schaffers10	any	continuous	13,14	☒
bbob-schaffers1000	any	continuous	13,14	☒
bbob-griewankrosenbrock	any	continuous	13,14	☒
bbob-schwefel	any	continuous	13,14	☒
bbob-gallagher101	any	continuous	13,14	☒
bbob-gallagher21	any	continuous	13,14	☒
bbob-katsuura	any	continuous	13,14	☒
bbob-lunacekbirastrigin	any	continuous	13,14	☒
pbo-onemax	any	binary	13	☒
pbo-leadingones	any	binary	13	☒
pbo-linear	any	binary	13	☒
pbo-onemaxdummy1	any	binary	13	☒
pbo-onemaxdummy2	any	binary	13	☒
pbo-onemaxneutrality	any	binary	13	☒
pbo-onemaxepistasis	any	binary	13	☒
pbo-onemaxruggedness1	any	binary	13	☒
pbo-onemaxruggedness2	any	binary	13	☒
pbo-onemaxruggedness3	any	binary	13	☒
pbo-leadingonesdummy1	any	binary	13	☒
pbo-leadingonesdummy2	any	binary	13	☒
pbo-leadingonesneutrality	any	binary	13	☒
pbo-leadingonesepistasis	any	binary	13	☒
pbo-leadingonesruggedness1	any	binary	13	☒
pbo-leadingonesruggedness2	any	binary	13	☒
pbo-leadingonesruggedness3	any	binary	13	☒
pbo-labs	any	binary	13	☒
pbo-isingring	any	binary	13	☒
pbo-isingtorus	any	binary	13	☒
pbo-isingtriangular	any	binary	13	☒
pbo-mis	any	binary	13	☒
pbo-nqueens	any	binary	13	☒
pbo-concatenatedtrap	any	binary	13	☒
pbo-nklandscapes	any	binary	13	☒
graph-maxcut2000	800	binary	13	☒
graph-maxcut2001	800	binary	13	☒
graph-maxcut2002	800	binary	13	☒
graph-maxcut2003	800	binary	13	☒
graph-maxcut2004	800	binary	13	☒
graph-maxcoverage2100	800	binary	13	☒
graph-maxcoverage2101	800	binary	13	☒

Citation

If you use this repository or the benchmarks in your research, please cite the following paper:

@misc{papenmeier2025bencher,
      title={Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization}, 
      author={Leonard Papenmeier and Luigi Nardi},
      year={2025},
      eprint={2505.21321},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2505.21321}, 
}

Building on MacOS (under development)

brew install swig gfortran openblas pkg-config glfw libomp

To allow the build tools to find OpenBLAS, you must run: brew info openblas | grep PKG_CONFIG_PATH

and set the PKG_CONFIG_PATH environment variable accordingly, e.g.: export PKG_CONFIG_PATH="/opt/homebrew/opt/openblas/lib/pkgconfig"

Mujoco

Download https://github.com/google-deepmind/mujoco/releases/download/2.1.1/mujoco-2.1.1-macos-universal2.dmg and mount it.

Then, copy the dynamic library and headers to ~/.mujoco/mujoco210/:

mkdir -p ~/.mujoco/mujoco210/bin
cp /Volumes/MuJoCo/MuJoCo.framework/Versions/Current/libmujoco.2.1.1.dylib ~/.mujoco/mujoco210/bin/
ln -sf ~/.mujoco/mujoco210/bin/libmujoco.2.1.1.dylib ~/.mujoco/mujoco210/bin/libmujoco.dylib
mkdir -p ~/.mujoco/mujoco210/bin/MuJoCo.framework/Versions/A/
ln -s ~/.mujoco/mujoco210/bin/libmujoco.2.1.1.dylib ~/.mujoco/mujoco210/bin/MuJoCo.framework/Versions/A/libmujoco.2.1.1.dylib
cp -r /Volumes/MuJoCo/MuJoCo.framework/Versions/Current/Headers ~/.mujoco/mujoco210/include

You probably have to allow access to the library in the Security & Privacy settings.

export CC=/opt/homebrew/opt/llvm/bin/clang 

Toubleshooting

One main problem during the compilation occurs if you use a x86_64 Python on an ARM Mac.

Footnotes

1. Ryu, Seunghee, et al. "BO4Mob: Bayesian Optimization Benchmarks for High-Dimensional Urban Mobility Problem." arXiv preprint arXiv:2510.18824 (2025). For 1ramp, values should be integers between 1 and 2500. For the other scenarios, values should be integers between 1 and 2000. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

2. LassoBench ( Šehić Kenan, Gramfort Alexandre, Salmon Joseph and Nardi Luigi, "LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso", AutoML conference, 2022.) ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹

3. BAxUS Leonard Papenmeier, Luigi Nardi, and Matthias Poloczek, "Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces", NeurIPS 2022 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹

4. The LassoBench paper states 19,959 features, but the number of features in the RCV1 dataset is 47,236. ↩

5. SAASBO David Eriksson and Martin Jankowiak, "High-dimensional Bayesian optimization with sparse axis-aligned subspaces", UAI 2021 ↩ ↩²

6. BODi Aryan Deshwal, Sebastian Ament, Maximilian Balandat, Eytan Bakshy, Janardhan Rao Doppa, and David Eriksson, "Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddings", AISTATS 2023 ↩ ↩²

7. Bounce Leonard Papenmeier, Luigi Nardi and Matthias Poloczek, "Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spaces", NeurIPS 2023 ↩ ↩² ↩³

8. TurBO ( David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner and Matthias Poloczek, "Scalable Global Optimization via Local Bayesian Optimization." NeurIPS 2019) ↩ ↩² ↩³

9. LA-MCTS Linnan Wang, Rodrigo Fonseca, and Yuandong Tian, "Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search", NeurIPS 2020 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

10. The SVM benchmark is not included in the repository and was obtained by corresponding with the authors of the paper. ↩

11. Oh, Changyong, et al. "Combinatorial bayesian optimization using the graph cartesian product." Advances in Neural Information Processing Systems 32 (2019). ↩

12. Each category has 5 possible values. The benchmark expects an integer between 0 and 4 for each category. ↩

13. de Nobel, Jacob, et al. "Iohexperimenter: Benchmarking platform for iterative optimization heuristics." Evolutionary Computation 32.3 (2024): 205-210. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰ ↩²¹ ↩²² ↩²³ ↩²⁴ ↩²⁵ ↩²⁶ ↩²⁷ ↩²⁸ ↩²⁹ ↩³⁰ ↩³¹ ↩³² ↩³³ ↩³⁴ ↩³⁵ ↩³⁶ ↩³⁷ ↩³⁸ ↩³⁹ ↩⁴⁰ ↩⁴¹ ↩⁴² ↩⁴³ ↩⁴⁴ ↩⁴⁵ ↩⁴⁶ ↩⁴⁷ ↩⁴⁸ ↩⁴⁹ ↩⁵⁰ ↩⁵¹ ↩⁵² ↩⁵³ ↩⁵⁴ ↩⁵⁵ ↩⁵⁶

14. Hansen, Nikolaus, et al. "COCO: A platform for comparing continuous optimizers in a black-box setting." Optimization Methods and Software 36.1 (2021): 114-144. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰ ↩²¹ ↩²² ↩²³ ↩²⁴