SeBS: Serverless Benchmark Suite

FaaS benchmarking suite for serverless functions with automatic build, deployment, and measurements.

SeBS is a diverse suite of FaaS benchmarks that allows automatic performance analysis of commercial and open-source serverless platforms. We provide a suite of benchmark applications and experiments and use them to test and evaluate different components of FaaS systems. See the installation instructions to learn how to configure SeBS to use selected commercial and open-source serverless systems. Then, take a look at usage instructions to see how SeBS can automatically launch serverless functions and entire experiments in the cloud!

SeBS provides support for automatic deployment and invocation of benchmarks on commercial and black-box platforms AWS Lambda, Azure Functions, and Google Cloud Functions. Furthermore, we support the open-source platform OpenWhisk and offer a custom, Docker-based local evaluation platform. See the documentation on cloud providers for details on configuring each platform in SeBS. The documentation describes in detail the design and implementation of our tool, and see the modularity section to learn how SeBS can be extended with new platforms, benchmarks, and experiments. Find out more about our project in a paper summary.

Do you have further questions not answered by our documentation? Did you encounter troubles with installing and using SeBS? Or do you want to use SeBS in your work and you need new features? Feel free to reach us through GitHub issues or by writing to marcin.copik@inf.ethz.ch.

For more information on how to configure, use, and extend SeBS, see our documentation:

• How to use SeBS?
• Which benchmark applications are offered?
• Which experiments can be launched to evaluate FaaS platforms?
• How to configure serverless platforms?
• How SeBS builds and deploys functions?
• How SeBS package is designed?
• How to extend SeBS with new benchmarks, experiments, and platforms?

Publication

When using SeBS, please cite our Middleware '21 paper. An extended version of our paper is available on arXiv, and you can find more details about research work in this paper summary. You can cite our software repository as well, using the citation button on the right.

@inproceedings{copik2021sebs,
  author = {Copik, Marcin and Kwasniewski, Grzegorz and Besta, Maciej and Podstawski, Michal and Hoefler, Torsten},
  title = {SeBS: A Serverless Benchmark Suite for Function-as-a-Service Computing},
  year = {2021},
  isbn = {9781450385343},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3464298.3476133},
  doi = {10.1145/3464298.3476133},
  booktitle = {Proceedings of the 22nd International Middleware Conference},
  pages = {64–78},
  numpages = {15},
  keywords = {benchmark, serverless, FaaS, function-as-a-service},
  location = {Qu\'{e}bec city, Canada},
  series = {Middleware '21}
}

Installation

Requirements:

• Docker (at least 19)
• Python 3.7+ with:
  • pip
  • venv
• libcurl and its headers must be available on your system to install pycurl
• Standard Linux tools and zip installed

... and that should be all. We currently support Linux and other POSIX systems with Bash available. On Windows, we recommend using WSL.

To install the benchmarks with a support for all platforms, use:

./install.py --aws --azure --gcp --openwhisk --local

It will create a virtual environment in python-venv, and install necessary Python dependencies and third-party dependencies. To use SeBS, you must first active the new Python virtual environment:

. python-venv/bin/activate

Now you can deploy serverless experiments :-)

The installation of additional platforms is controlled with the --{platform} and --no-{platform} switches. Currently, the default behavior for install.py is to install only the local environment.

To verify the correctness of installation, you can use our regression testing.

Warning

We use libcurl to make HTTP requests. pycurl will attempt to build its bindings and needs headers for that - make sure you have all development packages installed. If you see an error like this one: src/pycurl.h:206:13: fatal error: gnutls/gnutls.h: No such file or directory, it means that you are missing some of the dependencies.

Warning

Make sure that your Docker daemon is running and your user has sufficient permissions to use it. Otherwise, you might see many "Connection refused" and "Permission denied" errors when using SeBS.

Authors

• Marcin Copik (ETH Zurich) - main author.
• Michał Podstawski (Future Processing SA) - contributed graph and DNA benchmarks, and worked on Google Cloud support.
• Laurin Brandner (ETH Zurich) - contributed serverless workflows.
• Nico Graf (ETH Zurich) - contributed to the implementation of regression tests and bugfixes and helped with testing and documentation.
• Kacper Janda, Mateusz Knapik, JmmCz, AGH University of Science and Technology - contributed together Google Cloud support.
• Grzegorz Kwaśniewski (ETH Zurich) - worked on the modeling experiments.
• Paweł Żuk (University of Warsaw) - contributed OpenWhisk support.
• Sascha Kehrli (ETH Zurich) - contributed local measurement of Docker containers.
• Kaleab - contributed to SeBS local backend to make it portable between platforms and more robust on non-Linux systems.
• lawrence910426 - contributed color-coded output to SeBS CLI.
• Abhishek Kumar - contributed new language versions and Knative support.
• Prajin Khadka - contributed new language versions, container support, and ARM builds.