Set up Tremor benchmarking infrastructure

[anupdhml]

Describe the problem you are trying to solve

We would like to run various Tremor benchmarks periodically and use results from them to reliably measure how Tremor’s performance varies as we introduce new changes to the project.

Describe the solution you'd like

A setup that will be initially useful to catch any performance regressions that may occur between Tremor releases (this implies some storage for historical results). Over time, we should be able to evolve it to do the same on pull requests.

Ability to view historical benchmark results on-demand (eg: over a web browser) would be nice to have as well.

Notes

CNCF has provided us with a bare-metal machine for use here, via cncf/cluster#160. For folks working on this, we can help with the initial machine provisioning.

To run all the existing tremor benchmarks via the unified tremor cli, do: cargo run -p tremor-cli -- test bench tremor-cli/tests. Some old benchmarks (driven from shell-scripts) are also there in the codebase -- we can port all the useful ones from here to the newer tremor-cli based setup and clean these up.

We can also add net new benchmarks via the criterion lib:
https://github.com/bheisler/criterion.rs
https://github.com/bheisler/cargo-criterion

Also of interest, for CI use: https://github.com/bheisler/iai

[akshatagarwl]

A not so curated list of articles that might be helpful while working this:

• CI for performance: Reliable benchmarking in noisy environments
• Alternative measurements other than wall time bheisler/criterion.rs#130
• Personal Computer World Benchmark Tests
• Rust Performance: A story featuring perf and flamegraph on Linux
• https://www.brendangregg.com/activebenchmarking.html
• http://kellabyte.com/2014/02/12/create-benchmarks-and-results-that-have-value/
• Are Benchmarks From Cloud CI Services Reliable?
• https://perf.rust-lang.org/
• crate: criterion-cycles-per-byte
• crate: iai
• Criterion.rs v0.3.4 And Iai 0.1.0
• Measuring and Reducing CPU Usage in SQLite
• Building an Open Source, Continuous Benchmark System
• NixOS: FAQ/Bare Metal Deployment

[shivam-51]

Hey @humancalico , I would like to work on this issue under the LFX Mentorship Spring 2021. Can you please guide me on the next steps apart from official application?

[Licenser]

Hi @shivam-51 I'm sorry the application process is closed by now and selections already made. But keep an eye out there are likely to be more mentorship opportunities coming up :) or come by our discord and say hi if you would want to toy around with something outside of the LFX Mentorship.

[shivam-51]

Hi @shivam-51 I'm sorry the application process is closed by now and selections already made. But keep an eye out there are likely to be more mentorship opportunities coming up :)

Absolutely! I'll try again in the next cohort :)

[Licenser]

Awesome 👍 , keep an eye out on our Twitter and discord, we'll announce mentorship programs as soon as we can :)

[vielmetti]

Hi @humancalico how did this benchmarking infrastructure setup go? We have a maintenance event at Equinix Metal in our FRA2 data center which will necessitate some changes on your side, if the work is still underway.

cc @anupdhml

[Licenser]

👍 all good I suspect IP's are changing or something else?

[vielmetti]

@Licenser - Something else - data center migration, so it'll be a shiny new machine in a new data center, and you'll need to spin it up and set it up before we tear the old one down at the end of October.

[Licenser]

uhhh shiny :D, we'll migrate, that shouldn't be a big issue. Thanks for the heads up!

[vielmetti]

@Licenser Can I run communications through you? If so please expect an email with particulars, or if it's all the same to you, we can just take care of things right here.

[Licenser]

Yes absolutely, feel free to mail me any time or look into it from your side, really whatever is easier for you :) we'll adopt. I blocked out a day next week for migration just in case.

[vielmetti]

@Licenser - the suggestion is to spin up a new c3.small in our Frankfurt (FR) metro, set it up to your liking. When you're done and the new system is running, destroy the old one.

[darach]

This works great @vielmetti 👍

[Licenser]

@vielmetti the old server is gone :)