Rework benchmarks

This commit reworks our benchmark suite:

* Use criterion instead of std-lib bencher
* Benchmark more different sizes
* Add benchmarks for other rust db-crates as comparision
* Add a new insert benchmark

.github/workflows/benches.yml

@@ -0,0 +1,58 @@
+on:
+  pull_request:
+    types: [opened]
+  issue_comment:
+    types: [created]
+
+
+jobs:
+  benchmarks:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        backend: ["postgres", "sqlite", "mysql"]
+    steps:
+      - uses: khan/pull-request-comment-trigger@master
+        id: bench
+        with:
+          trigger: ':diesel run benches'
+
+      - name: Checkout sources
+        if: steps.bench.outputs.triggered == 'true'
+        uses: actions/checkout@v2
+
+      - name: Install postgres (Linux)
+        if: steps.bench.outputs.triggered == 'true' && matrix.backend == 'postgres'
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libpq-dev postgresql
+          echo "host    all             all             127.0.0.1/32            md5" > sudo tee -a /etc/postgresql/10/main/pg_hba.conf
+          sudo service postgresql restart && sleep 3
+          sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'postgres';"
+          sudo service postgresql restart && sleep 3
+          echo '::set-env name=PG_DATABASE_URL::postgres://postgres:postgres@localhost/'
+
+      - name: Install sqlite (Linux)
+        if: steps.bench.outputs.triggered == 'true' && matrix.backend == 'sqlite'
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libsqlite3-dev
+          echo '::set-env name=SQLITE_DATABASE_URL::/tmp/test.db'
+
+        
+      - name: Install mysql (Linux)
+        if: steps.bench.outputs.triggered == 'true' && matrix.backend == 'mysql'
+        run: |
+          sudo apt-get update
+          sudo apt-get -y install mysql-server libmysqlclient-dev
+          sudo /etc/init.d/mysql start
+          mysql -e "create database diesel_test; create database diesel_unit_test; grant all on \`diesel_%\`.* to 'root'@'localhost';" -uroot -proot
+          echo '::set-env name=MYSQL_DATABASE_URL::mysql://root:root@localhost/diesel_test'
+
+      - name: Run benches
+        if: steps.bench.outputs.triggered == 'true'
+        uses: jasonwilliams/criterion-compare-action@move_to_actions
+        with:
+          cwd: "diesel_bench"
+          token: ${{ secrets.GITHUB_TOKEN }}
+

.github/workflows/ci.yml

@@ -253,6 +253,12 @@ jobs:
           command: test
           args: --manifest-path diesel_tests/Cargo.toml --no-default-features --features "${{ matrix.backend }}"
 
+      - name: Run diesel_benches
+        uses: actions-rs/cargo@v1
+        with:
+          command: test
+          args: --manifest-path diesel_bench/Cargo.toml --no-default-features --features "${{ matrix.backend }}" --bench benchmarks
+
       - name: Run rustdoc (nightly)
         if: matrix.run == 'nightly'
         uses: actions-rs/cargo@v1

diesel_bench/Cargo.toml

@@ -0,0 +1,53 @@
+[package]
+name = "diesel_bench"
+version = "0.1.0"
+authors = []
+edition = "2018"
+build = "build.rs"
+autobenches = false
+
+[workspace]
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+dotenv = "0.15"
+criterion = "=0.3.2" # critcmp only supports 0.3.2
+sqlx = {version = "0.4.0-beta.1", optional = true}
+async-std = { version = "1.5", optional = true}
+rusqlite = {version = "0.23", optional = true}
+rust_postgres = {version = "0.17", optional = true, package = "postgres"}
+rust_mysql = {version = "18.2", optional = true, package = "mysql"}
+rustorm = {version = "0.17", optional = true}
+rustorm_dao = {version = "0.5", optional = true}
+quaint = {version = "0.2.0-alpha.12", optional = true}
+tokio = {version = "0.2", optional = true}
+serde = {version = "1", optional = true, features = ["derive"]}
+
+[dependencies.diesel]
+path = "../diesel"
+default-features = false
+features = []
+
+[build-dependencies]
+diesel = { path = "../diesel", default-features = false }
+diesel_migrations = { path = "../diesel_migrations" }
+dotenv = "0.15"
+
+
+[[bench]]
+name = "benchmarks"
+path = "benches/lib.rs"
+bench = true
+harness = false
+
+[features]
+default = []
+postgres = ["diesel/postgres"]
+sqlite = ["diesel/sqlite"]
+mysql = ["diesel/mysql"]
+
+[patch.crates-io]
+rustorm = {git = "https://github.com/weiznich/rustorm"}
+rustorm_dao = {git = "https://github.com/weiznich/rustorm"}
+quaint = {git = "https://github.com/weiznich/quaint"}

diesel_bench/Readme.md

@@ -0,0 +1,116 @@
+# A benchmark suite for relational database connection crates in rust
+
+This directory contains a basic benchmark suite that allows to compare different crates that can be used to connect to relational database systems. Those benchmarks are created with the following goals:
+
+a) To track diesels performance and find potential regressions
+b) To evaluate potential alternatives for the currently used C-dependencies
+c) To compare diesel with the competing crates
+
+It currently supports the following database systems:
+
+* PostgreSQL
+* MySQL/MariaDB
+* SQLite
+
+and the following crates:
+
+* Diesel
+* [SQLx](https://github.com/launchbadge/sqlx)
+* [Rustorm](https://github.com/ivanceras/rustorm)
+* [Quaint](https://github.com/prisma/quaint)
+* [Postgres](https://github.com/sfackler/rust-postgres)
+* [Rusqlite](https://github.com/rusqlite/rusqlite)
+* [Mysql](https://github.com/blackbeam/rust-mysql-simple)
+
+By default only diesels own benchmarks are executed. To run the benchmark do the following:
+
+```sh
+$ DATABASE_URL=your://database/url diesel migration run --migration-dir ../migrations/$backend
+$ DATABASE_URL=your://database/url cargo bench --features "$backend"
+```
+
+To enable other crates add the following features:
+
+* `SQLx: ` `sqlx sqlx/$backend async-std`
+* `Rustorm`: `rustorm rustorm/with-$backend rustorm_dao`
+* `Quaint`: `quaint quaint/single-$backend tokio quaint/serde-support serde`
+* `Postgres`: `rust-postgres`
+* `Rusqlite`: `rusqlite`
+* `Mysql`: `rust-mysql`
+
+## Benchmarks
+
+### Common data structures
+
+#### Table definitions
+
+The following schema definition was used. (For Mysql/Sqlite postgres specific types where replaced by their equivalent type).
+```sql
+CREATE TABLE users (
+  id SERIAL PRIMARY KEY,
+  name VARCHAR NOT NULL,
+  hair_color VARCHAR
+);
+
+CREATE TABLE posts (
+  id SERIAL PRIMARY KEY,
+  user_id INTEGER NOT NULL,
+  title VARCHAR NOT NULL,
+  body TEXT
+);
+
+CREATE TABLE comments (
+  id SERIAL PRIMARY KEY,
+  post_id INTEGER NOT NULL,
+  text TEXT NOT NULL
+);
+
+```
+
+
+
+
+#### Struct definitions
+
+```rust
+pub struct User {
+    pub id: i32,
+    pub name: String,
+    pub hair_color: Option<String>,
+}
+
+pub struct Post {
+    pub id: i32,
+    pub user_id: i32,
+    pub title: String,
+    pub body: Option<String>,
+}
+
+pub struct Comment {
+    id: i32,
+    post_id: i32,
+    text: String,
+}
+```
+
+Field types are allowed to differ, to whatever type is expected compatible by the corresponding crate.
+
+### `bench_trivial_query`
+
+
+This benchmark tests how entities from a single table are loaded. Before starting the benchmark 1, 10, 100, 1000 or 10000 entries are inserted into the `users` table. For this the `id` of the user is provided by the autoincrementing id, the `name` is set to `User {id}` and the `hair_color` is set to `Null`. An implementation of this benchmark is expected to return a list of the type `User.
+
+### `bench_medium_complex_query`
+
+This benchmark tests how entities from more than one table are loaded. Before starting the benchmark 1, 10, 1000, 10000 entries are inserted into the `users` table.  For this the `id` of the user is provided by the autoincrementing id, the `name` is set to `User {id}` and the `hair_color` is set to `"black"` for even id's, to `"brown"` otherwise. An implementation of this benchmark is expected to return a list of the type `(User, Option<Post>)` so that matching pairs of `User` and `Option<Post>` are returned. Though the `posts` table is empty the corresponding implementation needs to query both tables. 
+
+### `bench_insert`
+
+This benchmark tests how fast entities are inserted into the database. For this each implementation gets a size how many entries are scheduled to be inserted into the database. An implementation of this benchmark is expected to insert as many entries into the user table as the number provided by the benchmark framework. It is not required to clean up the already inserted entries at any time.  Newly inserted users are generated using the following rules: `id` of the user is provided by the autoincrementing id, the `name` is set to `User {batch_id}` and the `hair_color` is set to `"hair_color"`.
+
+### `bench_loading_associations_sequentially`
+
+This benchmark tests how fast a complex set of entities is received from the database. 
+Before starting the benchmark a large amount of data needs to be inserted into the database. The `users` table is required to contain 100 entries (or for sqlite 9 entries) based on the following rules: `id` is determined by the autoincrementing column, `name` is set to `User {batch_id}` and `hair_color` is set to `"black"` for even id's, otherwise to `"brown"`. For each entry in the `users` table, 10 entries in the `posts` table need to exist. Each entry in the `posts` table is based on the following rules: `id` is autogenerated by autoincrementing column, `title` is set to `Post {post_batch_id} for user {user_id}` where `post_batch_id` referees to number of the current post in relation to the user (so between 0 and 9), `user_id` is set to the corresponding user's id, and `body` is set to `NULL`. For each entry in the `posts` table 10 entries in the `comments` table are generated based on the following rules:
+`id` is set to the autoincrementing default value, `text` is set to `Comment {comment_batch_id} on post {post_id}` where `comment_batch_id` referees to the number of the current comment in relation to the post (so between 0 and 9), `post_id` is set tho the corresponding posts's id. 
+An implementation of the benchmark is expected to return a list of the type `(User, Vec<(Post, Vec<Comment>)>)`, that contains all users wit all corresponding comments and posts grouped by their corresponding associations.