docs: Adds CI benchmarks

.circleci/config.yml

@@ -142,6 +142,24 @@ jobs:
       - run: "cp ./docs/api.swagger.json ../docs/apis/hydra.json"
       - run: "(cd ../docs && git add -A && git commit -a -m \"Updates ORY Hydra Swagger definitions\" && git push origin) || exit 0"
 
+  benchmark:
+    docker:
+      - image: circleci/golang:1.10
+    working_directory: /go/src/github.com/ory/hydra
+    steps:
+      - checkout
+      - run: git config --global user.email "hi@ory.am"
+      - run: git config --global user.name "ORY Continuous Integration"
+      - run: curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
+      - run: go get -u github.com/rakyll/hey
+      - run: dep ensure -vendor-only
+      - run: go install .
+      - run: ./scripts/run-bench.sh
+      - run: git remote rm origin
+      - run: git remote add origin https://arekkas:$GITHUB_TOKEN@github.com/ory/hydra.git
+      - run: git fetch origin
+      - run: "if [[ $(git log --oneline -n 1) != *\"Updates benchmarks\"* ]]; then (git commit -m \"Updates benchmarks\" -- BENCHMARKS.md && git push -u origin $CIRCLE_BRANCH); fi"
+
 workflows:
   version: 2
   "test, build, push, and deploy":
@@ -158,6 +176,10 @@ workflows:
           filters:
             branches:
               only: master
+      - benchmark:
+          filters:
+            branches:
+              only: test-bench
       - swagger:
           filters:
             tags:

BENCHMARKS.md

@@ -0,0 +1,244 @@
+# Benchmarks
+
+In this document you will find benchmark results for different endpoints of ORY Hydra. All benchmarks are executed
+using [rakyll/hey](https://github.com/rakyll/hey). Please note that these benchmarks run against the in-memory storage
+adapter of ORY Hydra. These benchmarks represent what performance you would get with a zero-overhead database implementation.
+
+We do not include benchmarks against databases (e.g. MySQL or PostgreSQL) as the performance greatly differs between
+deployments (e.g. request latency, database configuration) and tweaking individual things may greatly improve performance.
+We believe, for that reason, that benchmark results for these database adapters are difficult to generalize and potentially
+deceiving. They are thus not included.
+
+This file is updated on every push to master. It thus represents the benchmark data for the latest version.
+
+All benchmarks run 10.000 requests in total, with 100 concurrent requests. All benchmarks run on Circle-CI with a
+["2 CPU cores and 4GB RAM"](https://support.circleci.com/hc/en-us/articles/360000489307-Why-do-my-tests-take-longer-to-run-on-CircleCI-than-locally-)
+configuration.
+
+## BCrypt
+
+ORY Hydra uses BCrypt to obfuscate secrets of OAuth 2.0 Clients. When using flows such as the OAuth 2.0 Client Credentials
+Grant, ORY Hydra validates the client credentials using BCrypt which causes (by design) CPU load. CPU load and performance
+depend on the BCrypt cost which can be set using the environment variable `BCRYPT_COST`. For these benchmarks,
+we have set `BCRYPT_COST=8`.
+
+## OAuth 2.0
+
+This section contains various benchmarks against OAuth 2.0 endpoints
+
+### Token Introspection
+
+```
+
+Summary:
+  Total:	1.1424 secs
+  Slowest:	0.0633 secs
+  Fastest:	0.0002 secs
+  Average:	0.0110 secs
+  Requests/sec:	8753.1552
+
+  Total data:	1550000 bytes
+  Size/request:	155 bytes
+
+Response time histogram:
+  0.000 [1]	|
+  0.007 [2565]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
+  0.013 [3521]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
+  0.019 [3182]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
+  0.025 [446]	|■■■■■
+  0.032 [147]	|■■
+  0.038 [77]	|■
+  0.044 [41]	|
+  0.051 [16]	|
+  0.057 [2]	|
+  0.063 [2]	|
+
+
+Latency distribution:
+  10% in 0.0016 secs
+  25% in 0.0064 secs
+  50% in 0.0119 secs
+  75% in 0.0140 secs
+  90% in 0.0177 secs
+  95% in 0.0214 secs
+  99% in 0.0351 secs
+
+Details (average, fastest, slowest):
+  DNS+dialup:	0.0000 secs, 0.0002 secs, 0.0633 secs
+  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0122 secs
+  req write:	0.0001 secs, 0.0000 secs, 0.0218 secs
+  resp wait:	0.0106 secs, 0.0002 secs, 0.0543 secs
+  resp read:	0.0002 secs, 0.0000 secs, 0.0235 secs
+
+Status code distribution:
+  [200]	10000 responses
+
+
+
+```
+
+### Client Credentials Grant
+
+This endpoint uses [BCrypt](#bcrypt).
+
+```
+
+Summary:
+  Total:	24.3045 secs
+  Slowest:	0.8962 secs
+  Fastest:	0.0205 secs
+  Average:	0.2366 secs
+  Requests/sec:	411.4469
+
+  Total data:	1570000 bytes
+  Size/request:	157 bytes
+
+Response time histogram:
+  0.020 [1]	|
+  0.108 [941]	|■■■■■■■■■■
+  0.196 [2380]	|■■■■■■■■■■■■■■■■■■■■■■■■■■
+  0.283 [3637]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
+  0.371 [1893]	|■■■■■■■■■■■■■■■■■■■■■
+  0.458 [752]	|■■■■■■■■
+  0.546 [261]	|■■■
+  0.634 [97]	|■
+  0.721 [23]	|
+  0.809 [12]	|
+  0.896 [3]	|
+
+
+Latency distribution:
+  10% in 0.1096 secs
+  25% in 0.1840 secs
+  50% in 0.2115 secs
+  75% in 0.2921 secs
+  90% in 0.3829 secs
+  95% in 0.4145 secs
+  99% in 0.5893 secs
+
+Details (average, fastest, slowest):
+  DNS+dialup:	0.0001 secs, 0.0205 secs, 0.8962 secs
+  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0139 secs
+  req write:	0.0001 secs, 0.0000 secs, 0.0742 secs
+  resp wait:	0.2360 secs, 0.0204 secs, 0.8961 secs
+  resp read:	0.0002 secs, 0.0000 secs, 0.0807 secs
+
+Status code distribution:
+  [200]	10000 responses
+
+
+
+```
+
+## OAuth 2.0 Client Management
+
+### Creating OAuth 2.0 Clients
+
+This endpoint uses [BCrypt](#bcrypt) and generates IDs and secrets by reading from  which negatively impacts
+performance. Performance will be better if IDs and secrets are set in the request as opposed to generated by ORY Hydra.
+
+```
+This test is currently disabled due to issues with /dev/urandom being inaccessible in the CI.
+```
+
+### Listing OAuth 2.0 Clients
+
+```
+
+Summary:
+  Total:	0.6554 secs
+  Slowest:	0.0331 secs
+  Fastest:	0.0002 secs
+  Average:	0.0064 secs
+  Requests/sec:	15257.4657
+
+  Total data:	2670000 bytes
+  Size/request:	267 bytes
+
+Response time histogram:
+  0.000 [1]	|
+  0.003 [2703]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
+  0.007 [2650]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
+  0.010 [3599]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
+  0.013 [626]	|■■■■■■■
+  0.017 [101]	|■
+  0.020 [77]	|■
+  0.023 [111]	|■
+  0.027 [94]	|■
+  0.030 [28]	|
+  0.033 [10]	|
+
+
+Latency distribution:
+  10% in 0.0005 secs
+  25% in 0.0029 secs
+  50% in 0.0062 secs
+  75% in 0.0087 secs
+  90% in 0.0102 secs
+  95% in 0.0123 secs
+  99% in 0.0241 secs
+
+Details (average, fastest, slowest):
+  DNS+dialup:	0.0001 secs, 0.0002 secs, 0.0331 secs
+  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0067 secs
+  req write:	0.0001 secs, 0.0000 secs, 0.0182 secs
+  resp wait:	0.0047 secs, 0.0001 secs, 0.0315 secs
+  resp read:	0.0008 secs, 0.0000 secs, 0.0205 secs
+
+Status code distribution:
+  [200]	10000 responses
+
+
+
+```
+
+### Fetching a specific OAuth 2.0 Client
+
+```
+
+Summary:
+  Total:	0.5133 secs
+  Slowest:	0.0345 secs
+  Fastest:	0.0002 secs
+  Average:	0.0050 secs
+  Requests/sec:	19480.9563
+
+  Total data:	2650000 bytes
+  Size/request:	265 bytes
+
+Response time histogram:
+  0.000 [1]	|
+  0.004 [3452]	|■■■■■■■■■■■■■■■■■■■■■■■■■■
+  0.007 [5398]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
+  0.010 [597]	|■■■■
+  0.014 [76]	|■
+  0.017 [112]	|■
+  0.021 [198]	|■
+  0.024 [44]	|
+  0.028 [39]	|
+  0.031 [78]	|■
+  0.034 [5]	|
+
+
+Latency distribution:
+  10% in 0.0007 secs
+  25% in 0.0026 secs
+  50% in 0.0046 secs
+  75% in 0.0055 secs
+  90% in 0.0075 secs
+  95% in 0.0125 secs
+  99% in 0.0268 secs
+
+Details (average, fastest, slowest):
+  DNS+dialup:	0.0000 secs, 0.0002 secs, 0.0345 secs
+  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0097 secs
+  req write:	0.0001 secs, 0.0000 secs, 0.0162 secs
+  resp wait:	0.0007 secs, 0.0001 secs, 0.0252 secs
+  resp read:	0.0021 secs, 0.0000 secs, 0.0290 secs
+
+Status code distribution:
+  [200]	10000 responses
+
+
+
+```

README.md

@@ -52,6 +52,7 @@ able to securely manage JSON Web Keys.
   - [ORY Keto: Access Control Policies as a Server](#ory-keto-access-control-policies-as-a-server)
 - [Security](#security)
   - [Disclosing vulnerabilities](#disclosing-vulnerabilities)
+- [Benchmarks](#benchmarks)
 - [Telemetry](#telemetry)
 - [Documentation](#documentation)
   - [Guide](#guide)
@@ -220,6 +221,10 @@ Read [the security guide now](https://www.ory.sh/docs/1-hydra/0-tutorial/0-readm
 If you think you found a security vulnerability, please refrain from posting it publicly on the forums, the chat, or GitHub
 and send us an email to [hi@ory.am](mailto:hi@ory.am) instead.
 
+## Benchmarks
+
+Our continuous integration runs a collection of benchmarks against ORY Hydra. You can find the results in [./BENCHMARKS.md](BENCHMARKS.md).
+
 ## Telemetry
 
 Our services collect summarized, anonymized data which can optionally be turned off. Click