-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: Adds auto-generated benchmarks (#897)
- Loading branch information
Showing
4 changed files
with
411 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,244 @@ | ||
| # Benchmarks | ||
|
|
||
| In this document you will find benchmark results for different endpoints of ORY Hydra. All benchmarks are executed | ||
| using [rakyll/hey](https://github.com/rakyll/hey). Please note that these benchmarks run against the in-memory storage | ||
| adapter of ORY Hydra. These benchmarks represent what performance you would get with a zero-overhead database implementation. | ||
|
|
||
| We do not include benchmarks against databases (e.g. MySQL or PostgreSQL) as the performance greatly differs between | ||
| deployments (e.g. request latency, database configuration) and tweaking individual things may greatly improve performance. | ||
| We believe, for that reason, that benchmark results for these database adapters are difficult to generalize and potentially | ||
| deceiving. They are thus not included. | ||
|
|
||
| This file is updated on every push to master. It thus represents the benchmark data for the latest version. | ||
|
|
||
| All benchmarks run 10.000 requests in total, with 100 concurrent requests. All benchmarks run on Circle-CI with a | ||
| ["2 CPU cores and 4GB RAM"](https://support.circleci.com/hc/en-us/articles/360000489307-Why-do-my-tests-take-longer-to-run-on-CircleCI-than-locally-) | ||
| configuration. | ||
|
|
||
| ## BCrypt | ||
|
|
||
| ORY Hydra uses BCrypt to obfuscate secrets of OAuth 2.0 Clients. When using flows such as the OAuth 2.0 Client Credentials | ||
| Grant, ORY Hydra validates the client credentials using BCrypt which causes (by design) CPU load. CPU load and performance | ||
| depend on the BCrypt cost which can be set using the environment variable `BCRYPT_COST`. For these benchmarks, | ||
| we have set `BCRYPT_COST=8`. | ||
|
|
||
| ## OAuth 2.0 | ||
|
|
||
| This section contains various benchmarks against OAuth 2.0 endpoints | ||
|
|
||
| ### Token Introspection | ||
|
|
||
| ``` | ||
| Summary: | ||
| Total: 1.1424 secs | ||
| Slowest: 0.0633 secs | ||
| Fastest: 0.0002 secs | ||
| Average: 0.0110 secs | ||
| Requests/sec: 8753.1552 | ||
| Total data: 1550000 bytes | ||
| Size/request: 155 bytes | ||
| Response time histogram: | ||
| 0.000 [1] | | ||
| 0.007 [2565] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.013 [3521] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.019 [3182] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.025 [446] |■■■■■ | ||
| 0.032 [147] |■■ | ||
| 0.038 [77] |■ | ||
| 0.044 [41] | | ||
| 0.051 [16] | | ||
| 0.057 [2] | | ||
| 0.063 [2] | | ||
| Latency distribution: | ||
| 10% in 0.0016 secs | ||
| 25% in 0.0064 secs | ||
| 50% in 0.0119 secs | ||
| 75% in 0.0140 secs | ||
| 90% in 0.0177 secs | ||
| 95% in 0.0214 secs | ||
| 99% in 0.0351 secs | ||
| Details (average, fastest, slowest): | ||
| DNS+dialup: 0.0000 secs, 0.0002 secs, 0.0633 secs | ||
| DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0122 secs | ||
| req write: 0.0001 secs, 0.0000 secs, 0.0218 secs | ||
| resp wait: 0.0106 secs, 0.0002 secs, 0.0543 secs | ||
| resp read: 0.0002 secs, 0.0000 secs, 0.0235 secs | ||
| Status code distribution: | ||
| [200] 10000 responses | ||
| ``` | ||
|
|
||
| ### Client Credentials Grant | ||
|
|
||
| This endpoint uses [BCrypt](#bcrypt). | ||
|
|
||
| ``` | ||
| Summary: | ||
| Total: 24.3045 secs | ||
| Slowest: 0.8962 secs | ||
| Fastest: 0.0205 secs | ||
| Average: 0.2366 secs | ||
| Requests/sec: 411.4469 | ||
| Total data: 1570000 bytes | ||
| Size/request: 157 bytes | ||
| Response time histogram: | ||
| 0.020 [1] | | ||
| 0.108 [941] |■■■■■■■■■■ | ||
| 0.196 [2380] |■■■■■■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.283 [3637] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.371 [1893] |■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.458 [752] |■■■■■■■■ | ||
| 0.546 [261] |■■■ | ||
| 0.634 [97] |■ | ||
| 0.721 [23] | | ||
| 0.809 [12] | | ||
| 0.896 [3] | | ||
| Latency distribution: | ||
| 10% in 0.1096 secs | ||
| 25% in 0.1840 secs | ||
| 50% in 0.2115 secs | ||
| 75% in 0.2921 secs | ||
| 90% in 0.3829 secs | ||
| 95% in 0.4145 secs | ||
| 99% in 0.5893 secs | ||
| Details (average, fastest, slowest): | ||
| DNS+dialup: 0.0001 secs, 0.0205 secs, 0.8962 secs | ||
| DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0139 secs | ||
| req write: 0.0001 secs, 0.0000 secs, 0.0742 secs | ||
| resp wait: 0.2360 secs, 0.0204 secs, 0.8961 secs | ||
| resp read: 0.0002 secs, 0.0000 secs, 0.0807 secs | ||
| Status code distribution: | ||
| [200] 10000 responses | ||
| ``` | ||
|
|
||
| ## OAuth 2.0 Client Management | ||
|
|
||
| ### Creating OAuth 2.0 Clients | ||
|
|
||
| This endpoint uses [BCrypt](#bcrypt) and generates IDs and secrets by reading from which negatively impacts | ||
| performance. Performance will be better if IDs and secrets are set in the request as opposed to generated by ORY Hydra. | ||
|
|
||
| ``` | ||
| This test is currently disabled due to issues with /dev/urandom being inaccessible in the CI. | ||
| ``` | ||
|
|
||
| ### Listing OAuth 2.0 Clients | ||
|
|
||
| ``` | ||
| Summary: | ||
| Total: 0.6554 secs | ||
| Slowest: 0.0331 secs | ||
| Fastest: 0.0002 secs | ||
| Average: 0.0064 secs | ||
| Requests/sec: 15257.4657 | ||
| Total data: 2670000 bytes | ||
| Size/request: 267 bytes | ||
| Response time histogram: | ||
| 0.000 [1] | | ||
| 0.003 [2703] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.007 [2650] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.010 [3599] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.013 [626] |■■■■■■■ | ||
| 0.017 [101] |■ | ||
| 0.020 [77] |■ | ||
| 0.023 [111] |■ | ||
| 0.027 [94] |■ | ||
| 0.030 [28] | | ||
| 0.033 [10] | | ||
| Latency distribution: | ||
| 10% in 0.0005 secs | ||
| 25% in 0.0029 secs | ||
| 50% in 0.0062 secs | ||
| 75% in 0.0087 secs | ||
| 90% in 0.0102 secs | ||
| 95% in 0.0123 secs | ||
| 99% in 0.0241 secs | ||
| Details (average, fastest, slowest): | ||
| DNS+dialup: 0.0001 secs, 0.0002 secs, 0.0331 secs | ||
| DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0067 secs | ||
| req write: 0.0001 secs, 0.0000 secs, 0.0182 secs | ||
| resp wait: 0.0047 secs, 0.0001 secs, 0.0315 secs | ||
| resp read: 0.0008 secs, 0.0000 secs, 0.0205 secs | ||
| Status code distribution: | ||
| [200] 10000 responses | ||
| ``` | ||
|
|
||
| ### Fetching a specific OAuth 2.0 Client | ||
|
|
||
| ``` | ||
| Summary: | ||
| Total: 0.5133 secs | ||
| Slowest: 0.0345 secs | ||
| Fastest: 0.0002 secs | ||
| Average: 0.0050 secs | ||
| Requests/sec: 19480.9563 | ||
| Total data: 2650000 bytes | ||
| Size/request: 265 bytes | ||
| Response time histogram: | ||
| 0.000 [1] | | ||
| 0.004 [3452] |■■■■■■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.007 [5398] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ | ||
| 0.010 [597] |■■■■ | ||
| 0.014 [76] |■ | ||
| 0.017 [112] |■ | ||
| 0.021 [198] |■ | ||
| 0.024 [44] | | ||
| 0.028 [39] | | ||
| 0.031 [78] |■ | ||
| 0.034 [5] | | ||
| Latency distribution: | ||
| 10% in 0.0007 secs | ||
| 25% in 0.0026 secs | ||
| 50% in 0.0046 secs | ||
| 75% in 0.0055 secs | ||
| 90% in 0.0075 secs | ||
| 95% in 0.0125 secs | ||
| 99% in 0.0268 secs | ||
| Details (average, fastest, slowest): | ||
| DNS+dialup: 0.0000 secs, 0.0002 secs, 0.0345 secs | ||
| DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0097 secs | ||
| req write: 0.0001 secs, 0.0000 secs, 0.0162 secs | ||
| resp wait: 0.0007 secs, 0.0001 secs, 0.0252 secs | ||
| resp read: 0.0021 secs, 0.0000 secs, 0.0290 secs | ||
| Status code distribution: | ||
| [200] 10000 responses | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.