perf: add Regorus ACI benchmark tests #7298
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Seeing the performance comparison of Regorus and OPA in the Regorus docs made me curious to try it out myself. This PR adds the test files from their repo (MIT licensed) and two benchmark tests that evaluates the policy using the same parameters as the
opa eval
command from their example.The benchmarks show that doing the full
opa eval
dance where we build/compile the bundle from scratch is about as fast/slow as their example shows (~30 milliseconds here, but that's without the overhead of the eval command line tool).However, building the bundle and query beforehand — as would be done in the case of a real OPA deployment — and measuring only evaluation time, shows something else entirely: ~0.09 milliseconds, or 90 microseconds.
So while the numbers they present aren't misleading in any way, they're not representative of OPA's evaluation performance as experienced running OPA as a server, or through the Go API.
That they compile the bundle in that short time is impressive nonetheless!
Saving the benchmark here as it's a good one to use for future improvements.