OPA Deployment Archtectural Decisions

[humbertoc-silva]

Hi, OPA community,

I have a topic related to deployment architectural decisions to discuss with you. I see orientations on OPA docs to deploy it as a sidecar container, to use it inside services through the Go module or SDK, and in some situations to use the OPA binary too. All these options make me think that the main way to deploy OPA is as a component distributed throughout the stack.

But did anyone think deploying OPA as a centralized authorization service? What would be the pros and cons of using OPA this way?

I know that, as always, it depends, so to help with this, I will describe an environment that we can use as a background to our discussion:

• It needs to authorize hundreds of millions of requests per day
• It will authorize critical services
• Many teams can produce authorization policies

I have an opinion but I would like to hear your experiences before sharing my own.

[charlieegan3]

Hi there @humbertoc-silva, this is a great question and it'd be good to have some more content on this topic in the OPA docs for sure.

The pros to this approach are:

• reduced total RAM and CPU footprint of all running OPAs as you can better share OPA with many, potentially 10,000s of containers which would have otherwise needed sidecars.
• Related to the above, suitable if large amounts of data is needed for policy evaluation and would be too much to replicate in every sidecar. (aside, this is a common use case for Enterprise OPA deployments on Kubernetes.

The cons to this approach are:

• higher latency due to cross-node (or further) hops.
• Single point of failure, if misconfigured, a central deployment can disable many dependant services.
• Traffic to OPA must be secured using mTLS and OPA must identify callers to ensure they're using the permitted endpoints.

[anderseknert]

Adding to Charlie's great summary:

Many teams can produce authorization policies

OPA does not provide any type of isolation between policies loaded on the same instance. So any policy can at least by default get any data from any system, query any rule or package, and so on.

You could probably prevent most of this with compile time checks that make sure that doesn't happen, e.g. have custom Regal rules to determine that no reference in data.team1.* points to anything under data.team2.* and so on. But OPA itself won't care.

[humbertoc-silva]

Hi @charlieegan3 and @anderseknert, thank you for the contribution.

I agree with the pros, but related to the high volume of data, imagine if we have one OPA container instance (enterprise or open) with 1GB of data and we scale out to 10 containers, here is my first point, OPA doesn't share a common cache and at the final we will have 10GB of duplicated data. The solution could be to deploy a cache layer and turn off http.send cache feature for example. OPA will be more light to scale and data will be shared.

About the cons, I agree too, latency, point of failure, and traffic security are all things that are difficult to handle and support. But here I have a question for you. Does OPA was designed to support hundred of thousands of policies and rules in the same instance? Why do I ask this? I did a test loading 10K, 50K, and 100k simple ABAC-based rules like the following example:

allow if {
    subject.attr_1 == input.subject.foo
    resource.attr_2 == input.resource.bar
    "action_1" == input.env.action
}

...
...
...

OPA wasn't able to load 50k+ rules like those. I waited minutes but the container didn't start. So the volume of rules in a centralized OPA could be a problem?

Good point about policy isolation, I didn't think about that, and this can be a factor in deciding to run certain policies in different OPA instances.

To finish, maybe it could be interesting to establish a template or some rules about the way people will write their policies because we can have poor policies that will affect the entire system's performance.

[charlieegan3]

Hey, this is an interesting policy to me:

allow if {
    subject.attr_1 == input.subject.foo
    resource.attr_2 == input.resource.bar
    "action_1" == input.env.action
}

It looks to me like a policy that could be based on a shared policy using data (which could then be supplemented with more involved policies where needed).

Would you be able to share an example bundle for testing? it'd also be interesting to know how much resource you allocated to OPA when running it.

Napkin math makes that around a 200mb bundle (the example rule is 4k), which should be doable. Though it is a large policy bundle, and I wonder if any of that could be data instead.

[anderseknert]

I was never good with math, but I know a little Rego :) To create a policy with 50k rules like that:

package p

import rego.v1

header := `package p

import rego.v1

subject := data.org.subject
resource := data.org.resource

`

template := `allow if {
    subject.attr_%[1]d == input.subject.foo
    resource.attr_%[1]d == input.resource.bar
    "action_%[1]d" == input.env.action
}
`

rules := concat("\n", [rule |
	some i in numbers.range(1, 50000)
	rule := sprintf(template, [i])
])

policy := concat("", [header, rules])

opa eval -f raw -d p.rego data.p.policy > large.rego

This takes about a second for OPA to load on my laptop, and once loaded, evaluation time is ~1 millisecond or below.

[humbertoc-silva]

@charlieegan3 I need to recover the project or create another one to build a bundle for the test. I ran the test on my laptop, a MacBook Air M1 with 8GB of RAM and I did not limit the OPA resources. I remember that OPA consumed about ~3GB when loaded 10K-25K.

@anderseknert I did something similar, but I used JS/Node.js to create a big Rego file, and then I started OPA pointing to this file.

I will try to reproduce my test and share it with you.

[humbertoc-silva]

Hi guys,

I put my code inside a Git repository to share with you. Let me know if the problem is on my machine or if I made a mistake when creating and loading the policy into OPA that caused the problem. With few rules, OPA loads the policy very fast, but with 50.000 it takes minutes here.

I put some instructions on README.md but feel free to ask me about anything.

Thank you!

https://github.com/humbertoc-silva/opa-policy-load-test

[charlieegan3]

Hey there, I have had a little play here, I think loading what is in effect data, into OPA in this way is not going to work well.

I'd suggest using JSON instead which can be loaded more quickly.

https://play.openpolicyagent.org/p/VhaynCJM3F

I tested this in your repo and saw OPA start instantly and respond locally in around 100ms

https://github.com/humbertoc-silva/opa-policy-load-test/compare/main...charlieegan3:opa-policy-load-test:main?expand=1

[humbertoc-silva]

Nice @charlieegan3. So we have some choices here:

• less expressive on Rego and more expressive on data
• More expressive on Rego and less expressive on data (or no data).

Using my version, OPA took minutes to start, but the authorization request was evaluated between 2 and 5 ms, maybe because of the index? With your version, OPA can load faster but it took more time to evaluate the rules.

I tried to write Rego more expressively because it is easier to read and understand but I need to pay attention to the limitations of loading a lot of rules into OPA in this way.

So, returning to the original topic, it is possible to use OPA in a centralized way. We have some benefits related to resources but we need to establish a way to put lots of rules to avoid problems, mainly in the server start phase. One of the strategies is to consider the use of data and create rules to process this data instead of writing verbose and explicit rules.

Do you agree? Have some other points or observations?

Thank you a lot!

[charlieegan3]

Out of interest, how much slower were the evaluations when loading the data? Can you also try with --optimize-store-for-read-speed and report on your timings with that too?

[charlieegan3]

https://www.openpolicyagent.org/docs/v0.70.0/policy-performance/#storage-optimization