Utilizing constraint framework for plain OPA running on K8S

[Gal-Zaidman]

My Use-case:
I started looking into OPA-envoy and Gatekeeper recently, for a policy enforcement solution.
I want to deploy Envoy API GW and OPA as a sidecar in a K8S cluster and I expect to have multiple policies that are similar in nature and might change frequently.
I really liked the way Gatekeeper manages policies (ConstraintTemplate and Constraint CRDs) within K8S but I still want the policy enforcement to occur on the Envoy API GW layer, meaning before reaching K8S API (so not as an admission controller).
My ideal solution would be to defined a set of constraints as K8S object (the way Gatekeeper works), but have an OPA-envoy instance running as a sidecar container to envoy that would get the rego policies from the constraints CRs and enforces them – so kind of like gatekeeper with all of its controllers but without the whole admission controller stuff.

Some background (according to my understanding):

• The gatekeeper project introduced the constraint framework[1] which is a really great way to manage and reuse policies in a K8S native way.
• The gatekeeper project includes a set of K8S controllers that watches over the ConstraintTemplate and Constraint CRDs, translates them into policies and pushes them to gatekeeper OPA.
• Gatekeeper works only with admission controller.

It is still very common to deploy plain OPA on a k8s environment, as a sidecar to envoy or different applications for policy enforcement. In that case you need to create the policies as config maps, or have a bundle server running, this is not great because it is hard to manage the policies that way, you can have a lot of code duplication between the policies, and you basically miss out on all the great things gatekeeper and constraint framework has to offer as a K8S user.

It would be great if there was a way to configure gatekeeper to integrate with a different OPA instances in the same K8S cluster, meaning have the gatekeeper controllers push the policy and data to different OPA instances in the cluster.
I looked briefly at gatekeeper and I get the feeling that most of the hard parts are implemented already, and I think that this could have huge value for K8S users.

P.S,

1. this is derived from a slack thread[2].
2. I don't think it falls under the umbrella of "Multi-Target Constraint Framework"[3]

[1] https://github.com/open-policy-agent/frameworks/tree/master/constraint
[2] https://openpolicyagent.slack.com/archives/CDTN970AX/p1658844157716909
[3] #204

[maxsmythe]

I really liked the way Gatekeeper manages policies (ConstraintTemplate and Constraint CRDs) within K8S

I'm glad you like it!

Before diving in to brainstorming, some clarification on the behavior/infrastructure of Gatekeeper/Constraint Framework:

• To improve performance/memory usage, and execution isolation, the constraint framework handles much of the coordination of the evaluation of templates at the Golang level, so whatever is evaluating the policies (in this case a sidecar?) would need to be running constraint framework (which would mean not being able to leverage OPA's compile-to-WASM behavior, if that's a requirement).

• Gatekeeper doesn't really work so much on a push-based model as a watch-based model. Each webhook/controller establishes a watch with the API server and independently keeps their policy configs up-to-date

I don't think it falls under the umbrella of "Multi-Target Constraint Framework"[3]

This is true in the strictest sense (since I don't think this leads to a single constraint template with multiple targets), but it would mean creating a new target to feed into the constraint framework (which has been done before, see terraform-validator.

It would be helpful to know a bit more about the specifics of your policy goals and maybe see some example content of what an inbound envoy authz request would look like to make this discussion more concrete.

Gatekeeper works only with admission controller.

This is true. I took a brief glance at Envoy authorization. It looks like this is the schema for Envoy's authorization requests? Assuming that's true, the rough work would be as follows:

• Add another webhook that is capable of parsing the envoy proxy
• Create a TargetHandler that understands Envoy requests
  • If useful, figuring out general-purpose filtering (match for a K8s constraint) could be helpful to avoid code duplication, uniformity of policy management, and improve speed
• Add this webhook as a sidecar and call it
• profit

The biggest questions would be as-follows:

• Is the controller model what's desired here? Is the intent to store the policy config on the API server?
• Where should the code live? Gatekeeper provides 99% of the most difficult operational code (maintaining watches, reporting readiness), so it wouldn't make sense to re-implement that, but is this in scope for the G8r project? If not how to maintain it easily.

@ritazh @sozercan @shomron In case you find this interesting.

[Gal-Zaidman]

Sorry it took me a while to reply.

Before diving in to brainstorming, some clarification on the behavior/infrastructure of Gatekeeper/Constraint Framework:

• To improve performance/memory usage, and execution isolation, the constraint framework handles much of the coordination of the evaluation of templates at the Golang level, so whatever is evaluating the policies (in this case a sidecar?) would need to be running constraint framework (which would mean not being able to leverage OPA's compile-to-WASM behavior, if that's a requirement).

I hope that I understood you here, but as I see it OPA is evaluating the policies, and yes it is running as a side car to envoy (External Authorization filter to envoy), nothing fancy exactly as you describe in your docs Link.

• Gatekeeper doesn't really work so much on a push-based model as a watch-based model. Each webhook/controller establishes a watch with the API server and independently keeps their policy configs up-to-date

Amm yes but when a constraint(policy) is changed (Created/updated/deleted) how does the controller pass that change to OPA?

It would be helpful to know a bit more about the specifics of your policy goals and maybe see some example content of what an inbound envoy authz request would look like to make this discussion more concrete.

Sure, basically the inbund request to OPA would be a parsed grpc request from envoy that would contain a CRD in json format in the parsed_body.content field of the input (he is an example of something I captured on my local test env https://controlc.com/1821f869).
Most of the policies will most likely apply to fields on the CRD, so something similar to the target that you have right now on gatekeeper (admission.k8s.gatekeeper.sh) but with the wrapper of the grpc request.

This is true. I took a brief glance at Envoy authorization. It looks like this is the schema for Envoy's authorization requests?

No this is the scheme for the configuration of the ext_authz in envoy... it is not relevant.

Assuming that's true, the rough work would be as follows:

• Add another webhook that is capable of parsing the envoy proxy

As I said webhook is to late, webhook is when the request reaches the K8S API, I want to evaluate the request on the Envoy API GW layer, meaning before reaching K8S API.
Just to clarify the usecase here is different gatekeeper validates requests from K8S users, creating things in the K8S API.
I want to evaluate requests from clients, meaning they can't interact with my K8S API, just call an API I expose to them from my running services.... Internally I want to utilize the fact that I work in a K8S ENV and manage my policies as K8S API objects with controllers and so on, but the client is unaware of that

• Create a TargetHandler that understands Envoy requests

Yep I guess we will need that, but since it will contain an object that the available target knows how to parse (CRD) than it would be nice if it could utilize the existing code.

• If useful, figuring out general-purpose filtering (match for a K8s constraint) could be helpful to avoid code duplication, uniformity of policy management, and improve speed
• Add this webhook as a sidecar and call it
• profit

The biggest questions would be as-follows:

• Is the controller model what's desired here? Is the intent to store the policy config on the API server?

Yes this is what I want to get from gatekeeper, otherwise I would just deploy the opa-envoy sidecar like you document already.

• Where should the code live? Gatekeeper provides 99% of the most difficult operational code (maintaining watches, reporting readiness), so it wouldn't make sense to re-implement that, but is this in scope for the G8r project? If not how to maintain it easily.

Ammm well I basically need all the core logic that gatekeeper has right now, controllers and so on, but with the ability to configure those controllers to talk to a different instance of OPA, so I'm not sure if this is an actual separate project, maybe some extenstion of gatekeeper configuration options and maybe opa-envoy-plugin ?

Maybe this can be done with gatekeeper as is, but instead of using webhooks and admission controllers to interact with it, create an envoy External Authorization filter that will talk with gatekeeper?

[Gal-Zaidman]

I looked into gatekeeper and opa-envoy-plugin source code a bit and now I understand that my request is more complicated than I assumed... The main complication is that gatekeeper runs OPA through the constraint framework, which embeds OPA as a library whereas opa-envoy-plugin runs opa as a runtime, meaning that we can’t simply extend gatekeepers opa with a plugin or redirect the requests to gatekeepers opa...

Still, I tried to think about the possible ways to implement it, again just to frame it, the end goal is to use gatekeeper controllers and constraint framework logic but with envoy target -- not webhooks.

Option1: Add a sidecar container to gatekeeper that implements the envoyExtAuthz gRPC server and use gatekeeper to check requests

• Probably the simplest and most generic solution
• Need to expose a gRPC endpoint from gatekeeper to check requests, since they will not come from the webhooks... or some other means of communication with it.
• Probably can be some variation of opa-envoy-plugin, but since it is tightly coupled with the plugin and runtime, it will be more as an example.

Option 2: Run an envoyExtAuthz gRPC server within gatekeeper instead of the webhooks (configurable):

• Will be less generic then option 1 since it will expose the what envoy expect, but will be easier to implement since like the webhooks implementation, the OPA constraint client is accessible.
• Probably can be some variation of opa-envoy-plugin, but since it is tightly coupled with the plugin and runtime, it will be more as an example.
• Similar in the amount of work to adding a webhook (I assume).

Option 3: Add a plugin to gatekeepers OPA for providing external authorization to envoy, like we do in opa-envoy-plugin:
Not possible at the moment, since gatekeeper runs OPA through the constraint framework, which embeds OPA as a library (through the rego package) that means that we can’t simply extend gatekeepers opa with a plugin, since plugins usage is only possible through the runtime package which is relevant only when using OPA as an executible. Does it make sense to extend the package usage to accept external plugins?

Any thoughts?