Add OCI support

[carabasdaniel]

What part of OPA would you like to see improved?

We would like OPA to be able to consume bundles from an OCI registry. This integration will allow us to take advantage of the richness of the OCI ecosystem including but not limited to image signing, versioning, caching, layering, etc.

Describe the ideal solution

Ideally once this is implemented a user could use a configuration similar to the example below to consume a bundle from an OCI registry.

Example:

services:
  github-registry:
    url: https://ghcr.io
    type: oci
    credentials:
      username: alice
      password: secret

bundles:
  authz:
    service: github-registry
    resource: open-policy-agent/my-bundle:1.0.1
    persist: true
    polling:
      min_delay_seconds: 60
      max_delay_seconds: 120

# The local OCI store would be persisted to /var/opa/oci
persistence_directory: /var/opa

Additional Context

To implement this we thought we should create a downloader interface and implement an OCIDownloader - download.NewOCI(...). The current downloader can remain as the default http downloader.

To drive the implementation of the new OCIDownloader we would like to use ORAS as it would offer higher level constructs that are easier to implement with.

For additional details please consult the following document that was presented in the OPA community call.

[tsandall]

Hi @carabasdaniel. Thanks for filing this issue. This topic has come up before over the years (e.g., #1464)

his integration will allow us to take advantage of the richness of the OCI ecosystem including but not limited to image signing, versioning, caching, layering, etc.

Could you elaborate on a few of these things?

• how does image signing compare to the existing bundle signing mechanism we have in OPA?
• what use cases are driving your versioning requirements and how are those not met by the existing bundle support?
• what use cases are driving your need for layered filesystems that images are based on?
• I'm assuming the caching benefit you described is related to layers but if not, please elaborate on how the existing HTTP-based caching mechanisms are lacking.

Overall, I'm in favour in adding OCI support to OPA for policy distribution, I just want to understand the motivation (beyond OCI being cool 😅 ).

The other thing to consider is the implementation and reliance on 3rdparty libraries. We're already starting to see difficulties due to the embedding of otel, wasmtime, and badger in OPA--if we use ORAS that would add another dependency; it looks like the oras-go package has a v1 release that promises stability, can the implementation rely on that?

EDIT: one more thing to consider... now that we have disk-store support in the server, we want to work on optimizing how data sets are loaded out of bundles and written into the store. One of the challenges there is to avoid paging all of the data into memory unnecessarily. We want to get away from deserializing the data sets into in-memory data structures unnecessarily. Whatever we do with the existing HTTP downloader will need to be done for the OCI downloader. This might influence whether we use ORAS or something else.

[carabasdaniel]

Hi @tsandall

how does image signing compare to the existing bundle signing mechanism we have in OPA?

According to their documentation, cosign offers support for:
- Hardware and KMS signing
- Bring-your-own PKI
- Our free OIDC PKI (Fulcio)
- Built-in binary transparency and timestamping service (Rekor)
As cosign is part of the sigstore project I think it makes sense to rely on this rich ecosystem for signing OCI artifacts, instead of using an internal implementation that only works for OPA.

e.g. as a security engineer, I'd like to be able to verify integrity of all OCI artifacts using the same tools

These two mechanisms don't need to be mutually exclusive though. It's possible to sign/verify layer contents using the current OPA mechanism as well.

what use cases are driving your versioning requirements and how are those not met by the existing bundle support?

Like any other artifact that we build and publish, we want to:

1. Version it using semver (semver based on the API it's protecting)
2. Distribution and consumption - here we prefer an OCI registry to a bucket store because they are ubiquitous. We can push to multiple vendors hassle free, as well as private registries.
3. Be able to list in a human readable format, all the versions that have been published
4. Be able to freeze a bundle in OPA to a particular version

These requirements are possible with http and the current bundle support, but they require investment in server side infrastructure, and at that point, you're building the features available in an OCI registry.

what use cases are driving your need for layered filesystems that images are based on?

Our current use case doesn't require multiple layers. The implementation we are using right now uses one layer per bundle. We don't have to use multiple layers in order to reap other benefits of the OCI ecosystem.

However, in the future we plan to use layers to easily manage different parts of a policy: manifest, rego specifications and data. And for data, you can imagine using child layers to only bring in changes to data, not update all of it.

I'm assuming the caching benefit you described is related to layers but if not, please elaborate on how the existing HTTP-based caching mechanisms are lacking.

The caching benefit is not related to layering in the sense that we'd only download changes from one version of a bundle to the next. The implementation of the local OCI store and its caching mechanism are very similar to what the HTTP implementation does. It uses the digest of a layer to determine whether it has to be downloaded or not (like the etag is currently used).
Here we mean that a disk-based cache is useful, so that we don't need to download bundles again on a restart of an OPA process.
But, as mentioned earlier, layering can help us in the future to only download changes.

I hope this clarifies a bit the motivation and I think this feature can open up multiple possibilities of improvements in the future.

[tsandall]

@carabasdaniel thanks for the detailed response!

Another area that comes to mind (in addition to the disk storage integration that's still a bit of an unknown...) is incremental updates. OPA recently gained support for delta updates on data contained in the bundle. This is particularly useful for cases where the bundles are relatively large, but only small sets of data change--and where it's desirable to propagate that change quickly/efficiently.

How does OCI support incremental updates?

[carabasdaniel]

Hi @tsandall,

With OCI, we'd have multiple layers, with each layer that is a child containing the delta from its parent. It sounds like the delta mechanism is there already, so this should work nicely. A layered approach can easily only grab new layers that contain new changes, and that would help with speed.

We'd like to implement this as a separate PR if that's ok, to get the basic functionality out there first. In the initial draft I put up, we're using a single layer approach to a bundle, and we currently still “feed” the tarball to the bundle loader directly. We'll have to test what this means for bundles that contain delta updates.

[tsandall]

@carabasdaniel agreed we don't have to implement the functionality in the initial PR but it would be good to spec out how it would work so that we know we have that option in the future--otherwise we'd be adding something to OPA that we know will lead to scalability issues that will make users unhappy.

With OCI, we'd have multiple layers, with each layer that is a child containing the delta from its parent. It sounds like the delta mechanism is there already, so this should work nicely. A layered approach can easily only grab new layers that contain new changes, and that would help with speed.

It would be useful to spec out what the image contents would look like (format) and the network requests to fetch updates would look like (protocol) and call out any limitations, e.g., # of layers, that users would run into. E.g., suppose you have a bundle containing data for 100,000 resources/objects. If 1% of those objects change, what do the image layers look like and what do the network requests look like? What does it look like if another 1% change a few minutes later?

[carabasdaniel]

Hi @tsandall,

It would be useful to spec out what the image contents would look like (format) and the network requests to fetch updates would look like (protocol) and call out any limitations, e.g., # of layers, that users would run into. E.g., suppose you have a bundle containing data for 100,000 resources/objects. If 1% of those objects change, what do the image layers look like and what do the network requests look like? What does it look like if another 1% change a few minutes later?

To get a better understanding can you please provide a few more details about the use case here?

• Is it a mechanism to continuously (forever) feed data updates to OPA?

• In OPA right now, if OPA restarts, how do I get back to my state? Do I have to consume all the deltas?

[tsandall]

Is it a mechanism to continuously (forever) feed data updates to OPA?

Yes, that's how bundles have always worked--OPA periodically downloads the latest version of the bundle from the service that it's configured to talk to. Over time we've extended the bundle format and protocol to support deltas, long-polling, etc.

In OPA right now, if OPA restarts, how do I get back to my state? Do I have to consume all the deltas?

That's up to the bundle server--the format and protocol allow for the server to send a full snapshot or a snapshot and a sequence of deltas. IF OPA was using in-memory storage then the server would have to transmit the entire contents of the bundle; with OPA v0.39.0 we now have disk storage in the server and w/ the latest change to persist the ETag, OPA will send a network request w/ the ETag of the most recent bundle activation--in which case the server may not have to retransmit anything.

[viovanov]

Hi @tsandall - we looked at the delta feature, and I don't think layers will be helpful for this use case.
It's not helpful to keep growing the number of layers in the image forever.
In this scenario, the behavior should remain as it is now.
The publisher of the bundle would replace the layer that opa downloads, with a delta bundle. (this works with the current PR)

[stale]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[anderseknert]

Can we close this and create follow-up issues for new features on top of what's there now?

[anderseknert]

Closing this since we have at least basic support implemented, and there seems to be follow-up issues created as well. Let me know if I missed something and I'll re-open.