conflicting rule for data path after POSTing a policy with a default value

[valbisson]

Hi OPA community

I'm writing a thin wrapper over OPA server in order to validate user-fed data & policies before posting them in the prod server. To do so, when receiving a new policy, I do the following steps:

• I first pull all policies locally (GET /v1/policies) and write a .rego file for each
• I write a data.json file filled with the result from GET /v1/data
• I run opa test . to make sure the new policy complies with everything (some policies contain tests)

This is fine (albeit open for much optimization) on paper, but what happens in real life is different.

When POSTing this policy to a brand new opa --server instance (->data is an empty {}):

package business_code

# passes when can==do
default can_do := false
can_do {
    input.can == `do`
}

To my surprise, the default line creates an entry into data:

curl -s http://localhost:8181/v1/data|jq
{
  "result": {
    "business_code": {
      "can_do": false
    }
  }
}

I can list the policy correctly.

But then when POSTing the next policy, my 3 steps procedure above creates files as expected, but the data.json file now has this can_do=false hardcoded, which keep tests from compiling:

$opa test .
1 error occurred: business_rule.rego:5: rego_compile_error: conflicting rule for data path business_code/can_do found

How comes POSTing a policy also updates data? What can I do do prevent this conflict?

To reproduce:

1. create policy file business_rule.rego

package business_code

# passes when can==do
default can_do := false
can_do {
    input.can == `do`
}

2. create data file data.json

{"business_code": {"can_do": false}}

3. run tests:
   opa test .

Edit: using opa 0.42, and had the issue on 0.41 as well.

[anderseknert]

Hi @valbisson! And thanks for reaching out 😃

The "Data API" should not be mistaken for serving only "static" JSON data such as that provided in a data.json file. In fact, the data API is what serves pretty much all decisions OPA makes! See the docs on the OPA document model to learn more about base documents vs. virtual documents, and how they both contribute to building the OPA data document.

With that said, I'm not sure I understand why you need to GET the policies and data from a running OPA instance? If your users provide their policies, data, and possibly tests as well.. why not just accept those files as they are, run opa check --strict, opa test, and so on, before packaging them into a bundle (i.e. opa build) which you'll then push somewhere to be used in your OPAs running in production?

[valbisson]

Hi @anderseknert , thanks for the quick answer!

I did misunderstand how the data API was serving both types of documents. It appears that posting the policy creates the virtual data, which I transform back into a base document when I pull & dump it in a file, causing the collision with the conflicting data path reported above.

As a solution, I'll try to store data (documents or patches) and policies upstream from the opa server, and validate from these, rather than pull them back each time. As you suggested, right? (I can't blindly push them as they may depend on data/policies from other users, ie not be testable out of context).
We have a requirement to support live data updates (possibly policies as well, if all goes well) in a somewhat low latency manner, so if I understand correctly, Option 4: Push Data is closer to what we want, rather than building & reloading a new bundle each time a delta arrives.

[anderseknert]

Pushing policy and data into OPA is fine of course, although I'd probably start with the bundle model and look into alternatives only when that (including delta bundles, and so on) doesn't cut it.

Still not sure I understand why you'd first need to make a roundtrip into OPA for testing, i.e. first upload something and then retrieve it back down to test? Even if you have user A depend on something user B uploaded in the past, would it not be simpler to just have a directory structure like, say:

├── a
│   ├── policy-a-test.rego
│   └── policy-a.rego
└── b
    ├── policy-b-test.rego
    └── policy-b.rego

And then run the tests, checks and so on, before pushing anything into OPA?

Either way, sounds like you're making progress 👍

[valbisson]

Yeah I thought that's what I was describing as well lol
All good, thanks a lot!