OPA data approach in policy takes too much time.

[somsom13]

Hi, I have a question about OPA's data approach in policy.

I have this data.

# data written in array
"permissions": {
    "my-host-domain": {
      "/path/to-resource": {
        "GET": ["client1", "client2"]
     }
   }
}

# data written in object
"permissions": {
    "my-host-domain": {
      "/path/to-resource": {
        "GET": {
            "client1":{},
            "client2":{}
       }
     }
   }
}

And in the policy, I look up the value with my_data := data.permissions[input.domain].
When I profile my policy, there is too much difference in the speed of executing this line (data.permissions[input.domain]), when the data is in the form of an object and in the form of an array.

• in form of array: 1 ms
• in form of object: 4 ms

Originally, I thought that after OPA loads the bundle, it doesn't load data to memory even if I approach to the data in policy.

So, here's my question.

• If there is data access by data.permissions[input.domain] inside the policy, does it load a new one every time the policy is executed?
• Why there is a difference in the inquiry speed when it is in the form of an object and in the form of an array?

[johanfylling]

Hi @somsom13!

Data from storage is unmarshalled on each policy execution. This is however currently investigated in open-policy-agent/opa#4147, and might get optimized in a future release.

The fact that you're seeing a large speed difference between when nested entities are arrays or objects is surprising. Is this consistent over multiple runs? Is it possible for you to share the full policy and data you're using in your tests?

[somsom13]

Hi, @johanfylling !

Data from storage is unmarshalled on each policy execution.

Thank you for your explanation. I understood that "unmarshall" as OPA making json data as a GO(REGO) parsable structure, am I right?

Is this consistent over multiple runs?

And yes, the result is almost same over multiple runs.

Is it possible for you to share the full policy and data you're using in your tests?

I created a private github repository that contains my policy and data, and added you as a collaborator. I hope you will be able to approach it. If there's someone else who needs to take a look at it, please let me know. (Or if possible, you can invite other collaborators as you want!)

I added the evaluation command in README, so that you can run the command easily.

I also found out that if I run eval command with --explain full option, time increase happens this line: data.permissions[input.domain].

But if I remove that --explain full option, time increase happens in this line: some path, _ in allowed_apis.

I hope this helps finding the cause. Thanks for the help!