Why are functions, as opposed to regular rules, loaded "greedily"?

[anderseknert]

If I have a policy like:

allow {
    data.some.other.package.x
}

then x won't be evaluated until allow is. But if x is a function, like:

allow {
    data.some.other.package.x(input.y)
}

then the file won't compile (undefined function error) unless the dependency has been pre-loaded..

What is the reason for this?

[tsandall]

The reason boils downs to the fact that base and virtual documents can be mixed at the package level. The type checker errors on the second example because it knows that x doesn't exist. On the other hand, it doesn't know that x is missing in the first example because that x could refer to a base document (and that base documents may not have been loaded.)

Related open-policy-agent/opa#491

[anderseknert]

The type checker might know x doesn't exist now, but just like with x possibly being a base document, x the function could be added later (as long as it is before allow is actually evaluated), right? The context here is that function x exists in a bundle ("helper-functions") that has yet to be loaded by OPA. Since OPA can't guarantee remote bundles being loaded in any particular order we'll see errors (as in, logging on the most severe error level) in the OPA logs if the "helper-functions" bundle has not been loaded when the the bundles that depends on it are loaded.

While we can use /health?bundles to wait for all bundles to load, it's not gonna help with the error logger situation, as we will see compile errors in the logs even without traffic being routed to our OPAs.

Thanks for the link! Funny how that seems to ask for precisely the opposite :)

[tsandall]

The type checker might know x doesn't exist now, but just like with x possibly being a base document, x the function could be added later (as long as it is before allow is actually evaluated), right?

Right, the type checker can only operate on what it's given--it can't know that x might exist in the future. My answer explains why the behaviour is the way it is.

The context here is that function x exists in a bundle ("helper-functions") that has yet to be loaded by OPA. Since OPA can't guarantee remote bundles being loaded in any particular order we'll see errors (as in, logging on the most severe error level) in the OPA logs if the "helper-functions" bundle has not been loaded when the the bundles that depends on it are loaded.

While we can use /health?bundles to wait for all bundles to load, it's not gonna help with the error logger situation, as we will see compile errors in the logs even without traffic being routed to our OPAs.

Using multple bundles to distribute helpers should be discouraged. We caution against using multiple bundles here where multi-bundle support is introduced. We could provide this use case as an anti-pattern example if that helps folks.

Thanks for the link! Funny how that seems to ask for precisely the opposite :)

Thinking about the future, I would want OPA to be more strict about this kind of thing (rather than less) which is why I linked to the issue.

[anderseknert]

The point is not to promote one way of working with bundles over another, merely showing a situation where functions getting evaluated early leads to an outcome which is unique to functions. Had we instead used a “helper-rules” library there would have been zero compile time errors, and nothing “discouraging” the pattern. The problem with this apparent inconsistency is of course that for most users won’t really consider whether they export/import rules or functions - hence why it would be useful to know why that distinction is important to the compiler.