mocking function responses via "with" statements

[rmetcalf9]

What part of OPA would you like to see improved?

I am aware that in OPA tests we can mock the input provided in each test.
I need to be able to write tests where

• different environment variables are set in the policy execution (e.g. output of opa.runtime()["env"]["OPA_ENVIRONMENT"])
• different http responses are received in the policy execution (e.g. output of http.send({ "method": "GET", "url": jwksUrl, "cache": true, "enable_redirect": true }))

Not being able to mock these responses means our tests can not be relied upon.
I have hacked rego in some ways so there is one entry point in test mode and another in real mode with different responses but this is unreliable and leaves me open to lots of mistakes.

Describe the ideal solution

In Python we can "patch" a response as part of a test so we can cause python not to execute the function at all and instead use a constant return value. I would love something similar in OPA.

Describe a "Good Enough" solution

A way to tag particular OPA tests so I can setup a complicated system which runs the opa test command multiple times with different values, plus a way to override url's so I can redirect the http requests to some kind of mock during testing.

Additional Context

OPA is a different style language to what our developers are used to. Our central policies are also a single point of failure and there is a danger that when global functions are changed we kill access to all our API's.
We are trying to use TDD in order to minimise the risks here but not having the ability to fully mock function outputs in OPA makes this very difficult.

[srenatus]

#2028 This is the issue to track, here, I believe. You've already seen it, since you've +999'ed it. 😄

I believe your examples could at least partly be served using the existing measures:

different environment variables are set in the policy execution (e.g. output of opa.runtime()["env"]["OPA_ENVIRONMENT"])

If you wrap that in a rule,

opa_env = opa.runtime().env # it's a complete rule

p {
  opa_env == "prod"
  # logic
}

p { 
  opa_env == "staging"
  # other logic
}

then you can use

test_p_prod {
  p with opa_env as "prod"
}

etc. You should be able to mock http.send responses, too, in a similar way.

[rmetcalf9]

I just implemented this. It works really well.
For my case I changed it slightly:

opa_environment_variable = opa.runtime()["env"]["OPA_ENVIRONMENT"]

I now have much better testing.
It would be great to see this documented here - https://www.openpolicyagent.org/docs/latest/policy-testing/
Maybe a section on mocking rules with examples of mocking http.send responses as well as environment variables as I think this use case will be very common.

[srenatus]

I'm glad you've gotten this sorted out. Since you've still got a fresh memory, a docs contribution would always be welcome and now might be a good time 😄

But don't feel pushed, it's also a good first issue for someone else coming by.

[jcmcken]

What would the workaround be for http.send (i.e. function with variable input)?

[srenatus]

@jcmcken There are straightforward cases and ugly ones. E.g. if your input comes from input, or some other piece of data, you can use a rule,

allow {
    resp := http.send({"url":sprintf("https://some/api?query=%s", [input.foo]), "method": "get"})
    resp.body.foo
}

⬇️

allow {
  resp_of_api
}

resp_of_api = resp.body.foo {
    resp := http.send({"url":sprintf("https://some/api?query=%s", [input.foo]), "method": "get"})
}

test_good {
    allow with resp_of_api as true
}

test_bad {
    not allow with resp_of_api as false
}

👉 It's straightforward if you don't need to use another function to encapsulate your http.send call.

If you do, there are other workarounds, I think... perhaps using with and some stand-in data ref... 🤔 I would help if you could share your example at hand, I suppose.

[rmetcalf9]

The http.send method isn't working for me.
I have the following

http_send(inp) = result {
    result := http.send(inp)
}

test_point(apiPath) = result {
    url := concat("", ["http://dummy.com", apiPath])
    result := http_send({ "method": "GET", "url": url, "cache": true, "enable_redirect": true })
}

and my test

test_callstudioservice_respond_200 () {
    expectedResp := { "A": "a" }
    test_point("/a/b/c") == expectedResp
    with http_send({ "method": "GET", "url": "http://dummy.com/a/b/c", "cache": true, "enable_redirect": true }) as expectedResp
}

but I get

1 error occurred during loading: grabyo/dataaccess/studio/v1/main_test.rego:16: rego_parse_error: unexpected as keyword: expected with target path
            with http_send({ "method": "GET", "url": "http://dummy.com/a/b/c", "cache": true, "enable_redirect": true }) as expectedResp

I tried knocking out the param passed to the http_send (not satisfactory since I need to test the params passed to the mocked service are correct - not just the response. I then get the following error:

1 error occurred: grabyo/dataaccess/studio/v1/main_test.rego:16: rego_compile_error: with keyword cannot replace functions

Can anyone help me write a test that tests both the params sent to a function and then mocks the response?

[srenatus]

Note that

test_callstudioservice_respond_200 () {
    expectedResp := { "A": "a" }
    test_point("/a/b/c") == expectedResp
    with http_send({ "method": "GET", "url": "http://dummy.com/a/b/c", "cache": true, "enable_redirect": true }) as expectedResp
}

again tries to mock a function: http_send(inp) = result { result := http.send(inp) } is a user-defined function.

As a workaround, you'd have to turn that into a rule:

http_send = result {
    inp := { ... } # <--- inp would need to be constructed from `input`, or other rules here
    result := http.send(inp)
}

can then use with http_send as expectedResp in your tests.

I'm not saying it's not cumbersome, but as long as you can construct your arguments to http.send using input and other rules, you can make this work.

[rmetcalf9]

Not being able to mock user defined functions is giving me some challenging problems. I have my logic divided up in such a way that I have nested user defined functions and I need to test each individually.
I am going to experiment with using input and forking the logic but it's going to get messy as I mock different levels.
I have done a simple version of this with my jwt functionality as follows:

# This version is used in real policy calls and will decode the jwt
access_result = result {
  not input.testOnlyJwtResult
  jwtResult := utils.jwt.v2.get_grabyo_validated_jwt(input.encodedJwt)
  result = get_access_result(input.method, input.path, input.queryParams, jwtResult[0], jwtResult[1], jwtResult[2])
}

# This is only matched in tests. It enables us to mock the jwt validation response
#  a real request from an agent should never include testOnlyJwtResult in it's input
access_result = result {
  input.testOnlyJwtResult
  result = get_access_result(input.method, input.path, input.queryParams, input.testOnlyJwtResult[0], input.testOnlyJwtResult[1], input.testOnlyJwtResult[2])
}

This method tests the output of the functions but not the input paramaters.

I think it would be a highly valuable feature for rego to implement a way to mock user defined functions.
Something like PATCH in python.

[srenatus]

I think it would be a highly valuable feature for rego to implement a way to mock user defined functions.

💯 Agreed. This is just a workaround.

Let's talk design, would you prefer something simple like a static replacement

mock_func("good-token", _) = [ true, {}, good_payload ]
mock_func("bad-token", _) = [ false, {}, {} ]

test_foo {
  allow with io.jwt.decode_verify as mock_func
}

or inline definitions?

test_foo {
  allow with io.jwt.decode_verify(_, _) as [true, {}, {}]
}

or, using their args,

test_foo {
  allow with check_header("Authorization", _) as true
        with check_header("X-Token", _) as false
}

[rmetcalf9]

From my first read I like the first syntax with static replacement.
I presume it would work with mocking of multiple user defined functions by using multiple with statements.
The inline definitions worries me because I want to be able to test the input params to the function are correct as well as the output. (Although could that by fixed by replacing the first '_' in your example?)

[srenatus]

I've started working on this, and the simple replacement does seem like a good, simple approach.

I presume it would work with mocking of multiple user defined functions by using multiple with statements.

Yes!

FWIW I've started with replacing built-ins in mocks, but user-defined functions are on my list, too. 🤞

[rmetcalf9]

Sounds good. We have been discussing it (I showed my workaround to other developers) and we are keen to not have to do it and have the ability to mock http.send.
What is the process to get the feature into a release?

[srenatus]

What is the process to get the feature into a release?

No process. I'm working on it. You can watch the progress on the (currently still) draft PR: #4540. Should become part of 0.40., sometime around the end of the month. 🤞

[srenatus]

@rmetcalf9 the part that allows mocking built-in functions is merged. If you build from source or grab the edge binaries, you can test drive it. 😃