[example] adds example for collecting response metrics

[nburkley]

We had a requirement to collect metrics on responses being returned to a user.

We couldn't find any existing solution to guide us so we build a custom module and config. We've created an example from our solution in the hopes that it helps someone else with a similar need.

I used examples/custom-config and examples/cors as templates when putting this together.

Any feedback is welcome!

[nburkley]

Ideally, we would not have to define a resolver here, but unlike the other conf blocks, the default resolver doesn't appear to be picked up here.

[nburkley]

We tried using $backend_endpoint here, but it's value was "backend_upstream" which resulted in a peering lookup error failed to set current backend peer: missing peers while connecting to upstream.

[nburkley]

Hi @mikz or other maintainers - any feedback on this PR?

[mikz]

I plan to take a look and suggest some improvements. Surely this week. Sorry for taking so long. 5. 9. 2017 v 16:00, Niall Burkley <notifications@github.com>:

…

Hi @mikz or other maintainers - any feedback on this PR? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nburkley]

No worries, thanks @mikz!

[mikz]

@nburkley I think this is very important use case and we could try to build it inside the gateway.

The plan would be following:

1. define header(s) that APIcast would read to see metrics to report
   For example X-APIcast-Report-Metrics: Documents, Size, Time.
   And send headers X-APIcast-Metric-Documents: 14, X-APIcast-Metric-Size: 10024, X-APIcast-Metric-Time: 234 .

2. define header_filter phase that will read those headers and add them to the context to be reported by the post_action

3. extend option to drop post_action #290 to support report call

4. change the post action to check if there are any reports and push them to the 3scale backend

But back to the example.
I'd suggest to try to touch the nginx configuration as little as possible and do everything in lua. There is already backend client that does authrep. Adding there a report method is trivial. Then this example could be simplified to just use the backend_client with some ngx.timer.at(0) to run it in the background.

I think this is good use case and we will have to improve the way of writing such customizations by improving the foundations.

I can either merge the example as it is and improve it later when those new APIs are available or try to work on this PR as I create those APIs and merge it as complete example.
I think the second approach could be more useful if you'd like to clean up your custom module.

With the backend_client all the nginx customization could go away and just leave the custom lua module.

[nburkley]

@mikz Thanks for the helpful feedback.

I'm very happy to hear you're considering integrating logic like this directly into the gateway. Ideally, we'd prefer to customize our 3scale gateway a little as possible and handle all our business logic in our application layer or the 3scale admin, so enhancements like this would be very welcome.

There is already backend client that does authrep.

We did look at using this initially, but we would like our API limits to be "soft limits" where API access is not disrupted when the limit is hit. The authrep call returns a 409 response for metrics that have reached a limit. While we can do this call in the background and ignore the response, it doesn't log metrics once the limit is reached - something that the Report endpoint does do.

From what I can see we would either need to modify the backend_clint to include a reporting call or have a custom nginx conf as we're currently doing.

I can either merge the example as it is and improve it later when those new APIs are available or try to work on this PR as I create those APIs and merge it as a complete example.

I don't mind waiting to include this as a complete example. As I mentioned, I created this PR because we were having trouble finding the 'correct' way to do this ourselves and thought it might be helpful for someone else in a similar situation. If you think this example will make more sense once the other API changes have been made, then I'm happy to wait for a more complete example.

In the meantime, if there are any other changes or improvements I can make to this, I'm happy to help.

[kevprice83]

@nburkley just a couple of comments based on Michal's initial feedback. The more generic we can make this whilst still fitting your use case (which is a good one) the better.

[kevprice83]

@nburkley as @mikz pointed out it would be better to do this in lua. You could make 2 requests here to 3scale, first an authorise and then a report. This is done in the post_action phase and doesn't impact the latency on the client request. If the authorize returns a 409 you can allow the call through but if it returns a 403 then the call should be rejected, then you can report based on the authorize response. The authorize already exists in the backend_client so you would just need to add the report call there.

[nburkley]

Sure, I can do that - I was trying to avoid modifying the backend_clint code - hence the call to an nginx conf.

I didn't have any auth-checks as I as going with the assumption that if the request made it thorugh our stack successfully it was already authorized - I can add this though.

[kevprice83]

Better if this regex is not hardcoded. One suggestion could be to define another environment variable that consists of the regex pattern(s) to match against. APICAST_REPORT_METRIC_MATCHER for example... or at the very least define a variable with the regex pattern outside the scope of the function which can then be used anywhere within this custom module.

[nburkley]

Sure, make sense, I'll tidy this up.

[nburkley]

thanks for the feedback @kevprice83 - I'll ping you when I've updated the PR.

[mikz]

This is now doable by a policy.
Our backend client supports :report method (f388d17) so a policy can call it in post_action phase for example.

[nburkley]

Thanks @mikz Closing this in favour of f388d17