How to suppress health check logs?

[kennethmyers]

Whenever the /health endpoint is hit, logs such as the following are produced (when running with the default log_level: INFO):

{"client_addr":"[::1]:63959","level":"info","msg":"Received request.","req_id":2,"req_method":"GET","req_path":"/health","time":"2020-07-10T12:42:31-05:00"}
{"client_addr":"[::1]:63959","level":"info","msg":"Sent response.","req_id":2,"req_method":"GET","req_path":"/health","resp_bytes":2,"resp_duration":0.318927,"resp_status":200,"time":"2020-07-10T12:42:31-05:00"}

While it is desirable to log HTTP requests in general, logging each and every health check requests floods the logs with useless data, especially when running OPA in a system with frequent health checks such as Kubernetes.

Is there a way to suppress these health check logs, but still keep all other HTTP request logs?

If there is not a way currently, would the maintainers be open to a PR to implement such a feature?

Thanks

[tsandall]

I tend to agree but I'm surprised that health check access logs outpace access logs from policy queries. How frequently do you have the health check invoked? Perhaps another question is what are you relying on the OPA server logs for? Do you have the console decision logger enabled (e.g., --set decision_logs.console=true)?

One thing we've wanted to do for a little while is get away from the low-level HTTP access logs. Instead, by default, OPA would log INFO & ERROR logs about async operations like bundle downloads and status reports as well as decision logs. The HTTP access logs would be relegated to DEBUG where they rightly belong IMO. Would this satisfy your requirements?

EDIT: @patrick-east WDYT?

Related: #2173

[kennethmyers]

The health check runs every 10 seconds, which is the default frequency for Kubernetes "liveness" probes.

We do have the decision logger enabled and that is the actual information we care about retrieving from the OPA server logs. So in our case, yes, relegating HTTP access logs to DEBUG would satisfy our requirements (I also agree it makes more sense for HTTP access logs to fall under DEBUG).

Thank you for the response and for your work on this great project!

[Lirt]

Totally agree with @kennethmyers. Health check logs are extremely annoying during troubleshooting even on empty cluster. I usually look there for decision logs or check the input format for Kubernetes operations.

@tsandall Your idea with splitting these messages by log levels sounds great to me.

[Treiblesschorle]

Hi all, I'm facing the same issue due to Kubernetes running a health check every 10 seconds. However, in my case I'm interested in the HTTP access logs only currently, and I'm already using DEBUG log level, i.e. splitting these messages by log levels would not solve it for me. A way to disable the health check HTTP logs or some way to define a filter for the logs would be great.

[tsandall]

@Treiblesschorle have you looked at the Decision Logging feature? If you run the OPA server with --set=decision_logs.console=true you'll get much better formatted logs describing the policy queries.

Unfortunately it looks like there was a regression in v0.29.4 around the log level controls but what I would suggest is running OPA with ERROR log level and enabling console decision logging:

opa run --server --log-level=ERROR --set=decision_logs.console=true

This way the decision logs are printed but other debug/info messages are suppressed.

We'll fix #3654 for the next release.

[Treiblesschorle]

@tsandall It seems that I missed that the decision logs contain the request input and results. In that case, that works for me. Thanks for the reminder.
I can confirm that nothing is logged when doing:
opa run --server --log-level=ERROR --set=decision_logs.console=true
using OPA 0.30.0.
However, after that is fixed, my issue should be resolved. Thanks 👍

[mvaal]

This does not appear to be working for me.
I am using the openpolicyagent/opa:0.30.0-rootless image. It gave me an error that it only accepts error not ERROR. And when I use error, I get zero logs, even with --set=decision_logs.console=true active. When I remove --log-level=error, I get both health check and decision logs again. Wondering why ERROR is working for you? Maybe I am using the wrong image?

[ashutosh-narkar]

#3654 is fixed with this PR. A new version of OPA with this change will be released this week. cc @Treiblesschorle

[Treiblesschorle]

@mvaal You are correct, the log level has to be lower case. In my post, I just copied the line from the post above. Also, it is currently expected that nothing is logged due to the mentioned regression (#3654).

[mvaal]

I saw the 0.31.0 version released today. I upgraded and can confirm that this is working as expected.

opa run --server --log-level=error --set=decision_logs.console=true

Returns only the decision logs and not the health check logs. Thanks!

[ashutosh-narkar]

Thanks for confirming this @mvaal ! Closing this issue.